I’m getting the following error trying to install a query:
(MetadataUpdateOperation.java:151) Failed executeInMemory for InstallQueryOperation
I shutdown the database, then restart it and then my query installation runs successful. This feels very much like a memory leak, since I can install queries several times, but after 5 or 6 compilations, the system will start failing my query compilations.
I’m running product release_3.3.0_11-01-2021
Hi, demit733. Could you send more information from gsql DEBUG log, around the timestamp of “Failed executeInMemory for InstallQueryOperation”, so we can know what’s going wrong? You can change the log level of gsql by
gadmin config set GSQL.BasicConfig.LogConfig.LogLevel DEBUG
I turned on the DEBUG log. The file is 1.5mb. I was able to compile my query 4 times, then I received an error. Here is a snippet of debug file before and after the error:
:36.215 tigergraph|127.0.0.1:56234|00000000207 (SessionManager.java:262) Add current thread ‘pool-4-thread-10’ into session ‘00000000207’.
I@20211116 22:23:36.216 tigergraph|127.0.0.1:56234|00000000207 (BaseHandler.java:209) Protocol: HTTP/1.1
I@20211116 22:23:36.216 tigergraph|127.0.0.1:56234|00000000207 (BaseHandler.java:210) BaseHandler: SimpleAuthHandler
I@20211116 22:23:36.216 tigergraph|127.0.0.1:56234|00000000207 (BaseHandler.java:212) Received|POST|/gsql/simpleauth|0
I@20211116 22:23:36.216 tigergraph|127.0.0.1:56234|00000000207 (BaseHandler.java:508) Reconcile users only
D@20211116 22:23:36.216 tigergraph|127.0.0.1:56234|00000000207 (SimpleAuthHandler.java:22) A simple authentication.
D@20211116 22:23:36.216 tigergraph|127.0.0.1:56234|00000000207 (Util.java:1641) {“privileges”:{“StatesCatalog”:[“READ_SCHEMA”,“WRITE_SCHEMA”,“READ_LOADINGJOB”,“EXECUTE_LOADINGJOB”,“WRITE_LOADINGJOB”,“READ_QUERY”,“WRITE_QUERY”,“READ_DATA”,“WRITE_DATA”,“WRITE_DATASOURCE”,“READ_ROLE”,“WRITE_ROLE”,“READ_USER”,“WRITE_USER”,“READ_PROXYGROUP”,“WRITE_PROXYGROUP”,“READ_FILE”,“WRITE_FILE”,“DROP_GRAPH”,“EXPORT_GRAPH”,“CLEAR_GRAPHSTORE”,“DROP_ALL”,“ACCESS_TAG”],“1”:[“READ_SCHEMA”,“WRITE_SCHEMA”,“READ_LOADINGJOB”,“EXECUTE_LOADINGJOB”,“WRITE_LOADINGJOB”,“READ_QUERY”,“WRITE_QUERY”,“READ_DATA”,“WRITE_DATA”,“WRITE_DATASOURCE”,“READ_ROLE”,“WRITE_ROLE”,“READ_USER”,“WRITE_USER”,“READ_PROXYGROUP”,“WRITE_PROXYGROUP”,“READ_FILE”,“WRITE_FILE”,“DROP_GRAPH”,“EXPORT_GRAPH”,“CLEAR_GRAPHSTORE”,“DROP_ALL”,“ACCESS_TAG”],“MyCalendarCatalog”:[“READ_SCHEMA”,“WRITE_SCHEMA”,“READ_LOADINGJOB”,“EXECUTE_LOADINGJOB”,“WRITE_LOADINGJOB”,“READ_QUERY”,“WRITE_QUERY”,“READ_DATA”,“WRITE_DATA”,“WRITE_DATASOURCE”,“READ_ROLE”,“WRITE_ROLE”,“READ_USER”,“WRITE_USER”,“READ_PROXYGROUP”,“WRITE_PROXYGROUP”,“READ_FILE”,“WRITE_FILE”,“DROP_GRAPH”,“EXPORT_GRAPH”,“CLEAR_GRAPHSTORE”,“DROP_ALL”,“ACCESS_TAG”],“BenefitPlans_Catalog”:[“READ_SCHEMA”,“WRITE_SCHEMA”,“READ_LOADINGJOB”,“EXECUTE_LOADINGJOB”,“WRITE_LOADINGJOB”,“READ_QUERY”,“WRITE_QUERY”,“READ_DATA”,“WRITE_DATA”,“WRITE_DATASOURCE”,“READ_ROLE”,“WRITE_ROLE”,“READ_USER”,“WRITE_USER”,“READ_PROXYGROUP”,“WRITE_PROXYGROUP”,“READ_FILE”,“WRITE_FILE”,“DROP_GRAPH”,“EXPORT_GRAPH”,“CLEAR_GRAPHSTORE”,“DROP_ALL”,“ACCESS_TAG”]},“roles”:{“StatesCatalog”:[“superuser”],“1”:[“superuser”],“MyCalendarCatalog”:[“superuser”],“BenefitPlans_Catalog”:[“superuser”]},“name”:“tigergraph”,“isSuperUser”:true,“secrets”:{}}
I@20211116 22:23:36.217 tigergraph|127.0.0.1:56234|00000000207 (BaseHandler.java:309) Successful|POST|/gsql/simpleauth|application/json; charset=UTF-8|1ms
I@20211116 22:23:36.217 tigergraph|127.0.0.1:56234|00000000207 (SessionManager.java:338) Remove current thread ‘pool-4-thread-10’ from session ‘00000000207’.
D@20211116 22:23:36.217 (SessionManager.java:356) Remove session ‘00000000207’.
E@20211116 22:23:36.246 tigergraph|127.0.0.1:55162|00000000200 (ExternalUtil.java:530) g++: internal compiler error: Killed (program cc1plus)
E@20211116 22:23:36.250 tigergraph|127.0.0.1:55162|00000000200 (ExternalUtil.java:530) Please submit a full bug report,
E@20211116 22:23:36.251 tigergraph|127.0.0.1:55162|00000000200 (ExternalUtil.java:530) with preprocessed source if appropriate.
E@20211116 22:23:36.251 tigergraph|127.0.0.1:55162|00000000200 (ExternalUtil.java:530) See <file:///usr/share/doc/gcc-4.8/README.Bugs> for instructions.
D@20211116 22:23:36.252 tigergraph|127.0.0.1:55162|00000000200 (Util.java:2739) TIMER for Finished compile query: BenefitPlans_Catalog_getAccountByAllStates, time elapsed millis: 23388
I@20211116 22:23:36.252 tigergraph|127.0.0.1:55162|00000000200 (ExternalUtil.java:546) BenefitPlans_Catalog_getAccountByAllStates finished 4
D@20211116 22:23:36.254 tigergraph|127.0.0.1:55162|00000000200 (CompileServerUtil.java:1233) This compile Server(m1) send request to my leader (m1). requst: ‘failedCompileTask’, data: {“name”:“getAccountByAllStates”,“graphname”:“BenefitPlans_Catalog”}
headers: {GSQL-Session={“fromGraphStudio”:false,“sessionParameters”:{},“compileThread”:0,“clientPath”:"/home/tigergraph/mydata/BenefitPlans",“auth”:“Basic dGlnZXJncmFwaDp0aWdlcmdyYXBo”,“clientCommit”:“f451d8a9a66c7ca0d8d4d9046f440f21097cdd03”,“gShellTest”:false,“sessionId”:“00000000200”,“fromGsqlClient”:true,“serverId”:“1_1637122188046”,“terminalWidth”:206,“graph”:“BenefitPlans_Catalog”,“loadingProgressAborted”:false,“sessionAborted”:false,“metadataUpdateSeqId”:0,“fromGsqlServer”:true}}
I@20211116 22:23:36.256 (SessionManager.java:234) Add session from gsql leader server: ‘00000000200’
I@20211116 22:23:36.256 tigergraph|127.0.0.1:56240|00000000200 (SessionManager.java:262) Add current thread ‘pool-4-thread-10’ into session ‘00000000200’.
D@20211116 22:23:36.256 tigergraph|127.0.0.1:56240|00000000200 (BaseInternalHandler.java:36) BaseInternalHandler: RequestFromCompileServerInternalHandler
D@20211116 22:23:36.256 tigergraph|127.0.0.1:56240|00000000200 (BaseInternalHandler.java:38) Receive request: /internal/compileserver_request
D@20211116 22:23:36.256 tigergraph|127.0.0.1:56240|00000000200 (RequestFromCompileServerInternalHandler.java:38) Get request from compile server:
{
“request”: “failedCompileTask”,
“replica”: 1,
“data”: “{“name”:“getAccountByAllStates”,“graphname”:“BenefitPlans_Catalog”}”
The Error log states:
E@20211116 22:23:41.313 tigergraph|127.0.0.1:55162|00000000200 (MetadataUpdateOperation.java:151) Failed executeInMemory for InstallQueryOperation
Let me know if you want the whole log files.
Kind Regards,
Mitch DeFelice
The error is “g++: internal compiler error: Killed (program cc1plus)”.
This is usually due to shortage of memory. When install query, each g++ process require at least 1.5 gb memory. If the query is complex, then g++ will need more. You can monitor the memory usage during install query to check it.
It’s possible that the memory is used by other process when install queries, for example run some queries or loading jobs.
OR some queries are too large that actually used much more than 1.5 gb memory. If that’s the case, you will need to increate the memory size on your machine.
I’m running with 3GB memory configuration. I shut down all the database processes and re-started them. I was able to compile my query 4 times before it starting to fail. It would seem to me that you may have a memory leak. I’m compiling a query that has 40 lines of code,which 10 lines are made up of either case statements or printlin() statements.
How much memory do you recommend when running TigerGraph on a laptop?
The TigerGraph system itself will require more than 3GB memory. So 3GB is not enough. You can monitor your memory usage.
The gsql will use at least one thread to install query, even the memory is not enough. I think that’s why you met the g++ error. I recommend use at least 8GB for memory configuration.