GSE is down when load larger graph

when the load graph size is 79G. GSE is down when loading process at 12%.
the disk space is much larger than graph size.

the same environment works ok when loading 8.7 GB graph.

df -h /home2/tigergraph/tigergraph/data/gstore/
Filesystem      Size  Used Avail Use% Mounted on
/dev/md0         11T   70G   11T   1% /data/dxgradb

free
              total        used        free      shared  buff/cache   available
Mem:      528300368    15089480   460436604       34444    52774284   510050828

from /home2/tigergraph/tigergraph/log/gse/GSE_1#1.out
It show bad allocation. is there any other limit for graph size?

EventId:"03a97c2af817466a9eddc222e5d3efa3" SpanId:"ReportKafkaStatusSpanId" TimestampNS:1660598526526846140 Source:<ServiceName:"GPE" Replica:1 Partition:1 > 
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
E0815 14:22:07.198987  2238 glogging.cpp:132] ============ Crashed with stacktrace ============
  0# 0x00000000006ADD38 in /home2/tigergraph/tigergraph/app/3.4.0/bin/tg_dbs_gsed
 1# 0x00007F0336C1E980 in /home2/tigergraph/tigergraph/app/3.4.0/.syspre/usr/lib_ld3/libpthread.so.0
 2# 0x00007F0335B93E87 in /home2/tigergraph/tigergraph/app/3.4.0/.syspre/usr/lib_ld3/libc.so.6
 3# 0x00007F0335B957F1 in /home2/tigergraph/tigergraph/app/3.4.0/.syspre/usr/lib_ld3/libc.so.6
 4# 0x00007F033655C535 in /home2/tigergraph/tigergraph/app/3.4.0/.syspre/usr/lib_ld4/libstdc++.so.6
 5# 0x00007F033655A6D6 in /home2/tigergraph/tigergraph/app/3.4.0/.syspre/usr/lib_ld4/libstdc++.so.6
 6# 0x00007F033655A703 in /home2/tigergraph/tigergraph/app/3.4.0/.syspre/usr/lib_ld4/libstdc++.so.6
 7# 0x00007F03365ADAF5 in /home2/tigergraph/tigergraph/app/3.4.0/.syspre/usr/lib_ld4/libstdc++.so.6
 8# 0x00007F0336C136DB in /home2/tigergraph/tigergraph/app/3.4.0/.syspre/usr/lib_ld3/libpthread.so.0
 9# 0x00007F0335C7661F in /home2/tigergraph/tigergraph/app/3.4.0/.syspre/usr/lib_ld3/libc.so.6

============ End of stacktrace ============
*** Aborted at 1660598527 (unix time) try "date -d @1660598527" if you are using GNU date ***
terminate called recursively
E0815 14:22:07.221910  3396 glogging.cpp:132] ============ Crashed with stacktrace ============
  0# 0x00000000006ADD38 in /home2/tigergraph/tigergraph/app/3.4.0/bin/tg_dbs_gsed
 1# 0x00007F0336C1E980 in /home2/tigergraph/tigergraph/app/3.4.0/.syspre/usr/lib_ld3/libpthread.so.0
 2# 0x00007F0335B93E87 in /home2/tigergraph/tigergraph/app/3.4.0/.syspre/usr/lib_ld3/libc.so.6
 3# 0x00007F0335B957F1 in /home2/tigergraph/tigergraph/app/3.4.0/.syspre/usr/lib_ld3/libc.so.6
 4# 0x00007F033655C4C5 in /home2/tigergraph/tigergraph/app/3.4.0/.syspre/usr/lib_ld4/libstdc++.so.6
 5# 0x00007F033655A6D6 in /home2/tigergraph/tigergraph/app/3.4.0/.syspre/usr/lib_ld4/libstdc++.so.6
 6# 0x00007F033655A703 in /home2/tigergraph/tigergraph/app/3.4.0/.syspre/usr/lib_ld4/libstdc++.so.6
 7# 0x00007F03365ADAF5 in /home2/tigergraph/tigergraph/app/3.4.0/.syspre/usr/lib_ld4/libstdc++.so.6
 8# 0x00007F0336C136DB in /home2/tigergraph/tigergraph/app/3.4.0/.syspre/usr/lib_ld3/libpthread.so.0
 9# 0x00007F0335C7661F in /home2/tigergraph/tigergraph/app/3.4.0/.syspre/usr/lib_ld3/libc.so.6

============ End of stacktrace ============
PC: @     0x7f0335b93e87 gsignal
*** SIGABRT (@0x3f30000085d) received by PID 2141 (TID 0x7f02c63ff700) from PID 2141; stack trace: ***
    @           0x6addc5 gutil::(anonymous namespace)::FailureSignalHandler()
    @     0x7f0336c1e980 (unknown)
    @     0x7f0335b93e87 gsignal
    @     0x7f0335b957f1 abort
FATAL ERROR: OUT OF MEMORY (src/yqueue.hpp:71)
terminate called recursively
terminate called recursively
terminate called recursively
terminate called recursively
terminate called recursively

load_job log is below:

MessageQueue|ZMQContext|Initialized
MessageQueue|ZMQContext|Initialized
TigerGraph RESTPP_LOADER: 
 --- Version --- 
TigerGraph version: 3.4.0
product          release_3.4.0_12-20-2021         d50cab57c7582aa64cc2036c5cd646176c18b949  2021-10-31 09:07:36 -0700
cqrs             release_3.4.0_12-20-2021         0ea55f12b1f34d80ac08efd1876aba1d25864403  2021-12-07 22:56:25 -0800
third_party      release_3.4.0_12-20-2021         317031418629dc08ef6a1a6ffeacf5d3f90852a3  2021-08-22 06:30:36 -0700
er               release_3.4.0_12-20-2021         ae4c979b7b9ad1021fbf2ce0216a13ee663cfe81  2021-12-15 04:25:01 -0800
gle              release_3.4.0_12-20-2021         ca46dde07b3a94055501ec91b04a47d4f1d255b0  2021-12-20 10:51:53 -0800
document         release_3.4.0_12-20-2021         9d65a7395d8db226a56ac0563a935f1998a8e191  2021-07-26 19:00:16 -0700
glive            release_3.4.0_12-20-2021         3053aed948c727b690d69417e7d589f514e2f054  2021-11-15 20:07:15 -0800
gap              release_3.4.0_12-20-2021         a1b613308181a8414c0b67fe9d1b24d23db29ef5  2021-11-30 11:58:14 -0800
gst              release_3.4.0_12-20-2021         10822174580b35a3983e6f3a2773ae4d11306a4f  2021-12-09 18:07:52 -0800
gus              release_3.4.0_12-20-2021         0321b36b92e0f265c75549ae87988b43044883fa  2021-10-28 00:32:31 -0700
engine           release_3.4.0_12-20-2021         e3d12d8b4d652cec6e7bbbe717d1c47d7d4b3427  2021-12-09 18:18:19 -0800

[INFO] EOL = 
(10)
==============args==================
sep_str: 
(char) sep = ,
eol_str: 
(char) eol = 

job_name_or_path: nhop_vivianw_load_job
path: 
input_files: 
is_directory: 160
ignoreheader: 0
progress_file: /home2/tigergraph/tigergraph/log/restpp/restpp_loader_logs/nhop_vivianw/nhop_vivianw.load_job.file.m1.1660597709147.progress
configFileName: /home2/tigergraph/tigergraph/data/configs/tg.cfg
partition: 1
replica: 1
gsql_pipe_fd: -1
concurrency: 256
batch_size: 8192
gsql_pipe_name: 
transactional: 0
skipNLines: -1
firstNLines: -1
====================================
Cannot bind to ipc:///var/tmp/tigergraph/tigergraph/gsql_6410 StatusHubAgentServer won't start
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0815 14:08:29.277676  6410 config.cpp:192] Queue infos:
I0815 14:08:29.277705  6410 config.cpp:194] queue name:GSE_journal_0 topic:Topic: GSE_journal_0
 source: GSE
 target: GSE

I0815 14:08:29.277709  6410 config.cpp:194] queue name:GSE_journal_1 topic:Topic: GSE_journal_1
 source: GSE
 target: GSE

I0815 14:08:29.277711  6410 config.cpp:194] queue name:GSE_journal_10 topic:Topic: GSE_journal_10
 source: GSE
 target: GSE

I0815 14:08:29.277714  6410 config.cpp:194] queue name:GSE_journal_11 topic:Topic: GSE_journal_11
 source: GSE
 target: GSE

I0815 14:08:29.277716  6410 config.cpp:194] queue name:GSE_journal_12 topic:Topic: GSE_journal_12
 source: GSE
 target: GSE

I0815 14:08:29.277719  6410 config.cpp:194] queue name:GSE_journal_13 topic:Topic: GSE_journal_13
 source: GSE
 target: GSE

I0815 14:08:29.277722  6410 config.cpp:194] queue name:GSE_journal_14 topic:Topic: GSE_journal_14
 source: GSE
 target: GSE

I0815 14:08:29.277724  6410 config.cpp:194] queue name:GSE_journal_15 topic:Topic: GSE_journal_15
 source: GSE
 target: GSE

I0815 14:08:29.277727  6410 config.cpp:194] queue name:GSE_journal_16 topic:Topic: GSE_journal_16
 source: GSE
 target: GSE

I0815 14:08:29.277730  6410 config.cpp:194] queue name:GSE_journal_17 topic:Topic: GSE_journal_17
 source: GSE
 target: GSE

I0815 14:08:29.277734  6410 config.cpp:194] queue name:GSE_journal_18 topic:Topic: GSE_journal_18
 source: GSE
 target: GSE

I0815 14:08:29.277735  6410 config.cpp:194] queue name:GSE_journal_19 topic:Topic: GSE_journal_19
 source: GSE
 target: GSE

I0815 14:08:29.277738  6410 config.cpp:194] queue name:GSE_journal_2 topic:Topic: GSE_journal_2
 source: GSE
 target: GSE

I0815 14:08:29.277741  6410 config.cpp:194] queue name:GSE_journal_20 topic:Topic: GSE_journal_20
 source: GSE
 target: GSE

I0815 14:08:29.277745  6410 config.cpp:194] queue name:GSE_journal_21 topic:Topic: GSE_journal_21
 source: GSE
 target: GSE

I0815 14:08:29.277756  6410 config.cpp:194] queue name:GSE_journal_22 topic:Topic: GSE_journal_22
 source: GSE
 target: GSE

I0815 14:08:29.277760  6410 config.cpp:194] queue name:GSE_journal_23 topic:Topic: GSE_journal_23
 source: GSE
 target: GSE

I0815 14:08:29.277762  6410 config.cpp:194] queue name:GSE_journal_24 topic:Topic: GSE_journal_24
 source: GSE
 target: GSE

I0815 14:08:29.277765  6410 config.cpp:194] queue name:GSE_journal_25 topic:Topic: GSE_journal_25
 source: GSE
 target: GSE

I0815 14:08:29.277767  6410 config.cpp:194] queue name:GSE_journal_26 topic:Topic: GSE_journal_26
 source: GSE
 target: GSE

I0815 14:08:29.277770  6410 config.cpp:194] queue name:GSE_journal_27 topic:Topic: GSE_journal_27
 source: GSE
 target: GSE

I0815 14:08:29.277773  6410 config.cpp:194] queue name:GSE_journal_28 topic:Topic: GSE_journal_28
 source: GSE
 target: GSE

I0815 14:08:29.277776  6410 config.cpp:194] queue name:GSE_journal_29 topic:Topic: GSE_journal_29
 source: GSE
 target: GSE

I0815 14:08:29.277778  6410 config.cpp:194] queue name:GSE_journal_3 topic:Topic: GSE_journal_3
 source: GSE
 target: GSE

I0815 14:08:29.277781  6410 config.cpp:194] queue name:GSE_journal_30 topic:Topic: GSE_journal_30
 source: GSE
 target: GSE

I0815 14:08:29.277783  6410 config.cpp:194] queue name:GSE_journal_31 topic:Topic: GSE_journal_31
 source: GSE
 target: GSE

I0815 14:08:29.277786  6410 config.cpp:194] queue name:GSE_journal_4 topic:Topic: GSE_journal_4
 source: GSE
 target: GSE

I0815 14:08:29.277789  6410 config.cpp:194] queue name:GSE_journal_5 topic:Topic: GSE_journal_5
 source: GSE
 target: GSE

I0815 14:08:29.277791  6410 config.cpp:194] queue name:GSE_journal_6 topic:Topic: GSE_journal_6
 source: GSE
 target: GSE

I0815 14:08:29.277794  6410 config.cpp:194] queue name:GSE_journal_7 topic:Topic: GSE_journal_7
 source: GSE
 target: GSE

I0815 14:08:29.277797  6410 config.cpp:194] queue name:GSE_journal_8 topic:Topic: GSE_journal_8
 source: GSE
 target: GSE

I0815 14:08:29.277799  6410 config.cpp:194] queue name:GSE_journal_9 topic:Topic: GSE_journal_9
 source: GSE
 target: GSE

I0815 14:08:29.277802  6410 config.cpp:194] queue name:delta_queue topic:Topic: deltaQ
 source: GPE
 target: GPE

I0815 14:08:29.277806  6410 config.cpp:194] queue name:get_request_queue topic:Topic: get_requestQ
 source: GPE
 target: GPE
GPE_1_1: tcp://127.0.0.1:7502

 source: RESTPP
 target: GPE
GPE_1_1: tcp://127.0.0.1:7502

 source: RESTPP-LOADER
 target: GPE
GPE_1_1: tcp://127.0.0.1:7502

 source: KAFKA-LOADER
 target: GPE
GPE_1_1: tcp://127.0.0.1:7502

I0815 14:08:29.277808  6410 config.cpp:194] queue name:gse_delta_queue topic:Topic: deltaQ
 source: GSE
 target: GPE

I0815 14:08:29.277812  6410 config.cpp:194] queue name:id_request_queue_query topic:Topic: id_requesQ_QUERY
 source: GPE
 target: GSE
GSE_1_1: tcp://127.0.0.1:6500

 source: RESTPP
 target: GSE
GSE_1_1: tcp://127.0.0.1:6500

 source: RESTPP-LOADER
 target: GSE
GSE_1_1: tcp://127.0.0.1:6500

 source: KAFKA-LOADER
 target: GSE
GSE_1_1: tcp://127.0.0.1:6500

I0815 14:08:29.277814  6410 config.cpp:194] queue name:id_response_queue_query topic:Topic: id_responseQ_QUERY
 source: GSE
 target: GPE
GPE_1_1: tcp://127.0.0.1:7500

 source: GSE
 target: RESTPP
RESTPP_1_1: tcp://127.0.0.1:5500

 source: GSE
 target: RESTPP-LOADER
RESTPP-LOADER_1_1: tcp://127.0.0.1:8501

 source: GSE
 target: KAFKA-LOADER
KAFKA-LOADER_1_1: tcp://127.0.0.1:9501

I0815 14:08:29.277818  6410 config.cpp:194] queue name:response_queue topic:Topic: responseQ
 source: GPE
 target: GPE
GPE_1_1: tcp://127.0.0.1:7501

 source: GPE
 target: RESTPP
RESTPP_1_1: tcp://127.0.0.1:5400

 source: GPE
 target: RESTPP-LOADER
RESTPP-LOADER_1_1: tcp://127.0.0.1:8401

 source: GPE
 target: KAFKA-LOADER
KAFKA-LOADER_1_1: tcp://127.0.0.1:9401

I0815 14:08:29.277918  6410 completion_queue_manager.cpp:18] Client CQManager(0x7fc36055c240) constructing
I0815 14:08:29.278414  6410 gdict.cpp:298] Dictionary initialize start
2022-08-15 14:08:29,278:6410(0x7fc366a1e000):ZOO_INFO@log_env@1102: Client environment:zookeeper.version=zookeeper C client 3.5.8
2022-08-15 14:08:29,278:6410(0x7fc366a1e000):ZOO_INFO@log_env@1106: Client environment:host.name=xsj-dxgradb01
2022-08-15 14:08:29,278:6410(0x7fc366a1e000):ZOO_INFO@log_env@1113: Client environment:os.name=Linux
2022-08-15 14:08:29,278:6410(0x7fc366a1e000):ZOO_INFO@log_env@1114: Client environment:os.arch=4.15.0-191-generic
2022-08-15 14:08:29,278:6410(0x7fc366a1e000):ZOO_INFO@log_env@1115: Client environment:os.version=#202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022
2022-08-15 14:08:29,278:6410(0x7fc366a1e000):ZOO_INFO@log_env@1123: Client environment:user.name=(null)
2022-08-15 14:08:29,278:6410(0x7fc366a1e000):ZOO_INFO@log_env@1131: Client environment:user.home=/home2/tigergraph
2022-08-15 14:08:29,278:6410(0x7fc366a1e000):ZOO_INFO@log_env@1143: Client environment:user.dir=/data/dxgradb/tigergraph/tigergraph/app/3.4.0/bin
2022-08-15 14:08:29,278:6410(0x7fc366a1e000):ZOO_INFO@zookeeper_init_internal@1186: Initiating client connection, host=127.0.0.1:19999 sessionTimeout=120000 watcher=0x7ad1c0 sessionId=0 sessionPasswd=<null> context=0x7fc36051bc80 flags=0
I0815 14:08:29.278551  6460 completion_queue_manager.cpp:87] Client CompletionQueue (0x7fc36055c1e0) begin processing.
2022-08-15 14:08:29,279:6410(0x7fc2fb9ff700):ZOO_INFO@check_events@2473: initiated connection to server [127.0.0.1:19999]
2022-08-15 14:08:29,280:6410(0x7fc2fb9ff700):ZOO_INFO@check_events@2525: session establishment complete on server [127.0.0.1:19999], sessionId=0x10000004425002c, negotiated timeout=120000 
I0815 14:08:29.281047  6462 zookeeper_context.cpp:206] Root Watcher SESSION_EVENT state = CONNECTED_STATE for path: NA
I0815 14:08:29.281076  6462 zookeeper_context.cpp:77] ZooKeeper Connection is setup. Session id: 10000004425002c, previous client id:0
I0815 14:08:29.281088  6462 zookeeper_watcher.cpp:288] Zk Session connected, notifying watchers
I0815 14:08:29.281095  6462 zookeeper_watcher.cpp:295]   --> Number of watchers: 0
I0815 14:08:29.281101  6462 zookeeper_watcher.cpp:296]   --> Callback time used(us): 6
I0815 14:08:29.329336  6410 heartbeat_client.cpp:437] CLIENT: resolved server address: 127.0.0.1:17797
I0815 14:08:29.329360  6410 channel_pool.cpp:13] Create channel for target: 127.0.0.1:17797
I0815 14:08:29.330699  6410 async_client.cpp:40] Connected to 127.0.0.1:17797. ChannelState:2
W0815 14:08:29.330929  6410 heartbeat_client.cpp:465] CLIENT: Detect server update, old server: , new server: 127.0.0.1:17797. Tried to start client session, session: 0x2f54328 state:CLIENT_SESSION_ISSUED rc: kOk
I0815 14:08:29.330952  6460 heartbeat_client.cpp:339] ClientSession is issued. Session 0x2f54328 Server:127.0.0.1:17797
I0815 14:08:29.331221  6463 single_thread_worker.cpp:83] SingleThreadWorker start: HeartbeatClient
I0815 14:08:29.331295  6463 client_watcher_manager.cpp:32] Reconnect session, re-watch all paths. size:0
I0815 14:08:29.341069  6410 gdict.cpp:343] Dictionary initialize succeed, took 63 milliseconds
E0815 14:08:29.347196  6410 gconfig_general.cpp:320] Graph nhop_vivianw failed to get end points. List endpoints files failed. rc:kNotFound
E0815 14:08:29.355362  6410 gconfig_general.cpp:320] Graph nhop_vivianw failed to get end points. List endpoints files failed. rc:kNotFound

14:08:29.341118 restppconfig.cpp:226] Engine_RefreshConfig|Start RefreshServerConfig
14:08:29.352248 restppconfig.cpp:346] Engine_RefreshGraphSchemaCatalog|| rc.Ok() = 1
14:08:29.354168 restppconfig.cpp:363] Engine_RefreshGraphSchema|| rc.Ok() = 1
14:08:29.354173 restppconfig.cpp:234] Engine_RefreshConfig|Start RefreshEndpoints
14:08:29.355383 restppconfig.cpp:241] Engine_RefreshConfig|Start RefreshLoadingJobs
14:08:29.355387 restppconfig.cpp:377] Engine_RefreshLoadingJobs|received start
14:08:29.355390 restppconfig.cpp:385] Engine_RefreshLoadingJobs|received end
14:08:29.355697 restppconfig.cpp:424] Engine_LJM|Registered 1 loading jobs with total parsing JSON time: 0 (ms) and total register job time : 0 (ms)
14:08:29.355700 restppconfig.cpp:248] Engine_RefreshConfig|Finish Catalog Refresh
14:08:29.355705 restppconfig.cpp:278] Engine_RefreshConfig|
Refresh graphCatalogYamlNode and schema_node success

14:08:29.355707 restppconfig.cpp:289] Engine_RefreshConfig|
Refresh gpe schema_node success

14:08:29.355709 restppconfig.cpp:309] Engine_RefreshConfig|
Refresh Auth start

14:08:29.355711 restppconfig.cpp:320] Engine_RefreshConfig|
Finish Refresh Auth

14:08:29.355713 restppconfig.cpp:321] Engine_RefreshConfig|Start Refresh Tags
14:08:29.356966 restppconfig.cpp:326] Engine_RefreshConfig|
Finish Refresh Tags

14:08:29.356971 restppconfig.cpp:327] Engine_RefreshConfig|
Finish All ConfigRefresh

14:08:29.357436 gtimer.cpp:82] MessageQueue|Kafka|CreateWriter|loading-logLog folder at /home2/tigergraph/tigergraph/log/fileLoader
ENTERPRISE_EDITION
0815 14:08:29.404 D kafka/kafka.go:229] KafkaProducer EventOutputProducer: Start sarama producer with addrs: [127.0.0.1:30002]
0815 14:08:29.404 D kafka/kafka.go:434] Start to create sarama partition consumer
0815 14:08:29.405 I kafka/kafka.go:449] KafkaConsumer EventInputConsumer, start offset: 258585, oldest: 220827, newest: 258585
0815 14:08:29.406 I kafka/kafka.go:474] KafkaConsumer EventInputConsumer: the real offset used to start kafka consumer is [258585]
E0815 14:08:29.428349  6410 zookeeper_context.cpp:1056] Recursive delete /tigergraph/dict/objects/__services/RESTPP-LOADER/_runtime_nodes/RESTPP-LOADER_1_1 failed. PathCount:1 Rc:no node
ENTERPRISE_EDITION
ENTERPRISE_EDITION
ENTERPRISE_EDITION
ENTERPRISE_EDITION
[INFO] yamlfromfile = 0, with args.job_name_or_path = nhop_vivianw_load_job
filename = /home2/tigergraph/tigergraph/log/restpp/restpp_loader_logs/nhop_vivianw/nhop_vivianw.load_job.file.m1.1660597709147.progress, progress size = 817
write: filename = /home2/tigergraph/tigergraph/log/restpp/restpp_loader_logs/nhop_vivianw/nhop_vivianw.load_job.file.m1.1660597709147.progress, mem = {"commandStr":"/home2/tigergraph/tigergraph/app/3.4.0/bin//tg_app_fileldr  --job nhop_vivianw_load_job --totalTask 1 --config /home2/tigergraph/tigergraph/data/configs/tg.cfg --progress /home2/tigergraph/tigergraph/log/restpp/restpp_loader_logs/nhop_vivianw/nhop_vivianw.load_job.file.m1.1660597709147.progress --jobid nhop_vivianw.load_job.file.m1.1660597709147","config_list":[{"EOL":"\\n","HEADER":"false","SEPARATOR":" ","TRANSACTION":"false","filename":"data_source","path":"/wrk/xsjhdnobkup2/yifei/git_graphanalytics/graphanalytics_dev/graphanalytics/L2/nhop_xnik_dv/cpu_dv/sk_29_16.0_0.45_0.20.mtx"}],"graph_name":"nhop_vivianw","machine_id":1,"version":"v1"}
/wrk/xsjhdnobkup2/yifei/git_graphanalytics/graphanalytics_dev/graphanalytics/L2/nhop_xnik_dv/cpu_dv/sk_29_16.0_0.45_0.20.mtx

offset_line = 0, offset_line_ = 0, skipNLines = -1, firstNLines = -1
line_count = 0
[INFO] Start loading /wrk/xsjhdnobkup2/yifei/git_graphanalytics/graphanalytics_dev/graphanalytics/L2/nhop_xnik_dv/cpu_dv/sk_29_16.0_0.45_0.20.mtx, LineBatch = 8192, LineOffset = 0, ByteOffset = 0
0815 14:08:36.456 I event/command_consumer.go:72] Received cmd. Name: ReportKafkaStatus, Id: 020cb7c72d5f46deaf1bd1cfd0da32a6, CreatedTimeStamp: 1660597716453260001, DeadlineTimeStamp: 1660597731453260001
0815 14:08:36.456 D event/command_consumer.go:74] Cmd Id: 020cb7c72d5f46deaf1bd1cfd0da32a6, Parameter: type_url:"type.googleapis.com/tigergraph.tutopia.common.pb.Bytes" value:"\n\n\010\001\020\0010\324\353\352\227\006" 
0815 14:09:06.459 I event/command_consumer.go:72] Received cmd. Name: ReportKafkaStatus, Id: 502ce4c5fa5b480893947082ab9facfa, CreatedTimeStamp: 1660597746456003120, DeadlineTimeStamp: 1660597761456003120
0815 14:09:06.459 D event/command_consumer.go:74] Cmd Id: 502ce4c5fa5b480893947082ab9facfa, Parameter: type_url:"type.googleapis.com/tigergraph.tutopia.common.pb.Bytes" value:"\n\r\010\001\020\001(\222\t0\362\353\352\227\006" 
0815 14:09:36.460 I event/command_consumer.go:72] Received cmd. Name: ReportKafkaStatus, Id: 6bc3ab72dffd480683cf0410ea5fc2ae, CreatedTimeStamp: 1660597776458535365, DeadlineTimeStamp: 1660597791458535365
0815 14:09:36.461 D event/command_consumer.go:74] Cmd Id: 6bc3ab72dffd480683cf0410ea5fc2ae, Parameter: type_url:"type.googleapis.com/tigergraph.tutopia.common.pb.Bytes" value:"\n\r\010\001\020\001(\235!0\220\354\352\227\006" 
0815 14:10:06.463 I event/command_consumer.go:72] Received cmd. Name: ReportKafkaStatus, Id: 97a1017214fc43cf86b1f17700ff4b9e, CreatedTimeStamp: 1660597806461008699, DeadlineTimeStamp: 1660597821461008699
0815 14:10:06.463 D event/command_consumer.go:74] Cmd Id: 97a1017214fc43cf86b1f17700ff4b9e, Parameter: type_url:"type.googleapis.com/tigergraph.tutopia.common.pb.Bytes" value:"\n\r\010\001\020\001(\30190\256\354\352\227\006" 
0815 14:10:36.511 I event/command_consumer.go:72] Received cmd. Name: ReportKafkaStatus, Id: 9e86457679024653b9ee2a524b7ba25f, CreatedTimeStamp: 1660597836463671169, DeadlineTimeStamp: 1660597851463671169
0815 14:10:36.512 D event/command_consumer.go:74] Cmd Id: 9e86457679024653b9ee2a524b7ba25f, Parameter: type_url:"type.googleapis.com/tigergraph.tutopia.common.pb.Bytes" value:"\n\r\010\001\020\001(\202L0\314\354\352\227\006" 
0815 14:11:06.469 I event/command_consumer.go:72] Received cmd. Name: ReportKafkaStatus, Id: caa908cf52b64a829da114f1fb389adf, CreatedTimeStamp: 1660597866466275607, DeadlineTimeStamp: 1660597881466275607
0815 14:11:06.469 D event/command_consumer.go:74] Cmd Id: caa908cf52b64a829da114f1fb389adf, Parameter: type_url:"type.googleapis.com/tigergraph.tutopia.common.pb.Bytes" value:"\n\r\010\001\020\001(\353e0\352\354\352\227\006" 
0815 14:11:36.471 I event/command_consumer.go:72] Received cmd. Name: ReportKafkaStatus, Id: 23aac9bfa2a54c2baae5e90a4d150207, CreatedTimeStamp: 1660597896468848057, DeadlineTimeStamp: 1660597911468848057
0815 14:11:36.471 D event/command_consumer.go:74] Cmd Id: 23aac9bfa2a54c2baae5e90a4d150207, Parameter: type_url:"type.googleapis.com/tigergraph.tutopia.common.pb.Bytes" value:"\n\r\010\001\020\001(\231y0\210\355\352\227\006" 
0815 14:12:06.475 I event/command_consumer.go:72] Received cmd. Name: ReportKafkaStatus, Id: 3b5f6ee14bbd4b458078d4e71a67c178, CreatedTimeStamp: 1660597926471645422, DeadlineTimeStamp: 1660597941471645422
0815 14:12:06.475 D event/command_consumer.go:74] Cmd Id: 3b5f6ee14bbd4b458078d4e71a67c178, Parameter: type_url:"type.googleapis.com/tigergraph.tutopia.common.pb.Bytes" value:"\n\016\010\001\020\001(\260\212\0010\246\355\352\227\006" 
0815 14:12:36.477 I event/command_consumer.go:72] Received cmd. Name: ReportKafkaStatus, Id: 49c27a4aeee44e7389ec0c54e9def0bd, CreatedTimeStamp: 1660597956474111631, DeadlineTimeStamp: 1660597971474111631
0815 14:12:36.477 D event/command_consumer.go:74] Cmd Id: 49c27a4aeee44e7389ec0c54e9def0bd, Parameter: type_url:"type.googleapis.com/tigergraph.tutopia.common.pb.Bytes" value:"\n\016\010\001\020\001(\355\243\0010\304\355\352\227\006" 
0815 14:13:06.479 I event/command_consumer.go:72] Received cmd. Name: ReportKafkaStatus, Id: 83627c19e88e434f80830910ed502b75, CreatedTimeStamp: 1660597986476846648, DeadlineTimeStamp: 1660598001476846648
0815 14:13:06.479 D event/command_consumer.go:74] Cmd Id: 83627c19e88e434f80830910ed502b75, Parameter: type_url:"type.googleapis.com/tigergraph.tutopia.common.pb.Bytes" value:"\n\016\010\001\020\001(\357\264\0010\342\355\352\227\006" 
0815 14:13:36.481 I event/command_consumer.go:72] Received cmd. Name: ReportKafkaStatus, Id: b4e63be920ec4452be46f0065d7d72a5, CreatedTimeStamp: 1660598016479356392, DeadlineTimeStamp: 1660598031479356392
0815 14:13:36.481 D event/command_consumer.go:74] Cmd Id: b4e63be920ec4452be46f0065d7d72a5, Parameter: type_url:"type.googleapis.com/tigergraph.tutopia.common.pb.Bytes" value:"\n\016\010\001\020\001(\256\310\0010\200\356\352\227\006" 
0815 14:14:06.484 I event/command_consumer.go:72] Received cmd. Name: ReportKafkaStatus, Id: 9545a16b2c644e59b5ac2b8d237b3f6a, CreatedTimeStamp: 1660598046481772245, DeadlineTimeStamp: 1660598061481772245
0815 14:14:06.484 D event/command_consumer.go:74] Cmd Id: 9545a16b2c644e59b5ac2b8d237b3f6a, Parameter: type_url:"type.googleapis.com/tigergraph.tutopia.common.pb.Bytes" value:"\n\016\010\001\020\001(\373\331\0010\236\356\352\227\006" 
0815 14:14:36.487 I event/command_consumer.go:72] Received cmd. Name: ReportKafkaStatus, Id: 9cc2f3db5e9b493cbf57c862ca03ec4a, CreatedTimeStamp: 1660598076484578808, DeadlineTimeStamp: 1660598091484578808
0815 14:14:36.487 D event/command_consumer.go:74] Cmd Id: 9cc2f3db5e9b493cbf57c862ca03ec4a, Parameter: type_url:"type.googleapis.com/tigergraph.tutopia.common.pb.Bytes" value:"\n\016\010\001\020\001(\375\352\0010\274\356\352\227\006" 
0815 14:15:06.491 I event/command_consumer.go:72] Received cmd. Name: ReportKafkaStatus, Id: e3c121e4289b417db301efd1ead53462, CreatedTimeStamp: 1660598106487304932, DeadlineTimeStamp: 1660598121487304932
0815 14:15:06.491 D event/command_consumer.go:74] Cmd Id: e3c121e4289b417db301efd1ead53462, Parameter: type_url:"type.googleapis.com/tigergraph.tutopia.common.pb.Bytes" value:"\n\016\010\001\020\001(\316\373\0010\332\356\352\227\006" 
0815 14:15:36.491 I event/command_consumer.go:72] Received cmd. Name: ReportKafkaStatus, Id: a83f5aed3e484aa89decc09270708007, CreatedTimeStamp: 1660598136489813124, DeadlineTimeStamp: 1660598151489813124
0815 14:15:36.491 D event/command_consumer.go:74] Cmd Id: a83f5aed3e484aa89decc09270708007, Parameter: type_url:"type.googleapis.com/tigergraph.tutopia.common.pb.Bytes" value:"\n\016\010\001\020\001(\311\216\0020\370\356\352\227\006" 
0815 14:16:06.494 I event/command_consumer.go:72] Received cmd. Name: ReportKafkaStatus, Id: d55c03e8ab604ddc91249314ff4d38aa, CreatedTimeStamp: 1660598166492407049, DeadlineTimeStamp: 1660598181492407049
0815 14:16:06.494 D event/command_consumer.go:74] Cmd Id: d55c03e8ab604ddc91249314ff4d38aa, Parameter: type_url:"type.googleapis.com/tigergraph.tutopia.common.pb.Bytes" value:"\n\016\010\001\020\001(\357\241\0020\226\357\352\227\006" 
0815 14:16:36.497 I event/command_consumer.go:72] Received cmd. Name: ReportKafkaStatus, Id: 975ae9150e6f4ea68cf4433078beaee7, CreatedTimeStamp: 1660598196495179230, DeadlineTimeStamp: 1660598211495179230
0815 14:16:36.497 D event/command_consumer.go:74] Cmd Id: 975ae9150e6f4ea68cf4433078beaee7, Parameter: type_url:"type.googleapis.com/tigergraph.tutopia.common.pb.Bytes" value:"\n\016\010\001\020\001(\352\264\0020\264\357\352\227\006" 
0815 14:17:06.502 I event/command_consumer.go:72] Received cmd. Name: ReportKafkaStatus, Id: 27410ae7df274688aa8ecbda8b26a019, CreatedTimeStamp: 1660598226497981866, DeadlineTimeStamp: 1660598241497981866
0815 14:17:06.502 D event/command_consumer.go:74] Cmd Id: 27410ae7df274688aa8ecbda8b26a019, Parameter: type_url:"type.googleapis.com/tigergraph.tutopia.common.pb.Bytes" value:"\n\016\010\001\020\001(\233\277\0020\322\357\352\227\006" 
0815 14:17:36.503 I event/command_consumer.go:72] Received cmd. Name: ReportKafkaStatus, Id: f43de4d3ec1e4b76bea4ba69e9782111, CreatedTimeStamp: 1660598256500553578, DeadlineTimeStamp: 1660598271500553578
0815 14:17:36.503 D event/command_consumer.go:74] Cmd Id: f43de4d3ec1e4b76bea4ba69e9782111, Parameter: type_url:"type.googleapis.com/tigergraph.tutopia.common.pb.Bytes" value:"\n\016\010\001\020\001(\325\323\0020\360\357\352\227\006" 
0815 14:18:06.507 I event/command_consumer.go:72] Received cmd. Name: ReportKafkaStatus, Id: ab9ec956a69e474badec061c1910142e, CreatedTimeStamp: 1660598286503125103, DeadlineTimeStamp: 1660598301503125103
0815 14:18:06.507 D event/command_consumer.go:74] Cmd Id: ab9ec956a69e474badec061c1910142e, Parameter: type_url:"type.googleapis.com/tigergraph.tutopia.common.pb.Bytes" value:"\n\016\010\001\020\001(\363\345\0020\216\360\352\227\006" 
%3|1660598309.386|ERROR|RESTPP-LOADER_1_1_CONSUMER#consumer-3| [thrd:127.0.0.1:30002/bootstrap]: 1/1 brokers are down
0815 14:18:36.508 I event/command_consumer.go:72] Received cmd. Name: ReportKafkaStatus, Id: dafcf8d186c74b0c914b76394789d3e2, CreatedTimeStamp: 1660598316506084710, DeadlineTimeStamp: 1660598331506084710
0815 14:18:36.508 D event/command_consumer.go:74] Cmd Id: dafcf8d186c74b0c914b76394789d3e2, Parameter: type_url:"type.googleapis.com/tigergraph.tutopia.common.pb.Bytes" value:"\n\016\010\001\020\001(\374\366\0020\254\360\352\227\006" 
0815 14:19:06.783 I event/command_consumer.go:72] Received cmd. Name: ReportKafkaStatus, Id: 7c1119f88b6141fcbc2bf670cc7d668a, CreatedTimeStamp: 1660598346508993209, DeadlineTimeStamp: 1660598361508993209
0815 14:19:06.783 D event/command_consumer.go:74] Cmd Id: 7c1119f88b6141fcbc2bf670cc7d668a, Parameter: type_url:"type.googleapis.com/tigergraph.tutopia.common.pb.Bytes" value:"\n\016\010\001\020\001(\311\212\0030\312\360\352\227\006" 
0815 14:19:36.514 I event/command_consumer.go:72] Received cmd. Name: ReportKafkaStatus, Id: 47769e43e82842d09e26e40a4e4c824f, CreatedTimeStamp: 1660598376511791918, DeadlineTimeStamp: 1660598391511791918
0815 14:19:36.514 D event/command_consumer.go:74] Cmd Id: 47769e43e82842d09e26e40a4e4c824f, Parameter: type_url:"type.googleapis.com/tigergraph.tutopia.common.pb.Bytes" value:"\n\016\010\001\020\001(\315\235\0030\350\360\352\227\006" 
0815 14:20:06.517 I event/command_consumer.go:72] Received cmd. Name: ReportKafkaStatus, Id: bec8c5e14e8c4ba2b94f6396668ff41c, CreatedTimeStamp: 1660598406514689473, DeadlineTimeStamp: 1660598421514689473
0815 14:20:06.517 D event/command_consumer.go:74] Cmd Id: bec8c5e14e8c4ba2b94f6396668ff41c, Parameter: type_url:"type.googleapis.com/tigergraph.tutopia.common.pb.Bytes" value:"\n\016\010\001\020\001(\216\254\0030\206\361\352\227\006" 
0815 14:20:36.519 I event/command_consumer.go:72] Received cmd. Name: ReportKafkaStatus, Id: 8b0fa802c787451196b10e3665005a1b, CreatedTimeStamp: 1660598436517361946, DeadlineTimeStamp: 1660598451517361946
0815 14:20:36.519 D event/command_consumer.go:74] Cmd Id: 8b0fa802c787451196b10e3665005a1b, Parameter: type_url:"type.googleapis.com/tigergraph.tutopia.common.pb.Bytes" value:"\n\016\010\001\020\001(\350\273\0030\244\361\352\227\006" 
0815 14:21:06.522 I event/command_consumer.go:72] Received cmd. Name: ReportKafkaStatus, Id: ae79ee17e4e94f4fb8db0a4aacde9278, CreatedTimeStamp: 1660598466520235088, DeadlineTimeStamp: 1660598481520235088
0815 14:21:06.522 D event/command_consumer.go:74] Cmd Id: ae79ee17e4e94f4fb8db0a4aacde9278, Parameter: type_url:"type.googleapis.com/tigergraph.tutopia.common.pb.Bytes" value:"\n\016\010\001\020\001(\303\317\0030\302\361\352\227\006" 
0815 14:21:36.525 I event/command_consumer.go:72] Received cmd. Name: ReportKafkaStatus, Id: ec53474ba9e64c1a9d5f4d1a38da7709, CreatedTimeStamp: 1660598496523220055, DeadlineTimeStamp: 1660598511523220055
0815 14:21:36.525 D event/command_consumer.go:74] Cmd Id: ec53474ba9e64c1a9d5f4d1a38da7709, Parameter: type_url:"type.googleapis.com/tigergraph.tutopia.common.pb.Bytes" value:"\n\016\010\001\020\001(\246\341\0030\340\361\352\227\006" 
0815 14:22:06.528 I event/command_consumer.go:72] Received cmd. Name: ReportKafkaStatus, Id: 68b0e14790944ccea9c09f6c63240737, CreatedTimeStamp: 1660598526526180885, DeadlineTimeStamp: 1660598541526180885
0815 14:22:06.528 D event/command_consumer.go:74] Cmd Id: 68b0e14790944ccea9c09f6c63240737, Parameter: type_url:"type.googleapis.com/tigergraph.tutopia.common.pb.Bytes" value:"\n\016\010\001\020\001(\345\360\0030\376\361\352\227\006" 
E0815 14:22:34.517230  6503 gbrain_active_address_resolver.cpp:136] [RefreshGSE] errors[0]: 205, msg: No leader is found
.......
 

Hi Vivian,
I’ve seen this issue before in a support ticket when working with others from Xilinx (@jimwu). My recommendation is that you change your memory allocator from jemalloc to tcmalloc.

To do this, you’ll need to modify the GSE environmental variables. First, we’ll find the location of tcmalloc, which is included in the TG package:

find $(gadmin config get System.AppRoot --file ~/.tg.cfg)  -name libtcmalloc.so | grep thirdparty

Then you can modify the EnvVars:

gadmin config entry GSE.BasicConfig.Env

You should also do the same for the GPE, using the config field GPE.BasicConfig.Env

LD_PRELOAD=PATH_FROM_ABOVE_HERE; LD_LIBRARY_PATH=$LD_LIBRARY_PATH
gadmin config apply 
gadmin restart gpe -y

This is usually caused by some incompatibility between your hardware and jemalloc. Both jemalloc and tcmalloc are popular open source projects, so there is low risk from this change.

2 Likes

By following your suggestions, the issue is solved. Thanks!!

1 Like