when the load graph size is 79G. GSE is down when loading process at 12%.
the disk space is much larger than graph size.
the same environment works ok when loading 8.7 GB graph.
df -h /home2/tigergraph/tigergraph/data/gstore/
Filesystem Size Used Avail Use% Mounted on
/dev/md0 11T 70G 11T 1% /data/dxgradb
free
total used free shared buff/cache available
Mem: 528300368 15089480 460436604 34444 52774284 510050828
from /home2/tigergraph/tigergraph/log/gse/GSE_1#1.out
It show bad allocation. is there any other limit for graph size?
EventId:"03a97c2af817466a9eddc222e5d3efa3" SpanId:"ReportKafkaStatusSpanId" TimestampNS:1660598526526846140 Source:<ServiceName:"GPE" Replica:1 Partition:1 >
terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc
E0815 14:22:07.198987 2238 glogging.cpp:132] ============ Crashed with stacktrace ============
0# 0x00000000006ADD38 in /home2/tigergraph/tigergraph/app/3.4.0/bin/tg_dbs_gsed
1# 0x00007F0336C1E980 in /home2/tigergraph/tigergraph/app/3.4.0/.syspre/usr/lib_ld3/libpthread.so.0
2# 0x00007F0335B93E87 in /home2/tigergraph/tigergraph/app/3.4.0/.syspre/usr/lib_ld3/libc.so.6
3# 0x00007F0335B957F1 in /home2/tigergraph/tigergraph/app/3.4.0/.syspre/usr/lib_ld3/libc.so.6
4# 0x00007F033655C535 in /home2/tigergraph/tigergraph/app/3.4.0/.syspre/usr/lib_ld4/libstdc++.so.6
5# 0x00007F033655A6D6 in /home2/tigergraph/tigergraph/app/3.4.0/.syspre/usr/lib_ld4/libstdc++.so.6
6# 0x00007F033655A703 in /home2/tigergraph/tigergraph/app/3.4.0/.syspre/usr/lib_ld4/libstdc++.so.6
7# 0x00007F03365ADAF5 in /home2/tigergraph/tigergraph/app/3.4.0/.syspre/usr/lib_ld4/libstdc++.so.6
8# 0x00007F0336C136DB in /home2/tigergraph/tigergraph/app/3.4.0/.syspre/usr/lib_ld3/libpthread.so.0
9# 0x00007F0335C7661F in /home2/tigergraph/tigergraph/app/3.4.0/.syspre/usr/lib_ld3/libc.so.6
============ End of stacktrace ============
*** Aborted at 1660598527 (unix time) try "date -d @1660598527" if you are using GNU date ***
terminate called recursively
E0815 14:22:07.221910 3396 glogging.cpp:132] ============ Crashed with stacktrace ============
0# 0x00000000006ADD38 in /home2/tigergraph/tigergraph/app/3.4.0/bin/tg_dbs_gsed
1# 0x00007F0336C1E980 in /home2/tigergraph/tigergraph/app/3.4.0/.syspre/usr/lib_ld3/libpthread.so.0
2# 0x00007F0335B93E87 in /home2/tigergraph/tigergraph/app/3.4.0/.syspre/usr/lib_ld3/libc.so.6
3# 0x00007F0335B957F1 in /home2/tigergraph/tigergraph/app/3.4.0/.syspre/usr/lib_ld3/libc.so.6
4# 0x00007F033655C4C5 in /home2/tigergraph/tigergraph/app/3.4.0/.syspre/usr/lib_ld4/libstdc++.so.6
5# 0x00007F033655A6D6 in /home2/tigergraph/tigergraph/app/3.4.0/.syspre/usr/lib_ld4/libstdc++.so.6
6# 0x00007F033655A703 in /home2/tigergraph/tigergraph/app/3.4.0/.syspre/usr/lib_ld4/libstdc++.so.6
7# 0x00007F03365ADAF5 in /home2/tigergraph/tigergraph/app/3.4.0/.syspre/usr/lib_ld4/libstdc++.so.6
8# 0x00007F0336C136DB in /home2/tigergraph/tigergraph/app/3.4.0/.syspre/usr/lib_ld3/libpthread.so.0
9# 0x00007F0335C7661F in /home2/tigergraph/tigergraph/app/3.4.0/.syspre/usr/lib_ld3/libc.so.6
============ End of stacktrace ============
PC: @ 0x7f0335b93e87 gsignal
*** SIGABRT (@0x3f30000085d) received by PID 2141 (TID 0x7f02c63ff700) from PID 2141; stack trace: ***
@ 0x6addc5 gutil::(anonymous namespace)::FailureSignalHandler()
@ 0x7f0336c1e980 (unknown)
@ 0x7f0335b93e87 gsignal
@ 0x7f0335b957f1 abort
FATAL ERROR: OUT OF MEMORY (src/yqueue.hpp:71)
terminate called recursively
terminate called recursively
terminate called recursively
terminate called recursively
terminate called recursively
load_job log is below:
MessageQueue|ZMQContext|Initialized
MessageQueue|ZMQContext|Initialized
TigerGraph RESTPP_LOADER:
--- Version ---
TigerGraph version: 3.4.0
product release_3.4.0_12-20-2021 d50cab57c7582aa64cc2036c5cd646176c18b949 2021-10-31 09:07:36 -0700
cqrs release_3.4.0_12-20-2021 0ea55f12b1f34d80ac08efd1876aba1d25864403 2021-12-07 22:56:25 -0800
third_party release_3.4.0_12-20-2021 317031418629dc08ef6a1a6ffeacf5d3f90852a3 2021-08-22 06:30:36 -0700
er release_3.4.0_12-20-2021 ae4c979b7b9ad1021fbf2ce0216a13ee663cfe81 2021-12-15 04:25:01 -0800
gle release_3.4.0_12-20-2021 ca46dde07b3a94055501ec91b04a47d4f1d255b0 2021-12-20 10:51:53 -0800
document release_3.4.0_12-20-2021 9d65a7395d8db226a56ac0563a935f1998a8e191 2021-07-26 19:00:16 -0700
glive release_3.4.0_12-20-2021 3053aed948c727b690d69417e7d589f514e2f054 2021-11-15 20:07:15 -0800
gap release_3.4.0_12-20-2021 a1b613308181a8414c0b67fe9d1b24d23db29ef5 2021-11-30 11:58:14 -0800
gst release_3.4.0_12-20-2021 10822174580b35a3983e6f3a2773ae4d11306a4f 2021-12-09 18:07:52 -0800
gus release_3.4.0_12-20-2021 0321b36b92e0f265c75549ae87988b43044883fa 2021-10-28 00:32:31 -0700
engine release_3.4.0_12-20-2021 e3d12d8b4d652cec6e7bbbe717d1c47d7d4b3427 2021-12-09 18:18:19 -0800
[INFO] EOL =
(10)
==============args==================
sep_str:
(char) sep = ,
eol_str:
(char) eol =
job_name_or_path: nhop_vivianw_load_job
path:
input_files:
is_directory: 160
ignoreheader: 0
progress_file: /home2/tigergraph/tigergraph/log/restpp/restpp_loader_logs/nhop_vivianw/nhop_vivianw.load_job.file.m1.1660597709147.progress
configFileName: /home2/tigergraph/tigergraph/data/configs/tg.cfg
partition: 1
replica: 1
gsql_pipe_fd: -1
concurrency: 256
batch_size: 8192
gsql_pipe_name:
transactional: 0
skipNLines: -1
firstNLines: -1
====================================
Cannot bind to ipc:///var/tmp/tigergraph/tigergraph/gsql_6410 StatusHubAgentServer won't start
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0815 14:08:29.277676 6410 config.cpp:192] Queue infos:
I0815 14:08:29.277705 6410 config.cpp:194] queue name:GSE_journal_0 topic:Topic: GSE_journal_0
source: GSE
target: GSE
I0815 14:08:29.277709 6410 config.cpp:194] queue name:GSE_journal_1 topic:Topic: GSE_journal_1
source: GSE
target: GSE
I0815 14:08:29.277711 6410 config.cpp:194] queue name:GSE_journal_10 topic:Topic: GSE_journal_10
source: GSE
target: GSE
I0815 14:08:29.277714 6410 config.cpp:194] queue name:GSE_journal_11 topic:Topic: GSE_journal_11
source: GSE
target: GSE
I0815 14:08:29.277716 6410 config.cpp:194] queue name:GSE_journal_12 topic:Topic: GSE_journal_12
source: GSE
target: GSE
I0815 14:08:29.277719 6410 config.cpp:194] queue name:GSE_journal_13 topic:Topic: GSE_journal_13
source: GSE
target: GSE
I0815 14:08:29.277722 6410 config.cpp:194] queue name:GSE_journal_14 topic:Topic: GSE_journal_14
source: GSE
target: GSE
I0815 14:08:29.277724 6410 config.cpp:194] queue name:GSE_journal_15 topic:Topic: GSE_journal_15
source: GSE
target: GSE
I0815 14:08:29.277727 6410 config.cpp:194] queue name:GSE_journal_16 topic:Topic: GSE_journal_16
source: GSE
target: GSE
I0815 14:08:29.277730 6410 config.cpp:194] queue name:GSE_journal_17 topic:Topic: GSE_journal_17
source: GSE
target: GSE
I0815 14:08:29.277734 6410 config.cpp:194] queue name:GSE_journal_18 topic:Topic: GSE_journal_18
source: GSE
target: GSE
I0815 14:08:29.277735 6410 config.cpp:194] queue name:GSE_journal_19 topic:Topic: GSE_journal_19
source: GSE
target: GSE
I0815 14:08:29.277738 6410 config.cpp:194] queue name:GSE_journal_2 topic:Topic: GSE_journal_2
source: GSE
target: GSE
I0815 14:08:29.277741 6410 config.cpp:194] queue name:GSE_journal_20 topic:Topic: GSE_journal_20
source: GSE
target: GSE
I0815 14:08:29.277745 6410 config.cpp:194] queue name:GSE_journal_21 topic:Topic: GSE_journal_21
source: GSE
target: GSE
I0815 14:08:29.277756 6410 config.cpp:194] queue name:GSE_journal_22 topic:Topic: GSE_journal_22
source: GSE
target: GSE
I0815 14:08:29.277760 6410 config.cpp:194] queue name:GSE_journal_23 topic:Topic: GSE_journal_23
source: GSE
target: GSE
I0815 14:08:29.277762 6410 config.cpp:194] queue name:GSE_journal_24 topic:Topic: GSE_journal_24
source: GSE
target: GSE
I0815 14:08:29.277765 6410 config.cpp:194] queue name:GSE_journal_25 topic:Topic: GSE_journal_25
source: GSE
target: GSE
I0815 14:08:29.277767 6410 config.cpp:194] queue name:GSE_journal_26 topic:Topic: GSE_journal_26
source: GSE
target: GSE
I0815 14:08:29.277770 6410 config.cpp:194] queue name:GSE_journal_27 topic:Topic: GSE_journal_27
source: GSE
target: GSE
I0815 14:08:29.277773 6410 config.cpp:194] queue name:GSE_journal_28 topic:Topic: GSE_journal_28
source: GSE
target: GSE
I0815 14:08:29.277776 6410 config.cpp:194] queue name:GSE_journal_29 topic:Topic: GSE_journal_29
source: GSE
target: GSE
I0815 14:08:29.277778 6410 config.cpp:194] queue name:GSE_journal_3 topic:Topic: GSE_journal_3
source: GSE
target: GSE
I0815 14:08:29.277781 6410 config.cpp:194] queue name:GSE_journal_30 topic:Topic: GSE_journal_30
source: GSE
target: GSE
I0815 14:08:29.277783 6410 config.cpp:194] queue name:GSE_journal_31 topic:Topic: GSE_journal_31
source: GSE
target: GSE
I0815 14:08:29.277786 6410 config.cpp:194] queue name:GSE_journal_4 topic:Topic: GSE_journal_4
source: GSE
target: GSE
I0815 14:08:29.277789 6410 config.cpp:194] queue name:GSE_journal_5 topic:Topic: GSE_journal_5
source: GSE
target: GSE
I0815 14:08:29.277791 6410 config.cpp:194] queue name:GSE_journal_6 topic:Topic: GSE_journal_6
source: GSE
target: GSE
I0815 14:08:29.277794 6410 config.cpp:194] queue name:GSE_journal_7 topic:Topic: GSE_journal_7
source: GSE
target: GSE
I0815 14:08:29.277797 6410 config.cpp:194] queue name:GSE_journal_8 topic:Topic: GSE_journal_8
source: GSE
target: GSE
I0815 14:08:29.277799 6410 config.cpp:194] queue name:GSE_journal_9 topic:Topic: GSE_journal_9
source: GSE
target: GSE
I0815 14:08:29.277802 6410 config.cpp:194] queue name:delta_queue topic:Topic: deltaQ
source: GPE
target: GPE
I0815 14:08:29.277806 6410 config.cpp:194] queue name:get_request_queue topic:Topic: get_requestQ
source: GPE
target: GPE
GPE_1_1: tcp://127.0.0.1:7502
source: RESTPP
target: GPE
GPE_1_1: tcp://127.0.0.1:7502
source: RESTPP-LOADER
target: GPE
GPE_1_1: tcp://127.0.0.1:7502
source: KAFKA-LOADER
target: GPE
GPE_1_1: tcp://127.0.0.1:7502
I0815 14:08:29.277808 6410 config.cpp:194] queue name:gse_delta_queue topic:Topic: deltaQ
source: GSE
target: GPE
I0815 14:08:29.277812 6410 config.cpp:194] queue name:id_request_queue_query topic:Topic: id_requesQ_QUERY
source: GPE
target: GSE
GSE_1_1: tcp://127.0.0.1:6500
source: RESTPP
target: GSE
GSE_1_1: tcp://127.0.0.1:6500
source: RESTPP-LOADER
target: GSE
GSE_1_1: tcp://127.0.0.1:6500
source: KAFKA-LOADER
target: GSE
GSE_1_1: tcp://127.0.0.1:6500
I0815 14:08:29.277814 6410 config.cpp:194] queue name:id_response_queue_query topic:Topic: id_responseQ_QUERY
source: GSE
target: GPE
GPE_1_1: tcp://127.0.0.1:7500
source: GSE
target: RESTPP
RESTPP_1_1: tcp://127.0.0.1:5500
source: GSE
target: RESTPP-LOADER
RESTPP-LOADER_1_1: tcp://127.0.0.1:8501
source: GSE
target: KAFKA-LOADER
KAFKA-LOADER_1_1: tcp://127.0.0.1:9501
I0815 14:08:29.277818 6410 config.cpp:194] queue name:response_queue topic:Topic: responseQ
source: GPE
target: GPE
GPE_1_1: tcp://127.0.0.1:7501
source: GPE
target: RESTPP
RESTPP_1_1: tcp://127.0.0.1:5400
source: GPE
target: RESTPP-LOADER
RESTPP-LOADER_1_1: tcp://127.0.0.1:8401
source: GPE
target: KAFKA-LOADER
KAFKA-LOADER_1_1: tcp://127.0.0.1:9401
I0815 14:08:29.277918 6410 completion_queue_manager.cpp:18] Client CQManager(0x7fc36055c240) constructing
I0815 14:08:29.278414 6410 gdict.cpp:298] Dictionary initialize start
2022-08-15 14:08:29,278:6410(0x7fc366a1e000):ZOO_INFO@log_env@1102: Client environment:zookeeper.version=zookeeper C client 3.5.8
2022-08-15 14:08:29,278:6410(0x7fc366a1e000):ZOO_INFO@log_env@1106: Client environment:host.name=xsj-dxgradb01
2022-08-15 14:08:29,278:6410(0x7fc366a1e000):ZOO_INFO@log_env@1113: Client environment:os.name=Linux
2022-08-15 14:08:29,278:6410(0x7fc366a1e000):ZOO_INFO@log_env@1114: Client environment:os.arch=4.15.0-191-generic
2022-08-15 14:08:29,278:6410(0x7fc366a1e000):ZOO_INFO@log_env@1115: Client environment:os.version=#202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022
2022-08-15 14:08:29,278:6410(0x7fc366a1e000):ZOO_INFO@log_env@1123: Client environment:user.name=(null)
2022-08-15 14:08:29,278:6410(0x7fc366a1e000):ZOO_INFO@log_env@1131: Client environment:user.home=/home2/tigergraph
2022-08-15 14:08:29,278:6410(0x7fc366a1e000):ZOO_INFO@log_env@1143: Client environment:user.dir=/data/dxgradb/tigergraph/tigergraph/app/3.4.0/bin
2022-08-15 14:08:29,278:6410(0x7fc366a1e000):ZOO_INFO@zookeeper_init_internal@1186: Initiating client connection, host=127.0.0.1:19999 sessionTimeout=120000 watcher=0x7ad1c0 sessionId=0 sessionPasswd=<null> context=0x7fc36051bc80 flags=0
I0815 14:08:29.278551 6460 completion_queue_manager.cpp:87] Client CompletionQueue (0x7fc36055c1e0) begin processing.
2022-08-15 14:08:29,279:6410(0x7fc2fb9ff700):ZOO_INFO@check_events@2473: initiated connection to server [127.0.0.1:19999]
2022-08-15 14:08:29,280:6410(0x7fc2fb9ff700):ZOO_INFO@check_events@2525: session establishment complete on server [127.0.0.1:19999], sessionId=0x10000004425002c, negotiated timeout=120000
I0815 14:08:29.281047 6462 zookeeper_context.cpp:206] Root Watcher SESSION_EVENT state = CONNECTED_STATE for path: NA
I0815 14:08:29.281076 6462 zookeeper_context.cpp:77] ZooKeeper Connection is setup. Session id: 10000004425002c, previous client id:0
I0815 14:08:29.281088 6462 zookeeper_watcher.cpp:288] Zk Session connected, notifying watchers
I0815 14:08:29.281095 6462 zookeeper_watcher.cpp:295] --> Number of watchers: 0
I0815 14:08:29.281101 6462 zookeeper_watcher.cpp:296] --> Callback time used(us): 6
I0815 14:08:29.329336 6410 heartbeat_client.cpp:437] CLIENT: resolved server address: 127.0.0.1:17797
I0815 14:08:29.329360 6410 channel_pool.cpp:13] Create channel for target: 127.0.0.1:17797
I0815 14:08:29.330699 6410 async_client.cpp:40] Connected to 127.0.0.1:17797. ChannelState:2
W0815 14:08:29.330929 6410 heartbeat_client.cpp:465] CLIENT: Detect server update, old server: , new server: 127.0.0.1:17797. Tried to start client session, session: 0x2f54328 state:CLIENT_SESSION_ISSUED rc: kOk
I0815 14:08:29.330952 6460 heartbeat_client.cpp:339] ClientSession is issued. Session 0x2f54328 Server:127.0.0.1:17797
I0815 14:08:29.331221 6463 single_thread_worker.cpp:83] SingleThreadWorker start: HeartbeatClient
I0815 14:08:29.331295 6463 client_watcher_manager.cpp:32] Reconnect session, re-watch all paths. size:0
I0815 14:08:29.341069 6410 gdict.cpp:343] Dictionary initialize succeed, took 63 milliseconds
E0815 14:08:29.347196 6410 gconfig_general.cpp:320] Graph nhop_vivianw failed to get end points. List endpoints files failed. rc:kNotFound
E0815 14:08:29.355362 6410 gconfig_general.cpp:320] Graph nhop_vivianw failed to get end points. List endpoints files failed. rc:kNotFound
14:08:29.341118 restppconfig.cpp:226] Engine_RefreshConfig|Start RefreshServerConfig
14:08:29.352248 restppconfig.cpp:346] Engine_RefreshGraphSchemaCatalog|| rc.Ok() = 1
14:08:29.354168 restppconfig.cpp:363] Engine_RefreshGraphSchema|| rc.Ok() = 1
14:08:29.354173 restppconfig.cpp:234] Engine_RefreshConfig|Start RefreshEndpoints
14:08:29.355383 restppconfig.cpp:241] Engine_RefreshConfig|Start RefreshLoadingJobs
14:08:29.355387 restppconfig.cpp:377] Engine_RefreshLoadingJobs|received start
14:08:29.355390 restppconfig.cpp:385] Engine_RefreshLoadingJobs|received end
14:08:29.355697 restppconfig.cpp:424] Engine_LJM|Registered 1 loading jobs with total parsing JSON time: 0 (ms) and total register job time : 0 (ms)
14:08:29.355700 restppconfig.cpp:248] Engine_RefreshConfig|Finish Catalog Refresh
14:08:29.355705 restppconfig.cpp:278] Engine_RefreshConfig|
Refresh graphCatalogYamlNode and schema_node success
14:08:29.355707 restppconfig.cpp:289] Engine_RefreshConfig|
Refresh gpe schema_node success
14:08:29.355709 restppconfig.cpp:309] Engine_RefreshConfig|
Refresh Auth start
14:08:29.355711 restppconfig.cpp:320] Engine_RefreshConfig|
Finish Refresh Auth
14:08:29.355713 restppconfig.cpp:321] Engine_RefreshConfig|Start Refresh Tags
14:08:29.356966 restppconfig.cpp:326] Engine_RefreshConfig|
Finish Refresh Tags
14:08:29.356971 restppconfig.cpp:327] Engine_RefreshConfig|
Finish All ConfigRefresh
14:08:29.357436 gtimer.cpp:82] MessageQueue|Kafka|CreateWriter|loading-logLog folder at /home2/tigergraph/tigergraph/log/fileLoader
ENTERPRISE_EDITION
0815 14:08:29.404 D kafka/kafka.go:229] KafkaProducer EventOutputProducer: Start sarama producer with addrs: [127.0.0.1:30002]
0815 14:08:29.404 D kafka/kafka.go:434] Start to create sarama partition consumer
0815 14:08:29.405 I kafka/kafka.go:449] KafkaConsumer EventInputConsumer, start offset: 258585, oldest: 220827, newest: 258585
0815 14:08:29.406 I kafka/kafka.go:474] KafkaConsumer EventInputConsumer: the real offset used to start kafka consumer is [258585]
E0815 14:08:29.428349 6410 zookeeper_context.cpp:1056] Recursive delete /tigergraph/dict/objects/__services/RESTPP-LOADER/_runtime_nodes/RESTPP-LOADER_1_1 failed. PathCount:1 Rc:no node
ENTERPRISE_EDITION
ENTERPRISE_EDITION
ENTERPRISE_EDITION
ENTERPRISE_EDITION
[INFO] yamlfromfile = 0, with args.job_name_or_path = nhop_vivianw_load_job
filename = /home2/tigergraph/tigergraph/log/restpp/restpp_loader_logs/nhop_vivianw/nhop_vivianw.load_job.file.m1.1660597709147.progress, progress size = 817
write: filename = /home2/tigergraph/tigergraph/log/restpp/restpp_loader_logs/nhop_vivianw/nhop_vivianw.load_job.file.m1.1660597709147.progress, mem = {"commandStr":"/home2/tigergraph/tigergraph/app/3.4.0/bin//tg_app_fileldr --job nhop_vivianw_load_job --totalTask 1 --config /home2/tigergraph/tigergraph/data/configs/tg.cfg --progress /home2/tigergraph/tigergraph/log/restpp/restpp_loader_logs/nhop_vivianw/nhop_vivianw.load_job.file.m1.1660597709147.progress --jobid nhop_vivianw.load_job.file.m1.1660597709147","config_list":[{"EOL":"\\n","HEADER":"false","SEPARATOR":" ","TRANSACTION":"false","filename":"data_source","path":"/wrk/xsjhdnobkup2/yifei/git_graphanalytics/graphanalytics_dev/graphanalytics/L2/nhop_xnik_dv/cpu_dv/sk_29_16.0_0.45_0.20.mtx"}],"graph_name":"nhop_vivianw","machine_id":1,"version":"v1"}
/wrk/xsjhdnobkup2/yifei/git_graphanalytics/graphanalytics_dev/graphanalytics/L2/nhop_xnik_dv/cpu_dv/sk_29_16.0_0.45_0.20.mtx
offset_line = 0, offset_line_ = 0, skipNLines = -1, firstNLines = -1
line_count = 0
[INFO] Start loading /wrk/xsjhdnobkup2/yifei/git_graphanalytics/graphanalytics_dev/graphanalytics/L2/nhop_xnik_dv/cpu_dv/sk_29_16.0_0.45_0.20.mtx, LineBatch = 8192, LineOffset = 0, ByteOffset = 0
0815 14:08:36.456 I event/command_consumer.go:72] Received cmd. Name: ReportKafkaStatus, Id: 020cb7c72d5f46deaf1bd1cfd0da32a6, CreatedTimeStamp: 1660597716453260001, DeadlineTimeStamp: 1660597731453260001
0815 14:08:36.456 D event/command_consumer.go:74] Cmd Id: 020cb7c72d5f46deaf1bd1cfd0da32a6, Parameter: type_url:"type.googleapis.com/tigergraph.tutopia.common.pb.Bytes" value:"\n\n\010\001\020\0010\324\353\352\227\006"
0815 14:09:06.459 I event/command_consumer.go:72] Received cmd. Name: ReportKafkaStatus, Id: 502ce4c5fa5b480893947082ab9facfa, CreatedTimeStamp: 1660597746456003120, DeadlineTimeStamp: 1660597761456003120
0815 14:09:06.459 D event/command_consumer.go:74] Cmd Id: 502ce4c5fa5b480893947082ab9facfa, Parameter: type_url:"type.googleapis.com/tigergraph.tutopia.common.pb.Bytes" value:"\n\r\010\001\020\001(\222\t0\362\353\352\227\006"
0815 14:09:36.460 I event/command_consumer.go:72] Received cmd. Name: ReportKafkaStatus, Id: 6bc3ab72dffd480683cf0410ea5fc2ae, CreatedTimeStamp: 1660597776458535365, DeadlineTimeStamp: 1660597791458535365
0815 14:09:36.461 D event/command_consumer.go:74] Cmd Id: 6bc3ab72dffd480683cf0410ea5fc2ae, Parameter: type_url:"type.googleapis.com/tigergraph.tutopia.common.pb.Bytes" value:"\n\r\010\001\020\001(\235!0\220\354\352\227\006"
0815 14:10:06.463 I event/command_consumer.go:72] Received cmd. Name: ReportKafkaStatus, Id: 97a1017214fc43cf86b1f17700ff4b9e, CreatedTimeStamp: 1660597806461008699, DeadlineTimeStamp: 1660597821461008699
0815 14:10:06.463 D event/command_consumer.go:74] Cmd Id: 97a1017214fc43cf86b1f17700ff4b9e, Parameter: type_url:"type.googleapis.com/tigergraph.tutopia.common.pb.Bytes" value:"\n\r\010\001\020\001(\30190\256\354\352\227\006"
0815 14:10:36.511 I event/command_consumer.go:72] Received cmd. Name: ReportKafkaStatus, Id: 9e86457679024653b9ee2a524b7ba25f, CreatedTimeStamp: 1660597836463671169, DeadlineTimeStamp: 1660597851463671169
0815 14:10:36.512 D event/command_consumer.go:74] Cmd Id: 9e86457679024653b9ee2a524b7ba25f, Parameter: type_url:"type.googleapis.com/tigergraph.tutopia.common.pb.Bytes" value:"\n\r\010\001\020\001(\202L0\314\354\352\227\006"
0815 14:11:06.469 I event/command_consumer.go:72] Received cmd. Name: ReportKafkaStatus, Id: caa908cf52b64a829da114f1fb389adf, CreatedTimeStamp: 1660597866466275607, DeadlineTimeStamp: 1660597881466275607
0815 14:11:06.469 D event/command_consumer.go:74] Cmd Id: caa908cf52b64a829da114f1fb389adf, Parameter: type_url:"type.googleapis.com/tigergraph.tutopia.common.pb.Bytes" value:"\n\r\010\001\020\001(\353e0\352\354\352\227\006"
0815 14:11:36.471 I event/command_consumer.go:72] Received cmd. Name: ReportKafkaStatus, Id: 23aac9bfa2a54c2baae5e90a4d150207, CreatedTimeStamp: 1660597896468848057, DeadlineTimeStamp: 1660597911468848057
0815 14:11:36.471 D event/command_consumer.go:74] Cmd Id: 23aac9bfa2a54c2baae5e90a4d150207, Parameter: type_url:"type.googleapis.com/tigergraph.tutopia.common.pb.Bytes" value:"\n\r\010\001\020\001(\231y0\210\355\352\227\006"
0815 14:12:06.475 I event/command_consumer.go:72] Received cmd. Name: ReportKafkaStatus, Id: 3b5f6ee14bbd4b458078d4e71a67c178, CreatedTimeStamp: 1660597926471645422, DeadlineTimeStamp: 1660597941471645422
0815 14:12:06.475 D event/command_consumer.go:74] Cmd Id: 3b5f6ee14bbd4b458078d4e71a67c178, Parameter: type_url:"type.googleapis.com/tigergraph.tutopia.common.pb.Bytes" value:"\n\016\010\001\020\001(\260\212\0010\246\355\352\227\006"
0815 14:12:36.477 I event/command_consumer.go:72] Received cmd. Name: ReportKafkaStatus, Id: 49c27a4aeee44e7389ec0c54e9def0bd, CreatedTimeStamp: 1660597956474111631, DeadlineTimeStamp: 1660597971474111631
0815 14:12:36.477 D event/command_consumer.go:74] Cmd Id: 49c27a4aeee44e7389ec0c54e9def0bd, Parameter: type_url:"type.googleapis.com/tigergraph.tutopia.common.pb.Bytes" value:"\n\016\010\001\020\001(\355\243\0010\304\355\352\227\006"
0815 14:13:06.479 I event/command_consumer.go:72] Received cmd. Name: ReportKafkaStatus, Id: 83627c19e88e434f80830910ed502b75, CreatedTimeStamp: 1660597986476846648, DeadlineTimeStamp: 1660598001476846648
0815 14:13:06.479 D event/command_consumer.go:74] Cmd Id: 83627c19e88e434f80830910ed502b75, Parameter: type_url:"type.googleapis.com/tigergraph.tutopia.common.pb.Bytes" value:"\n\016\010\001\020\001(\357\264\0010\342\355\352\227\006"
0815 14:13:36.481 I event/command_consumer.go:72] Received cmd. Name: ReportKafkaStatus, Id: b4e63be920ec4452be46f0065d7d72a5, CreatedTimeStamp: 1660598016479356392, DeadlineTimeStamp: 1660598031479356392
0815 14:13:36.481 D event/command_consumer.go:74] Cmd Id: b4e63be920ec4452be46f0065d7d72a5, Parameter: type_url:"type.googleapis.com/tigergraph.tutopia.common.pb.Bytes" value:"\n\016\010\001\020\001(\256\310\0010\200\356\352\227\006"
0815 14:14:06.484 I event/command_consumer.go:72] Received cmd. Name: ReportKafkaStatus, Id: 9545a16b2c644e59b5ac2b8d237b3f6a, CreatedTimeStamp: 1660598046481772245, DeadlineTimeStamp: 1660598061481772245
0815 14:14:06.484 D event/command_consumer.go:74] Cmd Id: 9545a16b2c644e59b5ac2b8d237b3f6a, Parameter: type_url:"type.googleapis.com/tigergraph.tutopia.common.pb.Bytes" value:"\n\016\010\001\020\001(\373\331\0010\236\356\352\227\006"
0815 14:14:36.487 I event/command_consumer.go:72] Received cmd. Name: ReportKafkaStatus, Id: 9cc2f3db5e9b493cbf57c862ca03ec4a, CreatedTimeStamp: 1660598076484578808, DeadlineTimeStamp: 1660598091484578808
0815 14:14:36.487 D event/command_consumer.go:74] Cmd Id: 9cc2f3db5e9b493cbf57c862ca03ec4a, Parameter: type_url:"type.googleapis.com/tigergraph.tutopia.common.pb.Bytes" value:"\n\016\010\001\020\001(\375\352\0010\274\356\352\227\006"
0815 14:15:06.491 I event/command_consumer.go:72] Received cmd. Name: ReportKafkaStatus, Id: e3c121e4289b417db301efd1ead53462, CreatedTimeStamp: 1660598106487304932, DeadlineTimeStamp: 1660598121487304932
0815 14:15:06.491 D event/command_consumer.go:74] Cmd Id: e3c121e4289b417db301efd1ead53462, Parameter: type_url:"type.googleapis.com/tigergraph.tutopia.common.pb.Bytes" value:"\n\016\010\001\020\001(\316\373\0010\332\356\352\227\006"
0815 14:15:36.491 I event/command_consumer.go:72] Received cmd. Name: ReportKafkaStatus, Id: a83f5aed3e484aa89decc09270708007, CreatedTimeStamp: 1660598136489813124, DeadlineTimeStamp: 1660598151489813124
0815 14:15:36.491 D event/command_consumer.go:74] Cmd Id: a83f5aed3e484aa89decc09270708007, Parameter: type_url:"type.googleapis.com/tigergraph.tutopia.common.pb.Bytes" value:"\n\016\010\001\020\001(\311\216\0020\370\356\352\227\006"
0815 14:16:06.494 I event/command_consumer.go:72] Received cmd. Name: ReportKafkaStatus, Id: d55c03e8ab604ddc91249314ff4d38aa, CreatedTimeStamp: 1660598166492407049, DeadlineTimeStamp: 1660598181492407049
0815 14:16:06.494 D event/command_consumer.go:74] Cmd Id: d55c03e8ab604ddc91249314ff4d38aa, Parameter: type_url:"type.googleapis.com/tigergraph.tutopia.common.pb.Bytes" value:"\n\016\010\001\020\001(\357\241\0020\226\357\352\227\006"
0815 14:16:36.497 I event/command_consumer.go:72] Received cmd. Name: ReportKafkaStatus, Id: 975ae9150e6f4ea68cf4433078beaee7, CreatedTimeStamp: 1660598196495179230, DeadlineTimeStamp: 1660598211495179230
0815 14:16:36.497 D event/command_consumer.go:74] Cmd Id: 975ae9150e6f4ea68cf4433078beaee7, Parameter: type_url:"type.googleapis.com/tigergraph.tutopia.common.pb.Bytes" value:"\n\016\010\001\020\001(\352\264\0020\264\357\352\227\006"
0815 14:17:06.502 I event/command_consumer.go:72] Received cmd. Name: ReportKafkaStatus, Id: 27410ae7df274688aa8ecbda8b26a019, CreatedTimeStamp: 1660598226497981866, DeadlineTimeStamp: 1660598241497981866
0815 14:17:06.502 D event/command_consumer.go:74] Cmd Id: 27410ae7df274688aa8ecbda8b26a019, Parameter: type_url:"type.googleapis.com/tigergraph.tutopia.common.pb.Bytes" value:"\n\016\010\001\020\001(\233\277\0020\322\357\352\227\006"
0815 14:17:36.503 I event/command_consumer.go:72] Received cmd. Name: ReportKafkaStatus, Id: f43de4d3ec1e4b76bea4ba69e9782111, CreatedTimeStamp: 1660598256500553578, DeadlineTimeStamp: 1660598271500553578
0815 14:17:36.503 D event/command_consumer.go:74] Cmd Id: f43de4d3ec1e4b76bea4ba69e9782111, Parameter: type_url:"type.googleapis.com/tigergraph.tutopia.common.pb.Bytes" value:"\n\016\010\001\020\001(\325\323\0020\360\357\352\227\006"
0815 14:18:06.507 I event/command_consumer.go:72] Received cmd. Name: ReportKafkaStatus, Id: ab9ec956a69e474badec061c1910142e, CreatedTimeStamp: 1660598286503125103, DeadlineTimeStamp: 1660598301503125103
0815 14:18:06.507 D event/command_consumer.go:74] Cmd Id: ab9ec956a69e474badec061c1910142e, Parameter: type_url:"type.googleapis.com/tigergraph.tutopia.common.pb.Bytes" value:"\n\016\010\001\020\001(\363\345\0020\216\360\352\227\006"
%3|1660598309.386|ERROR|RESTPP-LOADER_1_1_CONSUMER#consumer-3| [thrd:127.0.0.1:30002/bootstrap]: 1/1 brokers are down
0815 14:18:36.508 I event/command_consumer.go:72] Received cmd. Name: ReportKafkaStatus, Id: dafcf8d186c74b0c914b76394789d3e2, CreatedTimeStamp: 1660598316506084710, DeadlineTimeStamp: 1660598331506084710
0815 14:18:36.508 D event/command_consumer.go:74] Cmd Id: dafcf8d186c74b0c914b76394789d3e2, Parameter: type_url:"type.googleapis.com/tigergraph.tutopia.common.pb.Bytes" value:"\n\016\010\001\020\001(\374\366\0020\254\360\352\227\006"
0815 14:19:06.783 I event/command_consumer.go:72] Received cmd. Name: ReportKafkaStatus, Id: 7c1119f88b6141fcbc2bf670cc7d668a, CreatedTimeStamp: 1660598346508993209, DeadlineTimeStamp: 1660598361508993209
0815 14:19:06.783 D event/command_consumer.go:74] Cmd Id: 7c1119f88b6141fcbc2bf670cc7d668a, Parameter: type_url:"type.googleapis.com/tigergraph.tutopia.common.pb.Bytes" value:"\n\016\010\001\020\001(\311\212\0030\312\360\352\227\006"
0815 14:19:36.514 I event/command_consumer.go:72] Received cmd. Name: ReportKafkaStatus, Id: 47769e43e82842d09e26e40a4e4c824f, CreatedTimeStamp: 1660598376511791918, DeadlineTimeStamp: 1660598391511791918
0815 14:19:36.514 D event/command_consumer.go:74] Cmd Id: 47769e43e82842d09e26e40a4e4c824f, Parameter: type_url:"type.googleapis.com/tigergraph.tutopia.common.pb.Bytes" value:"\n\016\010\001\020\001(\315\235\0030\350\360\352\227\006"
0815 14:20:06.517 I event/command_consumer.go:72] Received cmd. Name: ReportKafkaStatus, Id: bec8c5e14e8c4ba2b94f6396668ff41c, CreatedTimeStamp: 1660598406514689473, DeadlineTimeStamp: 1660598421514689473
0815 14:20:06.517 D event/command_consumer.go:74] Cmd Id: bec8c5e14e8c4ba2b94f6396668ff41c, Parameter: type_url:"type.googleapis.com/tigergraph.tutopia.common.pb.Bytes" value:"\n\016\010\001\020\001(\216\254\0030\206\361\352\227\006"
0815 14:20:36.519 I event/command_consumer.go:72] Received cmd. Name: ReportKafkaStatus, Id: 8b0fa802c787451196b10e3665005a1b, CreatedTimeStamp: 1660598436517361946, DeadlineTimeStamp: 1660598451517361946
0815 14:20:36.519 D event/command_consumer.go:74] Cmd Id: 8b0fa802c787451196b10e3665005a1b, Parameter: type_url:"type.googleapis.com/tigergraph.tutopia.common.pb.Bytes" value:"\n\016\010\001\020\001(\350\273\0030\244\361\352\227\006"
0815 14:21:06.522 I event/command_consumer.go:72] Received cmd. Name: ReportKafkaStatus, Id: ae79ee17e4e94f4fb8db0a4aacde9278, CreatedTimeStamp: 1660598466520235088, DeadlineTimeStamp: 1660598481520235088
0815 14:21:06.522 D event/command_consumer.go:74] Cmd Id: ae79ee17e4e94f4fb8db0a4aacde9278, Parameter: type_url:"type.googleapis.com/tigergraph.tutopia.common.pb.Bytes" value:"\n\016\010\001\020\001(\303\317\0030\302\361\352\227\006"
0815 14:21:36.525 I event/command_consumer.go:72] Received cmd. Name: ReportKafkaStatus, Id: ec53474ba9e64c1a9d5f4d1a38da7709, CreatedTimeStamp: 1660598496523220055, DeadlineTimeStamp: 1660598511523220055
0815 14:21:36.525 D event/command_consumer.go:74] Cmd Id: ec53474ba9e64c1a9d5f4d1a38da7709, Parameter: type_url:"type.googleapis.com/tigergraph.tutopia.common.pb.Bytes" value:"\n\016\010\001\020\001(\246\341\0030\340\361\352\227\006"
0815 14:22:06.528 I event/command_consumer.go:72] Received cmd. Name: ReportKafkaStatus, Id: 68b0e14790944ccea9c09f6c63240737, CreatedTimeStamp: 1660598526526180885, DeadlineTimeStamp: 1660598541526180885
0815 14:22:06.528 D event/command_consumer.go:74] Cmd Id: 68b0e14790944ccea9c09f6c63240737, Parameter: type_url:"type.googleapis.com/tigergraph.tutopia.common.pb.Bytes" value:"\n\016\010\001\020\001(\345\360\0030\376\361\352\227\006"
E0815 14:22:34.517230 6503 gbrain_active_address_resolver.cpp:136] [RefreshGSE] errors[0]: 205, msg: No leader is found
.......