EXE and CTRL always panic

bgdsh · March 29, 2023, 9:10am

controller panic output:

fatal error: startm: p has runnable gs

runtime stack:
runtime.throw(0x1164acf, 0x19)
        /home/graphsql/.glocal/go/go/src/runtime/panic.go:1117 +0x72
runtime.startm(0x0, 0x100000001)
        /home/graphsql/.glocal/go/go/src/runtime/proc.go:2392 +0x2a5
runtime.wakep()
        /home/graphsql/.glocal/go/go/src/runtime/proc.go:2477 +0x66
runtime.ready(0xc000336480, 0x4, 0xc000915d01)
        /home/graphsql/.glocal/go/go/src/runtime/proc.go:790 +0xb6
runtime.goready.func1()
        /home/graphsql/.glocal/go/go/src/runtime/proc.go:347 +0x38
runtime.systemstack(0x7fc914000020)
        /home/graphsql/.glocal/go/go/src/runtime/asm_amd64.s:379 +0x66
runtime.mstart()
        /home/graphsql/.glocal/go/go/src/runtime/proc.go:1246

goroutine 148 [running]:
runtime.systemstack_switch()
        /home/graphsql/.glocal/go/go/src/runtime/asm_amd64.s:339 fp=0xc000915d10 sp=0xc000915d08 pc=0x471a80
runtime.goready(0xc000336480, 0x4)
        /home/graphsql/.glocal/go/go/src/runtime/proc.go:346 +0x5c fp=0xc000915d40 sp=0xc000915d10 pc=0x43cf1c
runtime.send(0xc0006cb620, 0xc0005d4d20, 0xc000915ef8, 0xc000915dd8, 0x3)
        /home/graphsql/.glocal/go/go/src/runtime/chan.go:320 +0x85 fp=0xc000915d70 sp=0xc000915d40 pc=0x407b65
runtime.chansend(0xc0006cb620, 0xc000915ef8, 0xd5d801, 0xd596fa, 0x56)
        /home/graphsql/.glocal/go/go/src/runtime/chan.go:210 +0x565 fp=0xc000915df8 sp=0xc000915d70 pc=0x407945
runtime.chansend1(0xc0006cb620, 0xc000915ef8)
        /home/graphsql/.glocal/go/go/src/runtime/chan.go:143 +0x35 fp=0xc000915e30 sp=0xc000915df8 pc=0x4073d5
github.com/Shopify/sarama.(*Broker).responseReceiver(0xc000261c00)
        /home/graphsql/product/src/cqrs/.thirdparty/pkg/mod/github.com/!shopify/sarama@v1.26.4/broker.go:893 +0x59a fp=0xc000915fa0 sp=0xc000915e30 pc=0xd596fa
github.com/Shopify/sarama.(*Broker).responseReceiver-fm()
        /home/graphsql/product/src/cqrs/.thirdparty/pkg/mod/github.com/!shopify/sarama@v1.26.4/broker.go:843 +0x33 fp=0xc000915fb8 sp=0xc000915fa0 pc=0xdaa113
github.com/Shopify/sarama.withRecover(0xc0006a6120)
        /home/graphsql/product/src/cqrs/.thirdparty/pkg/mod/github.com/!shopify/sarama@v1.26.4/utils.go:43 +0x49 fp=0xc000915fd8 sp=0xc000915fb8 pc=0xda3889
runtime.goexit()
        /home/graphsql/.glocal/go/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc000915fe0 sp=0xc000915fd8 pc=0x4738c1
created by github.com/Shopify/sarama.(*Broker).Open.func1
        /home/graphsql/product/src/cqrs/.thirdparty/pkg/mod/github.com/!shopify/sarama@v1.26.4/broker.go:211 +0x886

executor panic output:

fatal error: found pointer to free object

runtime stack:
runtime.throw(0xfa163f, 0x1c)
        /home/graphsql/.glocal/go/go/src/runtime/panic.go:1117 +0x72
runtime.(*mspan).reportZombies(0x7ff7546aa2a0)
        /home/graphsql/.glocal/go/go/src/runtime/mgcsweep.go:614 +0x376
runtime.(*mspan).sweep(0x7ff7546aa2a0, 0xc000600900, 0x0)
        /home/graphsql/.glocal/go/go/src/runtime/mgcsweep.go:447 +0x473
runtime.(*mcentral).uncacheSpan(0x1717c98, 0x7ff7546aa2a0)
        /home/graphsql/.glocal/go/go/src/runtime/mcentral.go:214 +0xcb
runtime.(*mcache).releaseAll(0x7ff7866ae5b8)
        /home/graphsql/.glocal/go/go/src/runtime/mcache.go:276 +0x136
runtime.(*mcache).prepareForSweep(0x7ff7866ae5b8)
        /home/graphsql/.glocal/go/go/src/runtime/mcache.go:310 +0x46
runtime.acquirep(0xc00004a800)
        /home/graphsql/.glocal/go/go/src/runtime/proc.go:4973 +0x3d
runtime.stopm()
        /home/graphsql/.glocal/go/go/src/runtime/proc.go:2302 +0xab
runtime.findrunnable(0xc00004d000, 0x0)
        /home/graphsql/.glocal/go/go/src/runtime/proc.go:2960 +0x72e
runtime.schedule()
        /home/graphsql/.glocal/go/go/src/runtime/proc.go:3169 +0x2d7
runtime.park_m(0xc00038e900)
        /home/graphsql/.glocal/go/go/src/runtime/proc.go:3318 +0x9d
runtime.mcall(0x800000)
        /home/graphsql/.glocal/go/go/src/runtime/asm_amd64.s:327 +0x5b

I am running the version 3.7.0 with free trial license.
Does anyone know what the root cause is?

bgdsh · March 29, 2023, 9:17am

more executor panic log:

fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?)

runtime stack:
runtime.throw(0xfc7540, 0x3e)
        /home/graphsql/.glocal/go/go/src/runtime/panic.go:1117 +0x72 fp=0x7f7dfd7f9bb0 sp=0x7f7dfd7f9b80 pc=0x43a1f2
runtime.badPointer(0x7f7e9463d4b8, 0xc000982429, 0x0, 0x0)
        /home/graphsql/.glocal/go/go/src/runtime/mbitmap.go:351 +0x235 fp=0x7f7dfd7f9bf8 sp=0x7f7dfd7f9bb0 pc=0x4175b5
runtime.findObject(0xc000982429, 0x0, 0x0, 0xc000180400, 0x7f7e9ed7c438, 0x1)
        /home/graphsql/.glocal/go/go/src/runtime/mbitmap.go:387 +0x9b fp=0x7f7dfd7f9c30 sp=0x7f7dfd7f9bf8 pc=0x41767b
runtime.wbBufFlush1(0xc000077800)
        /home/graphsql/.glocal/go/go/src/runtime/mwbbuf.go:260 +0xa8 fp=0x7f7dfd7f9c88 sp=0x7f7dfd7f9c30 pc=0x434ba8
runtime.gcMarkDone.func1.1(0xc000077800)
        /home/graphsql/.glocal/go/go/src/runtime/mgc.go:1527 +0x2b fp=0x7f7dfd7f9ca0 sp=0x7f7dfd7f9c88 pc=0x46956b
runtime.runSafePointFn()
        /home/graphsql/.glocal/go/go/src/runtime/proc.go:1711 +0x5d fp=0x7f7dfd7f9cb8 sp=0x7f7dfd7f9ca0 pc=0x43fddd
runtime.findrunnable(0xc000077800, 0x0)
        /home/graphsql/.glocal/go/go/src/runtime/proc.go:2612 +0x1115 fp=0x7f7dfd7f9de0 sp=0x7f7dfd7f9cb8 pc=0x442d35
runtime.schedule()
        /home/graphsql/.glocal/go/go/src/runtime/proc.go:3169 +0x2d7 fp=0x7f7dfd7f9e48 sp=0x7f7dfd7f9de0 pc=0x443697
runtime.park_m(0xc000683b00)
        /home/graphsql/.glocal/go/go/src/runtime/proc.go:3318 +0x9d fp=0x7f7dfd7f9e78 sp=0x7f7dfd7f9e48 pc=0x443c1d
runtime.mcall(0x800000)
        /home/graphsql/.glocal/go/go/src/runtime/asm_amd64.s:327 +0x5b fp=0x7f7dfd7f9e88 sp=0x7f7dfd7f9e78 pc=0x4721bb

bgdsh · March 29, 2023, 9:25am

Tons of errors in RestApp log:

E0328 18:36:33.589854 272525 dispatcher.cpp:512] Error parse DeltaWatermarkByte_:
 10 00 03 00 11 33 5f 31 35 32 36 30 36 30 39 5f 32 31 38 34 5f 30 94 02 02 0f 31 35 32 36 30 36 30 39 5f 32 31 38 34 5f 30 02 42 30 78 61 61 66 32 36 31 36 33 66 35 61 31 31 33 32 31 65 37 38 36 33 36 66 62 61 61 38 38 63 38 35 65 35 34 62 33 61 63 38 64 38 33 63 62 34 31 32 61 38 39 39 31 35 32 62 65 31 38 37 30 39 30 39 31 02 81 ef c6 0e 02 2a 30 78 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 02 2a 30 78 38 61 30 31 66 61 35 61 37 37 33 31 31 62 62 63 66 32 39 65 32 39 33 64 38 65 63 62 34 38 37 30 37 63 66 64 62 37 30 30 02 00 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 00 02 2a 30 78 39 32 39 38 33 32 62 31 66 31 35 31 35 63 66 30 32 63 39 35 34 38 61 30 66 66 34 35 34 66 31 62 30 65 32 31 36 62 31 38 02 06 32 36 34 31 30 36 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

bgdsh · March 29, 2023, 11:25am

Fatal error in RestApp:

fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?)

runtime stack:
runtime.throw(0x7f0c6606912b, 0x3e)
	/home/graphsql/.glocal/go/go/src/runtime/panic.go:1117 +0x74 fp=0x7f0c06ff9110 sp=0x7f0c06ff90e0 pc=0x7f0c64be6774
runtime.badPointer(0x7f0c5facae90, 0xc0003230e0, 0xc000078800, 0xed0)
	/home/graphsql/.glocal/go/go/src/runtime/mbitmap.go:351 +0x237 fp=0x7f0c06ff9158 sp=0x7f0c06ff9110 pc=0x7f0c64bc3877
runtime.findObject(0xc0003230e0, 0xc000078800, 0xed0, 0x0, 0x0, 0x0)
	/home/graphsql/.glocal/go/go/src/runtime/mbitmap.go:387 +0x9b fp=0x7f0c06ff9190 sp=0x7f0c06ff9158 pc=0x7f0c64bc393b
runtime.scanobject(0xc000078800, 0xc00004f698)
	/home/graphsql/.glocal/go/go/src/runtime/mgcmark.go:1286 +0x257 fp=0x7f0c06ff9220 sp=0x7f0c06ff9190 pc=0x7f0c64bcfaf7
runtime.gcDrain(0xc00004f698, 0x2)
	/home/graphsql/.glocal/go/go/src/runtime/mgcmark.go:1048 +0x245 fp=0x7f0c06ff9278 sp=0x7f0c06ff9220 pc=0x7f0c64bcf285
runtime.gcBgMarkWorker.func2()
	/home/graphsql/.glocal/go/go/src/runtime/mgc.go:1999 +0x12b fp=0x7f0c06ff92b8 sp=0x7f0c06ff9278 pc=0x7f0c64c132eb
runtime.systemstack(0x7f0c07000000)
	/home/graphsql/.glocal/go/go/src/runtime/asm_amd64.s:379 +0x63 fp=0x7f0c06ff92c0 sp=0x7f0c06ff92b8 pc=0x7f0c64c1b423
runtime.mstart()
	/home/graphsql/.glocal/go/go/src/runtime/proc.go:1246 fp=0x7f0c06ff92c8 sp=0x7f0c06ff92c0 pc=0x7f0c64bebb00

Jon_Herke · March 29, 2023, 4:08pm

@bgdsh Can you explain the steps leading up to the errors mentioned previously?

bgdsh · March 30, 2023, 4:19am

After installing it on my server, the EXE will randomly crash. If I load data to tigergraph heavily, the CTRL and EXE will crash simultaneously.
Below is my latest example of crash, I shows connection timeout to ZK, but I also find so many errors in ZK logs like this

2023-03-30 06:05:28,102 [myid:1] - WARN  [NIOWorkerThread-6:NIOServerCnxn@364] - Unexpected exception
EndOfStreamException: Unable to read additional data from client, it probably closed the socket: address = /127.0.0.1:57558, session = 0x10009ef2452000c
	at org.apache.zookeeper.server.NIOServerCnxn.handleFailedRead(NIOServerCnxn.java:163)
	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:326)
	at org.apache.zookeeper.server.NIOServerCnxnFactory$IOWorkRequest.doWork(NIOServerCnxnFactory.java:522)
	at org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:154)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)

tigergraph@nvme-tigergraph-02:~$ gadmin status
+--------------------+-------------------------+-------------------------+
|    Service Name    |     Service Status      |      Process State      |
+--------------------+-------------------------+-------------------------+
|       ADMIN        |      StatusUnknown      |      StateUnknown       |
|        CTRL        |      StatusUnknown      |      StateUnknown       |
|        DICT        |      StatusUnknown      |      StateUnknown       |
|        ETCD        |      StatusUnknown      |      StateUnknown       |
|        EXE         |          Down           |         Stopped         |
|        GPE         |      StatusUnknown      |      StateUnknown       |
|        GSE         |      StatusUnknown      |      StateUnknown       |
|        GSQL        |      StatusUnknown      |      StateUnknown       |
|        GUI         |      StatusUnknown      |      StateUnknown       |
|        IFM         |      StatusUnknown      |      StateUnknown       |
|       KAFKA        |      StatusUnknown      |      StateUnknown       |
|     KAFKACONN      |      StatusUnknown      |      StateUnknown       |
|    KAFKASTRM-LL    |      StatusUnknown      |      StateUnknown       |
|       NGINX        |      StatusUnknown      |      StateUnknown       |
|       RESTPP       |      StatusUnknown      |      StateUnknown       |
|        TS3         |      StatusUnknown      |      StateUnknown       |
|      TS3SERV       |      StatusUnknown      |      StateUnknown       |
|         ZK         |      StatusUnknown      |      StateUnknown       |
+--------------------+-------------------------+-------------------------+
tigergraph@nvme-tigergraph-02:~$ gadmin start all
[   Info] Starting EXE
[   Info] Starting CTRL
[   Info] Starting ZK ETCD DICT KAFKA ADMIN GSE NGINX GPE RESTPP KAFKASTRM-LL KAFKACONN TS3SERV GSQL TS3 IFM GUI
[  Error] Timeout (The StartExecutable cmd execution gets error in instance EXE_1; Timeout(1m0s) when Waiting executable ZK#1:check_ready to finish)
tigergraph@nvme-tigergraph-02:~$ gadmin start all
[   Info] Starting EXE
[   Info] Starting CTRL
[   Info] Starting ZK ETCD DICT KAFKA ADMIN GSE NGINX GPE RESTPP KAFKASTRM-LL KAFKACONN TS3SERV GSQL TS3 IFM GUI
[  Error] Timeout (The StartExecutable cmd execution gets error in instance EXE_1; Timeout(1m0s) when Waiting executable ZK#1:check_ready to finish)

bgdsh · March 30, 2023, 4:20am

I have checked the hardware so many times, no error can be found.

bgdsh · March 30, 2023, 4:35am

Here are the whole log and output of EXE from start to crash:

Log: https://drive.google.com/file/d/1YDh9U2LpDwhW0-7-eT45qIBEHGv336Jx/view?usp=share_link
Output: https://drive.google.com/file/d/1YDh9U2LpDwhW0-7-eT45qIBEHGv336Jx/view?usp=share_link