How To Continuously Feed Off Of Multiple Topics

TigerGraph Platform
#1

Hi all,

I have a few questions about multiple Kafka loaders running in TigerGraph.

Is it that currently only one Kafka loader can run per node?

I’ve tried starting multiple Kafka load jobs on a single node, but it seems that the second can only start after the first is complete and so on.

Basically my use case is that I gave several different vertex/edge types that each have their own topics,

and I would always like to have a Kafka loader running for each.

Is that a common thing or is it not a good practice regarding TigerGraph?

If it is not advisable to have multiple Kafka loaders always running continuously, then is a more orchestrated

approach the usual solution? One where load jobs are periodically kicked off by a CI/CD server or something?

In that case, how can I make sure that when a specific load job is done, that when is starts back up later, that it starts at the offset it left off at?

#2

Hi Brett,

There is a Kafka config for concurrent Kafka loader numbers.

  1. gadmin --config kafka

  2. change KAFKA-LOADER.count to a larger value

  3. save

  4. gadmin config-apply

  5. gadmin restart restpp -fy

  6. restart the loading jobs.

Thanks.