Kafka Topic Metadata Timeout

Hi,

It seems that there is a timeout of only 3 seconds when starting a Kafka load job for the job

to fetch topic metadata. I think there is a lot of topic metadata, 3 seconds might not be enough time.

How do I override that timeout? I didn’t see any option for it for RUN LOADING JOB.

I get no log output file either, just this:

GSQL > @load_jobs/v_id_load_job.gsql
Using graph ‘cool_data’
The job kafka_v_id_load is dropped!
The job kafka_v_id_load is created.
Try to list topic metadata from Kafka broker ‘kool-kafka.com:443’, timeout: 3 sec …
Timeout expired while fetching topic metadata
GSQL >

I know there is nothing wrong with the broker because that is getting used everywhere at our company.

The broker has SSL, so I have those props as well in the kafka config:

{
“broker”: “kool-kafka.com:443”,
“kafka_config”: {
group.id”: “some-group”,
“ssl.protocol”: “TLSv1.2”,
“ssl.truststore.location”: “some_truststore.jks”,
“ssl.truststore.password”: “some_password”,
“ssl.keystore.location”: “some_keystore.jks”,
“ssl.keystore.password”: “some_password”,
“ssl.key.password”: “some_password”
}
}

the topic config I’m using is:

{
“topic”: “kool.kafka.tg-vertex-id-type”,
“default_start_offset”: -2
}

and the load job is:

USE GRAPH cool_data

DROP DATA_SOURCE k1
CREATE DATA_SOURCE KAFKA k1 FOR GRAPH cool_data
SET k1 = “/some/path/kafka_config.json”

DROP JOB kafka_v_id_load

define the loading job

CREATE LOADING JOB kafka_v_id_load FOR GRAPH cool_data {
DEFINE FILENAME f1 = “$k1:/some/path/v_id_topic_config.json”;
LOAD f1
TO VERTEX Id VALUES ($0, $1, $2)
USING SEPARATOR="|";
}

load the data

RUN LOADING JOB kafka_v_id_load USING EOF=“true”

kool-kafka.com:443 is a load balancer F5, and the actual keystore password has a + and / in it, do those maybe need to be escaped?

Anyways, I think there might be a lot of metadata, and it can’t be obtained in 3 seconds before timing out,

is there any way to increase the timeout?

The timeout could just be because the properties are wrong and the F5 is not actually getting connected to?

If so, it would be nice to know with some form of error message explaining the host couldn’t be connected to.

Another observation, when I set security.protocol to SSL, the job will start (but fail),

And I get an error log. In the error log I get: StartService failed|KafkaGroupReader|Set ‘security.protocol’ in kafka conf with value ‘SSL’ failed with error: Invalid value for configuration property “security.protocol”

Is SSL not even supported in the TigerGraph Kafka loader?

This is the Enterprise version.

Kafka SSL is not supported in the current version.

Thanks.

That is good to know, thanks.

Any idea when SSL will be supported?

This feature will be supported in our 3.1 release. Which might be released in the next half of 2020.

Any updates on Kafka SSL? i cannot find any reference of Kafka SSL in 3.1 documentation.