Run loading job through Rest call

GSQL
1

Hi,

I was following the documentation on how to run a loading job through REST api. Here is my call. I have activated rest authentication on my instance.

curl -X POST 'http://localhost:9000/ddl/MyGraph?&tag=loadData'

My job doesn’t expect any parameter as i have hard coded paths to all the files in the job. when i run the job, i get below output without showing other information.

{
  "version":
    {
      "edition":"enterprise",
       "api":"v2",
       "schema":0
    },
  "error":false,
  "message":"",
  "results":[],
  "code":"REST-0000"
}

when i check the the data in the graph, no data was loaded.

Your help is appreciated here.

Khan

2

@Khan Is the file small or large?

If the data size is large, it is better to reference the data filename, using the --data-binary flag:

curl -X POST --data-binary @<data_filename> "http://<server_ip>:9000/ddl/<graph_name>?tag=<job_name>&filename=<filename_variable><&optional_parameters>"

Example:

curl -X POST --data-binary "@D:\temp\out\load.json" "http://localhost:9000/ddl/MyGraph?tag=loadData&filename=file1"```
3

Hi @Khan,

It seems that you cannot load local files (i.e. files uploaded to TigerGraph host machine) when you call the loading job via the REST API.

The reason for this is that you must specify the filename= parameter in the query. Its value must refer to filevar specified by a DEFINE FILENAME statement in your loader job. The REST API request will run the loading job with the USING clause, setting a new value to the referenced filevar. (I assume that this new value will be some temporary directory on the server’s local disk into which the data file has been uploaded by the REST API request). Thus the original hard-coded reference to a local file is lost/ignored.

Furthermore, according to the documentation, if a loading job is run with the USING clause, then only those steps in the loading job will be executed whose filevar is mentioned in the USING clause. Since you can specify only one filename= parameter in the REST API URL, you can load only one file in a single request, and that file will be the one that you just uploaded. The rest of filevars will be ignored and so will be the matching loading steps. The documentation also says that if the USING clause is omitted, then the entire loading job will be run, but you can’t trick the REST API into this, as you cannot omit the filename= parameter.

So, if you have to load a number of different files, you need to call the /ddl/ request once for each. The good news is that you can have a single loading job, all you need to do is to refer to a different filevar each time.

Summary:
If your loading job is something like this:

CREATE LOADING JOB MyJob FOR GRAPH MyGraph {

    DEFINE FILENAME f1;
    DEFINE FILENAME f2;
    DEFINE FILENAME f3;

    LOAD f1
        TO VERTEX v1 VALUES (...)
        USING SEPARATOR = ",";

    LOAD f2
        TO VERTEX v2 VALUES (...)
        USING SEPARATOR = ",";

    LOAD f3
        TO EDGE e1 VALUES (...)
        USING SEPARATOR = ",";

}

then you need to call the REST API three times:

curl -X POST --data-binary @file1.csv "http://localhost:9000/ddl/MyGraph?tag=MyJob&filename=f1"
curl -X POST --data-binary @file2.csv "http://localhost:9000/ddl/MyGraph?tag=MyJob&filename=f2"
curl -X POST --data-binary @file3.csv "http://localhost:9000/ddl/MyGraph?tag=MyJob&filename=f3"

NOTE: you must prefix the filename with “@” otherwise the filename itself will be considered the data to be loaded.

Or you can have separate loading jobs for each of your files.

3 Likes
4

@Szilard_Barany @Jon_Herke is there option to set header=true in the REST API? I currently don’t find it recorded in the documentation. https://docs.tigergraph.com/dev/restpp-api/built-in-endpoints#post-ddl-graph_name-run-a-loading-job

5

Looking into it. Indeed, it’s not mentioned the docs. Asked colleagues about it; will get back to you when I got fedback.

6

I tried header=true, but it is not recognized by the api

7

header presence or absence is defined in the loading job specification. I don’t think there is any way to do this dynamically: https://docs.tigergraph.com/dev/gsql-ref/ddl-and-loading/creating-a-loading-job#conversion-and-run-job-details

8

@Szilard_Barany,

Has any improvements been made regarding the original question? Mainly trying to see if we can trigger a loading job remotely with files local to the TG host machine.

Thanks,
Joe

10

Hi,
In this curl command -d part is missing.

it’s will not respond any, because API will expect -d [data part] part in the curl command, try to pass loading job and file location in -d parts in curl command.

Below approaches worked for me,

consider loading job
use graph MyGraph
CREATE LOADING JOB loadData {

// this below info we going to pass in runtime.		
//DEFINE FILENAME student = $my_gs_datasource_1:gs://folder_1/folder_2/student.csv;

DEFINE FILENAME student ;	
LOAD student TO VERTEX v1 VALUES (...)USING SEPARATOR = ",";

}

Approach 1

curl -H ‘Content-Type: application/json’ -u “<tigergraph_user>:” -X POST ‘http://localhost:14240/gsql/v1/loading-jobs/run?graph=MyGraph’
-d ‘[
{
“name”: “loadData”,
“sys.data_root”: “/tmp”,
“verbose”: true,
“dryrun”: true,
“interval”: 1,
“maxNumError”: 1,
“maxPercentError”: 1,
“dataSources”: [
{
//remove this comment line while running, ex:$my_gs_datasource_1:gs://folder_1/folder_2/student.csv
“filename”: “student”,
“path”: “gs://folder_1/folder_2”,
“name”: “my_gs_datasource_1”
}
]
}
]’

Tip : Try to use Git Bash command line. Windows CMD line also works, but Bash CMD will be more compatible.

Approach 2 - pass JSON info in separate json file.

if it’s complex to pass in -d[data part] in curl, we can put this info in JSON, we can refer those file in curl,
following comment.

curl -H ‘Content-Type: application/json’ -u “<tigergraph_user>:” -X POST ‘http://localhost:14240/gsql/v1/loading-jobs/run?graph=MyGraph’ -d “@D:\my_loading_job_info.json”

this file script looks likes below, without ’ ’ character literal

[
{
“name”: “loadData”,
“sys.data_root”: “/tmp”,
“verbose”: true,
“dryrun”: true,
“interval”: 1,
“maxNumError”: 1,
“maxPercentError”: 1,
“dataSources”: [
{
“filename”: “student”,
“path”: “gs://folder_1/folder_2”,
“name”: “my_gs_datasource_1”
}
]
}
]

TG docs: [GSQL Endpoints run loading job]

As I understand it, the REST API follows basic validation before running, including data parts, even when given at the loading job level.

Sharing my experience - please ignore if anything is wrong!