S3 Loader Permissions

marcosconceicao · May 16, 2022, 12:09pm

Hi
What are the necessary permissions to use the S3 Loader?
I`m using allowing all List/Get commands to our tigergraph buckets, but still getting error

IAM Policy attached to tigergraph AWS user.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:List*",
                "s3:Get*",
                "s3:PutObject"
            ],
            "Resource": [
                "arn:aws:s3:::tigergraph*/*",
                "arn:aws:s3:::tigergraph*"
            ]
        }
    ]
}

Error Log

E@20220516 11:44:15.971 marcosconceicao|127.0.0.1:59570|00000030047 (LoadingUtils.java:67) dataSrc: s3 s1 () configFile is not JSON string
E@20220516 11:44:16.720 marcosconceicao|127.0.0.1:59570|00000030047 (S3ListBucketsFiles.java:87) Failed to connect s3 data source.
E@20220516 11:44:16.720 marcosconceicao|127.0.0.1:59570|00000030047 (S3ListBucketsFiles.java:88) com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: XA7N8ZFJVMDVG3X1; S3 Extended Request ID: DefkXhYmvRKtXHmREgi3oX5htMsYxaoSikl7VOK0/O/N0u9kQ2H9ERTpVw3p/BhkzyY6hzPriqE=), S3 Extended Request ID: DefkXhYmvRKtXHmREgi3oX5htMsYxaoSikl7VOK0/O/N0u9kQ2H9ERTpVw3p/BhkzyY6hzPriqE=
com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: XA7N8ZFJVMDVG3X1; S3 Extended Request ID: DefkXhYmvRKtXHmREgi3oX5htMsYxaoSikl7VOK0/O/N0u9kQ2H9ERTpVw3p/BhkzyY6hzPriqE=), S3 Extended Request ID: DefkXhYmvRKtXHmREgi3oX5htMsYxaoSikl7VOK0/O/N0u9kQ2H9ERTpVw3p/BhkzyY6hzPriqE=
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1701)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1356)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1102)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:759)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:733)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:715)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:675)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:657)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:521)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4705)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4652)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4646)
	at com.amazonaws.services.s3.AmazonS3Client.getS3AccountOwner(AmazonS3Client.java:958)
	at com.amazonaws.services.s3.AmazonS3Client.getS3AccountOwner(AmazonS3Client.java:948)
	at com.tigergraph.schema.b.f.mD(S3ListBucketsFiles.java:83)
	at com.tigergraph.schema.operation.CreateDataSourceOperation.g(CreateDataSourceOperation.java:264)
	at com.tigergraph.schema.operation.CreateDataSourceOperation.f(CreateDataSourceOperation.java:205)
	at com.tigergraph.schema.operation.CreateDataSourceOperation.e(CreateDataSourceOperation.java:138)
	at com.tigergraph.schema.operation.CreateDataSourceOperation.c(CreateDataSourceOperation.java:88)
	at com.tigergraph.schema.operation.CreateDataSourceOperation.ae(CreateDataSourceOperation.java:20)
	at com.tigergraph.schema.operation.CreateOperation.af(CreateOperation.java:78)
	at com.tigergraph.schema.operation.CreateOperation.nz(CreateOperation.java:61)
	at com.tigergraph.schema.operation.MetadataUpdateOperation.nK(MetadataUpdateOperation.java:149)
	at com.tigergraph.schema.operation.BaseOperation.af(BaseOperation.java:51)
	at com.tigergraph.schema.operation.BaseOperation.run(BaseOperation.java:39)
	at com.tigergraph.schema.ast.ddl.create.CreateDataSourceQb.runOnCatalog(CreateDataSourceQb.java:60)
	at com.tigergraph.schema.handler.QueryBlockHandler.a(QueryBlockHandler.java:896)
	at com.tigergraph.schema.handler.QueryBlockHandler.a(QueryBlockHandler.java:192)
	at com.tigergraph.schema.handler.CommandHandler.a(CommandHandler.java:111)
	at com.tigergraph.schema.handler.BaseHandler.handle(BaseHandler.java:332)
	at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:77)
	at jdk.httpserver/sun.net.httpserver.AuthFilter.doFilter(AuthFilter.java:71)
	at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:80)
	at jdk.httpserver/sun.net.httpserver.ServerImpl$Exchange$LinkHandler.handle(ServerImpl.java:692)
	at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:77)
	at jdk.httpserver/sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:664)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
E@20220516 11:44:16.720 marcosconceicao|127.0.0.1:59570|00000030047 (MetadataUpdateOperation.java:151) Failed executeInMemory for CreateDataSourceOperation

Robert_Hardaway · May 16, 2022, 1:39pm

The S3 loader only requires read access to the source bucket, are you able to browse the bucket from the AWS Console UI, using the same IAM user?

marcosconceicao · May 16, 2022, 2:26pm

I’m not using the Console UI with this user, but, with de AWS SDK i’m able to list and get objects from the source bucket.

Robert_Hardaway · May 16, 2022, 5:08pm

can you provide what is in the S3 config file and/or what the DATA_SOURCE statement says (remove the S3 key info), examples are:

CREATE DATA_SOURCE S3 tg_stage = “/home/tigergraph/tg/S3_config.json” FOR GRAPH TGStage
CREATE DATA_SOURCE S3 tg_stage_key = “{“file.reader.settings.fs.s3a.access.key”:“AKIAU3S******************”,“file.reader.settings.fs.s3a.secret.key”:”************************8Zz0K"}" FOR GRAPH TGStage

if you are using the json config file syntax (first example), the json file should contain:

{
“file.reader.settings.fs.s3a.access.key”: “AKIA45*****************”,
“file.reader.settings.fs.s3a.secret.key”: “*********************************XVoMFB”,
“aws.region”:“us-west-1”
}

marcosconceicao · May 16, 2022, 5:49pm

Sure!

CREATE DATA_SOURCE S3 tiger_test = "/home/tigergraph/s3a.config" FOR GRAPH graph_test

{
"file.reader.settings.fs.s3a.access.key": "A*************GZ",
"file.reader.settings.fs.s3a.secret.key": "N***************B"
}

Screen Shot 2022-05-16 at 14.48.25

marcosconceicao · May 16, 2022, 9:45pm

@Robert_Hardaway it’s possible to use an EC2 IAM Role to access the bucket, without the access key/secret?
Actually the instance already has access to the S3.

Robert_Hardaway · May 17, 2022, 12:55am

Not today, there is a feature request with tigergraph engineering to add IAM user authentication in Q3 of 2022