IAM role authentication to Databricks AWS staging buckets
If you're on AWS and require authenticating using an IAM role rather than an access key and secret, switch the Authentication Method dropdown to IAM role as shown in this screenshot:
![](https://files.readme.io/9be8c64-databricks-iam.png)
The IAM role must have the following permissions for the staging bucket:
s3:GetObject
s3:PutObject
s3:DeleteObject
s3:ListBucket
s3:GetBucketLocation
Additionally, the role must be assumable by the Polytomic role arn:aws:iam::568237466542:role/convox/prod-polytomic-ServiceRole-1ELGH39L0GCHT
. For on premises ECS deployments the role will be ${prefix}-ecs-task-role
. Contact [email protected] for deployment-specific guidance.
A trust policy that allows both Polytomic and Databricks to assume the role will look like the following:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::568237466542:role/convox/prod-polytomic-ServiceRole-1ELGH39L0GCHT"
},
"Action": "sts:AssumeRole",
"Condition": {
"StringEquals": {
"sts:ExternalId": "0000"
}
}
},
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::414351767826:role/unity-catalog-prod-UCMasterRole-14S5ZJVKOTYTL"
},
"Action": "sts:AssumeRole",
"Condition": {
"StringEquals": {
"sts:ExternalId": "0000"
}
}
}
]
}
Note that the values for sts:ExternalId
are specific to your Polytomic and Databricks accounts.
In addition to configuring Polytomic, you may also need to configure a Storage Credential which Databricks will use when reading data from the staging bucket. See the Databricks documentation for information on creating storage credentials. If your staging bucket is not configured as an External Location in Databricks, you'll need to provide Polytomic with the name of the Storage Credential in the connection configuration.
The Databricks user Polytomic uses will need the following permissions for the external location:
READ FILES
WRITE FILES
CREATE EXTERNAL TABLE
CREATE MANAGED STORAGE
And the following permissions for the storage credential:
READ FILES
WRITE FILES
CREATE EXTERNAL TABLE
Updated 3 months ago