IAM role authentication to Databricks AWS staging buckets

When using Databricks on AWS with IAM role authentication, you need to configure both Polytomic and Databricks to access your S3 staging bucket. This requires setting up an IAM role with a trust policy that allows both services to assume the role.

Configuring Databricks Storage Credential

When Polytomic stages data in S3, Databricks needs its own credentials to read that data. This is done through a Storage Credential in Unity Catalog.

When You Need a Storage Credential

When using IAM role authentication, Polytomic does not embed AWS credentials in the COPY INTO statements it runs on Databricks. Databricks must therefore have its own way to read the staged files from your S3 bucket. With Unity Catalog, that means the staging bucket must be covered by either:

• an External Location covering the staging path, with READ FILES granted to the principal running the sync, or
• a Storage Credential whose name you provide to Polytomic in the connection configuration.

If neither is configured, syncs fail when loading staged data. See Troubleshooting for the specific error and how to resolve it.

📘

This is separate from the bulk sync External location setting

The connection-level Storage Credential (or a Unity Catalog External Location covering the staging bucket) is what lets Databricks read your staging data, and it is required for IAM role authentication. The External location field in a bulk sync's configuration is a different setting that controls writing the destination data to an external table — leaving it blank does not remove the requirement above.

Creating the Storage Credential

1. In your Databricks workspace, navigate to Catalog Explorer.
2. Click the External Data button, then go to the Credentials tab.
3. Click Create credential.
4. Select AWS IAM Role as the credential type.
5. Enter a name for the credential (e.g., polytomic-staging-credential).
6. Enter the same IAM Role ARN that you configured in Polytomic.
7. Click Create.
8. Copy the External ID that Databricks generates - you'll need this for the trust policy.

For detailed instructions, see the Databricks documentation on creating storage credentials.

Configuring Polytomic

In your Databricks connection configuration:

1. Switch the Authentication Method dropdown to IAM role.
2. Enter the IAM Role ARN that both Polytomic and Databricks will use.
3. Enter the Storage Credential name you created in Databricks, in applicable.
4. Click Save.

After saving, Polytomic will generate an External ID for your connection. You'll need this External ID when configuring the IAM role trust policy.

Configuring the IAM Role

S3 Permissions Policy

The IAM role needs the following permissions for the S3 staging bucket:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "PolytomicAndDatabricksS3Access",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:ListBucket",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:GetBucketLocation",
        "s3:ListBucketMultipartUploads",
        "s3:ListMultipartUploadParts",
        "s3:AbortMultipartUpload"
      ],
      "Resource": [
        "arn:aws:s3:::your-staging-bucket/*",
        "arn:aws:s3:::your-staging-bucket"
      ]
    },
    {
      "Sid": "AllowSelfAssume",
      "Effect": "Allow",
      "Action": "sts:AssumeRole",
      "Resource": "arn:aws:iam::<YOUR-AWS-ACCOUNT-ID>:role/<YOUR-ROLE-NAME>"
    }
  ]
}

Replace:

• your-staging-bucket with your S3 bucket name
• <YOUR-AWS-ACCOUNT-ID> with your AWS account ID
• <YOUR-ROLE-NAME> with the name of this IAM role

Note: The self-assume statement is required by Databricks for storage credentials. For more information, see the AWS blog post on IAM role trust policy updates.

Trust Policy for Dual Access

The trust policy must allow both Polytomic and Databricks to assume the role. This requires including both services' AWS principals and both External IDs:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::568237466542:root",
          "arn:aws:iam::414351767826:role/unity-catalog-prod-UCMasterRole-14S5ZJVKOTYTL",
          "arn:aws:iam::<YOUR-AWS-ACCOUNT-ID>:role/<YOUR-ROLE-NAME>"
        ]
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": [
            "<POLYTOMIC-EXTERNAL-ID>",
            "<DATABRICKS-EXTERNAL-ID>"
          ]
        }
      }
    }
  ]
}

Replace:

• <YOUR-AWS-ACCOUNT-ID> with your AWS account ID
• <YOUR-ROLE-NAME> with the name of this IAM role (for self-assuming)
• <POLYTOMIC-EXTERNAL-ID> with the External ID from your Polytomic connection
• <DATABRICKS-EXTERNAL-ID> with the External ID from your Databricks storage credential

Principal explanations:

• arn:aws:iam::568237466542:root - Polytomic's AWS account
• arn:aws:iam::414351767826:role/unity-catalog-prod-UCMasterRole-14S5ZJVKOTYTL - Databricks Unity Catalog's AWS role
• arn:aws:iam::<YOUR-AWS-ACCOUNT-ID>:role/<YOUR-ROLE-NAME> - Self-assuming capability (required by Databricks)

See the Databricks documentation for the correct service principal for GovCloud.

Alternative: More Restrictive Polytomic Principal

For additional security, you can restrict Polytomic's access to use a specific role instead of the root account:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::568237466542:role/convox/prod-polytomic-ServiceRole-1ELGH39L0GCHT",
          "arn:aws:iam::414351767826:role/unity-catalog-prod-UCMasterRole-14S5ZJVKOTYTL",
          "arn:aws:iam::<YOUR-AWS-ACCOUNT-ID>:role/<YOUR-ROLE-NAME>"
        ]
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": [
            "<POLYTOMIC-EXTERNAL-ID>",
            "<DATABRICKS-EXTERNAL-ID>"
          ]
        }
      }
    }
  ]
}

Troubleshooting

"User does not have permission SELECT on any file" (SQLSTATE 42501)

If a sync fails while loading staged data with:

[INSUFFICIENT_PERMISSIONS] Insufficient privileges:
User does not have permission SELECT on any file. SQLSTATE: 42501

Databricks cannot read the staged files from your S3 bucket. Under IAM role authentication, Polytomic does not pass AWS credentials in the COPY INTO statement, so the staging bucket must be covered by a Unity Catalog External Location (with READ FILES granted to the principal running the sync) or by a Storage Credential whose name is set on the Polytomic connection. To resolve:

• Enter the Storage Credential name in the Polytomic connection configuration, or
• Register an External Location in Databricks covering the staging bucket path and grant READ FILES on it.

Note this is unrelated to the bulk sync External location field, which controls writing to external tables, not reading staged data.

Storage Credential Validation Fails

If Databricks reports that the storage credential validation failed:

• Ensure the IAM role trust policy includes the Databricks Unity Catalog principal
• Verify the External ID matches what Databricks generated
• Confirm the self-assuming statement is present in both the trust policy and permissions policy

Polytomic Cannot Assume Role

If Polytomic cannot connect:

• Verify the Polytomic principal is in the trust policy
• Ensure the External ID from Polytomic is included
• Check that the IAM role has the required S3 permissions

"Non-self-assuming role" Error

As of January 2025, Databricks blocks storage credentials that don't use self-assuming IAM roles. Ensure your trust policy includes the self-assuming principal: arn:aws:iam::<YOUR-AWS-ACCOUNT-ID>:role/<YOUR-ROLE-NAME>

Additional Resources

• Polytomic: Using AWS IAM roles to access S3 buckets
• Databricks: Create a storage credential for AWS S3
• Databricks: External Locations