Google Cloud Storage

Polytomic can connect to Google Cloud Storage using a service account key:

1. In the Google Cloud console, go to IAM & Admin → Service Accounts and click Create service account. You don't need to grant it any project-level roles.

2. Grant the service account access to your bucket. In Cloud Storage → Buckets, open your bucket and go to the Permissions tab. Click Grant access, enter the service account's email address, and choose a role:

   • Storage Object Viewer (roles/storage.objectViewer) if Polytomic only reads from the bucket.
   • Storage Object User (roles/storage.objectUser) if Polytomic also writes to the bucket. Polytomic creates, overwrites, and deletes objects when writing.

3. Create and save a JSON service account key file by selecting the service account, then Keys → Add key → Create new key → JSON → Create. Your browser downloads a .json file.

   If your Google Cloud organization enforces the iam.disableServiceAccountKeyCreation policy, an administrator must allow key creation for this project.

4. In Polytomic, go to Connections → Add connection → Google Cloud Storage.

5. Upload your JSON key file from step 3 to Polytomic's Service account key field. Polytomic reads the project ID and service account identity from the key file.

6. Enter your Bucket name. The bucket name may include an optional path to limit Polytomic to a folder within the bucket (for example, my-bucket/exports).

7. Click Save.

Polytomic reads CSV, JSON (including JSON Lines), and Parquet files. Files may also be compressed with .gz, .bz2, or .zst.

Malformed CSVs

Polytomic ignores malformed CSVs. This means that, when using Polytomic to sync from CSV files on Google Cloud Storage, you may see CSVs missing from the source file list if they exhibit malformations like the following:

• Empty column names.
• Duplicate column names.
• Rows with fewer columns than the number of columns in the file header.

If you don't see an expected CSV file listed in Polytomic, please check it for these issues.

Concatenating multiple files into one SQL table

When you sync from Google Cloud Storage with a Bulk Sync, Polytomic can concatenate all CSV, JSON, or Parquet files in your bucket into one table in your data warehouse. To do this, turn on the Files are time-based snapshots setting in your connection configuration:

After you turn on this setting, you must also specify these settings:

• Collection name: the name of the resulting SQL table in your data warehouse.
• File format: the format of the files to concatenate: CSV, JSON, or Parquet.
• CSV files have headers: whether your CSV files have a header row with field names. Enabled by default.
• Skip first lines: the number of lines Polytomic skips at the top of each CSV before reading the header row.

Files spread across multiple directories

Polytomic can discover multiple tables from your bucket's directory structure and add values from the file path as columns. To do this, turn on Files are time-based snapshots and then Multi-directory multi-table. Then enter a Tables glob path such as {{ capture("call_id") }}/{{ table() }}.csv.

Google Cloud Storage supports the same directory patterns as S3. See Syncing from S3 for the pattern syntax.