Release 2021.12.23

New Features

• Add support for replacing Postgresql tables during sync execution
• Improve speed for syncs to Snowflake
• A maximum sync duration may be set via the environment

Fixes

• Display object values correctly in Query Runner results
• "contains" filters for BigQuery models resulted in an error in some situations

New Features

• Add support for reading Dialpad call logs
• Add support for syncing Airtable attachment fields
• Add support for using Airtable formula fields in identity mappings

Fixes

• Improved handling of Zendesk associations
• Increased performance for syncs to Snowflake destinations
• Invalid reference errors from Salesforce were not exposed sometimes
• Sync failure notifications could be sent without an error shown
• Some read-only fields were excluded from the sync record logs
• Clustering keys are set when creating Snowflake tables
• Newly created MySQL tables are prefixed with polytomic rather than being
  created in a schema

Fixes

• Don't fail partially successful BigQuery jobs

• More efficient fetching of syncs list

• Fix Snowflake batch inserter

• Report an error if all records in SQL job fail

• Support Snowflake VARIANT/ARRAY columns for create/update

• Explicitly report timeouts to user rather than silently failing

• Reduce over-fetching of SQL schemas

• Correctly handle null arrays coming from Snowflake

New features

• Model filters for Cosmos DB

Fixes

• More efficient Snowflake querying (async mode)

• Fix Snowflake writes for arrays and JSON objects for replace syncs

• Allow manual syncs to be selected for sync-after

• More stability in the face of errors when writing to SQL databases

• Fix 'Last modified' to only display on saved models

• Fix memory leak on sync status page

• Exclude read-only types from Airtable destinations

• Performance improvements for sync history page

New features

• Show 'Last modified by' info on models and syncs

Fixes

• Surface failures from Smartsheet jobs

• Fix duplicate header in record logs

• Correctly format dates for Google Sheet exports

• Fix bug with connecting to Airtable bases

• Improve performance of API model editor

• Fix bug with per-field 'create-only' mode when applied to HubSpot destinations

• Fix bug with some date formats when writing to Salesforce

Fixes

• More robust handling of Affinity source data

• More robust writing to SQL databases when some records fail

• Correctly handle embedded commas when syncing Google Sheets to Snowflake

• Show 'Run after' syncs list on hover on syncs page

• Remove 'Browse' button from saved Google Sheets connections

• Enforce model refresh if query changes

• Robust handling of large (300k+) simultaneous record errors when syncing

• Decode Hubspot array fields correctly

• More efficient browser memory usage when viewing sync history page

• Faster load time in Query Runner for large schemas

• Fix occassional 'unreachable fields' display bug in sync config

New features

• SQL Runner supports SOQL queries

• Add 'Contains' model filter with arrays

• Faster performance of SQL runner schema display with many tables

• Add body parameter for API source models on POST

Fixes

• Handle Hubspot array fields correctly

• Improve memory pressure on sync history page

• Correctly format dates for snowflake-destination replace syncs

• Fix occasional 'unreachable fields' display bug

New Features

• Ability to define models from SOQL queries

• Allow type overrides on API models

• When writing to databases, automatically generate target column when new model field is added to sync

• Add support for notifications on sync warnings

• Record duplicate identity values as sync warnings

Fixes

• More robust handling of nil values in Affinity

• More robust handling of Airtable dates

• Fix bug where per-field sync mode overrides weren't getting saved

• Migrate to new HubSpot OAuth scopes

• Remove limit parameter from API fieldsets

• More robust handling of empty identity values

• User setting toggle would sometimes be out of date

• Do not fail sync on missing identity value

• API sources now terminates pagination on duplicate results

• Support datetime strings where offset is +/-hhmm (#1465)

New Features

• New source integration: Databricks

• New destination integration: Smartsheet

• Log Salesforce inserted and updated records

• Log duplicate Salesforce records as warnings encountered during sync

• Allow use of readonly Salesforce fields in identity mapping

• Support for querying MongoDB

• Support for ARRAY and BIGNUMERIC types in BigQuery

Fixes

• Sync history record log now downloads as CSV rather then opening in browser as plaintext

• Fix error when saving sync with null overrides

• Fix connection error when connecting to Freshdesk

• Faster querying of Google Sheets with many empty rows

New Features

• Enable string target filters for Salesforce

• Enable logging to S3 of total records synced

• Support for Microsoft 365 SSO

Fixes

• Performance improvement for displaying large number of tables in SQL database model editor's table selector

• On prem deploys can specify valid HTTP origins

• Fail syncs with invalid configurations

• Do not fail when a single record cannot be encoded

• Performance improvement for large syncs to BigQuery

New Features

• CSV connections and models support headers and query params

• Expose non-company entities for Harmonic

Fixes

• Filter duplicate model mappings from override list

• Better performance for dropdown menus

• Fix rare case where a sync fails but still displays 'Running'

• Better displaying of long strings in banner alerts

New Features

• Partial sync failure notifications

• Syncing people objects from Harmonic

• Support for generating models from SQL views

• Expose Chargebee payment source fields in model editor

Fixes

• More edge-case stability for Salesforce syncs

• Fix parse error for SQL queries that contain URLs with a '?' character

• Fix Chargebee edge case type errors

• Smoother UI flow for Okta SAML

• Fix occasional display of field ID rather than field name in sync preview

New Features

• Okta SAML login support

• Much faster syncs from Stripe

• Show errored record details for HubSpot sync errors

• Expose Stripe Invoice and Invoice Line Item objects

Fixes

• More thorough exposure of available fields for API models

• More stability for HubSpot sync edge cases

Fixes

• Disable Salesforce composite API when a batch size is set

New Features

• New sources: AWS Athena, Azure Cosmos DB, Chargebee, Harmonic

• New destination: Snowflake

• Allow more than one sync override for the same model field

• Faster interactions and saves for many-field models in model editor

• When writing to SQL databases, Polytomic's auto-generated schema now includes created_at and updated_at columns

Fixes

• Sort sync list by destination object name

• Improve saving performance on model config

• Optimisations for HubSpot syncs

• Include Polytomic client ID when making Salesforce requests

Fixes

• Correct set HubSpot associations in more edge cases.

• BigQuery schema generation now maps Arrays and Objects to String instead of Bytes.

New Features

• New sources: AWS Athena, Chargebee

• Add ability to specify identity value in sync preview

• Add ability to specify target filters (and filter logic) in addition to existing model filters

• More efficient SFDC syncs

• Show SQL database types in model editor

• Include the execution type with webhook payloads

• Add automatic verification for JSON path in API models

Fixes

• Correct mapping of HubSpot associations in some cases

• More robustness in the face of Google Sheet rate limits

• Sort SQL table names in model editor dropdown

• Enforce a platform limit for Salesforce calls

• Fix mistaken identity values in some HubSpot cases

New Features

• New destination: Segment

• Ability to add sync filters and overrides for API model fields

• Ability to add filter logic for target fields, not just source ones

• Ability to use datetime expressions in string constants

• Ability to specify a minutes-threshold environment variable LONGSYNC_THRESHOLD for long-running-sync alerts

• Add "Only sync if different" option for Salesforce syncs

• Field-level create-only mode now will also write data if the record exists in the destination but the corresponding field is empty

• Remove default last-24-hours filter on sync history page

• Support for connecting to MySQL via SSH tunnel

Fixes

• Fix URL to download error log for on-prem

• Set a sync as failed if maximum number of retries is exceeded

• Allow Marketo jobs to retry indefinitely

New Features

• Add ability to specify a constant string value in sync mappings

• Add ability to override string values in sync mappings with user-entered value

• Add ability to write to MySQL and PostgreSQL with auto-generated schemas

Fixes

• History for continuous syncs now only lists syncs with non-zero updates

New Features

• New source: Azure Synapse

• Sync filter predicates for strings, numbers, and dates

• Sync filter logic

• Filters for sync history page

• Much faster loading of Snowflake tables in model editor

• In-app documentation link

Fixes

• Fix bug when syncing to Pardot checkboxes

• More detailed error reporting for Salesforce edge cases

Fixes

• "Write null values" setting was not respected for Salesforce syncs in some
  situations

Fixes

• Changed records could be missed in some edge cases

New Features

• New destination: MySQL

• Sync diffs now only consider the fields in a sync mapping rather than all fields in parent models

• New sync schedule option: run after other syncs complete

• Expose Hubspot enumerations as target values

• Add support for Hubspot Tickets

• Add additional types to Salesforce models (picklist, address, int)

Fixes

• More generous Snowflake timeout when checking its connection

• More generous SQL Runner timeouts for long-running queries

• Fix SQL Runner scrollbar bug on Chrome for macOS

• Fix SQL Runner empty-row results for BigQuery

• Validate unreachable fields and invalid filters on sync save

• More efficient connection health checks

• Unadulterated display of Byte[] types from BigQuery

• Expose Affinity ID for Person, Org identity

• Affinity rate-limiting improvements

• Fix weekly sync Sunday display bug

• Move Google Sheet tab selection to model config

• Extend Snowflake timeout when checking a connection

• More efficient handling of simultaneous SQL database connections

• More efficient handling of HubSpot rate limits

• More efficient handling of Pardot rate limits

• Normalize case of email addresses

• Format empty arrays for Hubspot

• Add additional types to salesforce mapping

• Syncs page now orders them deterministically

• Clearer error messages for API sources

New Features

• New-user onboarding flow

• New source: CSV URL

• New destinations: BigQuery, Klaviyo, and Google Ads

• Add field-detail view to filters

• Support non-equality filters on Stripe invoices

• More efficient Salesforce syncing

• More efficient syncing from Stripe

Fixes

• More stable Salesforce token refreshes

• More accurate detection of Salesforce external-ID errors

• Fix some cases where filtering on Salesforce fields was not applied

• Use correct encoding for Salesforce null values when mixing Bulk v2 and composite APIs

• Correctly handle Salesforce fields without labels

• Join target field in model query editor now also displays its model

• Show connected user's email for Google Sheets connections

• Show sync error count while hovering over partial-error icon

• Pardot bugfixes

• Sync history duration was inaccurate in some cases

• Sanitize API source pagination

• Support parameterizing arguments on API sources

• Add showFieldDetails to sync filters

• Only short-circuit processing w/no filters set

• Set IsAssociation for all SFDC relationship ids

• Sync summary mappings view now shows field details on hover

• Show error message on unreachable sync filter fields

New Features

• New destination: Pardot

• Display SFDC and Marketo API count even when user has not set a limit

• Less SFDC API usage

Fixes

• More efficient syncing from Stripe Customer record

• Show response status code in webhook request log

• Fix cases where error counts would be inaccurate

• Fix cases where completed time would not update for SFDC Bulk v1

• Store full request URL with webhook logs

• Remove duplicate Lead errors if 'Skip converted' is set

Fixes

• Correctly enable Salesforce integration based on configuration

New Features

• Expose calculated fields on Salesforce destinations

• Support boolean destination field filters for Salesforce

• Hide unreachable fields in sync mappings

• Use connection logos for sync summary field logos

• Add field API name to sync 'Show details' view

• Enable request logs for webhook syncs

• Ability to add metadata to webhook syncs

• Readable errors for failed HTTP API-source syncs

Fixes

• More stability for Stripe joins

• Nulls now copied correctly for SFDC Bulk V1 batches

• Security improvements

• Sync errors now consistently surface during retries

• Correctly display nested BigQuery schemas in table picker

• Decreased network load per connection

• Show local field label in "Inbound joins" UI entry

• Hide sync filters if mappings are empty

• Uploads to Marketo were erroneously marked as failed, regardless of final
  status.

• Log job & failure information from Marketo

New Features

• New destination: Freshdesk

• Ability to clone syncs

• Show timestamp of when a connections objects were last retrieved

• Token pagination for REST API sources

Fixes

• Correctly infer array and object model fields

• Fix instability when unable to retrieve Zendesk custom fields

• Platform support for denormalized records (#745)

• Add support for Token pagination for APIs

New Features

• Ability to sync from arbitrary APIs

• Ability to sync to arbitrary webhooks

• Ability to trigger syncs through API

• Ability to add all fields from a model in sync mapping in one click

• Show field types in model query editor

• Support for syncing multi-select strings to HubSpot

• New 'completed with errors' status in sync history page

Fixes

• Increased efficiency of cross-model joins

• Increased stability of MongoDB connections

• Order models by name in sync mapping dropdown

• Better support for Stripe sub-fields in model builder

New Features

• Add date filter to sync filters

• New destination: Webhooks

• New source: HTTP APIs

Fixes

• Upgrade Snowflake client library
• Add support for Big Query NUMERIC fields

Fixes

• Expose SQL errors which could lead to premature query termination
• Add support for setting region on bigquery clients
• Reduce memory consumption used for duplication record detection

Fixes

• Add diagnostic logging for records excluded from syncs based on filters,
  change detected, and de-duplication

Fixes

• Fix situation where a Salesforce error could result in the sync halting

New Features

• New destination: Facebook Ad Audiences

Fixes

• Fix sporadic instability with Salesforce syncs

• Better type handling for Iterable syncs

• Improved logging of Salesforce integration

• Fix occasional error with HubSpot associations

New Features

• New destinations: Iterable and Amplitude

• Weekly syncs summary email

• Add ability to associate records in HubSpot syncs

• Add support for 'does not equal' sync filtera

Fixes

• 8x speedup for Snowflake queries using multiple tracking fields

• Fix issue where scheduled syncs sometimes don't run

• More stability for HubSpot syncs

Fixes

• Fix daily-schedule bug that recorded wrong minute value

New Features

• New sources: HubSpot and Stripe

• Add null/not null sync filters for Salesforce target fields

• Add sync filters for Google Sheets model fields

• Show full database schema in SQL Runner

• Add create and update-only sync modes for HubSpot

• Expose equality sync filters for Hubspot model fields

Fixes

• Faster loading of model and sync pages

• Format dates properly when syncing to HubSpot

• HubSpot test syncs sometimes did not show the correct records

• Miscellaneous UI improvements

New Features

• Tracking fields are now optional

• New UI design

• SQL runner: run ad-hoc SQL queries on your databases

• Ability to set create-only mode per-field

• Show/hide field details in sync mapping config

Fixes

• Hard timeout for sync preview operations that take too long

• Show on-premises execution log URL for any started execution

Fixes

• Show on premises log link for any started execution

• Fix occassional stability issue with Postgres connections

• Better performance for large Airtable tables

• Fix bug where a connection would mistakenly save on successfull OAuth

• Expose Affinity CRM relationship fields for sync

• More stable recovery in the face of errors from Google Sheets's API

• Prevent occasional record duplicates in HubSpot syncs

• Better support for Airtable formula fields

New Features

• Add support for syncing to Intercom companies

• Add support for syncing from Airtable Views

Fixes

• Sync field mappings are now listed in model-field alphabetical order

• Faster fetching of Airtable bases and tables

• Improvements for syncing to Affinity CRM

• Redirect user to login on session expiration

• Improved display of BigQuery table names

• More stability when refreshing Google Sheets connections

New Features

• Cron sync scheduling via 'Custom' schedule option

• Ability to sync to Stripe Customers and Subscriptions

• Unhealthy-connection destinations are now marked as such on the syncs list page

Fixes

• Security upgrades

• Query speed improvements

• (on-prem) More efficient database space utilisation

• Handle additional errors during Salesforce bulk job creation

• Reduce the number of SQL queries needed when joining fieldsets

Fixes

• Optimize SQL queries when multiple tracking fields are used

• Improve caching behavior for queries executed during a sync execution

New Features

• New destinations: Intercom, Affinity CRM

• Syncs page is now grouped by destination

• Model page query fieldsets are minimized by default

• Expose support for sync field filters on Salesforce fields

Fixes

• Improve load speed for syncs and model pages

• Lower memory usage during syncs, especially for large ones

• More efficient related-object syncing in Salesforce

• Do not return model page error if we cannot enumerate SFDC schema

• Google Sheets syncs would fail in rare conditions

• Add ability to ignore invalid references

• Better automatic model validation when executing a sync

• Clean-up relationships from removed fields

• Do not attempt to sync records we were unable to map to destination

• Gracefully handle search errors from Hubspot

New Features

• New sync destination: Airtable

• Syncs page now groups syncs by destination

• Make query name sticky when scrolling through model fields selector

• Add join indicators on model page

Fixes

• Syncing MongoDB to Google Sheets would sometime fail

• Cancelling a Salesforce sync from Polytomic after Salesforce has already written all the data should not count as a complete job

• Show error message for the specific case of failing to update a Google Sheet

• Expose Salesforce IDs as identity mapping targets

• Security improvements

New Features

• Ability to set filters on model fields in a sync configuration: is null/not-null and true/false for boolean fields.

• Banner confirmation of invitation email when a new user is added

• Retry network requests in the event of unexpected hang up

• Username and password were not always passed during the Redis client
  construction

• Errors when searching Marketo could result in a stopped sync

• Connections to SQL databases were prematurely removed from the connection
  pool.

New Features

• Expose 'sync all records' setting for all targets, not just Salesforce

• New sync source: Airtable

• Support for syncing to and from very large Google Sheets

• Allow and respect update-only and create-only fields in a create-update sync

• Indicate a Salesforce connection as unhealthy if Salesforce has run out of API calls for the day, even if that has nothing to do with Polytomic

• Update weekly sync summary email to summarize the whole week

Fixes

• Occasional inability to change a connection name

• Edge cases that would result in inaccurate updated-record counts in the sync history

• Introduce restriction of query relationship targets to existing model fields only

• Speed optimisations for large-data syncs

• Occasional errors for calculated columns in Google Sheets

• Update upstream Redis library to incorporate Redis Cluster fixes

Fixes

• Tracking field names were not always quoted correctly

New Features

• Implemented a weekly sync report

Fixes

• Snowflake field labels are now title-cased rather than uppercase

• Accurate indication of whether Salesforce relationship target fields are creatable or updateable

• Fix some cases where cancelling a running sync would result in nothing

• Better handling of default Salesforce batch sizes

• More efficient Salesforce upserts in bulk mode

• More aggressively cache Marketo target definitions

Fixes

• Marketo syncs were not using the maximum available batch size

New Features

• Clickable download links for sync logs in on-premise deployments

• Allow setting syncs to always be full syncs

• Show Salesforce account user in its connection configuration

Fixes

• Speed optimisations when gathering data for a sync

• More efficient Salesforce API usage for some sync types

• Further reduction in logging volume

• Treat automatically-skipped syncs as failures

• Treat Salesforce identity keys as case insensitive

• Remove logging statements without level filter

Fixes

• Updating Salesforce related objects using an external ID would sometimes fail
  when serial concurrency or fixed batch size was set.

New Features

• Add support for update-only or create-only Marketo syncs

Fixes

• Creating a new Google Sheets connection would sometimes fail

• Security upgrades

Fixes

• Marketo syncs could fail to start due to overly aggressive locking.

• A sync could fail the first time it was run after changing referenced tracking
  fields in the model.

Fixes

• Fixed regression in model query editing

Fixes

• Relationships to related Salesforce objects were not set in some
  circumstances.

• Preview was unavailable when a sync included related records.

• Lookup fields on related records were erroneously flagged as required.

Fixes

• Corrects a bug which could cause duplicate records to be created in
  Marketo.

New Features

• Add ability to sync to any SFDC object's related-object's ID field

• Add AzureSQL support

Fixes

• Don't disappear 'full resync' button during sync

• Allow the same model field to sync to two different target fields

• Remove character limit on query titles

• Stability improvements for particular Redis environments

• Skip data fields for a particular sync if source system is unavailable

• Editing query fields propagates those changes to
  relationships using those fields

• Fix error preventing logging in to Polytomic Front widget in Front's desktop app

• Security updates

• Logging improvements

New Features

• Add support for Azure SQL connections

Fixes

• Namespace all Redis keys for cluster mode compatibility; this includes the
  contents of all rel2020.12.23 hot fixes.

• Update Docker base image to use Ubuntu 20.04.

• Removing a fieldset involved in a relationship could result in the Model
  refusing to load.

• An error during OAuth connection could result in a half-configured connection.

• Hubspot syncs could drop the final record from a batch.

Fixes:

• Route redis scripts to the correct cluster node.

Fixes:

• Reference frontend scripts using a domain relative URL

Fixes:

• Set the Redis namespace for registered task handlers for Redis Cluster
  compatibility

Fixes

• Scope event stream keys correctly for Redis Cluster compatibility

New Features

• Add support for Amplitude model

• Ability to scroll left and right sides of model edit view independently

Fixes

• Display logout button in application sidebar

• Edge-case stability for Zendesk syncs

• More generous timeout for testing new DB connections

• Empty Google Sheet results could fail the sync

• A deleted identity field could cause the sync list fetch to silently fail

• Allow adding a previously-removed user to an organization

• Remove extraneous "Google Sheets" default name prefix from Google Sheet connections

• Accept full domains or URLs for Zendesk connection domain

• More stable handling for query tests that return an error

New Features

• Add support for syncing to Google Sheets

• Add support for syncing to Zendesk

• Add option to force a fresh authentication for existing OAuth connections

• Expose Salesforce external ID fields for sync identity mapping

Fixes

• Only display writeable target fields in mapping drop-down

• Handle widget view empty state

• Refreshing a database table schema in a query after editing field properties would revert said properties

• Deleting fields used in an existing sync could prevent new syncs from being created

• Deleting errored SSH-tunnel database connections would fail

New Features

• Added support for 'views': the ability to display a live subset of the Universal Model based on context (support ticket or customer email) in widgets that can be embedded in third-party customer systems like Front, Salesforce, and Zendesk. Front support only for now.

• Added support for a substring-of identity mapping function in addition to the default equality one.

Fixes

• Salesforce syncing would fail for certain field mapping type combinations

• Under certain conditions, test syncs resulted in an inaccurate record log.

• Editing a connection name sometimes resulted in the old name persisting.

New Features

• Syncing an array of values or delimited string to a multi-picklist
  field in Salesforce will set the picklist values.

  When syncing a multi-picklist field in Salesforce, comma,
  semi-colon, and null delimited strings will select picklist values,
  if a match exists. Syncing arrays to a multi-picklist field will
  alow select the picklist values.

• Sync filters may be applied to Update syncs.

• Updated BigQuery integration to support schema browser.

  Previously only SQL queries were supported for BigQuery fieldsets.

Fixes

• Deleting a fieldset that is referenced by a sync caused the sync to
  disappear from the list.

• When running on premises, duplicate root users could be created at
  startup.