Application Connectors

The data that your company’s software services provide can offer much more insight than what you can access on their internal dashboards.

Many of these services offer APIs (Application Programming Interface) for accessing or extracting data from them via a direct and secure internet connection. Hippobyte’s integration platform continuously extracts the underlying data from your existing applications and centralizes it all in a petabyte-scale data warehouse.

Connectors

Hippobyte supports replicating data from several API Cloud Applications. For a full list of supported APIs see the list to the left. If there is an integration that you would like but that is not yet supported, please let us know.

Data Synchronization Overview

• Connection and validation: If available, OAuth is utilized to connect with your application. OAuth grants us restricted access to your account, and protects your sensitive information. We use it for all APIs that support it.
• Initial Replication: Hippobyte discovers all available standard and custom objects available, and automatically pulls data for all objects that it has access to. The initial sync time differs for each connection type, but can take anywhere from a few minutes to over a month — duration is solely dependant on the limits of the application API.
• Schema definition: Hippobyte parses through all data, typecasting and mapping every column in the source object to a column in a corresponding destination table. We transform any data types that are not natively supported in the destination, into data types that are accepted.
• Load: Hippobyte automatically creates Schemas (one per integration) and tables for each mapped source object. Hippobyte populates these tables with the initial state of source data.
• Sync: Hippobyte incrementally updates each integration in batches, using a merge operation (upsert & insert) to update changed or modified data. These batches run on different time intervals per integrations ranging from every 1 hour to 24 hours. Hippobyte’s unique system automatically recognizes schema changes in the source, and persists these changes to the target.

Enterprise and Premium customers have access to additional Sync Strategies allowing users to completely reload data from source and utilize CRON syntax to schedule Sync schedules.

There are some data types, for example, binary data, that does not make sense to be stored in a SQL-based data warehouse, and so Hippobyte actively does not sync this type of data. For a full list of the data that is synced for each cloud application integration, please visit the individual Application Connector page.

Excluding Source Data

Depending on whether an API allows you to do it, we give you the option of not syncing specific tables with Hippobyte. This can be done on the Dashboard while you are integrating.

Updating Data

Update Frequency

By default, application connectors sync all new and modified data every 1 hour. Depending on the size of each update, it may take longer. In that case, Hippobyte triggers a connector sync shortly after the previously scheduled sychronization completes. For example:

• Update A Start: 8:00am
• Update A Finish: 9:10am
• Update B Start: 9:10am (best case scenario, actual time may be within a few minutes)

You can change the sync schedule in the Console. An application connector sync that encounters an error is re-tried repeatedly for up to 10 days.

Deleted Data

In general, Hippobyte will never delete data from your data warehouse. Hippobyte handles deleted data differently for different connectors.

For connectors where the API supports it, we utilize the h_status column. We mark the row with status 30 is it has been deleted in the source application.

Some APIs do not allow us to detect or indicate when data has been deleted in the source application. Reach out if this is a point of friction for you, we’ve worked with many customers and may have a work-around for a specific connector.