Databricks Data Export

This topic explains how to create and test a Databricks destination for Data Export. Databricks is a cloud-based data processing and analysis platform that lets you work with large sets of data. By exporting your LaunchDarkly experiment data to the same Databricks warehouse as your other data, you can build custom reports in Databricks to answer product behavior questions.

You can also run experiments using warehouse native metrics. To learn more, read Creating experiments using warehouse native metrics.

Prerequisites

To configure the Databricks Data Export integration, you must have the following prerequisites:

• You must have a LaunchDarkly role that lets you add and edit integrations and add destinations

Create a service principal

We recommended that you use a Databricks Service principal to provide LaunchDarkly service access to your warehouse.

To create a new service principal, follow the Databricks instructions to Add service principals to your account.

Ensure that the principal has Workspace access and Databricks SQL access entitlements. To learn how, read the Databricks documentation about Access entitlements.

Create a principal access token

Next, generate and provide an access token for the service principal. You can create an access token using the Databricks REST API. To learn how, read the Databricks REST API documentation.

You must provide specific values to the REST API. In the following example, set these values:

• Replace databricks-instance with your Databricks workspace URL. For example, dbc-abcd1234-5678.cloud.databricks.com.
• Replace your-existing-access-token with an existing valid PAT (string) that has permissions to create new tokens.

Provide the values for these parameters:

$
curl -X POST https://<databricks-instance>/api/2.0/token/create \
>
  -H "Authorization: Bearer <your-existing-access-token>" \
>
  -H "Content-Type: application/json" \
>
  -d '{
>
    "comment": "New PAT using DB API",
>
    "lifetime_seconds": <lifetime-of-pat-in-seconds>
>
  }'

Omitting lifetime_seconds sets the lifetime to the maximum allowed by the workspace configuration.

You will provide the access token later when setting up the integration in LaunchDarkly.

Create a SQL warehouse

Now, follow the Databricks instructions to Create a new SQL warehouse.

Under permissions, assign the Can use permission to the service principal you created above.

Under Connection details, find and save the Server hostname and HTTP path of the warehouse. You will need these when setting up the integration in LaunchDarkly.

Give catalog and schema permissions

Then, give the service principal you created the appropriate permissions for your destination catalog and destination schema.

For your destination catalog, give the service principal the USE CATALOG permission.

For your destination schema, give the service principal the following grants:

• USE SCHEMA
• APPLY TAG
• MODIFY
• READ VOLUME
• SELECT
• WRITE VOLUME
• CREATE MATERIALIZED VIEW
• CREATE TABLE
• CREATE VOLUME

Databricks selects “Unity Catalog” for metastore type by default. If your workspace uses the legacy Hive metastore, select it instead and provide the required S3 bucket details and access keys.

Set up the integration

To set up Databricks Data Export, connect your LaunchDarkly environment to your SQL warehouse, choose the metastore, then test the connection and save the destination. Databricks has no separate script to run in your warehouse, so LaunchDarkly tests the connection and creates the destination in a single step.

To configure the integration in LaunchDarkly:

1. Click the gear icon in the left sidebar to view Organization settings.
2. Click Integrations and find “Databricks Data Export.”
3. Click Add integration. The Databricks Data Export configuration page appears.
4. Give the integration a human-readable Name.
5. Choose a Project and environment to export data from. Environments that already have a Data Export destination are unavailable.
6. Enter the Server hostname you saved when you created a SQL warehouse. Enter the hostname without a URL scheme, for example dbc-abcd1234-5678.cloud.databricks.com.
7. Enter the HTTP path you saved when you created a SQL warehouse. The path starts with /sql/, for example /sql/1.0/warehouses/abc123.
8. Enter the Catalog and Schema to export data to. Catalog and schema names can contain only letters, numbers, and underscores. Hyphens are not allowed.
9. Enter the principal Access token you created in a previous step. LaunchDarkly uses the access token to connect to Databricks but does not store it after setup.
10. Select a Metastore of “Unity Catalog” or “Hive.” LaunchDarkly selects “Unity Catalog” by default.
11. Enter the staging bucket details if you selected the “Hive” metastore: the S3 bucket name, the S3 bucket region, the Bucket access ID, and the Bucket secret key. LaunchDarkly uses the bucket secret key to connect but does not store it after setup.
12. Read the Integration Terms and Conditions, then select the I have read and agree to the Integration Terms and Conditions checkbox.
13. Click Test connection and save. LaunchDarkly tests the connection to Databricks and, if it succeeds, activates the destination and starts the initial transfer.

Your Databricks Data Export destination is now configured.

To learn about the different event kinds for Databricks Data Export destinations, read Warehouse Data Export schema reference.

View an existing destination

The destination setup is visible on the edit page so that you can confirm which host, HTTP path, catalog, schema, and metastore a destination uses. However, all fields on the edit page are read-only.

LaunchDarkly never stores the secrets you provide at setup, so it cannot display them again. The Access token field and the Bucket secret key field for a Hive metastore show a masked placeholder to indicate that you provided a secret when you created the destination.

To view an existing Databricks Data Export destination:

1. Click the gear icon in the left sidebar to view Organization settings.
2. Click Integrations and find “Databricks Data Export.”
3. Find the destination you want to view and click Edit. The Databricks Data Export configuration page appears with the destination’s settings.

To verify that the destination still works, click Send event to send a test event from LaunchDarkly to your warehouse. To change any connection value, delete the destination and configure a new one.

Health checks

If your warehouse Data Export integration develops connection problems or other errors, the integration displays a health check warning. To learn more, read Warehouse health checks.