Databricks

Sifflet monitors the health and quality of your Databricks data, from table updates down to individual field values and business metrics. Sifflet catalogs the tables, views, and volumes of your Unity Catalog, and builds lineage from Databricks system tables: when an issue occurs, you see its root cause upstream and the downstream tables and dashboards it affects.

For the full list of assets and metadata Sifflet imports, see Databricks Collected Data.

Permissions Required

Sifflet uses read-only access. It connects to a Databricks SQL warehouse with a personal access token belonging to a dedicated service principal (recommended) or to a user, and only runs SELECT, SHOW, and DESCRIBE statements.

If a required permission is missing, the connection test and refreshes fail. If the optional system.lakeflow.pipeline_update_timeline permission is missing, the test succeeds with a warning.

To monitor your Databricks jobs as well, the Databricks Workflows integration needs additional permissions on system.lakeflow and system.access tables, listed on its page.

Cost and load: Sifflet runs metadata queries during each refresh and monitor queries on each monitor run on the SQL warehouse you configure, so they consume Databricks compute. Use a dedicated SQL warehouse for Sifflet to isolate this cost: an X-Small warehouse is enough for environments with up to a few thousand tables. Sifflet identifies its connections with the Sifflet user agent.

Requirements

Required Roles for Setup

• In Databricks:
  • a workspace admin, to create the service principal, allow it to use personal access tokens, and grant it access to the SQL warehouse;
  • an account admin, to enable the system.access system schema if it isn't enabled yet, and optionally the system.query schema;
  • a metastore admin (or the owner of each catalog), to grant privileges on your catalogs and on the system catalog.
• In Sifflet: the Admin role to create the credential, and the Admin or System Editor role to create the source. See Access Control.

Supported Versions and Editions

• Databricks workspaces on AWS, Azure, and Google Cloud, with Unity Catalog enabled.
• A Databricks SQL warehouse (serverless, pro, or classic). Sifflet doesn't run queries on all-purpose or job clusters.
• Catalogs governed by Unity Catalog. The legacy hive_metastore catalog is not supported. For other configurations, contact Sifflet support.
• Authentication with a personal access token only. OAuth machine-to-machine (M2M) authentication is not supported.

Network Access

• Default: Sifflet connects to your Databricks workspace over the internet, on port 443, so the workspace must be reachable from the public internet.
• IP allowlisting: if you restrict access to your workspace with Databricks IP access lists, add the IP addresses of your Sifflet instance, listed in Settings > Integration Preferences.

Other Prerequisites

• The system.access system schema is enabled in your Databricks account. system.lakeflow is enabled by default. Enabling system.query is optional: it is only needed for the permissions kept for future features. To enable a system schema, see the Databricks documentation.
• A SQL warehouse for Sifflet. You can create a dedicated one (recommended) or use an existing one.

Connect Sifflet to Databricks

Step 1: Create a Service Principal in Databricks

A service principal is a Databricks identity dedicated to automated tools. Use it rather than a user's token, as Databricks recommends: the access doesn't depend on a person, and you grant it exactly the permissions Sifflet needs.

1. Create a service principal, from the account console or with the API, and add it to your workspace.

2. In the Configurations tab of the service principal, enable the Databricks SQL access entitlement. It is the only entitlement Sifflet needs.

3. Note the service principal's Application ID, shown in Settings > Identity and access > Service principals. The grants below use it as <APPLICATION_ID>.

   

4. Allow the service principal to use personal access tokens: go to Settings > Workspace admin > Advanced > Personal Access Tokens > Permission Settings, and add the Can Use permission for the service principal. See the Databricks documentation.

5. Allow the service principal to use the SQL warehouse: go to SQL Warehouses, select the warehouse, click Permissions, and give the service principal the Can use permission.

6. Generate a personal access token for the service principal. Databricks only lets you do it with the Databricks CLI or the API. For example, as a workspace admin:

   Shell

   databricks token-management create-obo-token <APPLICATION_ID> --lifetime-seconds <LIFETIME_IN_SECONDS> --comment "Sifflet"

   The token starts with dapi. Databricks shows it only once: keep it for step 2. When the token expires, Sifflet can't connect anymore: generate a new token and update the Sifflet credential before the expiry date.

Alternative: User Personal Access Token

You can use a personal access token of a Databricks user instead of a service principal. Sifflet then has the same access as this user, and the connection stops working if the user leaves.

1. Create the token by following Databricks personal access tokens for workspace users in the Databricks documentation.
2. Make sure the user has the Databricks SQL access entitlement and the Can use permission on the SQL warehouse.
3. In the grants below, replace <APPLICATION_ID> with the user's email address.

Keep the token for step 2.

Grant the Permissions

Run the following script in a SQL editor of your workspace, as a metastore admin. Repeat the first block for each catalog you want Sifflet to monitor. On these catalogs, grant the privileges at the catalog level: Test Connection checks them on the catalog's information_schema, and catalog-level privileges also apply to the schemas, tables, and volumes created later in the catalog. On the system catalog, schema-level and table-level grants are enough.

-- Repeat for each catalog to monitor
GRANT USE CATALOG ON CATALOG <CATALOG_NAME> TO `<APPLICATION_ID>`;
GRANT USE SCHEMA ON CATALOG <CATALOG_NAME> TO `<APPLICATION_ID>`;
GRANT SELECT ON CATALOG <CATALOG_NAME> TO `<APPLICATION_ID>`;

-- Lineage, from Databricks system tables
GRANT USE CATALOG ON CATALOG system TO `<APPLICATION_ID>`;
GRANT USE SCHEMA ON SCHEMA system.access TO `<APPLICATION_ID>`;
GRANT SELECT ON TABLE system.access.column_lineage TO `<APPLICATION_ID>`;

All users can read system.information_schema by default in Unity Catalog. If you revoked this access, grant USE SCHEMA on system.information_schema and SELECT on its tables and columns views back to the service principal.

Then, grant the optional permissions for the features you want:

-- Optional (volumes): repeat for each catalog to monitor
GRANT READ VOLUME ON CATALOG <CATALOG_NAME> TO `<APPLICATION_ID>`;

-- Optional (job lineage): link tables written by Lakeflow Declarative Pipelines to Databricks jobs
GRANT USE SCHEMA ON SCHEMA system.lakeflow TO `<APPLICATION_ID>`;
GRANT SELECT ON TABLE system.lakeflow.pipeline_update_timeline TO `<APPLICATION_ID>`;

-- Optional (future features): not used today, avoids new grants when Sifflet starts using them
-- Requires the system.query schema to be enabled
GRANT USE SCHEMA ON SCHEMA system.lakeflow TO `<APPLICATION_ID>`;
GRANT USE SCHEMA ON SCHEMA system.query TO `<APPLICATION_ID>`;
GRANT SELECT ON TABLE system.access.table_lineage TO `<APPLICATION_ID>`;
GRANT SELECT ON TABLE system.query.history TO `<APPLICATION_ID>`;
GRANT SELECT ON TABLE system.lakeflow.job_tasks TO `<APPLICATION_ID>`;
GRANT SELECT ON TABLE system.lakeflow.job_task_run_timeline TO `<APPLICATION_ID>`;

You can also grant the catalog privileges from Catalog Explorer: select the catalog, open the Permissions tab, click Grant, select the service principal, and select the Data Reader privilege preset. This preset includes USE CATALOG, USE SCHEMA, SELECT, and READ VOLUME.

Step 2: Add the Databricks Source in Sifflet

1. Create the credential: go to Integrations > Credentials, click New credential, and paste the personal access token from step 1 as-is, with no quotes or extra spaces:

   Text

   dapi0123456789abcdef0123456789abcdef

2. Collect the connection details: in Databricks, go to SQL Warehouses, select the warehouse Sifflet uses, and open the Connection details tab.

   

3. Create the source: go to Integrations > Sources, click New source, select Databricks, and fill in the form:

   Field	Description	Example
   Source name	The name of the source in Sifflet	Databricks Production
   Host	The Server hostname from the connection details, without https://	dbc-a1b2c3d4-e5f6.cloud.databricks.com
   Port	The Port from the connection details	443
   Http Path	The HTTP path from the connection details	/sql/1.0/warehouses/1234567890abcdef
   Credential	The credential created above	databricks-token

4. Select the scope: click Load assets. Sifflet lists the catalogs and schemas the token can access. Select the catalogs or schemas to include in the Data Catalog, and enable Automatically include new schemas so that Sifflet also catalogs the schemas created later. Create one source per Databricks workspace: see Integrations Management for the source model and refresh scheduling.

5. Test and save: click Test. Sifflet checks the catalog and lineage permissions, and shows a report with any missing permission. Then save the source. Sifflet then refreshes the source: follow the status and logs of the refresh in the source details, from Integrations > Sources. Your Databricks assets appear in the Data Catalog once the refresh completes.

Step 3 (Optional): Monitor Your Databricks Jobs

Add the Databricks Workflows integration to see your Databricks jobs and their latest run status in Sifflet, and to show in the lineage which job writes each table. It reuses the service principal and credential created above.

Troubleshooting

For connection, authentication, and permission errors, see Databricks Troubleshooting.