Cloud Composer

This page explains how to connect Sifflet to Cloud Composer, the managed Airflow service of Google Cloud. For an overview of the Airflow integration and its other deployments, see Apache Airflow; for what Sifflet imports, see Airflow Collected Data.

Permissions Required

Sifflet has read-only access to your Cloud Composer environment. It uses a dedicated Google Cloud service account, authenticated with a JSON key, to get the URL of the Airflow web server from the Cloud Composer API, then calls the Airflow REST API (/api/v1) to list DAGs and read their latest runs.

When a service account calls the Airflow REST API for the first time, Cloud Composer registers it as an Airflow user with the Op role by default, which can also trigger, edit, and delete DAG runs. To keep Sifflet's access read-only, register the service account with the Viewer role before you create the source, as described in Step 1.

Requirements

Required Roles for Setup

• In Google Cloud, permissions to create service accounts and keys and to grant IAM roles on the project (for example the Service Account Admin, Service Account Key Admin, and Project IAM Admin roles), and permission to run Airflow CLI commands in the environment with gcloud composer environments run.
• In Sifflet, the Admin role to create the credential. The System Editor role is enough to create the source with an existing credential. See Access Control.

Supported Versions and Editions

• Cloud Composer environments running Airflow 2.
• Environments running Airflow 3 aren't supported: Sifflet uses the Airflow REST API v1, which Airflow 3 removed.

Network Access

• Default: Sifflet calls the Cloud Composer API and the Airflow web server from the internet. The Airflow web server of a Cloud Composer environment is reachable from the internet and protected by IAM, even in environments with private IP networking.
• IP allowlisting: if you restrict access to the web server to specific IP ranges, add the IP addresses of your Sifflet instance, listed in Settings > Integration Preferences. See Configure web server network access in the Google Cloud documentation.

Other Prerequisites

• The service account must belong to the Google Cloud project of the environment: Sifflet looks up the environment in the project of the service account key.
• The name and the region of your Cloud Composer environment.

Connect Sifflet to Cloud Composer

Step 1: Create a Service Account in Google Cloud

1. In the Google Cloud console, select the project of your Cloud Composer environment, go to IAM & Admin > Service Accounts, and click Create service account.

2. Enter a name (for example sifflet-composer) and click Create and continue.

3. In Grant this service account access to project, select the Composer User role (roles/composer.user), then click Done.

4. Open the service account, go to the Keys tab, and click Add key > Create new key. Select JSON and click Create. The key file downloads to your computer: you can't download it again.

5. Get the numeric ID of the service account (its OAuth 2 client ID):

   Shell

   gcloud iam service-accounts describe <SERVICE_ACCOUNT_EMAIL> \
     --format="value(oauth2ClientId)"

6. Register the service account as an Airflow user with the Viewer role. Replace <ENVIRONMENT_NAME> and <REGION> with the name and region of your environment, <SERVICE_ACCOUNT_NUMERIC_ID> with the ID returned above, and <SERVICE_ACCOUNT_EMAIL> with the email address of the service account:

   Shell

   gcloud composer environments run <ENVIRONMENT_NAME> \
     --location <REGION> \
     users create -- \
     -u accounts.google.com:<SERVICE_ACCOUNT_NUMERIC_ID> \
     -e <SERVICE_ACCOUNT_EMAIL> \
     -f Sifflet \
     -l Integration \
     -r Viewer \
     --use-random-password

   For details, see Access the Airflow REST API using a service account in the Google Cloud documentation.

Keep the JSON key file for step 2.

Step 2: Add the Cloud Composer Source in Sifflet

Create the Credential

1. In Sifflet, go to Integrations > Credentials and click New credential.

2. Enter a name for the credential, and paste the full content of the JSON key file. It has this format:

   JSON

   {
     "type": "service_account",
     "project_id": "<PROJECT_ID>",
     "private_key_id": "<PRIVATE_KEY_ID>",
     "private_key": "<PRIVATE_KEY>",
     "client_email": "<SERVICE_ACCOUNT_EMAIL>",
     "client_id": "<SERVICE_ACCOUNT_NUMERIC_ID>",
     "auth_uri": "https://accounts.google.com/o/oauth2/auth",
     "token_uri": "https://oauth2.googleapis.com/token",
     "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
     "client_x509_cert_url": "<CLIENT_CERT_URL>"
   }

3. Click Save.

Create the Source

Go to Integrations > Sources, click New source, select Cloud Composer, and fill in the form:

Sifflet imports all the DAGs of the environment: there is no scope to select.

Test and Save

1. Click Test Connection. Sifflet gets the web server URL of the environment, then checks that the Airflow API is healthy (Airflow cluster is healthy.) and returns DAGs.
2. Save the source. You can follow the status and logs of its refreshes in the source details: see Integrations Management. Your DAGs appear in the Data Catalog once the first refresh succeeds.

To link your DAGs to the tables they update, see Linking Airflow DAGs to Data Assets.

Troubleshooting

Sifflet Can't Get the Environment Details

• Symptom: Test Connection fails before reaching Airflow, although the credential is valid.
• Cause: the environment doesn't exist in the project of the service account key or in this region, or the service account doesn't have the roles/composer.user role on the project.
• Fix:
  1. Check that the service account was created in the project of your environment: see Other Prerequisites.
  2. Check the Environment and GCP region fields, and the role of the service account: see Step 1.