Set up Databricks OAuth
This guide describes a feature of the dbt Cloud Enterprise plan. If you’re interested in learning more about an Enterprise plan, contact us at sales@getdbt.com.
dbt Cloud supports developer OAuth (OAuth for partner solutions) with Databricks, providing an additional layer of security for dbt enterprise users. When you enable Databricks OAuth for a dbt Cloud project, all dbt Cloud developers must authenticate with Databricks in order to use the dbt Cloud IDE. The project's deployment environments will still leverage the Databricks authentication method set at the environment level.
Databricks OAuth support in dbt Cloud is a beta feature and subject to change without notification. More updates to this feature coming soon.
Current limitations:
- Databrick's OAuth applications are in public preview
- The current experience requires the IDE to be restarted every hour (access tokens expire after 1 hour - workaround)
Configure Databricks OAuth (Databricks admin)
To get started, you will need to add dbt as an OAuth application with Databricks, in 2 steps:
- From your terminal, authenticate to the Databricks Account API with the Databricks CLI. You authenticate using:
- OAuth for users (prerequisites)
- Oauth for service principals (prerequisites)
- Username and password (must be account admin)
- In the same terminal, add dbt Cloud as an OAuth application using
curl
and the OAuth Custom App Integration API
For the second step, you can use this example curl
to authenticate with your username and password, replacing values as defined in the following table:
curl -u USERNAME:PASSWORD https://accounts.cloud.databricks.com/api/2.0/accounts/ACCOUNT_ID/oauth2/custom-app-integrations -d '{"redirect_urls": ["https://YOUR_ACCESS_URL", "https://YOUR_ACCESS_URL/complete/databricks"], "confidential": true, "name": "NAME", "scopes": ["sql", "offline_access"]}'
These parameters and descriptions will help you authenticate with your username and password:
Parameter | Description |
---|---|
USERNAME | Your Databricks username (account admin level) |
PASSWORD | Your Databricks password (account admin level) |
ACCOUNT_ID | Your Databricks account ID |
YOUR_ACCESS_URL | The appropriate Access URL for your dbt Cloud account region and plan |
NAME | The integration name (i.e 'databricks-dbt-cloud') |
After running the curl
, you'll get an API response that includes the client_id
and client_secret
required in the following section. At this time, this is the only way to retrieve the secret. If you lose the secret, then the integration needs to be deleted and re-created.
Configure the Connection in dbt Cloud (dbt Cloud project admin)
Now that you have an OAuth app set up in Databricks, you'll need to add the client ID and secret to dbt Cloud. To do so:
- go to Settings by clicking the gear in the top right.
- on the left, select Projects under Account Settings
- choose your project from the list
- select Connection to edit the connection details
- add the
OAuth Client ID
andOAuth Client Secret
from the Databricks OAuth app under the Optional Settings section
Authenticating to Databricks (dbt Cloud IDE developer)
Once the Databricks connection via OAuth is set up for a dbt Cloud project, each dbt Cloud user will need to authenticate with Databricks in order to use the IDE. To do so:
- Click the gear icon at the top right and select Profile settings.
- Select Credentials.
- Choose your project from the list
- Select
OAuth
as the authentication method, and click Save - Finalize by clicking the Connect Databricks Account button
You will then be redirected to Databricks and asked to approve the connection. This redirects you back to dbt Cloud. You should now be an authenticated Databricks user, ready to use the dbt Cloud IDE.