Dagster Cloud environment variables and secrets#

This guide is applicable to Dagster Cloud. Refer to the Using environment variables and secrets in Dagster code guide for info about Dagster Open Source.

Environment variables, which are key-value pairs configured outside your source code, allow you to dynamically modify application behavior depending on environment.

Using environment variables, you can define various configuration options for your Dagster application and securely set up secrets. For example, instead of hard-coding database credentials - which is bad practice and cumbersome for development - you can use environment variables to supply user details. This allows you to parameterize your pipeline without modifying code or insecurely storing sensitive data.

Understanding environment variables and secrets#

Overview
Management

Overview#

There are two types of environment variables in Dagster Cloud:

Built-in environment variables, which are automatically populated and contain metadata about deployments
User-provided environment variables, which are defined by you and scoped at the code location level.

Management#

In the following table, we compare the supported methods for managing and passing environment variables in Dagster Cloud:

Setup method - The method for declaring/managing environment variables
How it works - Summary of how environment variables are configured for the method
May be a good fit if... - A high-level summary of when the method may be a good fit

Setup method	How it works	May be a good fit if...
GitHub secrets	Environment variables are defined in a GitHub repository using Action Secrets. Supported for Serverless deployments only.	You're using Serverless deployment You're using Dagster's GitHub app
dagster-cloud CLI	Environment variables are passed in using the `--env` flag in the CLI. Supported for Serverless deployments only.	You're using Serverless deployment You're not using Dagster's GitHub app
Agent configuration	Environment variables are defined in the agent's configuration in your project's `dagster_cloud.yaml` file. Values are pulled from your user cluster, meaning they don't pass through Dagster Cloud.Supported for Hybrid deployments only.	You're using Hybrid deployment

Managing environment variables#

Only supported for Serverless deployments.

Step 1: Create GitHub Action Secrets

In the repository, click the Settings tab.
In the Security section of the sidebar, click Secrets > Actions.
Click New repository secret.
In the Name field, enter the name of the secret. For example, API_TOKEN
In the Value field, paste the value of the secret.
Click Add secret.

Step 2: Uncomment the env_vars property

To make the secrets you created available, you'll need to modify two files in your GitHub repository.

In the .github/workflows/branch_deployments.yml and .github/workflows/deploy.yml files, uncomment the env_vars property as follows:


# Uncomment to pass through Github Action secrets as a JSON string of key-value pairs
env_vars: ${{ toJson(secrets) }}

The secrets you created in the repository will now be available to your Dagster code as environment variables.

Only supported for Serverless deployments.

If you're using a git provider like Bitbucket, you can use the dagster-cloud CLI to pass in environment variables with the --env flag.

This flag accepts KEY=VALUE pairs. For example:


dagster-cloud serverless deploy --env DATABASE_USERNAME=hooli_testing

Accessing environment variables in Dagster code#

Ready to start using environment variables in your Dagster code? Refer to the Using environment variables and secrets in Dagster code guide for more info and examples.

Built-in environment variables#

Dagster Cloud provides a set of built-in, automatically populated environment variables, such as the name of a deployment or details about a branch deployment commit, that can be used to modify behavior based on environment.

All deployment variables#

The following variables are available in every deployment of your Dagster Cloud instance, including full (e.g., prod) and branch deployments.

Property	Description
DAGSTER_CLOUD_DEPLOYMENT_NAME	The name of the Dagster Cloud deployment. For example, `prod`.
DAGSTER_CLOUD_IS_BRANCH_DEPLOYMENT	If `1`, the deployment is a branch deployment. Refer to the Branch Deployment variables section for a list of variables available in branch deployments.

Branch Deployment variables#

The following environment variables are currently available only in a branch deployment.

For every commit made to a branch, the following metadata is available:

Property	Description
DAGSTER_CLOUD_GIT_SHA	The SHA of the commit.
DAGSTER_CLOUD_GIT_TIMESTAMP	The time the commit occurred.
DAGSTER_CLOUD_GIT_AUTHOR_EMAIL	The email of the git user who authored the commit.
DAGSTER_CLOUD_GIT_AUTHOR_NAME	The name of the git user who authored the commit.
DAGSTER_CLOUD_GIT_MESSAGE	The message associated with the commit.
DAGSTER_CLOUD_GIT_BRANCH	The name of the branch associated with the commit.
DAGSTER_CLOUD_GIT_REPO	The name of the repository associated with the commit.
DAGSTER_CLOUD_PULL_REQUEST_ID	The ID of the pull request associated with the commit.
DAGSTER_CLOUD_PULL_REQUEST_STATUS	The status of the pull request at the time of the commit. Possible values are `OPEN`, `CLOSED`, and `MERGED`.