Getting Started with Dagster Cloud using a Hybrid Deployment#
This guide is applicable to Dagster Cloud.
Welcome to Dagster Cloud!
In this guide, we'll give you everything you need to get up and running with Dagster Cloud using a Hybrid Deployment. With a Hybrid Deployment, your Dagster code is executed in your infrastructure. Refer to the Getting started with Serverless Deployment guide if you want to use Serverless.
We'll walk you through setting up your Dagster Cloud account, setting up an agent, loading your code, launching your first run, and configuring continuous integration.
Access to some Dagster code. You can either use your own code or a project scaffold. This guide uses a scaffold.
To have dagster and dagster-cloud installed. To install, run:
pip install dagster dagster-cloud
The ability to create and configure an agent in your infrastructure. Using a Hybrid Deployment requires setting up an agent for use with Dagster Cloud. The agent runs in your infrastructure, executes your code, and streams metadata about code execution over HTTPS back to Dagster Cloud.
Dagster Cloud supports a handful of agents, each with its own prerequisites for setup. Refer to the Agent documentation for more info.
For all but the local agent, you will need access to a container registry to store Docker images containing your Dagster code.
Step 1: Create a Dagster Cloud account and organization#
If you don't already have a Dagster Cloud account, sign up for one before continuing.
You'll also be asked to create a Dagster Cloud organization. When your organization is created, a single deployment, named prod, will also be created. Deployments are standalone environments, allowing you to operate independent instances of Dagster with separately managed permissions.
For most Hybrid deployments, with the exception of those using a local agent, you will need to create a Docker image containing your code. Then you will add a code location to Dagster Cloud that references the image. Typically, this build and deploy is done through CI/CD, but it is also possible to deploy a code location manually.
For continunous integration using GitHub actions:
Begin by copying the GitHub action deploy workflow in this repository.
Uncomment the step that is relevant to your Docker container registry.
Set the necessary GitHub action secrets, including your Dagster Cloud API Token and Organization ID.
For continuous integration using a CI/CD provider other than GitHub, your system should:
Build a Docker image containing your dagster code.
Push the image to a registry accessible by your agent.
Deploy the update to Dagster Cloud using the dagster-cloud CLI. Refer to the Adding a code location guide for more info.
For manual code deployments:
In Dagster Cloud, click Workspace.
Click the Add sample code location button:
A pre-filled YAML editor with configuration will display.
Verify that the configuration is correct, then click Add code location to save the config.
The agent will attempt to load your code. After the code is loaded, the new code location will show a Loaded status and a sample Dagster job will display in Dagster Cloud.
Branch Deployments allow you to quickly and collaboratively develop your Dagster jobs. When a branch is created or updated in your Dagster repository, Branch Deployments will create a testing environment in the current Dagster Cloud deployment. For example, prod.
Check out the Branch Deployment docs for more info. If you skip this step now, you can always set it up later.