Getting started with Dagster Cloud using Serverless Deployment#

In this guide, we'll give you everything you need to get up and running with Dagster Cloud using Serverless Deployment. Serverless Deployment is a fully-managed version of Dagster Cloud that allows you to run your Dagster jobs without spinning up any infrastructure.

We'll walk you through setting up your Dagster Cloud account, activating Serverless, loading your code, launching your first run, and configuring Continuous Integration (CI).

Before you get started, note that:

• Serverless Deployments are still in early access
• This deployment type works best with jobs that aren't computationally expensive. Refer to the Serverless guide for more info.
• If you find that Serverless isn't right for you, you can switch to a Hybrid deployment at any time

To use Hybrid Deployment instead, refer to the Getting started using Hybrid Deployment guide.

Prerequisites#

To complete the steps in this guide, you'll need access to some Dagster code.

You can either use your own code or our sample code (dagster-cloud-examples). This guide uses the dagster-cloud-examples Python package.

Step 1: Create a Dagster Cloud account and organization#

If you don't already have a Dagster Cloud account, sign up for one before continuing.

You'll also be asked to create a Dagster Cloud organization. When your organization is created, a single deployment, named prod, will also be created. Deployments are standalone environments, allowing you to operate independent instances of Dagster with separately managed permissions.

Step 2: Select Serverless as your deployment type#

Next, you'll be prompted to select a deployment type. Click Select under Serverless deployment and confirm when prompted.

Note: If you find that Serverless isn't right for you, you can switch to Hybrid Deployment at any time.

Step 3: Deploy your code#

Now that you've set up your deployment, you'll need to tell Dagster Cloud the location of your Dagster code.

A code location specifies a single Python package or file that defines your Dagster code. Each Dagster Cloud deployment can include code from one or more code locations. Refer to the Adding a code location guide for more info.

There are two ways to deploy your code to Dagster Cloud:

• Using GitHub - If you're a GitHub user, this is the easiest way to deploy your code.

• Using the dagster-cloud CLI - If you use a non-GitHub git provider or are developing locally, you can use our CLI to get started.

Using GitHub#

1. Click Choose account or organization and sign into your GitHub account.

2. Select the account or organization where Dagster should be installed.

3. After the installation and authorization is complete, you'll be redirected back to Dagster Cloud.

4. After Dagster Cloud is connected, you'll be prompted to either:

   • Clone the Dagster Cloud starter project - This option clones the dagster-cloud-template repo, which already has Serverless Deployment and Branch Deployments configured.

     1. Enter a Repository name.
     2. Select a Repository visibility.
     3. Click Clone and deploy.

   • Use an existing repo - This option uses an existing repository in your GitHub account.

     1. Select a repository.
     2. Click Deploy.

When finished, continue to the next step.

Using the dagster-cloud CLI#

To complete this step using the CLI, you can use your own Dagster code or the dagster-cloud-examples code. Our examples use the dagster-cloud-examples package.

1. To install the dagster-cloud CLI and the dagster-cloud-examples package, run:

   pip install dagster-cloud dagster-cloud-examples
   

2. Complete the setup:

   1. Complete the authentication step, including creating a user token.

   2. If using your own Dagster code, add dagster-cloud as a dependency in setup.py. Click here for an example.

3. Run the following to deploy your code to Dagster Cloud:

   dagster-cloud serverless deploy \
         --location-name "<LOCATION_NAME>" \
         --package-name <PACKAGE_NAME> \
   

When finished, continue to the next step.

Step 4: Launch a run#

In this step, you'll run your first job in Dagster Cloud. Our example uses the compute_cereal_properties job from the dagster-cloud-examples package.

1. Click the icon in the top-left corner of the page to expand the navigation.

2. In the nav, open a repository and click a job.

3. On the job's page, click the Launchpad tab.

4. Click Launch Run in the bottom right corner to launch the run:

   

Step 5: Set up Continuous Integration#

Branch Deployments allow you to quickly and collaboratively develop your Dagster jobs. When a branch is created or updated in your Dagster repository, Branch Deployments will create a testing environment in your Dagster Cloud instance.

Check out the Branch deployment docs for more info. If you skip this step now, you can always set it up later.

Next steps#

Congrats - your Dagster Cloud deployment is all set up! From here, you can:

• Invite your team
• Configure authentication for your account
• Set up monitoring and alerting
• Learn more about Branch Deployments