How to Migrate Your Instance#

Overview#

When upgrading your Dagster version, you may also need to migrate your Dagster instance. Migrations will only be required if you are upgrading your minor version.

Here, we walk you through the migration process in a Kubernetes environment.

Walkthrough#

For simplicity, we assume that you have only installed one release of the Dagster Helm chart in your Kubernetes cluster. All commands should be run in the proper namespace.

1. Before running the migration, check the migration document for any additional steps you may need to take as part of the migration.
2. Make sure you backup the data in your PostgreSQL database!
3. Run helm upgrade with your desired Dagster chart version and Helm values.
4. Scale down the Dagit and Daemon Deployments. We do not want any ongoing job runs to be writing to the database as the migration is happening. As we scale down the Deployments, we note each Deployment's replica count to scale each back up after the migration is complete.

# Get the names of Dagster's Dagit and Daemon Deployments created by Helm
export DAGIT_DEPLOYMENT_NAME=`kubectl get deploy \
    --selector=component=dagit -o jsonpath="{.items[0].metadata.name}"`
export DAEMON_DEPLOYMENT_NAME=`kubectl get deploy \
    --selector=component=dagster-daemon -o jsonpath="{.items[0].metadata.name}"`

# Save each deployment's replica count to scale back up after migrating
export DAGIT_DEPLOYMENT_REPLICA_COUNT=`kubectl get deploy \
    --selector=component=dagit -o jsonpath="{.items[0].status.replicas}"`
export DAEMON_DEPLOYMENT_REPLICA_COUNT=`kubectl get deploy \
    --selector=component=dagster-daemon -o jsonpath="{.items[0].status.replicas}"`

# Scale down the Deployments
kubectl scale deploy $DAGIT_DEPLOYMENT_NAME --replicas=0
kubectl scale deploy $DAEMON_DEPLOYMENT_NAME --replicas=0

5. Run a Kubernetes Job with the dagster instance migrate command. You can check if the migration succeeded by inspecting the the resulting Pod of the Job.

# Run `helm list` and save your Dagster Helm release name
export HELM_DAGSTER_RELEASE_NAME=<DAGSTER_RELEASE_NAME>

helm template $HELM_DAGSTER_RELEASE_NAME dagster/dagster \
    --set "migrate.enabled=true" \
    --show-only templates/job-instance-migrate.yaml \
    --values values.yaml \
    | kubectl apply -f -

6. Scale the Dagit and Daemon deployments back up.

kubectl scale deploy $DAGIT_DEPLOYMENT_NAME --replicas=$DAGIT_DEPLOYMENT_REPLICA_COUNT
kubectl scale deploy $DAEMON_DEPLOYMENT_NAME --replicas=$DAEMON_DEPLOYMENT_REPLICA_COUNT