Each Dagster Cloud full deployment (e.g., prod) needs to have at least one agent running. Since the agent is lightweight and can tolerate short downtimes such as during an upgrade, a single agent is adequate for most use cases.
If you want to distribute runs across multiple Kubernetes clusters, ECS clusters, EC2 instances, or other environments, you can run multiple agents. For example, one in each cluster, instance, etc.
It's recommended to only use multiple agents of the same type (e.g. multiple Kubernetes agents). Submitting runs to a specific agent isn't currently supported.
dagster-cloud version 0.13.0 or later for both agents and job code.
To use a Kubernetes, Docker, or local agent. These agent types can support running multiple agents. Additionally, note that:
Kubernetes agents must run in their own namespace or container. Running multiple agents within a single namespace isn't currently supported.
Docker agents must run in containers orchestrated by separate Docker daemons (e.g. on a separate host machine). Running multiple agents on one Docker daemon isn't currently supported.
In this step, you'll enable the agent_replicas option to turn on running multiple agents. You can also choose to label the agent, making it easily identifiable on Dagster Cloud's Status page.
Follow the instructions for your agent:
DOCKER AGENTS
Docker agents must run in containers orchestrated by separate Docker daemons (e.g. on a separate host machine). Running multiple agents on one Docker daemon isn't currently supported.
Add the following to the dagster.yaml file:
agent_replicas:enabled:truedagster_cloud_api:# <your other config>agent_label:"My agent"# optional
KUBERNETES AGENTS
Kubernetes agents must run in their own namespace or container. Running multiple agents within a single namespace isn't currently supported.
Add the following options to your Helm command:
helm upgrade \...
--set agentReplicas.enabled=true \
--set dagsterCloud.agentLabel="My agent"# optional, only supported on 0.13.14 and later
Or if you're using a values.yaml file:
agentReplicas:enabled:truedagsterCloud:agentLabel:"My agent"# optional, only supported on 0.13.14 and later
LOCAL AGENTS
Add the following to the dagster.yaml file:
agent_replicas:enabled:truedagster_cloud_api:# <your other config>agent_label:"My agent"# optional