See also the Dask deployment guide.
Connect to an existing scheduler.
Local cluster configuration.
YARN cluster configuration.
SSH cluster configuration.
PBS cluster configuration.
Moab cluster configuration.
SGE cluster configuration.
LSF cluster configuration.
SLURM cluster configuration.
OAR cluster configuration.
Kubernetes cluster configuration.
Dask-based executor.
The ‘cluster’ can be one of the following: (‘existing’, ‘local’, ‘yarn’, ‘ssh’, ‘pbs’, ‘moab’, ‘sge’, ‘lsf’, ‘slurm’, ‘oar’, ‘kube’).
If the Dask executor is used without providing executor-specific config, a local Dask cluster
will be created (as when calling dask.distributed.Client()
with dask.distributed.LocalCluster()
).
The Dask executor optionally takes the following config:
cluster:
{
local?: # takes distributed.LocalCluster parameters
{
timeout?: 5, # Timeout duration for initial connection to the scheduler
n_workers?: 4 # Number of workers to start
threads_per_worker?: 1 # Number of threads per each worker
}
}
To use the dask_executor, set it as the executor_def when defining a job:
from dagster import job
from dagster_dask import dask_executor
@job(executor_def=dask_executor)
def dask_enabled_job():
pass