Types¶

Built-in types¶

dagster.Nothing¶

Use this type only for inputs and outputs, in order to establish an execution dependency without communicating a value. Inputs of this type will not be passed to the op compute function, so it is necessary to use the explicit In API to define them rather than the Python 3 type hint syntax.

All values are considered to be instances of Nothing.

Examples:

@op
def wait(_) -> Nothing:
    time.sleep(1)
    return

@op(
    ins={"ready": In(dagster_type=Nothing)},
)
def done(_) -> str:
    return 'done'

@job
def nothing_job():
    done(wait())

# Any value will pass the type check for Nothing
@op
def wait_int(_) -> Int:
    time.sleep(1)
    return 1

@job
def nothing_int_job():
    done(wait_int())

Making New Types¶

class dagster.DagsterType(type_check_fn, key=None, name=None, is_builtin=False, description=None, loader=None, materializer=None, required_resource_keys=None, kind=DagsterTypeKind.REGULAR, typing_type=typing.Any, metadata_entries=None, metadata=None)[source]¶

Define a type in dagster. These can be used in the inputs and outputs of ops.

Parameters:

• type_check_fn (Callable[[TypeCheckContext, Any], [Union[bool, TypeCheck]]]) – The function that defines the type check. It takes the value flowing through the input or output of the op. If it passes, return either True or a TypeCheck with success set to True. If it fails, return either False or a TypeCheck with success set to False. The first argument must be named context (or, if unused, _, _context, or context_). Use required_resource_keys for access to resources.

• key (Optional[str]) –

  The unique key to identify types programmatically. The key property always has a value. If you omit key to the argument to the init function, it instead receives the value of name. If neither key nor name is provided, a CheckError is thrown.

  In the case of a generic type such as List or Optional, this is generated programmatically based on the type parameters.

  For most use cases, name should be set and the key argument should not be specified.

• name (Optional[str]) – A unique name given by a user. If key is None, key becomes this value. Name is not given in a case where the user does not specify a unique name for this type, such as a generic class.

• description (Optional[str]) – A markdown-formatted string, displayed in tooling.

• loader (Optional[DagsterTypeLoader]) – An instance of a class that inherits from DagsterTypeLoader and can map config data to a value of this type. Specify this argument if you will need to shim values of this type using the config machinery. As a rule, you should use the @dagster_type_loader decorator to construct these arguments.

• materializer (Optional[DagsterTypeMaterializer]) – An instance of a class that inherits from DagsterTypeMaterializer and can persist values of this type. As a rule, you should use the @dagster_type_materializer decorator to construct these arguments.

• required_resource_keys (Optional[Set[str]]) – Resource keys required by the type_check_fn.

• is_builtin (bool) – Defaults to False. This is used by tools to display or filter built-in types (such as String, Int) to visually distinguish them from user-defined types. Meant for internal use.

• kind (DagsterTypeKind) – Defaults to None. This is used to determine the kind of runtime type for InputDefinition and OutputDefinition type checking.

• typing_type – Defaults to None. A valid python typing type (e.g. Optional[List[int]]) for the value contained within the DagsterType. Meant for internal use.

property display_name¶

Either the name or key (if name is None) of the type, overridden in many subclasses

property unique_name¶

The unique name of this type. Can be None if the type is not unique, such as container types

dagster.PythonObjectDagsterType(python_type, key=None, name=None, **kwargs)[source]¶

Define a type in dagster whose typecheck is an isinstance check.

Specifically, the type can either be a single python type (e.g. int), or a tuple of types (e.g. (int, float)) which is treated as a union.

Examples

ntype = PythonObjectDagsterType(python_type=int)
assert ntype.name == 'int'
assert_success(ntype, 1)
assert_failure(ntype, 'a')

ntype = PythonObjectDagsterType(python_type=(int, float))
assert ntype.name == 'Union[int, float]'
assert_success(ntype, 1)
assert_success(ntype, 1.5)
assert_failure(ntype, 'a')

Parameters:

• python_type (Union[Type, Tuple[Type, ...]) – The dagster typecheck function calls instanceof on this type.

• name (Optional[str]) – Name the type. Defaults to the name of python_type.

• key (Optional[str]) – Key of the type. Defaults to name.

• description (Optional[str]) – A markdown-formatted string, displayed in tooling.

• loader (Optional[DagsterTypeLoader]) – An instance of a class that inherits from DagsterTypeLoader and can map config data to a value of this type. Specify this argument if you will need to shim values of this type using the config machinery. As a rule, you should use the @dagster_type_loader decorator to construct these arguments.

• materializer (Optional[DagsterTypeMaterializer]) – An instance of a class that inherits from DagsterTypeMaterializer and can persist values of this type. As a rule, you should use the @dagster_type_mate decorator to construct these arguments.

dagster.dagster_type_loader(config_schema, required_resource_keys=None, loader_version=None, external_version_fn=None)[source]¶

Create an dagster type loader that maps config data to a runtime value.

The decorated function should take the execution context and parsed config value and return the appropriate runtime value.

Parameters:

• config_schema (ConfigSchema) – The schema for the config that’s passed to the decorated function.

• loader_version (str) – (Experimental) The version of the decorated compute function. Two loading functions should have the same version if and only if they deterministically produce the same outputs when provided the same inputs.

• external_version_fn (Callable) – (Experimental) A function that takes in the same parameters as the loader function (config_value) and returns a representation of the version of the external asset (str). Two external assets with identical versions are treated as identical to one another.

Examples:

@dagster_type_loader(Permissive())
def load_dict(_context, value):
    return value

class dagster.DagsterTypeLoader[source]¶

Dagster type loaders are used to load unconnected inputs of the dagster type they are attached to.

The recommended way to define a type loader is with the @dagster_type_loader decorator.

class dagster.DagsterTypeLoaderContext(plan_data, execution_data, log_manager, step, output_capture, known_state)[source]¶

The context object provided to a @dagster_type_loader-decorated function during execution.

Users should not construct this object directly.

property job_def¶

The underlying job definition being executed.

property op_def¶

The op for which type loading is occurring.

property resources¶

The resources available to the type loader, specified by the required_resource_keys argument of the decorator.

dagster.dagster_type_materializer(config_schema, required_resource_keys=None)[source]¶

Create an output materialization hydration config that configurably materializes a runtime value.

The decorated function should take a :py:class:’dagster.DagsterTypeMaterializerContext`, the parsed config value, and the runtime value. It should materialize the runtime value, and should return an appropriate AssetMaterialization.

Parameters:

config_schema (object) – The type of the config data expected by the decorated function.

Examples:

# Takes a list of dicts such as might be read in using csv.DictReader, as well as a config
value, and writes
@dagster_type_materializer(str)
def materialize_df(_context, path, value):
    with open(path, 'w') as fd:
        writer = csv.DictWriter(fd, fieldnames=value[0].keys())
        writer.writeheader()
        writer.writerows(rowdicts=value)

    return AssetMaterialization.file(path)

class dagster.DagsterTypeMaterializer[source]¶

Dagster type materializers are used to materialize outputs of the dagster type they are attached to.

The recommended way to define a type loader is with the @dagster_type_materializer decorator.

class dagster.DagsterTypeMaterializerContext(plan_data, execution_data, log_manager, step, output_capture, known_state)[source]¶

The context object provided to a @dagster_type_materializer-decorated function during execution.

Users should not construct this object directly.

property job_def¶

The underlying job definition being executed.

property op_def¶

The op for which type materialization is occurring.

property resources¶

The resources available to the type materializer, specified by the required_resource_keys argument of the decorator.

dagster.usable_as_dagster_type(name=None, description=None, loader=None, materializer=None)[source]¶

Decorate a Python class to make it usable as a Dagster Type.

This is intended to make it straightforward to annotate existing business logic classes to make them dagster types whose typecheck is an isinstance check against that python class.

Parameters:

• python_type (cls) – The python type to make usable as python type.

• name (Optional[str]) – Name of the new Dagster type. If None, the name (__name__) of the python_type will be used.

• description (Optional[str]) – A user-readable description of the type.

• loader (Optional[DagsterTypeLoader]) – An instance of a class that inherits from DagsterTypeLoader and can map config data to a value of this type. Specify this argument if you will need to shim values of this type using the config machinery. As a rule, you should use the @dagster_type_loader decorator to construct these arguments.

• materializer (Optional[DagsterTypeMaterializer]) – An instance of a class that inherits from DagsterTypeMaterializer and can persist values of this type. As a rule, you should use the @dagster_type_materializer decorator to construct these arguments.

Examples:

# dagster_aws.s3.file_manager.S3FileHandle
@usable_as_dagster_type
class S3FileHandle(FileHandle):
    def __init__(self, s3_bucket, s3_key):
        self._s3_bucket = check.str_param(s3_bucket, 's3_bucket')
        self._s3_key = check.str_param(s3_key, 's3_key')

    @property
    def s3_bucket(self):
        return self._s3_bucket

    @property
    def s3_key(self):
        return self._s3_key

    @property
    def path_desc(self):
        return self.s3_path

    @property
    def s3_path(self):
        return 's3://{bucket}/{key}'.format(bucket=self.s3_bucket, key=self.s3_key)

dagster.make_python_type_usable_as_dagster_type(python_type, dagster_type)[source]¶

Take any existing python type and map it to a dagster type (generally created with DagsterType) This can only be called once on a given python type.