In this step, you'll load the dbt models into Dagster as assets using the dagster-dbt library.
Open the __init__.py file, located in /tutorial_template/tutorial_dbt_dagster/assets, and add the following code:
from dagster_dbt import load_assets_from_dbt_project
from dagster import file_relative_path
DBT_PROJECT_PATH = file_relative_path(__file__,"../../jaffle_shop")
DBT_PROFILES = file_relative_path(__file__,"../../jaffle_shop/config")
dbt_assets = load_assets_from_dbt_project(
project_dir=DBT_PROJECT_PATH, profiles_dir=DBT_PROFILES, key_prefix=["jaffle_shop"])
Let's discuss what this example is doing, specifically the load_assets_from_dbt_project function. This function loads dbt models into Dagster as assets, creating one Dagster asset for each model.
When invoked, this function:
Compiles your dbt project,
Parses the metadata provided by dbt, and
Generates a set of software-defined assets reflecting the models in the project. These assets share the same underlying op, which will invoke dbt to run the models represented by the loaded assets.
Let's take a look at the arguments we've supplied:
project_dir, which is the path to the dbt project
profiles_dir, which is the path to the dbt project's connection profiles
key_prefix, which is a prefix to apply to all models in the dbt project
Step 2: Add a Dagster repository and supply a dbt resource#
Next, you'll create a Dagster repository to supply resources to the dbt assets from the previous step.
Assets loaded from dbt require a dbt resource, which is responsible for firing off dbt CLI commands. Using the dbt_cli_resource resource, we can supply a dbt resource to the dbt project.
Open the repository.py file, located in /tutorial_template/tutorial_dbt_dagster, and add the following code:
import os
from dagster_dbt import dbt_cli_resource
from tutorial_dbt_dagster import assets
from tutorial_dbt_dagster.assets import DBT_PROFILES, DBT_PROJECT_PATH
from dagster import load_assets_from_package_module, repository, with_resources
@repositorydeftutorial_dbt_dagster():return with_resources(
load_assets_from_package_module(assets),{"dbt": dbt_cli_resource.configured({"project_dir": DBT_PROJECT_PATH,"profiles_dir": DBT_PROFILES,},),},)
Let's take a look at what's happening here:
Using with_resources, we've provided resources to the assets in the repository. In this example, that's the dbt_cli_resource resource.
Using load_assets_from_package_module, we've imported all assets in the assets module into the repository. This approach allows any new assets we create to be automatically added to the repository instead of needing to manually add them one by one.