build_asset_reconciliation_sensors , AutoMaterializePolicy works across code locations, as well as allow you to customize the conditions under which each asset is auto-materialized. [docs]FreshnessPolicys, Dagster’s UI now marks them as “overdue” instead of “late”.AthenaClientResource, ECRPublicResource, RedshiftClientResource, S3Resource, S3FileManagerResource, ConfigurablePickledObjectS3IOManager, SecretsManagerResource follow Pythonic resource system. The existing APIs remain supported.DatadogResource follows Pythonic resource system. The existing datadog_resource remains supported.GEContextResource follows Pythonic resource system. The existing ge_context_resource remains supported.GithubResource follows Pythonic resource system. The existing github_resource remains supported.MSTeamsResource follows Pythonic resource system. The existing msteams_resource remains supported.SlackResource follows Pythonic resource system. The existing slack_resource remains supported.pdb.set_trace no longer worked when running Dagster locally using dagster dev or dagit.@asset or Out caused an error to be thrown.pipelines or jobs could lead to blank pages.Definitions are now automatically bound to jobs. This will only result in a change in behavior if you a) have a job with no "io_manager" defined in its resource_defs and b) have supplied an IOManager with key "io_manager" to the resource_defs argument of your Definitions. Prior to 1.3.0, this would result in the job using the default filesystem-based IOManager for the key "io_manager". In 1.3.0, this will result in the "io_manager" supplied to your Definitions being used instead. The BindResourcesToJobs wrapper, introduced in 1.2 to simulate this behavior, no longer has any effect.CeleryK8sRunLauncher is now the same namespace as the Helm chart, instead of the default namespace. To restore the previous behavior, you can set the celeryK8sRunLauncher.jobNamespace field to the string default.SnowflakePandasIOManager has historically converted all timestamp data to strings before storing it in Snowflake. Now, it will instead ensure that timestamp data has a timezone, and if not, attach the UTC timezone. This allows the timestamp data to be stored as timestamps in Snowflake. If you have been storing timestamp data using the SnowflakePandasIOManager you can set the store_timestamps_as_strings=True configuration to continue storing timestamps as strings. For more information, and instructions for migrating Snowflake tables to use timestamp types, see the Migration Guide.Changes to experimental APIs
RunConfig to many APIs which previously would only accept a config dictionary.ResourceDefinition.execution config when constructing a RunConfig object.minutes_late and previous_minutes_late properties on the experimental FreshnesPolicySensorContext have been renamed to minutes_overdue and previous_minutes_overdue, respectively.Removal of deprecated APIs
metadata_entries arguments to event constructors have been removed. While MetadataEntry still exists and will only be removed in 2.0, it is no longer passable to any Dagster public API — users should always pass a dictionary of metadata values instead.latest_materialization_records_by_key function.BigQueryPandasIOManager and BigQueryPySparkIOManager.SnowflakePandasIOManager and SnowflakePySparkIOManager.required_resource_keys and Pythonic resources) are now supported in observable source assets.EcsRunLauncher now allows you to customize volumes and mount points for the launched ECS task. See the API docs for more information.DuckDBPandasIOManager and DuckDBPySparkIOManager follow Pythonic resource system. The existing duckdb_pandas_io_manager and duckdb_pyspark_io_manager remain supported.BigQueryPandasIOManager and BigQueryPySparkIOManager follow Pythonic resource system. The existing bigquery_pandas_io_manager and bigquery_pyspark_io_manager remain supported.gcp_crentials , a temporary file to store the credentials will be created and the GOOGLE_APPLICATION_CREDENTIALS environment variable will be set to the temporary file. When the BigQuery resource is garbage collected, the environment variable will be unset and the temporary file deleted.SnowflakePandasIOManager and SnowflakePySparkIOManager follow Pythonic resource system. The existing snowflake_pandas_io_manager and snowflake_pyspark_io_manager remain supported.dagster dev would periodically emit a harmless but annoying warning every few minutes about a gRPC server being shut down.RunRequest(partition_key=...) object.PartitionsDefinition.run_request_for_partition would incorrectly raise an error for a job with a DynamicPartitionsDefinition that was defined with a function.define_asset_job would raise an error.__ASSET_JOB has no op named ... errors when using automatic run retries.EcsRunLauncher would fail to launch runs if the use_current_ecs_task_config field was set to False but no task_definition field was set.EcsRunLauncher now allows you to set a capacity provider strategy and customize the ephemeral storage used for launched ECS tasks. See the docs for details. Thanks AranVinkItility!AssetsDefinition.from_op. Thanks @tghanken for the fix!minimum_interval_seconds parameter to enable customizing the evaluation interval on the slack run failure sensor, thanks @ldnicolasmay!Resource annotation for Pythonic resource inputs has been renamed to ResourceParam in preparation for the release of the feature in 1.3.materialize and materialize_to_memory now both accept a selection argument that allows specifying a subset of assets to materialize.MultiPartitionsDefinition is no longer marked experimental.MultiPartitionsDefinitions with a time dimension.load_assets_from_package_module and similar methods now accept a freshness_policy, which will be applied to all loaded assets.build_sensor_context and build_multi_asset_sensor_context can now take a Definitions object in place of a RepositoryDefinitions3_resource now accepts use_ssl and verify configurations.selection argument on define_asset_job.MultiPartitionsDefinition.MultiPartitionsMapping (experimental) is loaded.TimeWindowPartitionMappings to not be bubbled up to the UI.pdb stopped working when using dagster dev.AssetMaterialization objects with a null AssetKey.TimeWindowPartitionsDefinition with a non-standard cron schedule, and also provided a minute_of_hour or similar argument in build_schedule_from_partitioned_job. Dagster would silently create the wrong cron expression. It now raises an error.PartitionsDefinition. These partitions are now ignored.PersistentAirflowDatabase where versions of airflow from 2.0.0 till 2.3.0 would not use the correct connection environment variable name.poll_sync_run function ofdagster-census that prevented polling from working correctly (thanks @ldincolasmay!)run_request_for_partition method on JobDefinition and UnresolvedAssetJobDefinition is now deprecated and will be removed in 2.0.0. Instead, directly instantiate a run request with a partition key via RunRequest(partition_key=...).AssetsDefinition.to_source_assets to method convert a set of assets to SourceAsset objects.MultiPartitionsDefinitions.mlflow pin from the dagster-mlflow package.When using build_asset_reconciliation_sensor, in some cases duplicate runs could be produced for the same partition of an asset. This has been fixed.
When using Pythonic configuration for resources, aliased field names would cause an error. This has been fixed.
Fixed an issue where context.asset_partitions_time_window_for_output threw an error when an asset was directly invoked with build_op_context.
[dagster-dbt] In some cases, use of ephemeral dbt models could cause the dagster representation of the dbt dependency graph to become incorrect. This has been fixed.
[celery-k8s] Fixed a bug that caused JSON deserialization errors when an Op or Asset emitted JSON that doesn't represent a DagsterEvent.
Fixed an issue where launching a large backfill while running dagster dev would sometimes fail with a connection error after running for a few minutes.
Fixed an issue where dagster dev would sometimes hang when running Dagster code that attempted to read in input via stdin.
Fixed an issue where runs that take a long time to import code would sometimes continue running even after they were stopped by run monitoring for taking too long to start.
Fixed an issue where AssetSelection.groups() would simultaneously select both source and regular assets and consequently raise an error.
Fixed an issue where BindResourcesToJobs would raise errors encapsulating jobs which had config specified at definition-time.
Fixed Pythonic config objects erroring when omitting optional values rather than specifying None.
Fixed Pythonic config and resources not supporting Enum values.
DagsterInstance.local_temp and DagsterInstance.ephemeral now use object instance scoped local artifact storage temporary directories instead of a shared process scoped one, removing a class of thread safety errors that could manifest on initialization.
Improved direct invocation behavior for ops and assets which specify resource dependencies as parameters, for instance:
class MyResource(ConfigurableResource): pass @op def my_op(x: int, y: int, my_resource: MyResource) -> int: return x + y my_op(4, 5, my_resource=MyResource())
[dagster-azure] Fixed an issue with an AttributeError being thrown when using the async DefaultAzureCredential (thanks @mpicard)
[ui] Fixed an issue introduced in 1.2.3 in which no log levels were selected by default when viewing Run logs, which made it appear as if there were no logs at all.
environment_vars argument to ScheduleDefinition is deprecated (the argument is currently non-functional; environment variables no longer need to be whitelisted for schedules)--path-prefix can now be supplied via Helm chart (thanks @mpicard)