Releases: apache/airflow
Releases · apache/airflow
Airflow 1.10.13, 2020-11-25
New Features
- Add "already checked" to failed pods in K8sPodOperator (#11368)
- Pass SQLAlchemy engine options to FAB based UI (#11395)
- [AIRFLOW-4438] Add Gzip compression to S3_hook (#8571)
- Add permission "extra_links" for Viewer role and above (#10719)
- Add generate_yaml command to easily test KubernetesExecutor before deploying pods (#10677)
- Add Secrets backend for Microsoft Azure Key Vault (#10898)
Bug Fixes
- SkipMixin: Handle empty branches (#11120)
- [AIRFLOW-5274] dag loading duration metric name too long (#5890)
- Handle no Dagrun in DagrunIdDep (#8389) (#11343)
- Fix Kubernetes Executor logs for long dag names (#10942)
- Add on_kill support for the KubernetesPodOperator (#10666)
- KubernetesPodOperator template fix (#10963)
- Fix displaying of add serialized_dag table migration
- Fix Start Date tooltip on DAGs page (#10637)
- URL encode execution date in the Last Run link (#10595)
- Fixes issue with affinity backcompat in Airflow 1.10
- Fix KubernetesExecutor import in views.py
- Fix issues with Gantt View (#12419)
- Fix Entrypoint and _CMD config variables (#12411)
- Fix operator field update for SerializedBaseOperator (#10924)
- Limited cryptography to < 3.2 for python 2.7
- Install cattr on Python 3.7 - Fix docs build on RTD (#12045)
- Limit version of marshmallow-sqlalchemy
- Pin
kubernetes
to a max version of 11.0.0 (#11974) - Use snakebite-py3 for HDFS dependency for Python3 (#12340)
- Removes snakebite kerberos dependency (#10865)
- Fix failing dependencies for FAB and Celery (#10828)
- Fix pod_mutation_hook for 1.10.13 (#10850)
- Fix formatting of Host information
- Fix Logout Google Auth issue in Non-RBAC UI (#11890)
- Add missing imports to app.py (#10650)
- Show Generic Error for Charts & Query View in old UI (#12495)
- TimeSensor should respect the default_timezone config (#9699)
- TimeSensor should respect DAG timezone (#9882)
- Unify user session lifetime configuration (#11970)
- Handle outdated webserver session timeout gracefully. (#12332)
Improvements
- Add XCom.deserialize_value to Airflow 1.10.13 (#12328)
- Mount airflow.cfg to pod_template_file (#12311)
- All k8s object must comply with JSON Schema (#12003)
- Validate airflow chart values.yaml & values.schema.json (#11990)
- Pod template file uses custom custom env variable (#11480)
- Bump attrs and cattrs dependencies (#11969)
- Bump attrs to > 20.0 (#11799)
- [AIRFLOW-3607] Only query DB once per DAG run for TriggerRuleDep (#4751)
- Rename task with duplicate task_id
- Manage Flask AppBuilder Tables using Alembic Migrations (#12352)
airflow test
only works for tasks in 1.10, not whole dags (#11191)- Improve warning messaging for duplicate task_ids in a DAG (#11126)
- Pins moto to 1.3.14 (#10986)
- DbApiHook: Support kwargs in get_pandas_df (#9730)
- Make grace_period_seconds option on K8sPodOperator (#10727)
- Fix syntax error in Dockerfile 'maintainer' Label (#10899)
- The entrypoints in Docker Image should be owned by Airflow (#10853)
- Make dockerfiles Google Shell Guide Compliant (#10734)
- clean-logs script for Dockerfile: trim logs before sleep (#10685)
- When sending tasks to celery from a sub-process, reset signal handlers (#11278)
- SkipMixin: Add missing session.commit() and test (#10421)
- Webserver: Further Sanitize values passed to origin param (#12459)
- Security upgrade lodash from 4.17.19 to 4.17.20 (#11095)
- Log instead of raise an Error for unregistered OperatorLinks (#11959)
- Mask Password in Log table when using the CLI (#11468)
- [AIRFLOW-3607] Optimize dep checking when depends on past set and concurrency limit
- Execute job cancel HTTPRequest in Dataproc Hook (#10361)
- Use rst lexer to format airflow upgrade check output (#11259)
- Remove deprecation warning from contrib/kubernetes/pod.py
- adding body as templated field for CloudSqlImportOperator (#10510)
- Change log level for User's session to DEBUG (#12414)
Deprecations
- Deprecate importing Hooks from plugin-created module (#12133)
- Deprecate adding Operators and Sensors via plugins (#12069)
Doc only changes
- [Doc] Correct description for macro task_instance_key_str (#11062)
- Checks if all the libraries in setup.py are listed in installation.rst file (#12023)
- Revise "Project Focus" copy (#12011)
- Move Project focus and Principles higher in the README (#11973)
- Remove archived link from README.md (#11945)
- Update download url for Airflow Version (#11800)
- Add Project URLs for PyPI page (#11801)
- Move Backport Providers docs to our docsite (#11136)
- Refactor rebase copy (#11030)
- Add missing images for kubernetes executor docs (#11083)
- Fix identation in executor_config example (#10467)
- Enhanced the Kubernetes Executor doc (#10433)
- Refactor content to a markdown table (#10863)
- Rename "Beyond the Horizon" section and refactor content (#10802)
- Refactor official source section to use bullets (#10801)
- Add section for official source code (#10678)
- Add redbubble link to Airflow merchandise (#10359)
- README Doc: Link to Airflow directory in ASF Directory (#11137)
- Fix the default value for VaultBackend's config_path (#12518)
Airflow 2.0.0b3
Major features since beta2:
New features:
- Add description field to connection (#10873)
- Use different deserialization method in XCom init_on_load (#12327)
- Make nav fully accessible y keyboard, fix mobile nav menus (#12351)
- Add metric for scheduling delay between first run task & expected start time (#9544)
- Create DAG-level cluster policy (#12184)
Improvements:
- UI: Improve the layout of TI modal when browser at narrower widths (#12456)
- PR to add 'files' to template-fields in EmailOperator class (#12428)
- JSON Response is returned for invalid API requests (#12305)
- Make K8sPodOperator backwards compatible (#12384)
- Don't display DAG docs when None (#12415)
- Simplify using XComArg in jinja template string (#12405)
- Remove inapplicable configuration section [ldap] (since 2.0.0) (#12386)
- Add info log message about duration taken to load plugins (#12308)
- Manage Flask AppBuilder Tables using Alembic Migrations (#12352)
- UI: Improve presentation of DAG Docs (#12330)
- Wait option for dagrun operator (#12126)
- Use default view in TriggerDagRunLink (#11778)
- Handle naive datetimes in REST APIi (#12248)
- Unify user session lifetime configuration for webserver (#11970)
Bug fixes:
- Fix typo in migrations: RESOURCE_DAGS to RESOURCE_DAG. (#12460)
- UI: Fix bug in server timezone indicator (#12447)
- Fix issues with Gantt View (#12419)
- Proper exit status for failed CLI requests (#12375)
- Check for TaskGroup in _PythonDecoratedOperator (#12312)
- Reject 'connections add' CLI request if URI provided is invalid (#12370)
- Reorder Database Migrations (#12362)
- Refactor root logger handling in task run (#12342)
- Bugfix: REST API Variables update endpoint returns 204 No Content (#12321)
- UI: Fix/Enhancement: Disable forms and communicate to user when no DAG Runs (#12320)
- UI: Get all "tags" parameters not just one (#12324)
- Make dag_id, task_id, and execution_date nullable in event log schema (#12287)
- Don't wrap warrning messages when stderr is not a TTY (#12285)
- Don't treat warning message as rich formatting codes. (#12283)
- UI: Fix pause/unpause toggle to display failed state when unsuccessful (#12267)
- UI: Conditionally update button URL only when it is present (#12268)
Docs:
- Cleanup requirements in README a bit (#12446)
- Synchronization of supported K8S version (#12443)
- Show all Providers in Docs (#12363)
- Restructure documentation for releasing Airflow/Providers (#12350)
- Fix Sample CLI commands for upgrading to 2.0 (#12349)
- Docs installation improvements (#12304)
- Remove providers imports from core examples (#12252)
Misc/Internal:
- Move setup properties out of setup.py in to setup.cfg (#12417)
- Change log level for User's session to DEBUG (#12414)
- Adds mechanism for provider package discovery. (#12383)
- Add success/failed sets to State class (#12359)
- Enable protection for v1-10-stable branch (#12343)
- Fix prod image build (#12314)
- Remove deprecated Elasticsearch Configs (#12296)
- Remove deprecated BashTaskRunner (#12295)
- Added k9s as integrated tool to help with kubernetes testing (#12163)
- Providers in extras are properly configured and verified (#12265)
Airflow 2.0.0b2
- Update versions in UPDATING.md for 2.0.0b1 release (#12244)
- Remove Unnecessary comprehension (#12221)
- Sign release files with an apache.org key by default (#12241)
- Add back missing api_connextion/init.py file (#12240)
- Fixes automated provider installation with extras (#12233)
- Enable markdownlint rule - MD031/blanks-around-fences (#12238)
- Fix spelling in Python files (#12230)
- Add docs about security on GCP (#12187)
- Enable Markdownlint rule - MD022/blanks-around-headings (#12225)
- CSS Changes to adjust content width as per screen size and responsive table with multiline td. (#12227)
Airflow 1.10.12, 2020-08-25
New Features
- Add DateTimeSensor (#9697)
- Add ClusterPolicyViolation support to airflow local settings (#10282)
- Get Airflow configs with sensitive data from Secret Backends (#9645)
- [AIRFLOW-4734] Upsert functionality for PostgresHook.insert_rows() (#8625)
- Allow defining custom XCom class (#8560)
Bug Fixes
- Add pre 1.10.11 Kubernetes Paths back with Deprecation Warning (#10067)
- Fixes PodMutationHook for backwards compatibility (#9903)
- Fix bug in executor_config when defining resources (#9935)
- Respect DAG Serialization setting when running sync_perm (#10321)
- Show correct duration on graph view for running task (#8311) (#8675)
- Fix regression in SQLThresholdCheckOperator (#9312)
- [AIRFLOW-6931] Fixed migrations to find all dependencies for MSSQL (#9891)
- Avoid sharing session with RenderedTaskInstanceFields write and delete (#9993)
- Fix clear future recursive when ExternalTaskMarker is used (#9515)
- Handle IntegrityError while creating TIs (#10136)
- Fix airflow-webserver startup errors when using Kerberos Auth (#10047)
- Fixes treatment of open slots in scheduler (#9316) (#9505)
- Fix KubernetesPodOperator reattachment (#10230)
- Fix more PodMutationHook issues for backwards compatibility (#10084)
- [AIRFLOW-5391] Do not re-run skipped tasks when they are cleared (#7276)
- Fix task_instance_mutation_hook (#9910)
- Fixes failing formatting of DAG file containing {} in docstring (#9779)
- Fix is_terminal_support_colors function (#9734)
- Fix PythonVirtualenvOperator when using
provide_context=True
(#8256) - Fix issue with mounting volumes from secrets (#10366)
- BugFix: K8s Executor Multinamespace mode is evaluated to true by default (#10410)
- Make KubernetesExecutor recognize kubernetes_labels (#10412)
- Fix broken Kubernetes PodRuntimeInfoEnv (#10478)
Improvements
- Use Hash of Serialized DAG to determine DAG is changed or not (#10227)
- Update Serialized DAGs in Webserver when DAGs are Updated (#9851)
- Do not Update Serialized DAGs in DB if DAG did not change (#9850)
- Add repr to SerializedDagModel (#9862)
- Update JS packages to latest versions (#9811) (#9921)
- UI Graph View: Focus upstream / downstream task dependencies on mouseover (#9303)
- Allow
image
inKubernetesPodOperator
to be templated (#10068) - [AIRFLOW-6843] Add delete_option_kwargs to delete_namespaced_pod (#7523)
- Improve process terminating in scheduler_job (#8064)
- Replace deprecated base classes used in bigquery_check_operator (#10272)
- [AIRFLOW-5897] Allow setting -1 as pool slots value in webserver (#6550)
- Limit all google-cloud api to <2.0.0 (#10317)
- [AIRFLOW-6706] Lazy load operator extra links (#7327) (#10318)
- Add Snowflake support to SQL operator and sensor (#9843)
- Makes multi-namespace mode optional (#9570)
- Pin Pyarrow < 1.0
- Pin pymongo version to <3.11.0
- Pin google-cloud-container to <2 (#9901)
- Dockerfile: Remove package.json and yarn.lock from the prod image (#9814)
- Dockerfile: The group of embedded DAGs should be root to be OpenShift compatible (#9794)
- Update upper limit of flask-swagger, gunicorn & jinja2 (#9684)
- Webserver: Sanitize values passed to origin param (#10334)
- Sort connection type list in add/edit page alphabetically (#8692)
Doc only changes
- Add new committers: Ry Walker & Leah Cole to project.rst (#9892)
- Add Qingping Hou to committers list (#9725)
- Updated link to official documentation (#9629)
- Create a short-link for Airflow Slack Invites (#10034)
- Fix docstrings in BigQueryGetDataOperator (#10042)
- Set language on code-block on docs/howto/email-config.rst (#10238)
- Remove duplicate line from 1.10.10 CHANGELOG (#10289)
- Improve heading on Email Configuration page (#10175)
- Fix link for the Jinja Project in docs/tutorial.rst (#10245)
- Create separate section for Cron Presets (#10247)
- Add Syntax Highlights to code-blocks in docs/best-practices.rst (#10258)
- Fix docstrings in BigQueryGetDataOperator (#10042)
- Fix typo in Task Lifecycle section (#9867)
- Make Secret Backend docs clearer about Variable & Connection View (#8913)
Airflow 1.10.11, 2020-07-10
New Features
- Add task instance mutation hook (#8852)
- Allow changing Task States Colors (#9520)
- Add support for AWS Secrets Manager as Secrets Backend (#8186)
- Add airflow info command to the CLI (#8704)
- Add Local Filesystem Secret Backend (#8596)
- Add Airflow config CLI command (#8694)
- Add Support for Python 3.8 (#8836)(#8823)
- Allow K8S worker pod to be configured from JSON/YAML file (#6230)
- Add quarterly to crontab presets (#6873)
- Add support for ephemeral storage on KubernetesPodOperator (#6337)
- Add AirflowFailException to fail without any retry (#7133)
- Add SQL Branch Operator (#8942)
Bug Fixes
- Use NULL as dag.description default value (#7593)
- BugFix: DAG trigger via UI error in RBAC UI (#8411)
- Fix logging issue when running tasks (#9363)
- Fix JSON encoding error in DockerOperator (#8287)
- Fix alembic crash due to typing import (#6547)
- Correctly restore upstream_task_ids when deserializing Operators (#8775)
- Correctly store non-default Nones in serialized tasks/dags (#8772)
- Correctly deserialize dagrun_timeout field on DAGs (#8735)
- Fix tree view if config contains " (#9250)
- Fix Dag Run UI execution date with timezone cannot be saved issue (#8902)
- Fix Migration for MSSQL (#8385)
- RBAC ui: Fix missing Y-axis labels with units in plots (#8252)
- RBAC ui: Fix missing task runs being rendered as circles instead (#8253)
- Fix: DagRuns page renders the state column with artifacts in old UI (#9612)
- Fix task and dag stats on home page (#8865)
- Fix the trigger_dag api in the case of nested subdags (#8081)
- UX Fix: Prevent undesired text selection with DAG title selection in Chrome (#8912)
- Fix connection add/edit for spark (#8685)
- Fix retries causing constraint violation on MySQL with DAG Serialization (#9336)
- [AIRFLOW-4472] Use json.dumps/loads for templating lineage data (#5253)
- Restrict google-cloud-texttospeach to <v2 (#9137)
- Fix pickling failure when spawning processes (#8671)
- Pin Version of Azure Cosmos to <4 (#8956)
- Azure storage 0.37.0 is not installable any more (#8833)
- Fix modal_backdrop z-index in the UI (#7313)
- Fix Extra Links in Gannt View (#8308)
- Bug fix for EmrAddStepOperator init with cluster_name error (#9235)
- Fix KubernetesPodOperator pod name length validation (#8829)
- Fix non updating DAG code by checking against last modification time (#8266)
- BugFix: Unpausing a DAG with catchup=False creates an extra DAG run (#8776)
Improvements
- Improve add_dag_code_table migration (#8176)
- Persistent display/filtering of DAG status (#8106)
- Set unique logger names (#7330)
- Update the version of cattrs from 0.9 to 1.0 to support Python 3.8 (#7100)
- Reduce response payload size of /dag_stats and /task_stats (#8655)
- Add authenticator parameter to snowflake_hook (#8642)
- Show "Task Reschedule" table in Airflow Webserver (#9521)
- Change worker_refresh_interval fallback to default of 30 (#9588)
- Use pfromat instead of str to render arguments in WebUI (#9587)
- Simplify DagFileProcessorManager (#7521)
- Reload gunicorn when plugins has beeen changed (#8997)
- Fix the default value for store_dag_code (#9554)
- Add support for fetching logs from running pods (#8626)
- Persist start/end date and duration for DummyOperator Task Instance (#8663)
- Ensure "started"/"ended" in tooltips are not shown if job not started (#8667)
- Add context to execution_date_fn in ExternalTaskSensor (#8702)
- Avoid color info in response of
/dag_stats
&/task_stats
(#8742) - Make loading plugins from entrypoint fault-tolerant (#8732)
- Refactor Kubernetes worker config (#7114)
- Add default
conf
parameter to Spark JDBC Hook (#8787) - Allow passing backend_kwargs to AWS SSM client (#8802)
- Filter dags by clicking on tag (#8897)
- Support k8s auth method in Vault Secrets provider (#8640)
- Monitor pods by labels instead of names (#6377)
- Optimize count query on /home (#8729)
- Fix json string escape in tree view (#8551)
- Add TaskInstance state to TI Tooltip to be colour-blind friendlier (#8910)
- Add a tip to trigger DAG screen (#9049)
- Use Markup for htmlcontent for landing_times (#9242)
- Pinning max pandas version to 2.0 (lesser than) to allow pandas 1.0 (#7954)
- Update example webserver_config.py to show correct CSRF config (#8944)
- Fix displaying Executor Class Name in "Base Job" table (#8679)
- Use existing DagBag for 'dag_details' & 'trigger' Endpoints (#8501)
- Flush pending Sentry exceptions before exiting (#7232)
- Display DAG run conf in the list view (#6794)
- Fix performance degradation when updating dagrun state (#8435)
- Don't use the
|safe
filter in code, it's risky (#9180) - Validate only task commands are run by executors (#9178)
- Show Deprecation warning on duplicate Task ids (#8728)
- Move DAG._schedule_interval logic out of
DAG.__init__
(#8225) - Make retrieving Paused Dag ids a separate method (#7587)
- Bulk fetch paused_dag_ids (#7476)
- Add a configurable DAGs volume mount path for Kubernetes (#8147)
- Add schedulername option for KubernetesPodOperator (#6088)
- Support running git sync container as root (#6312)
- Add extra options for Slack Webhook operator and Slack hook (#9409)
- Monkey patch greenlet celery pools (#8559)
- Decrypt secrets from SystemsManagerParameterStoreBackend (#9214)
- Prevent clickable sorting on non sortable columns in TI view (#8681)
- Make hive macros py3 compatible (#8598)
- Fix SVG tooltip positioning with custom scripting (#8269)
- Avoid unnecessary sleep to maintain local task job heart rate (#6553)
- Include some missing RBAC roles on User and Viewer roles (#9133)
- Show Dag's Markdown docs on Tree View (#9448)
- Improved compatibility with Python 3.5+ - Convert signal.SIGTERM to int (#9207)
- Add 'main' param to template_fields in DataprocSubmitPySparkJobOperator (#9154)
- Make it possible to silence warnings from Airflow (#9208)
- Remove redundant count query in BaseOperator.clear() (#9362)
- Fix DB migration message (#8988)
- Fix awkward log info in dbapi_hook (#8482)
- Fix Celery default to no longer allow pickle (#7205)
- Further validation that only task commands are run by executors (#9240)
- Remove vendored nvd3 and slugify libraries (#9136)
- Enable configurable git sync depth (#9094)
- Reduce the required resources for the Kubernetes's sidecar (#6062)
- Refactor K8S codebase with k8s API models (#5481)
- Move k8sexecutor out of contrib to closer match master (#8904)
- Allow filtering using "event" and "owner" in "Log" view (#4881)
- Add Yandex.Cloud custom connection to 1.10 (#8791)
- Add table-hover css class to DAGs table (#5033)
- Show un/pause errors in dags view. (#7669)
- Restructure database queries on /home (#4872)
- Add Cross Site Scripting defence (#6913)
- Make Gantt tooltip the same as Tree and Graph view (#8220)
- Add config to only delete worker pod on task failure (#7507)(#8312)
- Remove duplicate error message on chart connection failure (#8476)
- Remove default value spark_binary (#8508)
- Expose Airflow Webserver Port in Production Docker Image (#8228)
- Commit after each alembic migration (#4797)
- KubernetesPodOperator fixes and test (#6524)
- Docker Image: Add ADDITIONAL_AIRFLOW_EXTRAS (#9032)
- Docker Image: Add ADDITIONAL_PYTHON_DEPS (#9031)
- Remove httplib2 from Google requirements (#9194)
- Merging multiple sql operators (#9124)
- Adds hive as extra in pyhive dependency (#9075)
- Change default auth for experimental backend to deny_all (#9611)
- Restrict changing XCom values from the Webserver (#9614)
- Add repr for DagTag so tags display properly in /dagmodel/show (#8719)
- Functionality to shuffle HMS connections used by HiveMetastoreHook facilitating load balancing (#9280)
- Expose SQLAlchemy's connect_args and make it configurable (#6478)
Doc only changes
- Add docs on using DAGRun.conf (#9578)
- Enforce code-block directives in doc (#9443)
- Carefully parse warning messages when building documentation (#8693)
- Make KubernetesPodOperator clear in docs (#8444)
- Improve language in Pod Mutation Hook docs (#8445)
- Fix formatting of Pool docs in concepts.rst (#8443)
- Make doc clearer about Airflow Variables using Environment Variables (#8427)
- Fix pools doc for LocalExecutor (#7643)
- Add Local and Sequential Executors to Doc (#8084)
- Add documentation for CLI command airflow dags test (#8251)
- Fix typo in DAG Serialization documentation (#8317)
- Add scheduler in production section (#7351)
- Add a structural dag validation example (#6727)
- Adding Task re-run documentation (#6295)
- Fix outdated doc on settings.policy (#7532)
- Add docs about reload_on_plugin_change opiton (#9575)
- Add copy button to Code Blocks in Airflow Docs (#9450)
- Update commands in docs for v1.10+ (#9585)
- Add more info on dry-run CLI option (#9582)
- Document default timeout value for SSHOperator (#8744)
- Fix docs on creating CustomOperator (#8678)
- Enhanced documentation around Cluster Policy (#8661)
- Use sphinx syntax in concepts.rst (#7729)
- Update README to remove Py 3.8 limitation for Master (#9451)
- Add note about using dag_run.conf in BashOperator (#9143)
- Improve tutorial - Include all imports statements (#8670)
- Added more precise Python requirements to README.md (#8455)
- Fix Airflow Stable version in README.md (#9360)
- Update AWS connection example to show how to set from env var (#9191)
- Fix list formatting of plugins doc. (#8873)
- Add 'Version Added' on Secrets Backend docs (#8264)
- Simplify language re roll-your-own secrets backend (#8257)
- Add installation description for repeatable PyPi installation (#8513)
- Add note extra links only render on when using RBAC webserver (#8788)
- Remove unused airflow import from docs (#9274)
- Add PR/issue note in Contribution Workflow Example (#9177)
- Don't use the term "whitelist" - lan...
Airflow 1.10.10, 2020-04-09
New Features
- [AIRFLOW-7048] Allow user to chose timezone to use in UI (#8046)
- Add Production Docker image support (#7832)
- Get Airflow Variables from Environment Variables (#7923)
- Get Airflow Variables from Hashicorp Vault (#7944)
- Get Airflow Variables from AWS Systems Manager Parameter Store (#7945)
- Get Airflow Variables from GCP Secrets Manager (#7946)
- [AIRFLOW-5705] Add secrets backend and support for AWS SSM / Get Airflow Connections from AWS Parameter Store(#6376)
- [AIRFLOW-7104] Add Secret backend for GCP Secrets Manager / Get Airflow Connections from GCP Secrets Manager (#7795)
- [AIRFLOW-7076] Add support for HashiCorp Vault as Secrets Backend / Get Airflow Connections from Hashicorp Vault (#7741)
- [AIRFLOW-6685] Add ThresholdCheckOperator (#7353)
- [AIRFLOW-7080] Add API endpoint to return a DAG's paused state (#7737)
Improvements
- Use same tooltip for Graph and Tree views for TaskInstances (#8043)
- Allow DateTimePicker in Webserver to actually pick times too (#8034)
- [AIRFLOW-5590] Add run_id to trigger DAG run API response (#6256)
- [AIRFLOW-6695] Can now pass dagrun conf when triggering dags via UI (#7312)
- [AIRFLOW-5336] Add ability to make updating FAB perms on webserver in it optional (#5940)
- [AIRFLOW-1467] Allow tasks to use more than one pool slot (#7160)
- [AIRFLOW-6987] Avoid creating default connections (#7629)
- [AIRFLOW-4175] S3Hook load_file should support ACL policy parameter (#7733)
- [AIRFLOW-4438] Add Gzip compression to S3_hook (#7680)
- Allow setting Airflow Variable values to empty string (#8021)
- Dont schedule dummy tasks (#7880)
- Prevent sequential scan of task instance table (#8014)
- [AIRFLOW-7017] Respect default dag view in trigger dag origin (#7667)
- [AIRFLOW-6837] Limit description length of a Dag on HomePage (#7457)
- [AIRFLOW-6989] Display Rendered template_fields without accessing Dag files (#7633)
- [AIRFLOW-5944] Rendering templated_fields without accessing DAG files (#6788)
- [AIRFLOW-5946] DAG Serialization: Store source code in db (#7217)
- [AIRFLOW-7079] Remove redundant code for storing template_fields (#7750)
- [AIRFLOW-7024] Add the verbose parameter support to SparkSqlOperator (#7676)
- [AIRFLOW-6733] Extend not replace template (#7366)
- [AIRFLOW-7001] Further fix for the MySQL 5.7 UtcDateTime (#7655)
- [AIRFLOW-6014] Handle pods which are preempted & deleted by kubernetes but not restarted (#6606)
- [AIRFLOW-6950] Remove refresh_executor_config from ti.refresh_from_db (#7577)
- [AIRFLOW-7016] Sort dag tags in the UI (#7661)
- [AIRFLOW-6762] Fix link to "Suggest changes on this page" (#7387)
- [AIRFLOW-6948] Remove ASCII Airflow from version command (#7572)
- [AIRFLOW-6767] Correct name for default Athena workgroup (#7394)
- [AIRFLOW-6905] Update pin.svg with new pinwheel (#7524)
- [AIRFLOW-6801] Make use of ImportError.timestamp (#7425)
- [AIRFLOW-6830] Add Subject/MessageAttributes to SNS hook and operator (#7451)
- [AIRFLOW-6630] Resolve handlebars advisory (#7284)
- [AIRFLOW-6945] MySQL 5.7 is used in v1-10-test as an option
- [AIRFLOW-6871] Optimize tree view for large DAGs (#7492)
- [AIRFLOW-7063] Fix dag.clear() slowness caused by count (#7723)
- [AIRFLOW-7023] Remove duplicated package definitions in setup.py (#7675)
- [AIRFLOW-7001] Time zone removed from MySQL TIMSTAMP field inserts
- [AIRFLOW-7105] Unify Secrets Backend method interfaces (#7830)
- Make BaseSecretsBackend.build_path generic (#7948)
- Allow hvac pakage installation using 'hashicorp' extra (#7915)
- Standardize SecretBackend class names (#7846)
- [AIRFLOW-5705] Make AwsSsmSecretsBackend consistent with VaultBackend (#7753)
- [AIRFLOW-7045] Update SQL query to delete RenderedTaskInstanceFields (#8051)
- Handle DST better in Task Instance tool tips (#8104)
Bug Fixes
- BugFix: Show task_id in the Graph View tooltip (#7859)
- [AIRFLOW-6730] Use total_seconds instead of seconds (#7363)
- [AIRFLOW-6167] Escape column name in create table in hive (#6741)
- [AIRFLOW-6628] DAG auto-complete now suggests from all acessible DAGs (#7251)
- [AIRFLOW-7113] Fix gantt render error (#7913)
- [AIRFLOW-6399] Add _access control to validate deserialized DAGs (#7896)
- [AIRFLOW-6399] Serialization: DAG access_control field should be decorated field in DAG serialization (#7879)
- [AIRFLOW-4453] Make behavior of
none_failed
consistent with documentation (#7464) - [AIRFLOW-4363] Fix JSON encoding error (#7628)
- [AIRFLOW-6683] Run REST API tests when DAGs are serialized (#7352)
- [AIRFLOW-6704] Copy common TaskInstance attributes from Task (#7324)
- [AIRFLOW-6734] Use configured base_template instead of hard-coding (#7367)
- [AIRFLOW-7098] Simple salesforce release 1.0.0 breaks the build (#7775)
- [AIRFLOW-6062] Executor would only delete workers in its own namespace (#7123)
- [AIRFLOW-7074] Add Permissions to view SubDAGs (#7752)
- [AIRFLOW-7025] Fix SparkSqlHook.run_query to handle its parameter properly (#7677)
- [AIRFLOW-6855] Escape project_dataset_table in SQL query in gcs to bq operator (#7475)
- [AIRFLOW-6949] Respect explicit conf to SparkSubmitOperator (#7575)
- [AIRFLOW-6588] write_stdout and json_format are boolean (#7199)
- [AIRFLOW-3439] Decode logs with 'utf-8' (#4474)
- [AIRFLOW-6878] Fix misconfigured default value for kube_client_request_args
- [AIRFLOW-5167] Update dependencies for GCP packages (#7116)
- [AIRFLOW-6821] Success callback not called when task marked as success from UI (#7447)
- [AIRFLOW-6740] Remove Undocumented, deprecated, dysfunctional PROXY_FIX_NUM_PROXIES (#7359)
- [AIRFLOW-6728] Change various DAG info methods to POST (#7364)
- [AIRFLOW-6997] Make sure worker pods initcontainers obtain env vars from config (#7663)
- [AIRFLOW-7062] Fix pydruid release breaking the build (#7720)
- [AIRFLOW-6040] ReadTimoutError in KubernetesExecutor should not raise exception (#7616)
- [AIRFLOW-6943] Fix utf-8 encodeed description in DAG in Python 2 (#7567)
- [AIRFLOW-6892] Fix broken non-wheel releases (#7514)
- [AIRFLOW-6789] BugFix: Fix Default Worker concurrency (#7494)
- [AIRFLOW-6840] Bump up version of future (#7471)
- [AIRFLOW-5705] Fix bugs in AWS SSM Secrets Backend (#7745)
- [AIRFLOW-5705] Fix bug in Secrets Backend (#7742)
- Fix CloudSecretsManagerBackend invalid connections_prefix (#7861)
- [AIRFLOW-7045] BugFix: DebugExecutor fails to change task state. (#8073)
- BugFix: Datetimepicker is stuck on the UI (#8092)
- [AIRFLOW-5277] Gantt chart respects per-user the Timezone UI setting (#8096)
- Fix timezones displayed in Task Instance tooltip (#8103)
- BugFix: Fix writing & deleting Dag Code for Serialized DAGs (#8151)
- Make the default TI pool slots '1' (#8153)
- Fix 500 error in Security screens (#8165)
- Fix Viewing Dag Code for Stateless Webserver (#8159)
- Fix issue with SQLAlchemy 1.3.16 (#8230)
Misc/Internal
- Fix Flaky TriggerDAG UI test (#8022)
- Remove unnecessary messages in CI (#7951)
- Fixes too high parallelism in CI (#7947)
- Install version is not persistent in breeze (#7914)
- Fixed automated check for image rebuild (#7912)
- Move Dockerfile to Dockerfile.ci (#7829)
- Generate requirements are now sorted (#8040)
- Change name of the common environment initialization function (#7805)
- Requirements now depend on python version (#7841)
- Bring back reset db explicitly called at CI entry (#7798)
- Fixes unclean installation of Airflow 1.10 (#7796)
- Change name of the common environment initialization function (#7805)
- [AIRFLOW-7029] Use separate docker image for running license check (#7678)
- [AIRFLOW-5842] Swtch to Debian buster image as a base (#7647)
- [AIRFLOW-5828] Move build logic out from hooks/build (#7618)
- [AIRFLOW-6839] Even more mypy speed improvements (#7460)
- [AIRFLOW-6820] split breeze into functions (#7433)
- [AIRFLOW-7097] Install gcloud beta componensts in CI image (#7772)
- [AIRFLOW-7018] fixing travis's job name escaping problem (#7668)
- [AIRFLOW-7054] Breeze has an option now to reset db at entry (#7710)
- [AIRFLOW-7005] Added exec command to Breeze (#7649)
- [AIRFLOW-7015] Detect Dockerhub repo/user when building on Dockerhub (#7673)
- [AIRFLOW-6727] Fix minor bugs in Release Management scripts (#7355)
- [AIRFLOW-7013] Automated check if Breeze image needs to be pulled (#7656)
- [AIRFLOW-7010] Skip in-container checks for Dockerhub builds (#7652)
- [AIRFLOW-7011] Pin JPype release to allow to build 1.10 images
- [AIRFLOW-7006] Fix missing +e in Breeze script (#7648)
- [AIRFLOW-6979] Fix breeze test-target specific test param issue (#7614)
- [AIRFLOW-6932] Add restart-environment command to Breeze
- [AIRFLOW-6919] Make Breeze DAG-test friedly (#7539)
- [AIRFLOW-6838] Introduce real subcommands for Breeze (#7515)
- [AIRFLOW-6763] Make systems tests ready for backport tests (#7389)
- [AIRFLOW-6866] Fix wrong export for Mac on Breeze (#7485)
- [AIRFLOW-6842] Skip fixing ownership on Mac (#7469)
- [AIRFLOW-6841] Fixed unbounded variable on Mac (#7465)
- [AIRFLOW-7067] Pinned version of Apache Airflow (#7730)
- [AIRFLOW-7058] Add support for different DB versions (#7717)
- [AIRFLOW-7002] Get rid of yaml "parser" in bash (#7646)
- [AIRFLOW-6972] Shorter frequently used commands in Breeze (#7608)
Doc only changes
- Fix typo for store_serialized_dags config (#7952)
- Fix broken link in README.md (#7893)
- Separate supported Postgres versions with comma (#7892)
- Fix grammar in setup.py (#7877)
- Add Jiajie Zhong to committers list (#8047)
- Update Security doc for 1.10.* for auth backends (#8072)
- Fix Example in config_templates for Secrets Backend (#8074)
- Add backticks in IMAGES.rst command description (#8075)
- Change version_added for store_dag_code config (#8076)
- [AIRFLOW-XXXX] Remove duplicate docs
- [AIRFLOW-XXXX] Remove the defunct limitation of Dag Serialization (#7716)
- [AIRFLOW-XXXX] Add prerequisite tasks for ...
Airflow 1.10.9, 2020-02-07
Bug Fixes
- [AIRFLOW-6751] Pin Werkzeug (dependency of a number of our dependencies) to < 1.0.0 (#7377)
Airflow 1.10.8, 2020-02-07
New Features
- [AIRFLOW-4026] Add filter by DAG tags (#6489)
- [AIRFLOW-6613] Center dag on graph view load (#7238)
- [AIRFLOW-5843] Add conf option to Add DAG Run view (#7281)
- [AIRFLOW-4495] Allow externally triggered dags to run for future exec dates (#7038)
Improvements
- [AIRFLOW-6438] Filter DAGs returned by blocked (#7019)
- [AIRFLOW-6666] Resolve js-yaml advisory (#7283)
- [AIRFLOW-6632] Bump dagre-d3 to resolve lodash CVE advisory (#7280)
- [AIRFLOW-6667] Resolve serialize-javascript advisory (#7282)
- [AIRFLOW-6451] self._print_stat() in dag_processing.py should be skippable (#7134)
- [AIRFLOW-6495] Load DAG only once when running a task using StandardTaskRunner (#7090)
- [AIRFLOW-6319] Add support for AWS Athena workgroups (#6871)
- [AIRFLOW-6677] Remove deprecation warning from SQLAlchmey (#7289)
- [AIRFLOW-6428] Fix import path for airflow.utils.dates.days_ago in Example DAGs (#7007)
- [AIRFLOW-6595] Use TaskNotFound exception instead of AirflowException (#7210)
- [AIRFLOW-6620] Mock celery in worker cli test (#7243)
- [AIRFLOW-6608] Change logging level for Bash & PyOperator Env exports
- [AIRFLOW-2279] Clear tasks across DAGs if marked by ExternalTaskMarker (#6633)
- [AIRFLOW-6359] Make Spark status_poll_interval explicit (#6978)
- [AIRFLOW-6359] spark_submit_hook.py status polling interval config (#6909)
- [AIRFLOW-6316] Use exampleinclude directives in tutorial.rst (#6868)
- [AIRFLOW-6519] Make TI logs constants in Webserver configurable (#7113)
- [AIRFLOW-6327] http_hook: Accept json= parameter for payload (#6886)
- [AIRFLOW-6261] flower_basic_auth eligible to _cmd (#6825)
- [AIRFLOW-6238] Filter dags returned by dag_stats
- [AIRFLOW-5616] Switch PrestoHook from pyhive to prestosql-client to s
- [AIRFLOW-6611] Add proxy_fix configs to default_airflow.cfg (#7236)
- [AIRFLOW-6557] Add test for newly added fields in BaseOperator (#7162)
- [AIRFLOW-6584] Pin cassandra driver (#7194)
- [AIRFLOW-6537] Fix backticks in rst files (#7140)
- [AIRFLOW-4428] Error if exec_date before default_args.start_date in trigger_dag (#6948)
- [AIRFLOW-6330] Show cli help when param blank or typo (#6883)
- [AIRFLOW-4113] Unpin boto3 (#6884)
- [AIRFLOW-6181] Add DebugExecutor (#6740)
- [AIRFLOW-6504] Allow specifying configmap for Airflow Local Setting (#7097)
- [AIRFLOW-6436] Cleanup for Airflow configs doc generator code (#7036)
- [AIRFLOW-6436] Add x_frame_enabled config in config.yml (#7024)
- [AIRFLOW-6436] Create & Automate docs on Airflow Configs (#7015)
- [AIRFLOW-6527] Make send_task_to_executor timeout configurable (#7143)
- [AIRFLOW-6272] Switch from npm to yarnpkg for managing front-end dependencies (#6844)
- [AIRFLOW-6350] Security - spark submit operator logging+exceptions should mask passwords
- [AIRFLOW-6358] Log details of failed task (#6908)
- [AIRFLOW-5149] Skip SLA checks config (#6923)
- [AIRFLOW-6057] Update template_fields of the PythonSensor (#6656)
- [AIRFLOW-4445] Mushroom cloud errors too verbose (#6952)
- [AIRFLOW-6394] Simplify github PR template (#6955)
- [AIRFLOW-5385] spark hook does not work on spark 2.3/2.4 (#6976)
Bug Fixes
- [AIRFLOW-6345] Ensure arguments to ProxyFix are integers (#6901)
- [AIRFLOW-6576] Fix scheduler crash caused by deleted task with sla misses (#7187)
- [AIRFLOW-6686] Fix syntax error constructing list of process ids (#7298)
- [AIRFLOW-6683] REST API respects store_serialized_dag setting (#7296)
- [AIRFLOW-6553] Add upstream_failed in instance state filter to WebUI (#7159)
- [AIRFLOW-6357] Highlight nodes in Graph UI if task id contains dots (#6904)
- [AIRFLOW-3349] Use None instead of False as value for encoding in StreamLogWriter (#7329)
- [AIRFLOW-6627] Email with incorrect DAG not delivered (#7250)
- [AIRFLOW-6637] Fix airflow test command in 1.10.x
- [AIRFLOW-6636] Avoid exceptions when printing task instance
- [AIRFLOW-6522] Clear task log file before starting to fix duplication in S3TaskHandler (#7120)
- [AIRFLOW-5501] Make default
in_cluster
value in KubernetesPodOperator respect config (#6124) - [AIRFLOW-6514] Use RUNNING_DEPS to check run from UI (#6367)
- [AIRFLOW-6381] Remove styling based on DAG id from DAGs page (#6985)
- [AIRFLOW-6434] Add return statement back to DockerOperator.execute (#7013)
- [AIRFLOW-2516] Fix mysql deadlocks (#6988)
- [AIRFLOW-6528] Disable flake8 W503 line break before binary operator (#7124)
- [AIRFLOW-6517] Make merge_dicts function recursive (#7111)
- [AIRFLOW-5621] Failure callback is not triggered when marked Failed on UI (#7025)
- [AIRFLOW-6353] Security - ui - add click jacking defence (#6995)
- [AIRFLOW-6348] Security - cli.py is currently printing logs with password (#6915)
- [AIRFLOW-6323] Remove non-ascii letters from default config (#6878)
- [AIRFLOW-6506] Fix do_xcom_push defaulting to True in KubernetesPodOperator (#7122)
- [AIRFLOW-6516] BugFix: airflow.cfg does not exist in Volume Mounts (#7109)
- [AIRFLOW-6427] Fix broken example_qubole_operator dag (#7005)
- [AIRFLOW-6385] BugFix: SlackAPIPostOperator fails when blocks not set (#7022)
- [AIRFLOW-6347] BugFix: Can't get task logs when serialization is enabled (#7092)
- [AIRFLOW-XXXX] Fix downgrade of db migration 0e2a74e0fc9f (#6859)
- [AIRFLOW-6366] Fix migrations for MS SQL Server (#6920)
- [AIRFLOW-5406] Allow spark without kubernetes (#6921)
- [AIRFLOW-6229] SparkSubmitOperator polls forever if status json can't… (#6918)
- [AIRFLOW-6352] Security - ui - add login timeout (#6912)
- [AIRFLOW-6397] Ensure sub_process attribute exists before trying to kill it (#6958)
- [AIRFLOW-6400] Fix pytest not working on Windows (#6964)
- [AIRFLOW-6418] Remove SystemTest.skip decorator (#6991)
- [AIRFLOW-6425] Serialization: Add missing DAG parameters to Json Schema (#7002)
Misc/Internal
- [AIRFLOW-6467] Use self.dag i/o creating a new one (#7067)
- [AIRFLOW-6490] Improve time delta comparison in local task job tests (#7083)
- [AIRFLOW-5814] Implementing Presto hook tests (#6491)
- [AIRFLOW-5704] Improve Kind Kubernetes scripts for local testing (#6516)
- [AIRFLOW-XXXX] Move airflow-config-yaml pre-commit before pylint (#7108)
- [AIRFLOW-XXXX] Improve clarity of confirm message (#7110)
- [AIRFLOW-6662] install dumb init (#7300)
- [AIRFLOW-6705] One less chatty message at breeze initialisation (#7326)
- [AIRFLOW-6705] Less chatty integration/backend checks (#7325)
- [AIRFLOW-6662] Switch to --init docker flag for signal propagation (#7278)
- [AIRFLOW-6661] Fail after 50 failing tests (#7277)
- [AIRFLOW-6607] Get rid of old local scripts for Breeze (#7225)
- [AIRFLOW-6589] BAT tests run in pre-commit on bash script changes (#7203)
- [AIRFLOW-6592] Doc build is moved to test phase (#7208)
- [AIRFLOW-6641] Better diagnostics for kubernetes flaky tests (#7261)
- [AIRFLOW-6642] Make local task job test less flaky (#7262)
- [AIRFLOW-6643] Fix flakiness of kerberos tests
- [AIRFLOW-6638] Remove flakiness test from test_serialized_db remove
- [AIRFLOW-6701] Rat is downloaded from stable backup/mirrors (#7323)
- [AIRFLOW-6702] Dumping kind logs to file.io. (#7319)
- [AIRFLOW-6491] Improve handling of Breeze parameters (#7084)
- [AIRFLOW-6470] Avoid pipe to file when do curl (#7063)
- [AIRFLOW-6471] Add pytest-instafail plugin (#7064)
- [AIRFLOW-6462] Limit exported variables in Dockerfile/Breeze (#7057)
- [AIRFLOW-6465] Add bash autocomplete for airflow in Breeze (#7060)
- [AIRFLOW-6464] Add cloud providers CLI tools in Breeze (#7059)
- [AIRFLOW-6461] Remove silent flags in Dockerfile (#7052)
- [AIRFLOW-6459] Increase verbosity of pytest (#7049)
- [AIRFLOW-6370] Skip Cassandra tests if cluster is not up (#6926)
- [AIRFLOW-6511] Remove BATS docker containers (#7103)
- [AIRFLOW-6475] Remove duplication of volume mount specs in Breeze.. (#7065)
- [AIRFLOW-6489] Add BATS support for Bash unit testing (#7081)
- [AIRFLOW-6387] print details of success/skipped task (#6956)
- [AIRFLOW-6568] Add Emacs related files to .gitignore (#7175)
- [AIRFLOW-6575] Entropy source for CI tests is changed to unblocking (#7185)
- [AIRFLOW-6496] Separate integrations in tests (#7091)
- [AIRFLOW-6634] Set PYTHONPATH in interactive Breeze
- [AIRFLOW-6564] Additional diagnostics information on CI check failure (#7172)
- [AIRFLOW-6383] Add no trailing-whitespace pre-commit hook (#6941)
Doc only changes
- [AIRFLOW-XXXX] Consistency fixes in new documentation (#7207)
- [AIRFLOW-XXXX] Improve grammar and structure in FAQ doc (#7291)
- [AIRFLOW-XXXX] Fix email configuration link in CONTRIBUTING.rst (#7311)
- [AIRFLOW-XXXX] Update docs with new BranchPythonOperator behaviour (#4682)
- [AIRFLOW-XXXX] Fix Typo in scripts/ci/ci_run_airflow_testing.sh (#7235)
- [AIRFLOW-XXXX] Screenshot showing disk space configuration for OSX (#7226)
- [AIRFLOW-XXXX] Add mentoring information to contributing docs (#7202)
- [AIRFLOW-XXXX] Add rebase info to contributing (#7201)
- [AIRFLOW-XXXX] Increase verbosity of static checks in CI (#7200)
- [AIRFLOW-XXXX] Adds branching strategy to documentation (#7193)
- [AIRFLOW-XXXX] Move email configuration from the concept page (#7189)
- [AIRFLOW-XXXX] Update task lifecycle diagram (#7161)
- [AIRFLOW-XXXX] Fix reference in concepts doc (#7135)
- [AIRFLOW-XXXX] Clear debug docs (#7104)
- [AIRFLOW-XXXX] Fix typos and broken links in development docs (#7086)
- [AIRFLOW-XXXX] Clarify wait_for_downstream and execution_date (#6999)
- [AIRFLOW-XXXX] Add
airflow dags show
command guide (#7014) - [AIRFLOW-XXXX] Update operation chaining documentation (#7018)
- [AIRFLOW-XXXX] Add
.autoenv_leave.zsh
to .gitignore (#6986) - [AIRFLOW-XXXX] Fix development packages installation instructions (#6942)
- [AIRFLOW-XXXX] Update committers list (#7212)
- [AIRFLOW-XXXX] Move UPDATING changes into correct versions (#7166)
- [AIRFLOW-XXXX] Add Documentation for check_slas flag (#6974)
- [AIRFLOW-XXXX] Fix gcp keyfile_d...
Airflow 1.10.7, 2019-12-24
New Features
- [AIRFLOW-5088] [AIP-24] Persisting serialized DAG in DB for webserver scalability (#5992)
- [AIRFLOW-6083] Adding ability to pass custom configuration to AWS Lambda client. (#6678)
- [AIRFLOW-5117] Automatically refresh EKS API tokens when needed (#5731)
- [AIRFLOW-5118] Add ability to specify optional components in DataprocClusterCreateOperator (#5821)
- [AIRFLOW-5681] Allow specification of a tag or hash for the git_sync init container (#6350)
- [AIRFLOW-6025] Add label to uniquely identify creator of Pod (#6621)
- [AIRFLOW-4843] Allow orchestration via Docker Swarm (SwarmOperator) (#5489)
- [AIRFLOW-5751] add get_uri method to Connection (#6426)
- [AIRFLOW-6056] Allow EmrAddStepsOperator to accept job_flow_name as alternative to job_flow_id (#6655)
- [AIRFLOW-2694] Declare permissions in DAG definition (#4642)
- [AIRFLOW-4940] Add DynamoDB to S3 operator (#5663)
- [AIRFLOW-4161] BigQuery to Mysql Operator (#5711)
- [AIRFLOW-6041] Add user agent to the Discovery API client (#6636)
- [AIRFLOW-6089] Reorder setup.py dependencies and add ci (#6681)
- [AIRFLOW-5921] Add bulk_load_custom to MySqlHook (#6575)
- [AIRFLOW-5854] Add support for
tty
parameter in Docker related operators (#6542) - [AIRFLOW-4758] Add GcsToGDriveOperator operator (#5822)
Improvements
- [AIRFLOW-3656] Show doc link for the current installed version (#6690)
- [AIRFLOW-5665] Add path_exists method to SFTPHook (#6344)
- [AIRFLOW-5729] Make InputDataConfig optional in Sagemaker's training config (#6398)
- [AIRFLOW-5045] Add ability to create Google Dataproc cluster with custom image from a different project (#5752)
- [AIRFLOW-6132] Allow to pass in tags for the AzureContainerInstancesOperator (#6694)
- [AIRFLOW-5945] Make inbuilt OperatorLinks work when using Serialization (#6715)
- [AIRFLOW-5947] Make the json backend pluggable for DAG Serialization (#6630)
- [AIRFLOW-6239] Filter dags return by last_dagruns (to only select visible dags, not all dags) (#6804)
- [AIRFLOW-6095] Filter dags returned by task_stats (to only select visible dags, not all dags) (#6684)
- [AIRFLOW-4482] Add execution_date to "trigger DagRun" API response (#5260)
- [AIRFLOW-1076] Add get method for template variable accessor (#6793)
- [AIRFLOW-5194] Add error handler to action log (#5883)
- [AIRFLOW-5936] Allow explicit get_pty in SSHOperator (#6586)
- [AIRFLOW-5474] Add Basic auth to Druid hook (#6095)
- [AIRFLOW-5726] Allow custom filename in RedshiftToS3Transfer (#6396)
- [AIRFLOW-5834] Option to skip serve_logs process with
airflow worker
(#6709) - [AIRFLOW-5583] Extend the 'DAG Details' page to display the start_date / end_date (#6235)
- [AIRFLOW-6250] Ensure on_failure_callback always has a populated context (#6812)
- [AIRFLOW-6222] http hook logs response body for any failure (#6779)
- [AIRFLOW-6260] Drive _cmd config option by env var (
AIRFLOW__CORE__SQL_ALCHEMY_CONN_CMD
for example) (#6801) - [AIRFLOW-6168] Allow proxy_fix middleware of webserver to be configurable (#6723)
- [AIRFLOW-5931] Use os.fork when appropriate to speed up task execution. (#6627)
- [AIRFLOW-4145] Allow RBAC roles permissions, ViewMenu to be over-ridable (#4960)
- [AIRFLOW-5928] Hive hooks load_file short circuit (#6582)
- [AIRFLOW-5313] Add params support for awsbatch_operator (#5900)
- [AIRFLOW-2227] Add delete method to Variable class (#4963)
- [AIRFLOW-5082] Add subject in AwsSnsHook (#5694)
- [AIRFLOW-5715] Make email, owner context available (#6385)
- [AIRFLOW-5345] Allow SqlSensor's hook to be customized by subclasses (#5946)
- [AIRFLOW-5417] Fix DB disconnects during webserver startup (#6023)
- [AIRFLOW-5730] Enable get_pandas_df on PinotDbApiHook (#6399)
- [AIRFLOW-3235] Add list function in AzureDataLakeHook (#4070)
- [AIRFLOW-5442] implementing get_pandas_df method for druid broker hook (#6057)
- [AIRFLOW-5883] Improve count() queries in a few places (#6532)
- [AIRFLOW-5811] Add metric for externally killed task count (#6466)
- [AIRFLOW-5758] Support the custom cursor classes for the PostgreSQL hook (#6432)
- [AIRFLOW-5766] Use httpbin.org in http_default (#6438)
- [AIRFLOW-5798] Set default ExternalTaskSensor.external_task_id (#6431)
- [AIRFLOW-5643] Reduce duplicated logic in S3Hook (#6313)
- [AIRFLOW-5562] Skip grant single DAG permissions for Admin role. (#6199)
- [AIRFLOW-6192] Stop creating Hook from SFTPSensor.init (#6748)
- [AIRFLOW-5749][AIRFLOW-4162] Support the "blocks" component for the Slack operators (#6418)
- [AIRFLOW-5693] Support the "blocks" component for the Slack messages (#6364)
- [AIRFLOW-5714] Collect SLA miss emails only from tasks missed SLA (#6384)
- [AIRFLOW-5049] Add validation for src_fmt_configs in bigquery hook (#5671)
- [AIRFLOW-6177] Log DAG processors timeout event at error level, not info (#6731)
- [AIRFLOW-6180] Improve kerberos init in pytest conftest (#6735)
- [AIRFLOW-6159] Change logging level of the heartbeat message to DEBUG (#6716)
- [AIRFLOW-6144] Improve the log message of airflow scheduler (#6710)
- [AIRFLOW-6099] Add host name to task runner log (#6688)
- [AIRFLOW-6045] Error on failed execution of compile_assets (#6640)
- [AIRFLOW-5144] Add confirmation on delete button click (#6745)
- [AIRFLOW-6099] Add host name to task runner log (#6688)
- [AIRFLOW-5915] Add support for the new documentation theme (#6563)
- [AIRFLOW-5897] Allow setting -1 as pool slots value in webserver (#6550)
- [AIRFLOW-5888] Use psycopg2-binary for postgres operations (#6533)
- [AIRFLOW-5870] Allow -1 for pool size and optimise pool query (#6520)
Bug Fixes
- [AIRFLOW-XXX] Bump Jira version to fix issue with async
- [AIRFLOW-XXX] Add encoding to fix Cyrillic output when reading back task logs (#6631)
- [AIRFLOW-5304] Fix extra links in BigQueryOperator with multiple queries (#5906)
- [AIRFLOW-6268] Prevent (expensive) ajax calls on home page when no dags visible (#6839)
- [AIRFLOW-6259] Reset page to 1 with each new search for dags (#6828)
- [AIRFLOW-6185] SQLAlchemy Connection model schema not aligned with Alembic schema (#6754)
- [AIRFLOW-3632] Only replace microseconds if execution_date is None in trigger_dag REST API (#6380)
- [AIRFLOW-5458] Bump Flask-AppBuilder to 2.2.0 (for Python >= 3.6) (#6607)
- [AIRFLOW-5072] gcs_hook should download files once (#5685)
- [AIRFLOW-5744] Environment variables not correctly set in Spark submit operator (#6796)
- [AIRFLOW-3189] Remove schema from DbHook.get_uri response if None (#6833)
- [AIRFLOW-6195] Fixed TaskInstance attrs not correct on UI (#6758)
- [AIRFLOW-5889] Make polling for AWS Batch job status more resillient (#6765)
- [AIRFLOW-6043] Fix bug in UI when "filtering by root" to display section of dag (#6638)
- [AIRFLOW-6033] Fix UI Crash at "Landing Times" when task_id is changed (#6635)
- [AIRFLOW-3745] Fix viewer not able to view dag details (#4569)
- [AIRFLOW-6175] Fixes bug when tasks get stuck in "scheduled" state (#6732)
- [AIRFLOW-5463] Make Variable.set when replacing an atomic operation (#6807)
- [AIRFLOW-5582] Add get_autocommit to JdbcHook (#6232)
- [AIRFLOW-5867] Fix webserver unit_test_mode data type (#6517)
- [AIRFLOW-5819] Update AWSBatchOperator default value (#6473)
- [AIRFLOW-5709] Fix regression in setting custom operator resources. (#6331)
- [AIRFLOW-5658] Fix broken navigation links (#6374)
- [AIRFLOW-5727] SqoopHook: Build --connect parameter only if port/schema are defined (#6397)
- [AIRFLOW-5695] use RUNNING_DEPS to check run from UI (#6367)
- [AIRFLOW-6254] obscure conn extra in logs (#6817)
- [AIRFLOW-4824] Add charset handling for SqlAlchemy engine for MySqlHook (#6816)
- [AIRFLOW-6091] Add flushing in execute method for BigQueryCursor (#6683)
- [AIRFLOW-6256] Ensure Jobs table is cleared when resetting DB (#6818)
- [AIRFLOW-5224] Add encoding parameter to GoogleCloudStorageToBigQuery (#6297)
- [AIRFLOW-5179] Remove top level init.py (#5818)
- [AIRFLOW-5660] Attempt to find the task in DB from Kubernetes pod labels (#6340)
- [AIRFLOW-6241] Fix typo in airflow/gcp/operator/dataflow.py (#6806)
- [AIRFLOW-6171] Apply .airflowignore to correct subdirectories (#6784)
- [AIRFLOW-6018] Display task instance in table during backfilling (#6612)
- [AIRFLOW-6189] Reduce the maximum test duration to 8 minutes (#6744)
- [AIRFLOW-6141] Remove ReadyToRescheduleDep if sensor mode == poke (#6704)
- [AIRFLOW-6054] Add a command that starts the database consoles (#6653)
- [AIRFLOW-6047] Simplify the logging configuration template (#6644)
- [AIRFLOW-6017] Exclude PULL_REQUEST_TEMPLATE.md from RAT check (#6611)
- [AIRFLOW-4560] Fix Tez queue parameter name in mapred_queue (#5315)
- [AIRFLOW-2143] Fix TaskTries graph counts off-by-1 (#6526)
- [AIRFLOW-5873] KubernetesPodOperator fixes and test (#6523)
- [AIRFLOW-5869] BugFix: Some Deserialized tasks have no start_date (#6519)
- [AIRFLOW-4020] Remove DAG edit permissions from Viewer role (#4845)
- [AIRFLOW-6263] Fix broken WinRM integration (#6832)
- [AIRFLOW-5836] Pin azure-storage-blob version to <12 (#6486)
- [AIRFLOW-4488] Fix typo for non-RBAC UI in max_active_runs_per_dag (#6778)
- [AIRFLOW-5942] Pin PyMSSQL to <3.0 (#6592)
- [AIRFLOW-5451] SparkSubmitHook don't set default namespace (#6072)
- [AIRFLOW-6271] Printing log files read during load_test_config (#6842)
- [AIRFLOW-6308] Unpin Kombu for Python 3
Misc/Internal
- [AIRFLOW-6009] Switch off travis_wait for regular tests (#6600)
- [AIRFLOW-6226] Always reset warnings in tests
- [AIRFLOW-XXX] Remove cyclic imports and pylint hacks in Serialization (#6601)
- [AIRFLOW-XXX] Bump npm from 6.4.1 to 6.13.4 in /airflow/www (#6815)
- [AIRFLOW-XXX] Remove executable permission from file
- [AIRFLOW-XXX] Group AWS & Azure dependencies (old
[emr]
etc. extra still work) - [AIRFLOW-5487] Fix unused warning var (#6111)
- [AIRFLOW-5925] Relax funcsigs and psutil version requirements (#6580)
- [AIRFLOW-...
Airflow 1.10.6rc1, 2019-10-18
New Features
- [AIRFLOW-4908] Implement BigQuery Hooks/Operators for update_dataset, patch_dataset and get_dataset (#5546)
- [AIRFLOW-4741] Optionally report task errors to Sentry (#5407)
- [AIRFLOW-4939] Add default_task_retries config (#5570)
- [AIRFLOW-5508] Add config setting to limit which StatsD metrics are emitted (#6130)
- [AIRFLOW-4222] Add cli autocomplete for bash & zsh (#5789)
- [AIRFLOW-3871] Operators template fields can now render fields inside objects (#4743)
Improvements
- [AIRFLOW-5127] Gzip support for CassandraToGoogleCloudStorageOperator (#5738)
- [AIRFLOW-5125] Add gzip support for AdlsToGoogleCloudStorageOperator (#5737)
- [AIRFLOW-5124] Add gzip support for S3ToGoogleCloudStorageOperator (#5736)
- [AIRFLOW-5653] Log AirflowSkipException in task instance log to make it clearer why tasks might be skipped (#6330)
- [AIRFLOW-5343] Remove legacy SQLAlchmey pessimistic pool disconnect handling (#6034)
- [AIRFLOW-5561] Relax httplib2 version required for gcp extra (#6194)
- [AIRFLOW-5657] Update the upper bound for dill dependency (#6334)
- [AIRFLOW-5292] Allow ECSOperator to tag tasks (#5891)
- [AIRFLOW-4939] Simplify Code for Default Task Retries (#6233)
- [AIRFLOW-5126] Read
aws_session_token
in extra_config of the aws hook (#6303) - [AIRFLOW-5636] Allow adding or overriding existing Operator Links (#6302)
- [AIRFLOW-4965] Handle quote exceptions in GCP AI operators (v1.10) (#6304)
- [AIRFLOW-3783] Speed up Redshift to S3 UNload with HEADERs (#6309)
- [AIRFLOW-3388] Add support to Array Jobs for AWS Batch Operator (#6153)
- [AIRFLOW-4574] add option to provide private_key in SSHHook (#6104) (#6163)
- [AIRFLOW-5530] Fix typo in AWS SQS sensors (#6012)
- [AIRFLOW-5445] Reduce the required resources for the Kubernetes's sidecar (#6062)
- [AIRFLOW-5443] Use alpine image in Kubernetes's sidecar (#6059)
- [AIRFLOW-5344] Add --proxy-user parameter to SparkSubmitOperator (#5948)
- [AIRFLOW-3888] HA for Hive metastore connection (#4708)
- [AIRFLOW-5269] Reuse session in Scheduler Job from health endpoint (#5873)
- [AIRFLOW-5153] Option to force delete non-empty BQ datasets (#5768)
- [AIRFLOW-4443] Document LatestOnly behavior for external trigger (#5214)
- [AIRFLOW-2891] Make DockerOperator container_name be templateable (#5696)
- [AIRFLOW-2891] allow configurable docker_operator container name (#5689)
- [AIRFLOW-4285] Update task dependency context definition and usage (#5079)
- [AIRFLOW-5142] Fixed flaky Cassandra test (#5758)
- [AIRFLOW-5218] Less polling of AWS Batch job status (#5825)
- [AIRFLOW-4956] Fix LocalTaskJob heartbeat log spamming (#5589)
- [AIRFLOW-3160] Load latest_dagruns asynchronously on home page (#5339)
- [AIRFLOW-5560] Allow no confirmation on reset dags in
airflow backfill
command (#6195) - [AIRFLOW-5280] conn: Remove aws_default's default region name (#5879)
- [AIRFLOW-5528] end_of_log_mark should not be a log record (#6159)
- [AIRFLOW-5526] Update docs configuration due to migration of GCP docs (#6154)
- [AIRFLOW-4835] Refactor operator render_template (#5461)
Bug Fixes
- [AIRFLOW-5459] Use a dynamic tmp location in Dataflow operator (#6078)
- [Airflow 4923] Fix Databricks hook leaks API secret in logs (#5635)
- [AIRFLOW-5133] Keep original env state in provide_gcp_credential_file (#5747)
- [AIRFLOW-5497] Update docstring in
airflow/utils/dag_processing.py
(#6314) - Revert/and then rework "[AIRFLOW-4797] Improve performance and behaviour of zombie detection (#5511)" to improve performance (#5908)
- [AIRFLOW-5634] Don't allow editing of DagModelView (#6308)
- [AIRFLOW-4309] Remove Broken Dag error after Dag is deleted (#6102)
- [AIRFLOW-5387] Fix "show paused" pagination bug (#6100)
- [AIRFLOW-5489] Remove unneeded assignment of variable (#6106)
- [AIRFLOW-5491] mark_tasks pydoc is incorrect (#6108)
- [AIRFLOW-5492] added missing docstrings (#6107)
- [AIRFLOW-5503] Fix tree view layout on HDPI screen (#6125)
- [AIRFLOW-5481] Allow Deleting Renamed DAGs (#6101)
- [AIRFLOW-3857] spark_submit_hook cannot kill driver pod in Kubernetes (#4678)
- [AIRFLOW-4391] Fix tooltip for None-State Tasks in 'Recent Tasks' (#5909)
- [AIRFLOW-5554] Require statsd 3.3.0 minimum (#6185)
- [AIRFLOW-5306] Fix the display of links when they contain special characters (#5904)
- [AIRFLOW-3705] Fix PostgresHook get_conn to use conn_name_attr (#5841)
- [AIRFLOW-5581] Cleanly shutdown KubernetesJobWatcher for safe Scheduler shutdown on SIGTERM (#6237)
- [AIRFLOW-5634] Don't allow disabled fields to be edited in DagModelView (#6307)
- [AIRFLOW-4833] Allow to set Jinja env options in DAG declaration (#5943)
- [AIRFLOW-5408] Fix env variable name in Kubernetes template (#6016)
- [AIRFLOW-5102] Worker jobs should terminate themselves if they can't heartbeat (#6284)
- [AIRFLOW-5572] Clear task reschedules when clearing task instances (#6217)
- [AIRFLOW-5543] Fix tooltip disappears in tree and graph view (RBAC UI) (#6174)
- [AIRFLOW-5444] Fix action_logging so that request.form for POST is logged (#6064)
- [AIRFLOW-5484] fix PigCliHook has incorrect named parameter (#6112)
- [AIRFLOW-5342] Fix MSSQL breaking task_instance db migration (#6014)
- [AIRFLOW-5556] Add separate config for timeout from scheduler dag processing (#6186)
- [AIRFLOW-4858] Deprecate "Historical convenience functions" in airflow.configuration (#5495) (#6144)
- [AIRFLOW-774] Fix long-broken DAG parsing Statsd metrics (#6157)
- [AIRFLOW-5419] Use
sudo
to kill cleared tasks when running with impersonation (#6026) (#6176) - [AIRFLOW-5537] Yamllint is not needed as dependency on host
- [AIRFLOW-5536] Better handling of temporary output files
- [AIRFLOW-5535] Fix name of VERBOSE parameter
- [AIRFLOW-5519] Fix sql_to_gcs operator missing multi-level default args by adding apply_defaults decorator (#6146)
- [AIRFLOW-5210] Make finding template files more efficient (#5815)
- [AIRFLOW-5447] Scheduler stalls because second watcher thread in default args (#6129)
Doc-only changes
- [AIRFLOW-5574] Fix Google Analytics script loading (#6218)
- [AIRFLOW-5588] Add Celery's architecture diagram (#6247)
- [AIRFLOW-5521] Fix link to GCP documentation (#6150)
- [AIRFLOW-5398] Update contrib example DAGs to context manager (#5998)
- [AIRFLOW-5268] Apply same DAG naming conventions as in literature (#5874)
- [AIRFLOW-5101] Fix inconsistent owner value in examples (#5712)
- [AIRFLOW-XXX] Fix typo - AWS DynamoDB Hook (#6319)
- [AIRFLOW-XXX] Fix Documentation for adding extra Operator Links (#6301)
- [AIRFLOW-XXX] Add section on task lifecycle & correct casing in docs (#4681)
- [AIRFLOW-XXX] Make it clear that 1.10.5 wasn't accidentally omitted from UPDATING.md (#6240)
- [AIRFLOW-XXX] Improve format in code-block directives (#6242)
- [AIRFLOW-XXX] Format Sendgrid docs (#6245)
- [AIRFLOW-XXX] Update to new logo (#6066)
- [AIRFLOW-XXX] Typo in FAQ - schedule_interval (#6291)
- [AIRFLOW-XXX] Add message about breaking change in DAG#get_task_instances in 1.10.4 (#6226)
- [AIRFLOW-XXX] Fix incorrect units in docs for metrics using Timers (#6152)
- [AIRFLOW-XXX] Fix backtick issues in .rst files & Add Precommit hook (#6162)
- [AIRFLOW-XXX] Update documentation about variables forcing answer (#6158)
- [AIRFLOW-XXX] Add a third way to configure authorization (#6134)
- [AIRFLOW-XXX] Add example of running pre-commit hooks on single file (#6143)
- [AIRFLOW-XXX] Add information about default pool to docs (#6019)
- [AIRFLOW-XXX] Make Breeze The default integration test environment (#6001)
Misc/Internal
- [AIRFLOW-5687] Upgrade pip to 19.0.2 in CI build pipeline (#6358) (#6361)
- [AIRFLOW-5533] Fixed failing CRON build (#6167)
- [AIRFLOW-5130] Use GOOGLE_APPLICATION_CREDENTIALS constant from library (#5744)
- [AIRFLOW-5369] Adds interactivity to pre-commits (#5976)
- [AIRFLOW-5531] Replace deprecated log.warn() with log.warning() (#6165)
- [AIRFLOW-4686] Make dags Pylint compatible (#5753)
- [AIRFLOW-4864] Remove calls to load_test_config (#5502)
- [AIRFLOW-XXX] Pin version of mypy so we are stable over time (#6198)
- [AIRFLOW-XXX] Add tests that got missed from #5127
- [AIRFLOW-4928] Move config parses to class properties inside DagBag (#5557)
- [AIRFLOW-5003] Making AWS Hooks pylint compatible (#5627)
- [AIRFLOW-5580] Add base class for system test (#6229)