Skip to content

Commit

Permalink
Merge pull request #3 from JDarDagran/update-previous-versions-to-doc…
Browse files Browse the repository at this point in the history
…usaurus-3-6

Update previous versions to docusaurus 3.6.
  • Loading branch information
julienledem authored Nov 7, 2024
2 parents f76e90b + 12c529e commit 75ac853
Show file tree
Hide file tree
Showing 518 changed files with 187 additions and 27,102 deletions.
1 change: 1 addition & 0 deletions .github/workflows/site_deploy.yml
Original file line number Diff line number Diff line change
Expand Up @@ -35,4 +35,5 @@ jobs:
path: './build'
- name: Deploy to GitHub Pages
id: deployment
if: github.event_name == 'push'
uses: actions/deploy-pages@v4
Original file line number Diff line number Diff line change
Expand Up @@ -101,7 +101,8 @@ spark.openlineage.transport.headers.X-Some-Extra-Header=abc
spark.openlineage.transport.compression=gzip
```

<details><summary>URL parsing within Spark integration</summary>
<details>
<summary>URL parsing within Spark integration</summary>
<p>

You can supply http parameters using values in url, the parsed `spark.openlineage.*` properties are located in url as follows:
Expand Down
3 changes: 3 additions & 0 deletions versioned_docs/version-1.21.1/client/python.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,9 @@
sidebar_position: 5
---

import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';

# Python

## Overview
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ title: Apache Airflow
---

:::caution
This page is about Airflow's external integration that works mainly for Airflow versions <2.7.
This page is about Airflow's external integration that works mainly for Airflow versions \<2.7.
[If you're using Airflow 2.7+, look at native Airflow OpenLineage provider documentation.](https://airflow.apache.org/docs/apache-airflow-providers-openlineage/stable/index.html) <br /><br />

The ongoing development and enhancements will be focused on the `apache-airflow-providers-openlineage` package,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ title: Exposing Lineage in Airflow Operators
---

:::caution
This page is about Airflow's external integration that works mainly for Airflow versions <2.7.
This page is about Airflow's external integration that works mainly for Airflow versions \<2.7.
[If you're using Airflow 2.7+, look at native Airflow OpenLineage provider documentation.](https://airflow.apache.org/docs/apache-airflow-providers-openlineage/stable/index.html) <br /><br />

The ongoing development and enhancements will be focused on the `apache-airflow-providers-openlineage` package,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ title: Custom Extractors
---

:::caution
This page is about Airflow's external integration that works mainly for Airflow versions <2.7.
This page is about Airflow's external integration that works mainly for Airflow versions \<2.7.
[If you're using Airflow 2.7+, look at native Airflow OpenLineage provider documentation.](https://airflow.apache.org/docs/apache-airflow-providers-openlineage/stable/index.html) <br /><br />

The ongoing development and enhancements will be focused on the `apache-airflow-providers-openlineage` package,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ title: Testing Custom Extractors
---

:::caution
This page is about Airflow's external integration that works mainly for Airflow versions <2.7.
This page is about Airflow's external integration that works mainly for Airflow versions \<2.7.
[If you're using Airflow 2.7+, look at native Airflow OpenLineage provider documentation.](https://airflow.apache.org/docs/apache-airflow-providers-openlineage/stable/index.html) <br /><br />

The ongoing development and enhancements will be focused on the `apache-airflow-providers-openlineage` package,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ title: Job Hierarchy
---

:::caution
This page is about Airflow's external integration that works mainly for Airflow versions <2.7.
This page is about Airflow's external integration that works mainly for Airflow versions \<2.7.
[If you're using Airflow 2.7+, look at native Airflow OpenLineage provider documentation.](https://airflow.apache.org/docs/apache-airflow-providers-openlineage/stable/index.html) <br /><br />

The ongoing development and enhancements will be focused on the `apache-airflow-providers-openlineage` package,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ title: Manually Annotated Lineage
---

:::caution
This page is about Airflow's external integration that works mainly for Airflow versions <2.7.
This page is about Airflow's external integration that works mainly for Airflow versions \<2.7.
[If you're using Airflow 2.7+, look at native Airflow OpenLineage provider documentation.](https://airflow.apache.org/docs/apache-airflow-providers-openlineage/stable/index.html) <br /><br />

The ongoing development and enhancements will be focused on the `apache-airflow-providers-openlineage` package,
Expand Down
6 changes: 3 additions & 3 deletions versioned_docs/version-1.21.1/integrations/airflow/older.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ title: Supported Airflow versions
---

:::caution
This page is about Airflow's external integration that works mainly for Airflow versions <2.7.
This page is about Airflow's external integration that works mainly for Airflow versions \<2.7.
[If you're using Airflow 2.7+, look at native Airflow OpenLineage provider documentation.](https://airflow.apache.org/docs/apache-airflow-providers-openlineage/stable/index.html) <br /><br />

The ongoing development and enhancements will be focused on the `apache-airflow-providers-openlineage` package,
Expand All @@ -16,7 +16,7 @@ while the `openlineage-airflow` will primarily be updated for bug fixes.
##### Airflow 2.7+

This package **should not** be used starting with Airflow 2.7.0 and **can not** be used with Airflow 2.8+.
It was designed as Airflow's external integration that works mainly for Airflow versions <2.7.
It was designed as Airflow's external integration that works mainly for Airflow versions \<2.7.
For Airflow 2.7+ use the native Airflow OpenLineage provider
[package](https://airflow.apache.org/docs/apache-airflow-providers-openlineage) `apache-airflow-providers-openlineage`.

Expand Down Expand Up @@ -44,6 +44,6 @@ openlineage.lineage_backend.OpenLineageBackend

The OpenLineageBackend does not take into account manually configured inlets and outlets.

##### Airflow <2.1
##### Airflow \<2.1

OpenLineage does not work with versions older than Airflow 2.1.
4 changes: 2 additions & 2 deletions versioned_docs/version-1.21.1/integrations/airflow/usage.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ title: Using the Airflow Integration
---

:::caution
This page is about Airflow's external integration that works mainly for Airflow versions <2.7.
This page is about Airflow's external integration that works mainly for Airflow versions \<2.7.
[If you're using Airflow 2.7+, look at native Airflow OpenLineage provider documentation.](https://airflow.apache.org/docs/apache-airflow-providers-openlineage/stable/index.html) <br /><br />

The ongoing development and enhancements will be focused on the `apache-airflow-providers-openlineage` package,
Expand All @@ -14,7 +14,7 @@ while the `openlineage-airflow` will primarily be updated for bug fixes. See [al
#### PREREQUISITES

- [Python 3.8](https://www.python.org/downloads)
- [Airflow >= 2.1,<2.8](https://pypi.org/project/apache-airflow)
- [Airflow >= 2.1,\<2.8](https://pypi.org/project/apache-airflow)

To use the OpenLineage Airflow integration, you'll need a running [Airflow instance](https://airflow.apache.org/docs/apache-airflow/stable/start.html). You'll also need an OpenLineage-compatible [backend](https://github.com/OpenLineage/OpenLineage#scope).

Expand Down
10 changes: 5 additions & 5 deletions versioned_docs/version-1.21.1/integrations/flink.md
Original file line number Diff line number Diff line change
Expand Up @@ -118,11 +118,11 @@ and allows all the configuration features present there to be used. The configur

The following parameters can be specified:

| Parameter | Definition | Example |
|------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------|
| openlineage.transport.type | The transport type used for event emit, default type is `console` | http |
| openlineage.facets.disabled | List of facets to disable, enclosed in `[]` (required from 0.21.x) and separated by `;`, default is `[spark_unknown;spark.logicalPlan;]` (currently must contain `;`) | \[some_facet1;some_facet1\] |
| openlineage.job.owners.<ownership-type\> | Specifies ownership of the job. Multiple entries with different types are allowed. Config key name and value are used to create job ownership type and name (available since 1.13). | openlineage.job.owners.team="Some Team" |
| Parameter | Definition | Example |
|-------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------|
| openlineage.transport.type | The transport type used for event emit, default type is `console` | http |
| openlineage.facets.disabled | List of facets to disable, enclosed in `[]` (required from 0.21.x) and separated by `;`, default is `[spark_unknown;spark.logicalPlan;]` (currently must contain `;`) | \[some_facet1;some_facet1\] |
| openlineage.job.owners.\<ownership-type\> | Specifies ownership of the job. Multiple entries with different types are allowed. Config key name and value are used to create job ownership type and name (available since 1.13). | openlineage.job.owners.team="Some Team" |

## Transports

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,4 +22,4 @@ The following parameters can be specified:
| spark.openlineage.jobName.appendDatasetName | Decides whether output dataset name should be appended to job name. By default `true`. | false |
| spark.openlineage.jobName.replaceDotWithUnderscore | Replaces dots in job name with underscore. Can be used to mimic legacy behaviour on Databricks platform. By default `false`. | false |
| spark.openlineage.debugFacet | Determines whether debug facet shall be generated and included within the event. Set `enabled` to turn it on. By default, facet is disabled. | enabled |
| spark.openlineage.job.owners.<ownership-type\> | Specifies ownership of the job. Multiple entries with different types are allowed. Config key name and value are used to create job ownership type and name (available since 1.13). | spark.openlineage.job.owners.team="Some Team" |
| spark.openlineage.job.owners.\<ownership-type\> | Specifies ownership of the job. Multiple entries with different types are allowed. Config key name and value are used to create job ownership type and name (available since 1.13). | spark.openlineage.job.owners.team="Some Team" |
Loading

0 comments on commit 75ac853

Please sign in to comment.