Skip to content

Commit

Permalink
Refreshing website content from main repo.
Browse files Browse the repository at this point in the history
  • Loading branch information
GitHub Action Website Snapshot committed Dec 9, 2024
1 parent c41da04 commit b5c680c
Show file tree
Hide file tree
Showing 24 changed files with 1,798 additions and 473 deletions.
105 changes: 104 additions & 1 deletion docs/client/java/partials/java_transport.md
Original file line number Diff line number Diff line change
Expand Up @@ -484,7 +484,7 @@ The `CompositeTransport` is designed to combine multiple transports, allowing ev

#### Behavior

- The configured transports will be initialized and used in sequence to emit OpenLineage events.
- The configured transports will be initialized and used in sequence (sorted by transport name) to emit OpenLineage events.
- If `continueOnFailure` is set to `false`, a failure in one transport will stop the event emission process, and an exception will be raised.
- If `continueOnFailure` is `true`, the failure will be logged, but the remaining transports will still attempt to send the event.

Expand Down Expand Up @@ -617,6 +617,109 @@ OpenLineageClient client = OpenLineageClient.builder()
</TabItem>
</Tabs>

### [Transform](https://github.com/OpenLineage/OpenLineage/tree/main/client/java/src/main/java/io/openlineage/client/transports/transform/TransformTransport.java)

The `TransformTransport` is designed to enable event manipulation before emitting the event. Together with `CompositeTransport`, it can be used to
send different events into multiple backends.

#### Configuration

- `type` - string, must be "transform". Required.
- `transformerClass` - class name of the event transformer. Class has to implement `io.openlineage.client.transports.transform.EventTransformer` interface and provide public no-arg constructor. Class needs to be available on the classpath. Required.
- `transformerProperties` - Extra properties that can be passed into `transformerClass` based on the configuration. Optional.
- `transport` - Transport configuration to emit modified events. Required.

#### Behavior

- The configured `transformerClass` will be used to alter events before the emission.
- Modified events will be passed into the configured `transport` for further processing.

#### `EventTransformer` interface

```java
public class CustomEventTransformer implements EventTransformer {
@Override
public void initialize(Map<String, String> properties) { ... }

@Override
public RunEvent transform(RunEvent event) { ... }

@Override
public DatasetEvent transform(DatasetEvent event) { .. }

@Override
public JobEvent transform(JobEvent event) { ... }
}
```

#### Examples

<Tabs groupId="integrations">
<TabItem value="yaml" label="Yaml Config">

```yaml
transport:
type: transport
transformerClass: io.openlineage.CustomEventTransformer
transformerProperties:
key1: value1
key2: value2
transport:
type: http
url: http://example.com/api
name: my_http
```
</TabItem>
<TabItem value="spark" label="Spark Config">
```ini
spark.openlineage.transport.type=transform
spark.openlineage.transport.transformerClass=io.openlineage.CustomEventTransformer
spark.openlineage.transport.transformerProperties.key1=value1
spark.openlineage.transport.transformerProperties.key2=value2
spark.openlineage.transport.transport.type=http
spark.openlineage.transport.transport.url=http://example.com/api
```

</TabItem>
<TabItem value="flink" label="Flink Config">

```ini
openlineage.transport.type=transform
openlineage.transport.transformerClass=io.openlineage.CustomEventTransformer
openlineage.transport.transformerProperties.key1=value1
openlineage.transport.transformerProperties.key2=value2
openlineage.transport.transport.type=http
openlineage.transport.transport.url=http://example.com/api
```

</TabItem>
<TabItem value="java" label="Java Code">

```java
import java.util.Arrays;
import io.openlineage.client.OpenLineageClient;
import io.openlineage.client.transports.TransformConfig;
import io.openlineage.client.transports.HttpConfig;
import io.openlineage.client.transports.HttpTransport;

HttpConfig httpConfig = new HttpConfig();
httpConfig.setUrl(URI.create("http://example.com/api"));

TransformConfig transformConfig = new TransformConfig();
transformConfig.setTransformerClass(CustomEventTransformer.class.getName());
transformConfig.setTransport(httpConfig);

OpenLineageClient client = OpenLineageClient
.builder()
.transport(new TransformTransport(transformConfig))
.build();
```

</TabItem>
</Tabs>

### [GcpLineage](https://github.com/OpenLineage/OpenLineage/blob/main/client/transports-dataplex/src/main/java/io/openlineage/client/transports/gcplineage/GcpLineageTransport.java)

To use this transport in your project, you need to include `io.openlineage:transports-gcplineage` artifact in
Expand Down
732 changes: 373 additions & 359 deletions static/apidocs/javadoc/allclasses-index.html

Large diffs are not rendered by default.

84 changes: 84 additions & 0 deletions static/apidocs/javadoc/index-all.html

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ <h1 title="Interface MergeConfig" class="title">Interface MergeConfig&lt;T&gt;</
<section class="class-description" id="class-description">
<dl class="notes">
<dt>All Known Implementing Classes:</dt>
<dd><code><a href="transports/CompositeConfig.html" title="class in io.openlineage.client.transports">CompositeConfig</a></code>, <code><a href="dataset/DatasetConfig.html" title="class in io.openlineage.client.dataset">DatasetConfig</a></code>, <code><a href="transports/FacetsConfig.html" title="class in io.openlineage.client.transports">FacetsConfig</a></code>, <code><a href="dataset/namespace/resolver/HostListNamespaceResolverConfig.html" title="class in io.openlineage.client.dataset.namespace.resolver">HostListNamespaceResolverConfig</a></code>, <code><a href="transports/HttpConfig.html" title="class in io.openlineage.client.transports">HttpConfig</a></code>, <code><a href="circuitBreaker/JavaRuntimeCircuitBreakerConfig.html" title="class in io.openlineage.client.circuitBreaker">JavaRuntimeCircuitBreakerConfig</a></code>, <code><a href="transports/KafkaConfig.html" title="class in io.openlineage.client.transports">KafkaConfig</a></code>, <code><a href="OpenLineageConfig.html" title="class in io.openlineage.client">OpenLineageConfig</a></code>, <code><a href="dataset/namespace/resolver/PatternMatchingGroupNamespaceResolverConfig.html" title="class in io.openlineage.client.dataset.namespace.resolver">PatternMatchingGroupNamespaceResolverConfig</a></code>, <code><a href="dataset/namespace/resolver/PatternNamespaceResolverConfig.html" title="class in io.openlineage.client.dataset.namespace.resolver">PatternNamespaceResolverConfig</a></code>, <code><a href="circuitBreaker/SimpleMemoryCircuitBreakerConfig.html" title="class in io.openlineage.client.circuitBreaker">SimpleMemoryCircuitBreakerConfig</a></code></dd>
<dd><code><a href="transports/CompositeConfig.html" title="class in io.openlineage.client.transports">CompositeConfig</a></code>, <code><a href="dataset/DatasetConfig.html" title="class in io.openlineage.client.dataset">DatasetConfig</a></code>, <code><a href="transports/FacetsConfig.html" title="class in io.openlineage.client.transports">FacetsConfig</a></code>, <code><a href="dataset/namespace/resolver/HostListNamespaceResolverConfig.html" title="class in io.openlineage.client.dataset.namespace.resolver">HostListNamespaceResolverConfig</a></code>, <code><a href="transports/HttpConfig.html" title="class in io.openlineage.client.transports">HttpConfig</a></code>, <code><a href="circuitBreaker/JavaRuntimeCircuitBreakerConfig.html" title="class in io.openlineage.client.circuitBreaker">JavaRuntimeCircuitBreakerConfig</a></code>, <code><a href="transports/KafkaConfig.html" title="class in io.openlineage.client.transports">KafkaConfig</a></code>, <code><a href="OpenLineageConfig.html" title="class in io.openlineage.client">OpenLineageConfig</a></code>, <code><a href="dataset/namespace/resolver/PatternMatchingGroupNamespaceResolverConfig.html" title="class in io.openlineage.client.dataset.namespace.resolver">PatternMatchingGroupNamespaceResolverConfig</a></code>, <code><a href="dataset/namespace/resolver/PatternNamespaceResolverConfig.html" title="class in io.openlineage.client.dataset.namespace.resolver">PatternNamespaceResolverConfig</a></code>, <code><a href="circuitBreaker/SimpleMemoryCircuitBreakerConfig.html" title="class in io.openlineage.client.circuitBreaker">SimpleMemoryCircuitBreakerConfig</a></code>, <code><a href="transports/TransformConfig.html" title="class in io.openlineage.client.transports">TransformConfig</a></code></dd>
</dl>
<hr>
<div class="type-signature"><span class="modifiers">public interface </span><span class="element-name type-name-label">MergeConfig&lt;T&gt;</span></div>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ <h1 title="Class CompositeConfig" class="title">Class CompositeConfig</h1>
<section class="class-description" id="class-description">
<dl class="notes">
<dt>All Implemented Interfaces:</dt>
<dd><code><a href="../MergeConfig.html" title="interface in io.openlineage.client">MergeConfig</a>&lt;<a href="CompositeConfig.html" title="class in io.openlineage.client.transports">CompositeConfig</a>&gt;</code>, <code><a href="TransportConfig.html" title="interface in io.openlineage.client.transports">TransportConfig</a></code></dd>
<dd><code><a href="../MergeConfig.html" title="interface in io.openlineage.client">MergeConfig</a>&lt;<a href="CompositeConfig.html" title="class in io.openlineage.client.transports">CompositeConfig</a>&gt;</code>, <code><a href="TransportConfig.html" title="interface in io.openlineage.client.transports">TransportConfig</a></code>, <code><a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Comparable.html" title="class or interface in java.lang" class="external-link">Comparable</a>&lt;<a href="TransportConfig.html" title="interface in io.openlineage.client.transports">TransportConfig</a>&gt;</code></dd>
</dl>
<hr>
<div class="type-signature"><span class="modifiers">public final class </span><span class="element-name type-name-label">CompositeConfig</span>
Expand Down Expand Up @@ -158,7 +158,7 @@ <h3 id="methods-inherited-from-class-io.openlineage.client.MergeConfig">Methods
<code><a href="../MergeConfig.html#mergePropertyWith(java.util.Map,java.util.Map)">mergePropertyWith</a>, <a href="../MergeConfig.html#mergePropertyWith(T,T)">mergePropertyWith</a>, <a href="../MergeConfig.html#mergeWith(T)">mergeWith</a>, <a href="../MergeConfig.html#mergeWithDefaultValue(T,T,T)">mergeWithDefaultValue</a></code></div>
<div class="inherited-list">
<h3 id="methods-inherited-from-class-io.openlineage.client.transports.TransportConfig">Methods inherited from interface&nbsp;io.openlineage.client.transports.<a href="TransportConfig.html" title="interface in io.openlineage.client.transports">TransportConfig</a></h3>
<code><a href="TransportConfig.html#getName()">getName</a>, <a href="TransportConfig.html#setName(java.lang.String)">setName</a></code></div>
<code><a href="TransportConfig.html#compareTo(io.openlineage.client.transports.TransportConfig)">compareTo</a>, <a href="TransportConfig.html#getName()">getName</a>, <a href="TransportConfig.html#setName(java.lang.String)">setName</a></code></div>
</section>
</li>
</ul>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@ <h1 title="Class ConsoleConfig" class="title">Class ConsoleConfig</h1>
<section class="class-description" id="class-description">
<dl class="notes">
<dt>All Implemented Interfaces:</dt>
<dd><code><a href="TransportConfig.html" title="interface in io.openlineage.client.transports">TransportConfig</a></code></dd>
<dd><code><a href="TransportConfig.html" title="interface in io.openlineage.client.transports">TransportConfig</a></code>, <code><a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Comparable.html" title="class or interface in java.lang" class="external-link">Comparable</a>&lt;<a href="TransportConfig.html" title="interface in io.openlineage.client.transports">TransportConfig</a>&gt;</code></dd>
</dl>
<hr>
<div class="type-signature"><span class="modifiers">public final class </span><span class="element-name type-name-label">ConsoleConfig</span>
Expand Down Expand Up @@ -113,7 +113,7 @@ <h3 id="methods-inherited-from-class-java.lang.Object">Methods inherited from cl
<code><a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Object.html#clone()" title="class or interface in java.lang" class="external-link">clone</a>, <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Object.html#equals(java.lang.Object)" title="class or interface in java.lang" class="external-link">equals</a>, <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Object.html#finalize()" title="class or interface in java.lang" class="external-link">finalize</a>, <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Object.html#getClass()" title="class or interface in java.lang" class="external-link">getClass</a>, <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Object.html#hashCode()" title="class or interface in java.lang" class="external-link">hashCode</a>, <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Object.html#notify()" title="class or interface in java.lang" class="external-link">notify</a>, <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Object.html#notifyAll()" title="class or interface in java.lang" class="external-link">notifyAll</a>, <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Object.html#toString()" title="class or interface in java.lang" class="external-link">toString</a>, <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Object.html#wait()" title="class or interface in java.lang" class="external-link">wait</a>, <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Object.html#wait(long)" title="class or interface in java.lang" class="external-link">wait</a>, <a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Object.html#wait(long,int)" title="class or interface in java.lang" class="external-link">wait</a></code></div>
<div class="inherited-list">
<h3 id="methods-inherited-from-class-io.openlineage.client.transports.TransportConfig">Methods inherited from interface&nbsp;io.openlineage.client.transports.<a href="TransportConfig.html" title="interface in io.openlineage.client.transports">TransportConfig</a></h3>
<code><a href="TransportConfig.html#getName()">getName</a>, <a href="TransportConfig.html#setName(java.lang.String)">setName</a></code></div>
<code><a href="TransportConfig.html#compareTo(io.openlineage.client.transports.TransportConfig)">compareTo</a>, <a href="TransportConfig.html#getName()">getName</a>, <a href="TransportConfig.html#setName(java.lang.String)">setName</a></code></div>
</section>
</li>
</ul>
Expand Down
Loading

0 comments on commit b5c680c

Please sign in to comment.