openlineage.client.client module
\n- \n
- \nclass openlineage.client.client.OpenLineageClientOptions(timeout=5.0, verify=True, api_key=None, adapter=None) \n
Bases:
\nobject- \n
- Parameters: \n
- \n
timeout (
float) \nverify (
bool) \napi_key (
Optional[str]) \nadapter (
Optional[HTTPAdapter]) \n
\n
- \n
- \ntimeout: float \n
- \n
- \nverify: bool \n
- \n
- \napi_key: str \n
- \n
- \nadapter: HTTPAdapter \n
- \n
- \nclass openlineage.client.client.OpenLineageConfig(transport=_Nothing.NOTHING, facets=_Nothing.NOTHING, filters=_Nothing.NOTHING) \n
Bases:
\nobject- \n
- Parameters: \n
- \n
transport (dict[str, Any] | None)
\nfacets (FacetsConfig)
\nfilters (list[FilterConfig])
\n
\n
- \n
- \ntransport: dict[str, Any] | None \n
- \n
- \nfacets: FacetsConfig \n
- \n
- \nfilters: list[FilterConfig] \n
- \n
- \nclassmethod from_dict(params) \n
- \n
- Parameters: \n
params (dict[str, Any])
\n \n- Return type: \n
- \n \n
- \n
- \nclass openlineage.client.client.OpenLineageClient(url=None, options=None, session=None, transport=None, factory=None, *, config=None) \n
Bases:
\nobject- \n
- Parameters: \n
- \n
url (str | None)
\noptions (OpenLineageClientOptions | None)
\nsession (Session | None)
\ntransport (Transport | None)
\nfactory (TransportFactory | None)
\nconfig (dict[str, str] | None)
\n
\n
- \n
- \nDYNAMIC_ENV_VARS_PREFIX = 'OPENLINEAGE__' \n
- \n
- \nDEFAULT_URL_TRANSPORT_NAME = 'default_http' \n
- \n
- \nclassmethod from_environment() \n
- \n
- Return type: \n
_T
\n \n
- \n
- \nclassmethod from_dict(config) \n
- \n
- Parameters: \n
config (dict[str, str])
\n \n- Return type: \n
_T
\n \n
- \n
- \nfilter_event(event) \n
Filters jobs according to config-defined events
\n- \n
- Parameters: \n
event (Event)
\n \n- Return type: \n
Event | None
\n \n
- \n
- \nemit(event) \n
- \n
- Parameters: \n
event (
\nUnion[RunEvent,DatasetEvent,JobEvent,RunEvent,DatasetEvent,JobEvent]) \n- Return type: \n
\nNone\n
- \n
- \nproperty config: OpenLineageConfig \n
Retrieves the OpenLineage configuration.
\nThis property method returns the content of the OpenLineage YAML config file.\nThe configuration is determined by merging sources in the following order of precedence:\n1. User-defined configuration passed to the client constructor.\n2. YAML config file located in one of the following paths:\n- Path specified by the OPENLINEAGE_CONFIG environment variable.\n- Current working directory.\n- $HOME/.openlineage.\n3. Environment variables with the OPENLINEAGE__ prefix.\nIf the configuration is not already loaded, it will be constructed by merging the above sources.\nIn case of a TypeError during the parsing of the configuration, a ValueError will be raised indicating\nthat the structure of the config does not match the expected format.
\n
- \n
- \nadd_environment_facets(event) \n
Adds environment variables as facets to the event object.
\n- \n
- Parameters: \n
event (
\nUnion[RunEvent,DatasetEvent,JobEvent,RunEvent,DatasetEvent,JobEvent]) \n- Return type: \n
\nUnion[RunEvent,DatasetEvent,JobEvent,RunEvent,DatasetEvent,JobEvent] \n
openlineage.client.event_v2 module
\n- \n
- \nclass openlineage.client.event_v2.BaseEvent(*, eventTime, producer='') \n
Bases:
\nRedactMixin- \n
- Parameters: \n
- \n
eventTime (
str) \nproducer (
str) \n
\n
- \n
- \neventTime: str \n
the time the event occurred at
\n
- \n
- \nproducer: str \n
- \n
- \nschemaURL: str \n
- \n
- \nproperty skip_redact: list[str] \n
- \n
- \neventtime_check(attribute, value) \n
- \n
- Parameters: \n
- \n
attribute (
str) \nvalue (
str) \n
\n- Return type: \n
\nNone\n
- \n
- \nproducer_check(attribute, value) \n
- \n
- Parameters: \n
- \n
attribute (
str) \nvalue (
str) \n
\n- Return type: \n
\nNone\n
- \n
- \nschemaurl_check(attribute, value) \n
- \n
- Parameters: \n
- \n
attribute (
str) \nvalue (
str) \n
\n- Return type: \n
\nNone\n
- \n
- \nclass openlineage.client.event_v2.RunEvent(*, eventTime, producer='', run, job, eventType=None, inputs=_Nothing.NOTHING, outputs=_Nothing.NOTHING) \n
Bases:
\nBaseEvent- \n
- Parameters: \n
- \n
eventTime (str)
\nproducer (str)
\nrun (Run)
\njob (Job)
\neventType (EventType | None)
\ninputs (list[InputDataset] | None)
\noutputs (list[OutputDataset] | None)
\n
\n
- \n
- \nrun: Run \n
- \n
- \njob: Job \n
- \n
- \neventType: EventType | None \n
the current transition of the run state. It is required to issue 1 START event and 1 of [ COMPLETE,\nABORT, FAIL ] event per run. Additional events with OTHER eventType can be added to the same run.\nFor example to send additional metadata after the run is complete
\n
- \n
- \ninputs: list[InputDataset] | None \n
The set of input datasets.
\n
- \n
- \noutputs: list[OutputDataset] | None \n
The set of output datasets.
\n
- \n
- \nclass openlineage.client.event_v2.JobEvent(*, eventTime, producer='', job, inputs=_Nothing.NOTHING, outputs=_Nothing.NOTHING) \n
Bases:
\nBaseEvent- \n
- Parameters: \n
- \n
eventTime (str)
\nproducer (str)
\njob (Job)
\ninputs (list[InputDataset] | None)
\noutputs (list[OutputDataset] | None)
\n
\n
- \n
- \njob: Job \n
- \n
- \ninputs: list[InputDataset] | None \n
The set of input datasets.
\n
- \n
- \noutputs: list[OutputDataset] | None \n
The set of output datasets.
\n
- \n
- \nclass openlineage.client.event_v2.DatasetEvent(*, eventTime, producer='', dataset) \n
Bases:
\nBaseEvent- \n
- Parameters: \n
- \n
eventTime (str)
\nproducer (str)
\ndataset (StaticDataset)
\n
\n
- \n
- \ndataset: StaticDataset \n
- \n
- \nopenlineage.client.event_v2.RunState \n
alias of
\nEventType
- \n
- \nclass openlineage.client.event_v2.Dataset(namespace, name, *, facets=_Nothing.NOTHING) \n
Bases:
\nRedactMixin- \n
- Parameters: \n
- \n
namespace (str)
\nname (str)
\nfacets (dict[str, DatasetFacet] | None)
\n
\n
- \n
- \nnamespace: str \n
The namespace containing that dataset
\n
- \n
- \nname: str \n
The unique name for that dataset within that namespace
\n
- \n
- \nfacets: dict[str, DatasetFacet] | None \n
The facets for this dataset
\n
- \n
- \nclass openlineage.client.event_v2.InputDataset(namespace, name, inputFacets=_Nothing.NOTHING, *, facets=_Nothing.NOTHING) \n
Bases:
\nDatasetAn input dataset
\n- \n
- Parameters: \n
- \n
namespace (str)
\nname (str)
\ninputFacets (dict[str, InputDatasetFacet] | None)
\nfacets (dict[str, DatasetFacet] | None)
\n
\n
- \n
- \ninputFacets: dict[str, InputDatasetFacet] | None \n
The input facets for this dataset.
\n
- \n
- \nclass openlineage.client.event_v2.OutputDataset(namespace, name, outputFacets=_Nothing.NOTHING, *, facets=_Nothing.NOTHING) \n
Bases:
\nDatasetAn output dataset
\n- \n
- Parameters: \n
- \n
namespace (str)
\nname (str)
\noutputFacets (dict[str, OutputDatasetFacet] | None)
\nfacets (dict[str, DatasetFacet] | None)
\n
\n
- \n
- \noutputFacets: dict[str, OutputDatasetFacet] | None \n
The output facets for this dataset
\n
- \n
- \nclass openlineage.client.event_v2.Run(runId, facets=_Nothing.NOTHING) \n
Bases:
\nRedactMixin- \n
- Parameters: \n
- \n
runId (str)
\nfacets (dict[str, RunFacet] | None)
\n
\n
- \n
- \nrunId: str \n
The globally unique ID of the run associated with the job.
\n
- \n
- \nfacets: dict[str, RunFacet] | None \n
The run facets.
\n
- \n
- \nrunid_check(attribute, value) \n
- \n
- Parameters: \n
- \n
attribute (
str) \nvalue (
str) \n
\n- Return type: \n
\nNone\n
- \n
- \nclass openlineage.client.event_v2.Job(namespace, name, facets=_Nothing.NOTHING) \n
Bases:
\nRedactMixin- \n
- Parameters: \n
- \n
namespace (str)
\nname (str)
\nfacets (dict[str, JobFacet] | None)
\n
\n
- \n
- \nnamespace: str \n
The namespace containing that job
\n
- \n
- \nname: str \n
The unique name for that job within that namespace
\n
- \n
- \nfacets: dict[str, JobFacet] | None \n
The job facets.
\n
- \n
- \nopenlineage.client.event_v2.set_producer(producer) \n
- \n
- Parameters: \n
producer (
\nstr) \n- Return type: \n
\nNone\n
openlineage.client.facet module
\n- \n
- \nopenlineage.client.facet.set_producer(producer) \n
- \n
- Parameters: \n
producer (
\nstr) \n- Return type: \n
\nNone\n
- \n
- \nclass openlineage.client.facet.BaseFacet \n
Bases:
\nRedactMixin- \n
- \n
- \nproperty skip_redact: List[str] \n
- \n
- \nclass openlineage.client.facet.NominalTimeRunFacet(nominalStartTime, nominalEndTime=None) \n
Bases:
\nBaseFacet- \n
- Parameters: \n
- \n
nominalStartTime (
str) \nnominalEndTime (
Optional[str]) \n
\n
- \n
- \nnominalStartTime:
str\n
- \n
- \nnominalEndTime:
Optional[str] \n
- \n
- \nclass openlineage.client.facet.ParentRunFacet(run, job) \n
Bases:
\nBaseFacet- \n
- Parameters: \n
- \n
run (
Dict[Any,Any]) \njob (
Dict[Any,Any]) \n
\n
- \n
- \nrun:
Dict[Any,Any] \n
- \n
- \njob:
Dict[Any,Any] \n
- \n
- \nclassmethod create(runId, namespace, name) \n
- \n
- Parameters: \n
- \n
runId (
str) \nnamespace (
str) \nname (
str) \n
\n- Return type: \n
- \n \n
- \n
- \nclass openlineage.client.facet.DocumentationJobFacet(description) \n
Bases:
\nBaseFacet- \n
- Parameters: \n
description (
\nstr) \n
- \n
- \ndescription:
str\n
- \n
- \nclass openlineage.client.facet.SourceCodeLocationJobFacet(type, url) \n
Bases:
\nBaseFacet- \n
- Parameters: \n
- \n
type (
str) \nurl (
str) \n
\n
- \n
- \ntype:
str\n
- \n
- \nurl:
str\n
- \n
- \nclass openlineage.client.facet.SqlJobFacet(query) \n
Bases:
\nBaseFacet- \n
- Parameters: \n
query (
\nstr) \n
- \n
- \nquery:
str\n
- \n
- \nclass openlineage.client.facet.DocumentationDatasetFacet(description) \n
Bases:
\nBaseFacet- \n
- Parameters: \n
description (
\nstr) \n
- \n
- \ndescription:
str\n
- \n
- \nclass openlineage.client.facet.SchemaField(name, type, description=None) \n
Bases:
\nRedactMixin- \n
- Parameters: \n
- \n
name (
str) \ntype (
str) \ndescription (
Optional[str]) \n
\n
- \n
- \nname:
str\n
- \n
- \ntype:
str\n
- \n
- \ndescription:
Optional[str] \n
- \n
- \nclass openlineage.client.facet.SchemaDatasetFacet(fields) \n
Bases:
\nBaseFacet- \n
- Parameters: \n
fields (
\nList[SchemaField]) \n
- \n
- \nfields:
List[SchemaField] \n
- \n
- \nclass openlineage.client.facet.DataSourceDatasetFacet(name, uri) \n
Bases:
\nBaseFacet- \n
- Parameters: \n
- \n
name (
str) \nuri (
str) \n
\n
- \n
- \nname:
str\n
- \n
- \nuri:
str\n
- \n
- \nclass openlineage.client.facet.OutputStatisticsOutputDatasetFacet(rowCount=None, size=None, fileCount=None) \n
Bases:
\nBaseFacet- \n
- Parameters: \n
- \n
rowCount (
Optional[int]) \nsize (
Optional[int]) \nfileCount (
Optional[int]) \n
\n
- \n
- \nrowCount:
Optional[int] \n
- \n
- \nsize:
Optional[int] \n
- \n
- \nfileCount:
Optional[int] \n
- \n
- \nclass openlineage.client.facet.ColumnMetric(nullCount=None, distinctCount=None, sum=None, count=None, min=None, max=None, quantiles=None) \n
Bases:
\nobject- \n
- Parameters: \n
- \n
nullCount (
Optional[int]) \ndistinctCount (
Optional[int]) \nsum (
Optional[int]) \ncount (
Optional[int]) \nmin (
Optional[float]) \nmax (
Optional[float]) \nquantiles (
Optional[Dict[str,float]]) \n
\n
- \n
- \nnullCount:
Optional[int] \n
- \n
- \ndistinctCount:
Optional[int] \n
- \n
- \nsum:
Optional[int] \n
- \n
- \ncount:
Optional[int] \n
- \n
- \nmin:
Optional[float] \n
- \n
- \nmax:
Optional[float] \n
- \n
- \nquantiles:
Optional[Dict[str,float]] \n
- \n
- \nclass openlineage.client.facet.DataQualityMetricsInputDatasetFacet(rowCount=None, bytes=None, fileCount=None, columnMetrics=_Nothing.NOTHING) \n
Bases:
\nBaseFacet- \n
- Parameters: \n
- \n
rowCount (
Optional[int]) \nbytes (
Optional[int]) \nfileCount (
Optional[int]) \ncolumnMetrics (
Dict[str,ColumnMetric]) \n
\n
- \n
- \nrowCount:
Optional[int] \n
- \n
- \nbytes:
Optional[int] \n
- \n
- \nfileCount:
Optional[int] \n
- \n
- \ncolumnMetrics:
Dict[str,ColumnMetric] \n
- \n
- \nclass openlineage.client.facet.Assertion(assertion, success, column=None) \n
Bases:
\nRedactMixin- \n
- Parameters: \n
- \n
assertion (
str) \nsuccess (
bool) \ncolumn (
Optional[str]) \n
\n
- \n
- \nassertion:
str\n
- \n
- \nsuccess:
bool\n
- \n
- \ncolumn:
Optional[str] \n
- \n
- \nclass openlineage.client.facet.DataQualityAssertionsDatasetFacet(assertions) \n
Bases:
\nBaseFacetThis facet represents asserted expectations on dataset or it\u2019s column.
\n- \n
- Parameters: \n
assertions (
\nList[Assertion]) \n
- \n
- \nassertions:
List[Assertion] \n
- \n
- \nclass openlineage.client.facet.SourceCodeJobFacet(language, source) \n
Bases:
\nBaseFacetThis facet represents source code that the job executed.
\n- \n
- Parameters: \n
- \n
language (
str) \nsource (
str) \n
\n
- \n
- \nlanguage:
str\n
- \n
- \nsource:
str\n
- \n
- \nclass openlineage.client.facet.ExternalQueryRunFacet(externalQueryId, source) \n
Bases:
\nBaseFacet- \n
- Parameters: \n
- \n
externalQueryId (
str) \nsource (
str) \n
\n
- \n
- \nexternalQueryId:
str\n
- \n
- \nsource:
str\n
- \n
- \nclass openlineage.client.facet.ErrorMessageRunFacet(message, programmingLanguage, stackTrace=None) \n
Bases:
\nBaseFacetThis facet represents an error message that was the result of a job run.
\n- \n
- Parameters: \n
- \n
message (
str) \nprogrammingLanguage (
str) \nstackTrace (
Optional[str]) \n
\n
- \n
- \nmessage:
str\n
- \n
- \nprogrammingLanguage:
str\n
- \n
- \nstackTrace:
Optional[str] \n
- \n
- \nclass openlineage.client.facet.SymlinksDatasetFacetIdentifiers(namespace, name, type) \n
Bases:
\nobject- \n
- Parameters: \n
- \n
namespace (
str) \nname (
str) \ntype (
str) \n
\n
- \n
- \nnamespace:
str\n
- \n
- \nname:
str\n
- \n
- \ntype:
str\n
- \n
- \nclass openlineage.client.facet.SymlinksDatasetFacet(identifiers=_Nothing.NOTHING) \n
Bases:
\nBaseFacetThis facet represents dataset symlink names.
\n- \n
- Parameters: \n
identifiers (
\nList[SymlinksDatasetFacetIdentifiers]) \n
- \n
- \nidentifiers:
List[SymlinksDatasetFacetIdentifiers] \n
- \n
- \nclass openlineage.client.facet.StorageDatasetFacet(storageLayer, fileFormat) \n
Bases:
\nBaseFacetThis facet represents dataset symlink names.
\n- \n
- Parameters: \n
- \n
storageLayer (
str) \nfileFormat (
str) \n
\n
- \n
- \nstorageLayer:
str\n
- \n
- \nfileFormat:
str\n
- \n
- \nclass openlineage.client.facet.OwnershipJobFacetOwners(name, type=None) \n
Bases:
\nobject- \n
- Parameters: \n
- \n
name (
str) \ntype (
Optional[str]) \n
\n
- \n
- \nname:
str\n
- \n
- \ntype:
Optional[str] \n
- \n
- \nclass openlineage.client.facet.OwnershipJobFacet(owners=_Nothing.NOTHING) \n
Bases:
\nBaseFacetThis facet represents ownership of a job.
\n- \n
- Parameters: \n
owners (
\nList[OwnershipJobFacetOwners]) \n
- \n
- \nowners:
List[OwnershipJobFacetOwners] \n
- \n
- \nclass openlineage.client.facet.JobTypeJobFacet(processingType, integration, jobType) \n
Bases:
\nBaseFacetThis facet represents job type properties.
\n- \n
- Parameters: \n
- \n
processingType (
str) \nintegration (
str) \njobType (
str) \n
\n
- \n
- \nprocessingType:
str\n
- \n
- \nintegration:
str\n
- \n
- \njobType:
str\n
- \n
- \nclass openlineage.client.facet.DatasetVersionDatasetFacet(datasetVersion) \n
Bases:
\nBaseFacetThis facet represents version of a dataset.
\n- \n
- Parameters: \n
datasetVersion (
\nstr) \n
- \n
- \ndatasetVersion:
str\n
- \n
- \nclass openlineage.client.facet.LifecycleStateChange(value) \n
Bases:
\nEnumAn enumeration.
\n- \n
- \nALTER = 'ALTER' \n
- \n
- \nCREATE = 'CREATE' \n
- \n
- \nDROP = 'DROP' \n
- \n
- \nOVERWRITE = 'OVERWRITE' \n
- \n
- \nRENAME = 'RENAME' \n
- \n
- \nTRUNCATE = 'TRUNCATE' \n
- \n
- \nclass openlineage.client.facet.LifecycleStateChangeDatasetFacetPreviousIdentifier(name, namespace) \n
Bases:
\nobject- \n
- Parameters: \n
- \n
name (
str) \nnamespace (
str) \n
\n
- \n
- \nname:
str\n
- \n
- \nnamespace:
str\n
- \n
- \nclass openlineage.client.facet.LifecycleStateChangeDatasetFacet(lifecycleStateChange, previousIdentifier) \n
Bases:
\nBaseFacetThis facet represents information of lifecycle changes of a dataset.
\n- \n
- Parameters: \n
- \n
lifecycleStateChange (
LifecycleStateChange) \npreviousIdentifier (
LifecycleStateChangeDatasetFacetPreviousIdentifier) \n
\n
- \n
- \nlifecycleStateChange:
LifecycleStateChange\n
- \n
- \npreviousIdentifier:
LifecycleStateChangeDatasetFacetPreviousIdentifier\n
- \n
- \nclass openlineage.client.facet.OwnershipDatasetFacetOwners(name, type) \n
Bases:
\nobject- \n
- Parameters: \n
- \n
name (
str) \ntype (
str) \n
\n
- \n
- \nname:
str\n
- \n
- \ntype:
str\n
- \n
- \nclass openlineage.client.facet.OwnershipDatasetFacet(owners=_Nothing.NOTHING) \n
Bases:
\nBaseFacetThis facet represents ownership of a dataset.
\n- \n
- Parameters: \n
owners (
\nList[OwnershipDatasetFacetOwners]) \n
- \n
- \nowners:
List[OwnershipDatasetFacetOwners] \n
- \n
- \nclass openlineage.client.facet.ColumnLineageDatasetFacetFieldsAdditionalInputFields(namespace, name, field) \n
Bases:
\nRedactMixin- \n
- Parameters: \n
- \n
namespace (
str) \nname (
str) \nfield (
str) \n
\n
- \n
- \nnamespace:
str\n
- \n
- \nname:
str\n
- \n
- \nfield:
str\n
- \n
- \nclass openlineage.client.facet.ColumnLineageDatasetFacetFieldsAdditional(inputFields, transformationDescription, transformationType) \n
Bases:
\nobject- \n
- Parameters: \n
- \n
inputFields (
ClassVar[List[ColumnLineageDatasetFacetFieldsAdditionalInputFields]]) \ntransformationDescription (
str) \ntransformationType (
str) \n
\n
- \n
- \ninputFields:
ClassVar[List[ColumnLineageDatasetFacetFieldsAdditionalInputFields]] \n
- \n
- \ntransformationDescription:
str\n
- \n
- \ntransformationType:
str\n
- \n
- \nclass openlineage.client.facet.ColumnLineageDatasetFacet(fields=_Nothing.NOTHING) \n
Bases:
\nBaseFacetThis facet contains column lineage of a dataset.
\n- \n
- Parameters: \n
fields (
\nDict[str,ColumnLineageDatasetFacetFieldsAdditional]) \n
- \n
- \nfields:
Dict[str,ColumnLineageDatasetFacetFieldsAdditional] \n
- \n
- \nclass openlineage.client.facet.ProcessingEngineRunFacet(version, name, openlineageAdapterVersion) \n
Bases:
\nBaseFacet- \n
- Parameters: \n
- \n
version (
str) \nname (
str) \nopenlineageAdapterVersion (
str) \n
\n
- \n
- \nversion:
str\n
- \n
- \nname:
str\n
- \n
- \nopenlineageAdapterVersion:
str\n
- \n
- \nclass openlineage.client.facet.ExtractionError(errorMessage, stackTrace, task, taskNumber) \n
Bases:
\nBaseFacet- \n
- Parameters: \n
- \n
errorMessage (
str) \nstackTrace (
Optional[str]) \ntask (
Optional[str]) \ntaskNumber (
Optional[int]) \n
\n
- \n
- \nerrorMessage:
str\n
- \n
- \nstackTrace:
Optional[str] \n
- \n
- \ntask:
Optional[str] \n
- \n
- \ntaskNumber:
Optional[int] \n
- \n
- \nclass openlineage.client.facet.ExtractionErrorRunFacet(totalTasks, failedTasks, errors) \n
Bases:
\nBaseFacet- \n
- Parameters: \n
- \n
totalTasks (
int) \nfailedTasks (
int) \nerrors (
List[ExtractionError]) \n
\n
- \n
- \ntotalTasks:
int\n
- \n
- \nfailedTasks:
int\n
- \n
- \nerrors:
List[ExtractionError] \n
openlineage.client.facet_v2 module
\n- \n
- \nclass openlineage.client.facet_v2.BaseFacet(*, producer='') \n
Bases:
\nRedactMixinall fields of the base facet are prefixed with _ to avoid name conflicts in facets
\n- \n
- Parameters: \n
producer (
\nstr) \n
- \n
- \nproperty skip_redact: list[str] \n
- \n
- \nclass openlineage.client.facet_v2.DatasetFacet(*, producer='', deleted=None) \n
Bases:
\nBaseFacetA Dataset Facet
\n- \n
- Parameters: \n
- \n
producer (str)
\ndeleted (bool | None)
\n
\n
- \n
- \nclass openlineage.client.facet_v2.InputDatasetFacet(*, producer='') \n
Bases:
\nBaseFacetAn Input Dataset Facet
\n- \n
- Parameters: \n
producer (
\nstr) \n
- \n
- \nclass openlineage.client.facet_v2.JobFacet(*, producer='', deleted=None) \n
Bases:
\nBaseFacetA Job Facet
\n- \n
- Parameters: \n
- \n
producer (str)
\ndeleted (bool | None)
\n
\n
- \n
- \nclass openlineage.client.facet_v2.OutputDatasetFacet(*, producer='') \n
Bases:
\nBaseFacetAn Output Dataset Facet
\n- \n
- Parameters: \n
producer (
\nstr) \n
- \n
- \nclass openlineage.client.facet_v2.RunFacet(*, producer='') \n
Bases:
\nBaseFacetA Run Facet
\n- \n
- Parameters: \n
producer (
\nstr) \n
- \n
- \nopenlineage.client.facet_v2.set_producer(producer) \n
- \n
- Parameters: \n
producer (
\nstr) \n- Return type: \n
\nNone\n
openlineage.client.filter module
\n- \n
- \nclass openlineage.client.filter.FilterConfig(type=None, match=None, regex=None) \n
Bases:
\nobject- \n
- Parameters: \n
- \n
type (str | None)
\nmatch (str | None)
\nregex (str | None)
\n
\n
- \n
- \ntype: str | None \n
- \n
- \nmatch: str | None \n
- \n
- \nregex: str | None \n
- \n
- \nclass openlineage.client.filter.Filter \n
Bases:
\nobject- \n
- \nfilter_event(event) \n
- \n
- Parameters: \n
event (RunEventType)
\n \n- Return type: \n
RunEventType | None
\n \n
- \n
- \nclass openlineage.client.filter.ExactMatchFilter(match) \n
Bases:
\nFilter- \n
- Parameters: \n
match (
\nstr) \n
- \n
- \nfilter_event(event) \n
- \n
- Parameters: \n
event (RunEventType)
\n \n- Return type: \n
RunEventType | None
\n \n
- \n
- \nclass openlineage.client.filter.RegexFilter(regex) \n
Bases:
\nFilter- \n
- Parameters: \n
regex (
\nstr) \n
- \n
- \nfilter_event(event) \n
- \n
- Parameters: \n
event (RunEventType)
\n \n- Return type: \n
RunEventType | None
\n \n
- \n
- \nopenlineage.client.filter.create_filter(conf) \n
- \n
- Parameters: \n
conf (FilterConfig)
\n \n- Return type: \n
Filter | None
\n \n
openlineage.client.run module
\n- \n
- \nclass openlineage.client.run.RunState(value) \n
Bases:
\nEnumAn enumeration.
\n- \n
- \nSTART = 'START' \n
- \n
- \nRUNNING = 'RUNNING' \n
- \n
- \nCOMPLETE = 'COMPLETE' \n
- \n
- \nABORT = 'ABORT' \n
- \n
- \nFAIL = 'FAIL' \n
- \n
- \nOTHER = 'OTHER' \n
- \n
- \nclass openlineage.client.run.Dataset(namespace, name, facets=_Nothing.NOTHING) \n
Bases:
\nRedactMixin- \n
- Parameters: \n
- \n
namespace (
str) \nname (
str) \nfacets (
Dict[Any,Any]) \n
\n
- \n
- \nnamespace:
str\n
- \n
- \nname:
str\n
- \n
- \nfacets:
Dict[Any,Any] \n
- \n
- \nclass openlineage.client.run.InputDataset(namespace, name, facets=_Nothing.NOTHING, inputFacets=_Nothing.NOTHING) \n
Bases:
\nDataset- \n
- Parameters: \n
- \n
namespace (
str) \nname (
str) \nfacets (
Dict[Any,Any]) \ninputFacets (
Dict[Any,Any]) \n
\n
- \n
- \ninputFacets:
Dict[Any,Any] \n
- \n
- \nclass openlineage.client.run.OutputDataset(namespace, name, facets=_Nothing.NOTHING, outputFacets=_Nothing.NOTHING) \n
Bases:
\nDataset- \n
- Parameters: \n
- \n
namespace (
str) \nname (
str) \nfacets (
Dict[Any,Any]) \noutputFacets (
Dict[Any,Any]) \n
\n
- \n
- \noutputFacets:
Dict[Any,Any] \n
- \n
- \nclass openlineage.client.run.DatasetEvent(eventTime, producer, schemaURL, dataset) \n
Bases:
\nRedactMixin- \n
- Parameters: \n
- \n
eventTime (
str) \nproducer (
str) \nschemaURL (
str) \ndataset (
Dataset) \n
\n
- \n
- \neventTime:
str\n
- \n
- \nproducer:
str\n
- \n
- \nschemaURL:
str\n
- \n
- \ndataset:
Dataset\n
- \n
- \nclass openlineage.client.run.Job(namespace, name, facets=_Nothing.NOTHING) \n
Bases:
\nRedactMixin- \n
- Parameters: \n
- \n
namespace (
str) \nname (
str) \nfacets (
Dict[Any,Any]) \n
\n
- \n
- \nnamespace:
str\n
- \n
- \nname:
str\n
- \n
- \nfacets:
Dict[Any,Any] \n
- \n
- \nclass openlineage.client.run.JobEvent(eventTime, producer, schemaURL, job, inputs=_Nothing.NOTHING, outputs=_Nothing.NOTHING) \n
Bases:
\nRedactMixin- \n
- Parameters: \n
- \n \n
- \n
- \neventTime:
str\n
- \n
- \nproducer:
str\n
- \n
- \nschemaURL:
str\n
- \n
- \njob:
Job\n
- \n
- \ninputs:
Optional[List[Dataset]] \n
- \n
- \noutputs:
Optional[List[Dataset]] \n
- \n
- \nclass openlineage.client.run.Run(runId, facets=_Nothing.NOTHING) \n
Bases:
\nRedactMixin- \n
- Parameters: \n
- \n
runId (
str) \nfacets (
Dict[Any,Any]) \n
\n
- \n
- \nrunId:
str\n
- \n
- \nfacets:
Dict[Any,Any] \n
- \n
- \ncheck(attribute, value) \n
- \n
- Parameters: \n
- \n
attribute (
str) \nvalue (
str) \n
\n- Return type: \n
\nNone\n
- \n
- \nclass openlineage.client.run.RunEvent(eventType, eventTime, run, job, producer, inputs=_Nothing.NOTHING, outputs=_Nothing.NOTHING, schemaURL='https://openlineage.io/spec/1-0-5/OpenLineage.json#/definitions/RunEvent') \n
Bases:
\nRedactMixin- \n
- Parameters: \n
- \n \n
- \n
- \neventType:
RunState\n
- \n
- \neventTime:
str\n
- \n
- \nrun:
Run\n
- \n
- \njob:
Job\n
- \n
- \nproducer:
str\n
- \n
- \ninputs:
Optional[List[Dataset]] \n
- \n
- \noutputs:
Optional[List[Dataset]] \n
- \n
- \nschemaURL:
str\n
- \n
- \ncheck(attribute, value) \n
- \n
- Parameters: \n
- \n
attribute (
str) \nvalue (
str) \n
\n- Return type: \n
\nNone\n
openlineage.client.serde module
\n- \n
- \nclass openlineage.client.serde.Serde \n
Bases:
\nobject- \n
- \nclassmethod remove_nulls_and_enums(obj) \n
- \n
- Parameters: \n
obj (
\nAny) \n- Return type: \n
\nAny\n
- \n
- \nclassmethod to_dict(obj) \n
- \n
- Parameters: \n
obj (Any)
\n \n- Return type: \n
dict[Any, Any]
\n \n
- \n
- \nclassmethod to_json(obj) \n
- \n
- Parameters: \n
obj (
\nAny) \n- Return type: \n
\nstr\n
openlineage.client.utils module
\n- \n
- \nopenlineage.client.utils.import_from_string(path) \n
- \n
- Parameters: \n
path (str)
\n \n- Return type: \n
type[Any]
\n \n
- \n
- \nopenlineage.client.utils.try_import_from_string(path) \n
- \n
- Parameters: \n
path (str)
\n \n- Return type: \n
type[Any] | None
\n \n
- \n
- \nopenlineage.client.utils.get_only_specified_fields(clazz, params) \n
- \n
- Parameters: \n
- \n
clazz (type[Any])
\nparams (dict[str, Any])
\n
\n- Return type: \n
dict[str, Any]
\n \n
- \n
- \nopenlineage.client.utils.deep_merge_dicts(dict1, dict2) \n
Deep merges two dictionaries.
\nThis function merges two dictionaries while handling nested dictionaries.\nFor keys that exist in both dictionaries, the values from dict2 take precedence.\nIf a key exists in both dictionaries and the values are dictionaries themselves,\nthey are merged recursively.\nThis function merges only dictionaries. If key is of different type, e.g. list\nit does not work properly.
\n- \n
- Parameters: \n
- \n
dict1 (dict[Any, Any])
\ndict2 (dict[Any, Any])
\n
\n- Return type: \n
dict[Any, Any]
\n \n
- \n
- \nclass openlineage.client.utils.RedactMixin \n
Bases:
\nobject- \n
- \nproperty skip_redact: list[str] \n
openlineage.client.uuid module
\n- \n
- \nopenlineage.client.uuid.generate_new_uuid(instant=None) \n
Generate new UUID for an instant of time. Each function call returns a new UUID value.
\nUUID version is an implementation detail, and should not be relied on.\nFor now it is [UUIDv7](https://datatracker.ietf.org/doc/rfc9562/), so for increasing instant values,\nreturned UUID is always greater than previous one.
\nUsing uuid6 lib implementation (MIT License), with few changes:\n* oittaa/uuid6-python\n* oittaa/uuid6-python
\nAdded in v1.15.0
\n- \n
- Parameters: \n
instant (datetime | None) \u2013 instant of time used to generate UUID. If not provided, current time is used.
\n \n- Return type: \n
UUID
\n \n- Returns: \n
UUID
\n \n
- \n
- \nopenlineage.client.uuid.generate_static_uuid(instant, data) \n
Generate UUID for instant of time and input data.\nCalling function with same arguments always produces the same result.
\nUUID version is an implementation detail, and **should not* be relied on.\nFor now it is [UUIDv7](https://datatracker.ietf.org/doc/rfc9562/), so for increasing instant values,\nreturned UUID is always greater than previous one. The only difference from RFC 9562 is that\nleast significant bytes are not random, but instead a SHA-1 hash of input data.
\nUsing uuid6 lib implementation (MIT License), with few changes:\n* oittaa/uuid6-python\n* oittaa/uuid6-python
\nAdded in v1.15.0
\n- \n
- Parameters: \n
- \n
instant (
datetime) \u2013 instant of time used to generate UUID. If not provided, current time is used. \ndata (
bytes) \u2013 input data to generate random part from. \n
\n- Return type: \n
\nUUID\n- Returns: \n
UUID
\n \n
openlineage.client.generated.base module
\n- \n
- \nopenlineage.client.generated.base.set_producer(producer) \n
- \n
- Parameters: \n
producer (
\nstr) \n- Return type: \n
\nNone\n
- \n
- \nclass openlineage.client.generated.base.BaseEvent(*, eventTime, producer='') \n
Bases:
\nRedactMixin- \n
- Parameters: \n
- \n
eventTime (
str) \nproducer (
str) \n
\n
- \n
- \neventTime: str \n
the time the event occurred at
\n
- \n
- \nproducer: str \n
- \n
- \nschemaURL: str \n
- \n
- \nproperty skip_redact \n
- \n
- \neventtime_check(attribute, value) \n
- \n
- Parameters: \n
- \n
attribute (
str) \nvalue (
str) \n
\n- Return type: \n
\nNone\n
- \n
- \nproducer_check(attribute, value) \n
- \n
- Parameters: \n
- \n
attribute (
str) \nvalue (
str) \n
\n- Return type: \n
\nNone\n
- \n
- \nschemaurl_check(attribute, value) \n
- \n
- Parameters: \n
- \n
attribute (
str) \nvalue (
str) \n
\n- Return type: \n
\nNone\n
- \n
- \nclass openlineage.client.generated.base.BaseFacet(*, producer='') \n
Bases:
\nRedactMixinall fields of the base facet are prefixed with _ to avoid name conflicts in facets
\n- \n
- Parameters: \n
producer (
\nstr) \n
- \n
- \nproperty skip_redact \n
- \n
- \nclass openlineage.client.generated.base.Dataset(namespace, name, *, facets=_Nothing.NOTHING) \n
Bases:
\nRedactMixin- \n
- Parameters: \n
- \n
namespace (str)
\nname (str)
\nfacets (dict[str, DatasetFacet] | None)
\n
\n
- \n
- \nnamespace: str \n
The namespace containing that dataset
\n
- \n
- \nname: str \n
The unique name for that dataset within that namespace
\n
- \n
- \nfacets: dict[str, DatasetFacet] | None \n
The facets for this dataset
\n
- \n
- \nclass openlineage.client.generated.base.DatasetEvent(*, eventTime, producer='', dataset) \n
Bases:
\nBaseEvent- \n
- Parameters: \n
- \n
eventTime (str)
\nproducer (str)
\ndataset (StaticDataset)
\n
\n
- \n
- \ndataset: StaticDataset \n
- \n
- \nclass openlineage.client.generated.base.DatasetFacet(*, producer='', deleted=None) \n
Bases:
\nBaseFacetA Dataset Facet
\n- \n
- Parameters: \n
- \n
producer (str)
\ndeleted (bool | None)
\n
\n
- \n
- \nclass openlineage.client.generated.base.EventType(value) \n
Bases:
\nEnumthe current transition of the run state. It is required to issue 1 START event and 1 of [ COMPLETE,\nABORT, FAIL ] event per run. Additional events with OTHER eventType can be added to the same run.\nFor example to send additional metadata after the run is complete
\n- \n
- \nSTART = 'START' \n
- \n
- \nRUNNING = 'RUNNING' \n
- \n
- \nCOMPLETE = 'COMPLETE' \n
- \n
- \nABORT = 'ABORT' \n
- \n
- \nFAIL = 'FAIL' \n
- \n
- \nOTHER = 'OTHER' \n
- \n
- \nclass openlineage.client.generated.base.InputDataset(namespace, name, inputFacets=_Nothing.NOTHING, *, facets=_Nothing.NOTHING) \n
Bases:
\nDatasetAn input dataset
\n- \n
- \ninputFacets: dict[str, InputDatasetFacet] | None \n
The input facets for this dataset.
\n
- \n
- \nclass openlineage.client.generated.base.InputDatasetFacet(*, producer='') \n
Bases:
\nBaseFacetAn Input Dataset Facet
\n- \n
- Parameters: \n
producer (
\nstr) \n
- \n
- \nclass openlineage.client.generated.base.Job(namespace, name, facets=_Nothing.NOTHING) \n
Bases:
\nRedactMixin- \n
- Parameters: \n
- \n
namespace (str)
\nname (str)
\nfacets (dict[str, JobFacet] | None)
\n
\n
- \n
- \nnamespace: str \n
The namespace containing that job
\n
- \n
- \nname: str \n
The unique name for that job within that namespace
\n
- \n
- \nfacets: dict[str, JobFacet] | None \n
The job facets.
\n
- \n
- \nclass openlineage.client.generated.base.JobEvent(*, eventTime, producer='', job, inputs=_Nothing.NOTHING, outputs=_Nothing.NOTHING) \n
Bases:
\nBaseEvent- \n
- \njob: Job \n
- \n
- \ninputs: list[InputDataset] | None \n
The set of input datasets.
\n
- \n
- \noutputs: list[OutputDataset] | None \n
The set of output datasets.
\n
- \n
- \nclass openlineage.client.generated.base.JobFacet(*, producer='', deleted=None) \n
Bases:
\nBaseFacetA Job Facet
\n- \n
- Parameters: \n
- \n
producer (str)
\ndeleted (bool | None)
\n
\n
- \n
- \nclass openlineage.client.generated.base.OutputDataset(namespace, name, outputFacets=_Nothing.NOTHING, *, facets=_Nothing.NOTHING) \n
Bases:
\nDatasetAn output dataset
\n- \n
- \noutputFacets: dict[str, OutputDatasetFacet] | None \n
The output facets for this dataset
\n
- \n
- \nclass openlineage.client.generated.base.OutputDatasetFacet(*, producer='') \n
Bases:
\nBaseFacetAn Output Dataset Facet
\n- \n
- Parameters: \n
producer (
\nstr) \n
- \n
- \nclass openlineage.client.generated.base.Run(runId, facets=_Nothing.NOTHING) \n
Bases:
\nRedactMixin- \n
- Parameters: \n
- \n
runId (str)
\nfacets (dict[str, RunFacet] | None)
\n
\n
- \n
- \nrunId: str \n
The globally unique ID of the run associated with the job.
\n
- \n
- \nfacets: dict[str, RunFacet] | None \n
The run facets.
\n
- \n
- \nrunid_check(attribute, value) \n
- \n
- Parameters: \n
- \n
attribute (
str) \nvalue (
str) \n
\n- Return type: \n
\nNone\n
- \n
- \nclass openlineage.client.generated.base.RunEvent(*, eventTime, producer='', run, job, eventType=None, inputs=_Nothing.NOTHING, outputs=_Nothing.NOTHING) \n
Bases:
\nBaseEvent- \n
- Parameters: \n
- \n
eventTime (str)
\nproducer (str)
\nrun (Run)
\njob (Job)
\neventType (EventType | None)
\ninputs (list[InputDataset] | None)
\noutputs (list[OutputDataset] | None)
\n
\n
- \n
- \nrun: Run \n
- \n
- \njob: Job \n
- \n
- \neventType: EventType | None \n
the current transition of the run state. It is required to issue 1 START event and 1 of [ COMPLETE,\nABORT, FAIL ] event per run. Additional events with OTHER eventType can be added to the same run.\nFor example to send additional metadata after the run is complete
\n
- \n
- \ninputs: list[InputDataset] | None \n
The set of input datasets.
\n
- \n
- \noutputs: list[OutputDataset] | None \n
The set of output datasets.
\n
- \n
- \nclass openlineage.client.generated.base.RunFacet(*, producer='') \n
Bases:
\nBaseFacetA Run Facet
\n- \n
- Parameters: \n
producer (
\nstr) \n
- \n
- \nclass openlineage.client.generated.base.StaticDataset(namespace, name, *, facets=_Nothing.NOTHING) \n
Bases:
\nDatasetA Dataset sent within static metadata events
\n- \n
- Parameters: \n
- \n
namespace (str)
\nname (str)
\nfacets (dict[str, DatasetFacet] | None)
\n
\n
openlineage.client.generated.column_lineage_dataset module
\n- \n
- \nclass openlineage.client.generated.column_lineage_dataset.ColumnLineageDatasetFacet(fields, dataset=_Nothing.NOTHING, *, producer='', deleted=None) \n
Bases:
\nDatasetFacet- \n
- Parameters: \n
- \n
fields (dict[str, Fields])
\ndataset (list[InputField] | None)
\nproducer (str)
\ndeleted (bool | None)
\n
\n
- \n
- \nfields: dict[str, Fields] \n
Column level lineage that maps output fields into input fields used to evaluate them.
\n
- \n
- \ndataset: list[InputField] | None \n
Column level lineage that affects the whole dataset. This includes filtering, sorting, grouping\n(aggregates), joining, window functions, etc.
\n
- \n
- \nclass openlineage.client.generated.column_lineage_dataset.Fields(inputFields, transformationDescription=None, transformationType=None) \n
Bases:
\nRedactMixin- \n
- Parameters: \n
- \n
inputFields (list[InputField])
\ntransformationDescription (str | None)
\ntransformationType (str | None)
\n
\n
- \n
- \ninputFields: list[InputField] \n
- \n
- \ntransformationDescription: str | None \n
a string representation of the transformation applied
\n
- \n
- \ntransformationType: str | None \n
no\noriginal data available (like a hash of PII for example)
\n- \n
- Type: \n
IDENTITY|MASKED reflects a clearly defined behavior. IDENTITY
\n \n- Type: \n
exact same as input; MASKED
\n \n
- \n
- \nclass openlineage.client.generated.column_lineage_dataset.InputField(namespace, name, field, transformations=_Nothing.NOTHING) \n
Bases:
\nRedactMixinRepresents a single dependency on some field (column).
\n- \n
- Parameters: \n
- \n
namespace (str)
\nname (str)
\nfield (str)
\ntransformations (list[Transformation] | None)
\n
\n
- \n
- \nnamespace: str \n
The input dataset namespace
\n
- \n
- \nname: str \n
The input dataset name
\n
- \n
- \nfield: str \n
The input field
\n
- \n
- \ntransformations: list[Transformation] | None \n
- \n
- \nclass openlineage.client.generated.column_lineage_dataset.Transformation(type, subtype=None, description=None, masking=None) \n
Bases:
\nRedactMixin- \n
- Parameters: \n
- \n
type (str)
\nsubtype (str | None)
\ndescription (str | None)
\nmasking (bool | None)
\n
\n
- \n
- \ntype: str \n
DIRECT, INDIRECT
\n- \n
- Type: \n
The type of the transformation. Allowed values are
\n \n
- \n
- \nsubtype: str | None \n
The subtype of the transformation
\n
- \n
- \ndescription: str | None \n
a string representation of the transformation applied
\n
- \n
- \nmasking: bool | None \n
is transformation masking the data or not
\n
openlineage.client.generated.data_quality_assertions_dataset module
\n- \n
- \nclass openlineage.client.generated.data_quality_assertions_dataset.Assertion(assertion, success, column=None) \n
Bases:
\nRedactMixin- \n
- Parameters: \n
- \n
assertion (str)
\nsuccess (bool)
\ncolumn (str | None)
\n
\n
- \n
- \nassertion: str \n
Type of expectation test that dataset is subjected to
\n
- \n
- \nsuccess: bool \n
- \n
- \ncolumn: str | None \n
Column that expectation is testing. It should match the name provided in SchemaDatasetFacet. If\ncolumn field is empty, then expectation refers to whole dataset.
\n
- \n
- \nclass openlineage.client.generated.data_quality_assertions_dataset.DataQualityAssertionsDatasetFacet(assertions, *, producer='') \n
Bases:
\nInputDatasetFacetlist of tests performed on dataset or dataset columns, and their results
\n- \n
- Parameters: \n
- \n
assertions (list[Assertion])
\nproducer (str)
\n
\n
- \n
- \nassertions: list[Assertion] \n
openlineage.client.generated.data_quality_metrics_input_dataset module
\n- \n
- \nclass openlineage.client.generated.data_quality_metrics_input_dataset.ColumnMetrics(nullCount=None, distinctCount=None, sum=None, count=None, min=None, max=None, quantiles=_Nothing.NOTHING) \n
Bases:
\nRedactMixin- \n
- Parameters: \n
- \n
nullCount (int | None)
\ndistinctCount (int | None)
\nsum (float | None)
\ncount (float | None)
\nmin (float | None)
\nmax (float | None)
\nquantiles (dict[str, float] | None)
\n
\n
- \n
- \nnullCount: int | None \n
The number of null values in this column for the rows evaluated
\n
- \n
- \ndistinctCount: int | None \n
The number of distinct values in this column for the rows evaluated
\n
- \n
- \nsum: float | None \n
The total sum of values in this column for the rows evaluated
\n
- \n
- \ncount: float | None \n
The number of values in this column
\n
- \n
- \nmin: float | None \n
- \n
- \nmax: float | None \n
- \n
- \nquantiles: dict[str, float] | None \n
0.1 0.25 0.5 0.75 1
\n- \n
- Type: \n
The property key is the quantile. Examples
\n \n
- \n
- \nclass openlineage.client.generated.data_quality_metrics_input_dataset.DataQualityMetricsInputDatasetFacet(columnMetrics, rowCount=None, bytes=None, fileCount=None, *, producer='') \n
Bases:
\nInputDatasetFacet- \n
- Parameters: \n
- \n
columnMetrics (dict[str, ColumnMetrics])
\nrowCount (int | None)
\nbytes (int | None)
\nfileCount (int | None)
\nproducer (str)
\n
\n
- \n
- \ncolumnMetrics: dict[str, ColumnMetrics] \n
The property key is the column name
\n
- \n
- \nrowCount: int | None \n
The number of rows evaluated
\n
- \n
- \nbytes: int | None \n
The size in bytes
\n
- \n
- \nfileCount: int | None \n
The number of files evaluated
\n
openlineage.client.generated.dataset_version_dataset module
\n- \n
- \nclass openlineage.client.generated.dataset_version_dataset.DatasetVersionDatasetFacet(datasetVersion, *, producer='', deleted=None) \n
Bases:
\nDatasetFacet- \n
- Parameters: \n
- \n
datasetVersion (str)
\nproducer (str)
\ndeleted (bool | None)
\n
\n
- \n
- \ndatasetVersion: str \n
The version of the dataset.
\n
openlineage.client.generated.datasource_dataset module
\n- \n
- \nclass openlineage.client.generated.datasource_dataset.DatasourceDatasetFacet(name=None, uri=None, *, producer='', deleted=None) \n
Bases:
\nDatasetFacet- \n
- Parameters: \n
- \n
name (str | None)
\nuri (str | None)
\nproducer (str)
\ndeleted (bool | None)
\n
\n
- \n
- \nname: str | None \n
- \n
- \nuri: str | None \n
- \n
- \nuri_check(attribute, value) \n
- \n
- Parameters: \n
- \n
attribute (
str) \nvalue (
str) \n
\n- Return type: \n
\nNone\n
openlineage.client.generated.documentation_dataset module
\n- \n
- \nclass openlineage.client.generated.documentation_dataset.DocumentationDatasetFacet(description, *, producer='', deleted=None) \n
Bases:
\nDatasetFacet- \n
- Parameters: \n
- \n
description (str)
\nproducer (str)
\ndeleted (bool | None)
\n
\n
- \n
- \ndescription: str \n
The description of the dataset.
\n
openlineage.client.generated.documentation_job module
\n- \n
- \nclass openlineage.client.generated.documentation_job.DocumentationJobFacet(description, *, producer='', deleted=None) \n
Bases:
\nJobFacet- \n
- Parameters: \n
- \n
description (str)
\nproducer (str)
\ndeleted (bool | None)
\n
\n
- \n
- \ndescription: str \n
The description of the job.
\n
openlineage.client.generated.error_message_run module
\n- \n
- \nclass openlineage.client.generated.error_message_run.ErrorMessageRunFacet(message, programmingLanguage, stackTrace=None, *, producer='') \n
Bases:
\nRunFacet- \n
- Parameters: \n
- \n
message (str)
\nprogrammingLanguage (str)
\nstackTrace (str | None)
\nproducer (str)
\n
\n
- \n
- \nmessage: str \n
A human-readable string representing error message generated by observed system
\n
- \n
- \nprogrammingLanguage: str \n
Programming language the observed system uses.
\n
- \n
- \nstackTrace: str | None \n
A language-specific stack trace generated by observed system
\n
openlineage.client.generated.external_query_run module
\n- \n
- \nclass openlineage.client.generated.external_query_run.ExternalQueryRunFacet(externalQueryId, source, *, producer='') \n
Bases:
\nRunFacet- \n
- Parameters: \n
- \n
externalQueryId (
str) \nsource (
str) \nproducer (
str) \n
\n
- \n
- \nexternalQueryId: str \n
Identifier for the external system
\n
- \n
- \nsource: str \n
source of the external query
\n
openlineage.client.generated.extraction_error_run module
\n- \n
- \nclass openlineage.client.generated.extraction_error_run.Error(errorMessage, stackTrace=None, task=None, taskNumber=None) \n
Bases:
\nRedactMixin- \n
- Parameters: \n
- \n
errorMessage (str)
\nstackTrace (str | None)
\ntask (str | None)
\ntaskNumber (int | None)
\n
\n
- \n
- \nerrorMessage: str \n
Text representation of extraction error message.
\n
- \n
- \nstackTrace: str | None \n
Stack trace of extraction error message
\n
- \n
- \ntask: str | None \n
Text representation of task that failed. This can be, for example, SQL statement that parser could\nnot interpret.
\n
- \n
- \ntaskNumber: int | None \n
Order of task (counted from 0).
\n
- \n
- \nclass openlineage.client.generated.extraction_error_run.ExtractionErrorRunFacet(totalTasks, failedTasks, errors, *, producer='') \n
Bases:
\nRunFacet- \n
- Parameters: \n
- \n
totalTasks (int)
\nfailedTasks (int)
\nerrors (list[Error])
\nproducer (str)
\n
\n
- \n
- \ntotalTasks: int \n
The number of distinguishable tasks in a run that were processed by OpenLineage, whether\nsuccessfully or not. Those could be, for example, distinct SQL statements.
\n
- \n
- \nfailedTasks: int \n
The number of distinguishable tasks in a run that were processed not successfully by OpenLineage.\nThose could be, for example, distinct SQL statements.
\n
- \n
- \nerrors: list[Error] \n
openlineage.client.generated.job_type_job module
\n- \n
- \nclass openlineage.client.generated.job_type_job.JobTypeJobFacet(processingType, integration, jobType=None, *, producer='', deleted=None) \n
Bases:
\nJobFacet- \n
- Parameters: \n
- \n
processingType (str)
\nintegration (str)
\njobType (str | None)
\nproducer (str)
\ndeleted (bool | None)
\n
\n
- \n
- \nprocessingType: str \n
BATCH or STREAMING
\n- \n
- Type: \n
Job processing type like
\n \n
- \n
- \nintegration: str \n
for example SPARK|DBT|AIRFLOW|FLINK
\n- \n
- Type: \n
OpenLineage integration type of this job
\n \n
- \n
- \njobType: str | None \n
QUERY|COMMAND|DAG|TASK|JOB|MODEL. This is an integration-specific field.
\n- \n
- Type: \n
Run type, for example
\n \n
openlineage.client.generated.lifecycle_state_change_dataset module
\n- \n
- \nclass openlineage.client.generated.lifecycle_state_change_dataset.LifecycleStateChange(value) \n
Bases:
\nEnumThe lifecycle state change.
\n- \n
- \nALTER = 'ALTER' \n
- \n
- \nCREATE = 'CREATE' \n
- \n
- \nDROP = 'DROP' \n
- \n
- \nOVERWRITE = 'OVERWRITE' \n
- \n
- \nRENAME = 'RENAME' \n
- \n
- \nTRUNCATE = 'TRUNCATE' \n
- \n
- \nclass openlineage.client.generated.lifecycle_state_change_dataset.LifecycleStateChangeDatasetFacet(lifecycleStateChange, previousIdentifier=None, *, producer='', deleted=None) \n
Bases:
\nDatasetFacet- \n
- Parameters: \n
- \n
lifecycleStateChange (LifecycleStateChange)
\npreviousIdentifier (PreviousIdentifier | None)
\nproducer (str)
\ndeleted (bool | None)
\n
\n
- \n
- \nlifecycleStateChange: LifecycleStateChange \n
The lifecycle state change.
\n
- \n
- \npreviousIdentifier: PreviousIdentifier | None \n
Previous name of the dataset in case of renaming it.
\n
- \n
- \nclass openlineage.client.generated.lifecycle_state_change_dataset.PreviousIdentifier(name, namespace) \n
Bases:
\nRedactMixinPrevious name of the dataset in case of renaming it.
\n- \n
- Parameters: \n
- \n
name (
str) \nnamespace (
str) \n
\n
- \n
- \nname: str \n
- \n
- \nnamespace: str \n
openlineage.client.generated.nominal_time_run module
\n- \n
- \nclass openlineage.client.generated.nominal_time_run.NominalTimeRunFacet(nominalStartTime, nominalEndTime=None, *, producer='') \n
Bases:
\nRunFacet- \n
- Parameters: \n
- \n
nominalStartTime (str)
\nnominalEndTime (str | None)
\nproducer (str)
\n
\n
- \n
- \nnominalStartTime: str \n
//en.wikipedia.org/wiki/ISO_8601) timestamp representing the nominal start time\n(included) of the run. AKA the schedule time
\n- \n
- Type: \n
An [ISO-8601](https
\n \n
- \n
- \nnominalEndTime: str | None \n
//en.wikipedia.org/wiki/ISO_8601) timestamp representing the nominal end time\n(excluded) of the run. (Should be the nominal start time of the next run)
\n- \n
- Type: \n
An [ISO-8601](https
\n \n
- \n
- \nnominalstarttime_check(attribute, value) \n
- \n
- Parameters: \n
- \n
attribute (
str) \nvalue (
str) \n
\n- Return type: \n
\nNone\n
- \n
- \nnominalendtime_check(attribute, value) \n
- \n
- Parameters: \n
- \n
attribute (
str) \nvalue (
str) \n
\n- Return type: \n
\nNone\n
openlineage.client.generated.output_statistics_output_dataset module
\n- \n
- \nclass openlineage.client.generated.output_statistics_output_dataset.OutputStatisticsOutputDatasetFacet(rowCount=None, size=None, fileCount=None, *, producer='') \n
Bases:
\nOutputDatasetFacet- \n
- Parameters: \n
- \n
rowCount (int | None)
\nsize (int | None)
\nfileCount (int | None)
\nproducer (str)
\n
\n
- \n
- \nrowCount: int | None \n
The number of rows written to the dataset
\n
- \n
- \nsize: int | None \n
The size in bytes written to the dataset
\n
- \n
- \nfileCount: int | None \n
The number of files written to the dataset
\n
openlineage.client.generated.ownership_dataset module
\n- \n
- \nclass openlineage.client.generated.ownership_dataset.Owner(name, type=None) \n
Bases:
\nRedactMixin- \n
- Parameters: \n
- \n
name (str)
\ntype (str | None)
\n
\n
- \n
- \nname: str \n
the identifier of the owner of the Dataset. It is recommended to define this as a URN. For example\napplication:foo, user:jdoe, team:data
\n
- \n
- \ntype: str | None \n
The type of ownership (optional)
\n
- \n
- \nclass openlineage.client.generated.ownership_dataset.OwnershipDatasetFacet(owners=_Nothing.NOTHING, *, producer='', deleted=None) \n
Bases:
\nDatasetFacet- \n
- Parameters: \n
- \n
owners (list[Owner] | None)
\nproducer (str)
\ndeleted (bool | None)
\n
\n
- \n
- \nowners: list[Owner] | None \n
The owners of the dataset.
\n
openlineage.client.generated.ownership_job module
\n- \n
- \nclass openlineage.client.generated.ownership_job.Owner(name, type=None) \n
Bases:
\nRedactMixin- \n
- Parameters: \n
- \n
name (str)
\ntype (str | None)
\n
\n
- \n
- \nname: str \n
the identifier of the owner of the Job. It is recommended to define this as a URN. For example\napplication:foo, user:jdoe, team:data
\n
- \n
- \ntype: str | None \n
The type of ownership (optional)
\n
- \n
- \nclass openlineage.client.generated.ownership_job.OwnershipJobFacet(owners=_Nothing.NOTHING, *, producer='', deleted=None) \n
Bases:
\nJobFacet- \n
- Parameters: \n
- \n
owners (list[Owner] | None)
\nproducer (str)
\ndeleted (bool | None)
\n
\n
- \n
- \nowners: list[Owner] | None \n
The owners of the job.
\n
openlineage.client.generated.parent_run module
\n- \n
- \nclass openlineage.client.generated.parent_run.Job(namespace, name) \n
Bases:
\nRedactMixin- \n
- Parameters: \n
- \n
namespace (
str) \nname (
str) \n
\n
- \n
- \nnamespace: str \n
The namespace containing that job
\n
- \n
- \nname: str \n
The unique name for that job within that namespace
\n
- \n
- \nclass openlineage.client.generated.parent_run.ParentRunFacet(run, job, *, producer='') \n
Bases:
\nRunFacetthe id of the parent run and job, iff this run was spawn from an other run (for example, the Dag run\nscheduling its tasks)
\n- \n
- Parameters: \n
- \n
run (Run)
\njob (Job)
\nproducer (str)
\n
\n
- \n
- \nrun: Run \n
- \n
- \njob: Job \n
- \n
- \nclassmethod create(runId, namespace, name) \n
- \n
- Parameters: \n
- \n
runId (
str) \nnamespace (
str) \nname (
str) \n
\n- Return type: \n
- \n \n
- \n
- \nclass openlineage.client.generated.parent_run.Run(runId) \n
Bases:
\nRedactMixin- \n
- Parameters: \n
runId (
\nstr) \n
- \n
- \nrunId: str \n
The globally unique ID of the run associated with the job.
\n
- \n
- \nrunid_check(attribute, value) \n
- \n
- Parameters: \n
- \n
attribute (
str) \nvalue (
str) \n
\n- Return type: \n
\nNone\n
openlineage.client.generated.processing_engine_run module
\n- \n
- \nclass openlineage.client.generated.processing_engine_run.ProcessingEngineRunFacet(version, name=None, openlineageAdapterVersion=None, *, producer='') \n
Bases:
\nRunFacet- \n
- Parameters: \n
- \n
version (str)
\nname (str | None)
\nopenlineageAdapterVersion (str | None)
\nproducer (str)
\n
\n
- \n
- \nversion: str \n
Processing engine version. Might be Airflow or Spark version.
\n
- \n
- \nname: str | None \n
Processing engine name, e.g. Airflow or Spark
\n
- \n
- \nopenlineageAdapterVersion: str | None \n
OpenLineage adapter package version. Might be e.g. OpenLineage Airflow integration package version
\n
openlineage.client.generated.schema_dataset module
\n- \n
- \nclass openlineage.client.generated.schema_dataset.SchemaDatasetFacet(fields=_Nothing.NOTHING, *, producer='', deleted=None) \n
Bases:
\nDatasetFacet- \n
- Parameters: \n
- \n
fields (list[SchemaDatasetFacetFields] | None)
\nproducer (str)
\ndeleted (bool | None)
\n
\n
- \n
- \nfields: list[SchemaDatasetFacetFields] | None \n
The fields of the data source.
\n
- \n
- \nclass openlineage.client.generated.schema_dataset.SchemaDatasetFacetFields(name, type=None, description=None, fields=_Nothing.NOTHING) \n
Bases:
\nRedactMixin- \n
- Parameters: \n
- \n
name (str)
\ntype (str | None)
\ndescription (str | None)
\nfields (list[SchemaDatasetFacetFields] | None)
\n
\n
- \n
- \nname: str \n
The name of the field.
\n
- \n
- \ntype: str | None \n
The type of the field.
\n
- \n
- \ndescription: str | None \n
The description of the field.
\n
- \n
- \nfields: list[SchemaDatasetFacetFields] | None \n
Nested struct fields.
\n
openlineage.client.generated.source_code_job module
\n- \n
- \nclass openlineage.client.generated.source_code_job.SourceCodeJobFacet(language, sourceCode, *, producer='', deleted=None) \n
Bases:
\nJobFacet- \n
- Parameters: \n
- \n
language (str)
\nsourceCode (str)
\nproducer (str)
\ndeleted (bool | None)
\n
\n
- \n
- \nlanguage: str \n
Language in which source code of this job was written.
\n
- \n
- \nsourceCode: str \n
Source code of this job.
\n
openlineage.client.generated.source_code_location_job module
\n- \n
- \nclass openlineage.client.generated.source_code_location_job.SourceCodeLocationJobFacet(type, url, repoUrl=None, path=None, version=None, tag=None, branch=None, *, producer='', deleted=None) \n
Bases:
\nJobFacet- \n
- Parameters: \n
- \n
type (str)
\nurl (str)
\nrepoUrl (str | None)
\npath (str | None)
\nversion (str | None)
\ntag (str | None)
\nbranch (str | None)
\nproducer (str)
\ndeleted (bool | None)
\n
\n
- \n
- \ntype: str \n
the source control system
\n
- \n
- \nurl: str \n
the full http URL to locate the file
\n
- \n
- \nrepoUrl: str | None \n
the URL to the repository
\n
- \n
- \npath: str | None \n
the path in the repo containing the source files
\n
- \n
- \nversion: str | None \n
the current version deployed (not a branch name, the actual unique version)
\n
- \n
- \ntag: str | None \n
optional tag name
\n
- \n
- \nbranch: str | None \n
optional branch name
\n
- \n
- \nurl_check(attribute, value) \n
- \n
- Parameters: \n
- \n
attribute (
str) \nvalue (
str) \n
\n- Return type: \n
\nNone\n
openlineage.client.generated.sql_job module
\n- \n
- \nclass openlineage.client.generated.sql_job.SQLJobFacet(query, *, producer='', deleted=None) \n
Bases:
\nJobFacet- \n
- Parameters: \n
- \n
query (str)
\nproducer (str)
\ndeleted (bool | None)
\n
\n
- \n
- \nquery: str \n
openlineage.client.generated.storage_dataset module
\n- \n
- \nclass openlineage.client.generated.storage_dataset.StorageDatasetFacet(storageLayer, fileFormat=None, *, producer='', deleted=None) \n
Bases:
\nDatasetFacet- \n
- Parameters: \n
- \n
storageLayer (str)
\nfileFormat (str | None)
\nproducer (str)
\ndeleted (bool | None)
\n
\n
- \n
- \nstorageLayer: str \n
iceberg, delta.
\n- \n
- Type: \n
Storage layer provider with allowed values
\n \n
- \n
- \nfileFormat: str | None \n
parquet, orc, avro, json, csv, text, xml.
\n- \n
- Type: \n
File format with allowed values
\n \n
openlineage.client.generated.symlinks_dataset module
\n- \n
- \nclass openlineage.client.generated.symlinks_dataset.Identifier(namespace, name, type) \n
Bases:
\nRedactMixin- \n
- Parameters: \n
- \n
namespace (
str) \nname (
str) \ntype (
str) \n
\n
- \n
- \nnamespace: str \n
The dataset namespace
\n
- \n
- \nname: str \n
The dataset name
\n
- \n
- \ntype: str \n
Identifier type
\n
- \n
- \nclass openlineage.client.generated.symlinks_dataset.SymlinksDatasetFacet(identifiers=_Nothing.NOTHING, *, producer='', deleted=None) \n
Bases:
\nDatasetFacet- \n
- Parameters: \n
- \n
identifiers (list[Identifier] | None)
\nproducer (str)
\ndeleted (bool | None)
\n
\n
- \n
- \nidentifiers: list[Identifier] | None \n
openlineage.client.transport.composite module
\n- \n
- \nclass openlineage.client.transport.composite.CompositeConfig(transports, continue_on_failure=True) \n
Bases:
\nConfigCompositeConfig is a configuration class for CompositeTransport.
\n- \n
- Parameters: \n
- \n
transports (list[dict[str, Any]] | dict[str, dict[str, Any]])
\ncontinue_on_failure (bool)
\n
\n
- \n
- \ntransports \n
A list of dictionaries, where each dictionary represents the configuration\nfor a child transport. Each dictionary should contain the necessary parameters\nto initialize a specific transport instance.
\n
- \n
- \ncontinue_on_failure \n
If set to True, the CompositeTransport will attempt to emit the event using\nall configured transports, regardless of whether any previous transport\nin the list failed to emit the event. If set to False, an error in one\ntransport will halt the emission process for subsequent transports.
\n
- \n
- \ntransports: list[dict[str, Any]] | dict[str, dict[str, Any]] \n
- \n
- \ncontinue_on_failure: bool \n
- \n
- \nclassmethod from_dict(params) \n
Create a CompositeConfig object from a dictionary.
\n- \n
- Parameters: \n
params (dict[str, Any])
\n \n- Return type: \n
- \n \n
- \n
- \nclass openlineage.client.transport.composite.CompositeTransport(config) \n
Bases:
\nTransportCompositeTransport is a transport class that emits events using multiple transports.
\n- \n
- Parameters: \n
config (
\nCompositeConfig) \n
- \n
- \nkind: str | None = 'composite' \n
- \n
- \nconfig_class \n
alias of
\nCompositeConfig
- \n
- \nproperty transports: list[Transport] \n
Create and return a list of transports based on the config.
\n
- \n
- \nemit(event) \n
Emit an event using all transports in the config.
\n- \n
- Parameters: \n
event (Event)
\n \n- Return type: \n
None
\n \n
openlineage.client.transport.console module
\n- \n
- \nclass openlineage.client.transport.console.ConsoleConfig \n
Bases:
\nConfig- \n
- \n
- \nclass openlineage.client.transport.console.ConsoleTransport(config) \n
Bases:
\nTransport- \n
- Parameters: \n
config (
\nConsoleConfig) \n
- \n
- \nkind: str | None = 'console' \n
- \n
- \nconfig_class \n
alias of
\nConsoleConfig
- \n
- \nemit(event) \n
- \n
- Parameters: \n
event (
\nUnion[RunEvent,DatasetEvent,JobEvent,RunEvent,DatasetEvent,JobEvent]) \n- Return type: \n
\nNone\n
openlineage.client.transport.factory module
\n- \n
- \nclass openlineage.client.transport.factory.DefaultTransportFactory \n
Bases:
\nTransportFactory- \n
- \n
- \nregister_transport(of_type, clazz) \n
- \n
- Parameters: \n
- \n
of_type (str)
\nclazz (type[Transport] | str)
\n
\n- Return type: \n
None
\n \n
- \n
- \ncreate(config=None) \n
Initializes and returns a transport mechanism based on the provided configuration.
\nIf \u2018OPENLINEAGE_DISABLED\u2019 is set to \u2018true\u2019, a NoopTransport instance is returned,\neffectively disabling transport.\nIf a configuration dictionary is provided, transport specified by the config is initialized.\nIf no configuration is provided, the function defaults to a console-based transport, logging\na warning and printing events to the console.
\n- \n
- Parameters: \n
config (dict[str, str] | None)
\n \n- Return type: \n
- \n \n
openlineage.client.transport.file module
\n- \n
- \nclass openlineage.client.transport.file.FileConfig(log_file_path, append=False) \n
Bases:
\nConfig- \n
- Parameters: \n
- \n
log_file_path (
str) \nappend (
bool) \n
\n
- \n
- \nlog_file_path:
str\n
- \n
- \nappend:
bool= False \n
- \n
- \nclassmethod from_dict(params) \n
- \n
- Parameters: \n
params (dict[str, Any])
\n \n- Return type: \n
- \n \n
- \n
- \nclass openlineage.client.transport.file.FileTransport(config) \n
Bases:
\nTransport- \n
- Parameters: \n
config (
\nFileConfig) \n
- \n
- \nkind: str | None = 'file' \n
- \n
- \nconfig_class \n
alias of
\nFileConfig
- \n
- \nemit(event) \n
- \n
- Parameters: \n
event (
\nUnion[RunEvent,DatasetEvent,JobEvent,RunEvent,DatasetEvent,JobEvent]) \n- Return type: \n
\nNone\n
openlineage.client.transport.http module
\n- \n
- \nclass openlineage.client.transport.http.TokenProvider(config) \n
Bases:
\nobject- \n
- Parameters: \n
config (dict[str, str])
\n \n
- \n
- \nget_bearer() \n
- \n
- Return type: \n
str | None
\n \n
- \n
- \nclass openlineage.client.transport.http.HttpCompression(value) \n
Bases:
\nEnumAn enumeration.
\n- \n
- \nGZIP = 'gzip' \n
- \n
- \nclass openlineage.client.transport.http.ApiKeyTokenProvider(config) \n
Bases:
\nTokenProvider- \n
- Parameters: \n
config (dict[str, str])
\n \n
- \n
- \nget_bearer() \n
- \n
- Return type: \n
str | None
\n \n
- \n
- \nopenlineage.client.transport.http.create_token_provider(auth) \n
- \n
- Parameters: \n
auth (dict[str, str])
\n \n- Return type: \n
- \n \n
- \n
- \nopenlineage.client.transport.http.get_session() \n
- \n
- Return type: \n
\nSession\n
- \n
- \nclass openlineage.client.transport.http.HttpConfig(url, endpoint='api/v1/lineage', timeout=5.0, verify=True, auth=_Nothing.NOTHING, compression=None, session=None, adapter=None, custom_headers=_Nothing.NOTHING) \n
Bases:
\nConfig- \n
- Parameters: \n
- \n
url (str)
\nendpoint (str)
\ntimeout (float)
\nverify (bool)
\nauth (TokenProvider)
\ncompression (HttpCompression | None)
\nsession (Session | None)
\nadapter (HTTPAdapter | None)
\ncustom_headers (dict[str, str])
\n
\n
- \n
- \nurl: str \n
- \n
- \nendpoint: str \n
- \n
- \ntimeout: float \n
- \n
- \nverify: bool \n
- \n
- \nauth: TokenProvider \n
- \n
- \ncompression: HttpCompression | None \n
- \n
- \nsession: Session | None \n
- \n
- \nadapter: HTTPAdapter | None \n
- \n
- \ncustom_headers: dict[str, str] \n
- \n
- \nclassmethod from_dict(params) \n
- \n
- Parameters: \n
params (dict[str, Any])
\n \n- Return type: \n
- \n \n
- \n
- \nclassmethod from_options(url, options, session) \n
- \n
- Parameters: \n
- \n
url (str)
\noptions (OpenLineageClientOptions)
\nsession (Session | None)
\n
\n- Return type: \n
HttpConfig
\n \n
- \n
- \nclass openlineage.client.transport.http.HttpTransport(config) \n
Bases:
\nTransport- \n
- Parameters: \n
config (
\nHttpConfig) \n
- \n
- \nkind: str | None = 'http' \n
- \n
- \nconfig_class \n
alias of
\nHttpConfig
- \n
- \nset_adapter(adapter) \n
- \n
- Parameters: \n
adapter (
\nHTTPAdapter) \n- Return type: \n
\nNone\n
- \n
- \nemit(event) \n
- \n
- Parameters: \n
event (
\nUnion[RunEvent,DatasetEvent,JobEvent,RunEvent,DatasetEvent,JobEvent]) \n- Return type: \n
\nResponse\n
openlineage.client.transport.kafka module
\n- \n
- \nclass openlineage.client.transport.kafka.KafkaConfig(config, topic, messageKey=None, flush=True) \n
Bases:
\nConfig- \n
- Parameters: \n
- \n
config (dict[str, str])
\ntopic (str)
\nmessageKey (str | None)
\nflush (bool)
\n
\n
- \n
- \nconfig: dict[str, str] \n
- \n
- \ntopic: str \n
- \n
- \nmessageKey: str | None \n
- \n
- \nflush: bool \n
- \n
- \nclassmethod from_dict(params) \n
- \n
- Parameters: \n
params (dict[str, Any])
\n \n- Return type: \n
_T
\n \n
- \n
- \nopenlineage.client.transport.kafka.on_delivery(err, msg) \n
- \n
- Parameters: \n
- \n
err (KafkaError)
\nmsg (Message)
\n
\n- Return type: \n
None
\n \n
- \n
- \nclass openlineage.client.transport.kafka.KafkaTransport(config) \n
Bases:
\nTransport- \n
- Parameters: \n
config (
\nKafkaConfig) \n
- \n
- \nkind: str | None = 'kafka' \n
- \n
- \nconfig_class \n
alias of
\nKafkaConfig
- \n
- \nemit(event) \n
- \n
- Parameters: \n
event (Event)
\n \n- Return type: \n
None
\n \n
openlineage.client.transport.msk_iam module
\n- \n
- \nclass openlineage.client.transport.msk_iam.MSKIAMConfig(config, topic, messageKey=None, flush=True, region=None, aws_profile=None, role_arn=None, aws_debug_creds=False) \n
Bases:
\nKafkaConfig- \n
- Parameters: \n
- \n
config (dict[str, str])
\ntopic (str)
\nmessageKey (str | None)
\nflush (bool)
\nregion (str)
\naws_profile (None | str)
\nrole_arn (None | str)
\naws_debug_creds (bool)
\n
\n
- \n
- \nregion: str \n
- \n
- \naws_profile: None | str \n
- \n
- \nrole_arn: None | str \n
- \n
- \naws_debug_creds: bool \n
- \n
- \nclass openlineage.client.transport.msk_iam.MSKIAMTransport(config) \n
Bases:
\nKafkaTransport- \n
- Parameters: \n
config (
\nMSKIAMConfig) \n
- \n
- \nkind: str | None = 'msk-iam' \n
- \n
- \nconfig_class \n
alias of
\nMSKIAMConfig
openlineage.client.transport.noop module
\n- \n
- \nclass openlineage.client.transport.noop.NoopConfig \n
Bases:
\nConfig- \n
- \n
- \nclass openlineage.client.transport.noop.NoopTransport(config) \n
Bases:
\nTransport- \n
- Parameters: \n
config (
\nNoopConfig) \n
- \n
- \nkind: str | None = 'noop' \n
- \n
- \nconfig_class \n
alias of
\nNoopConfig
- \n
- \nemit(event) \n
- \n
- Parameters: \n
event (
\nUnion[RunEvent,DatasetEvent,JobEvent,RunEvent,DatasetEvent,JobEvent]) \n- Return type: \n
\nNone\n
openlineage.client.transport.transport module
\nTo implement custom Transport implement Config and Transport classes.
\n- \n
- Transport needs to
- \n
specify class variable config that will point to Config class that Transport requires
\n__init__ that will accept specified Config class instance
\nimplement emit method that will accept RunEvent
\n
\n
Config file is read and parameters there are passed to from_dict classmethod.\nThe config class can have more complex attributes, but needs to be able to\ninstantiate them in from_dict method.
\nDefaultTransportFactory instantiates custom transports by looking at type field in\nclass config.
\n- \n
- \nclass openlineage.client.transport.transport.Config \n
Bases:
\nobject- \n
- \n
- \nclassmethod from_dict(params) \n
- \n
- Parameters: \n
params (dict[str, Any])
\n \n- Return type: \n
_T
\n \n
- \n
- \nclass openlineage.client.transport.transport.Transport \n
Bases:
\nobject- \n
- \nkind: str | None = None \n
- \n
- \nname: str | None = None \n
- \n
- \nconfig_class \n
alias of
\nConfig
- \n
- \nemit(event) \n
- \n
- Parameters: \n
event (Event)
\n \n- Return type: \n
Any
\n \n
- \n
- \nclass openlineage.client.transport.transport.TransportFactory \n
Bases:
\nobject- \n
- \ncreate(config=None) \n
- \n
- Parameters: \n
config (dict[str, str] | None)
\n \n- Return type: \n
- \n \n