From 649a60d51ee2abcb395c53f7984ac58fc0ffa6ae Mon Sep 17 00:00:00 2001 From: merobi-hub Date: Tue, 27 Feb 2024 11:12:32 -0500 Subject: [PATCH 1/2] Feb 24 update. Signed-off-by: merobi-hub --- channel/boston-meetup/index.html | 112 +- channel/dagster-integration/index.html | 26 +- channel/data-council-meetup/index.html | 50 + channel/dev-discuss/index.html | 9412 +++- channel/general/index.html | 11876 ++++- channel/github-discussions/index.html | 5997 +++ channel/github-notifications/index.html | 39823 ++++++++++++++++ channel/gx-integration/index.html | 34 +- channel/jobs/index.html | 248 +- channel/london-meetup/index.html | 52 +- channel/mark-grover/index.html | 24 + channel/nyc-meetup/index.html | 52 +- channel/open-lineage-plus-bacalhau/index.html | 24 + channel/providence-meetup/index.html | 32 +- channel/sf-meetup/index.html | 24 + .../index.html | 24482 ++++++++++ channel/spec-compliance/index.html | 28 +- channel/toronto-meetup/index.html | 28 +- channel/user-generated-metadata/index.html | 713 + index.html | 11876 ++++- 20 files changed, 103788 insertions(+), 1125 deletions(-) create mode 100644 channel/github-discussions/index.html create mode 100644 channel/github-notifications/index.html create mode 100644 channel/spark-support-multiple-scala-versions/index.html create mode 100644 channel/user-generated-metadata/index.html diff --git a/channel/boston-meetup/index.html b/channel/boston-meetup/index.html index 67d3cf1..9bc8e7f 100644 --- a/channel/boston-meetup/index.html +++ b/channel/boston-meetup/index.html @@ -46,6 +46,18 @@

Public Channels

+
  • + + # github-discussions + +
    • + +
  • + + # github-notifications + +
    • +
  • # gx-integration @@ -94,6 +106,12 @@

    Public Channels
    • +
  • + + # spark-support-multiple-scala-versions + +
    • +
  • # spec-compliance @@ -106,6 +124,12 @@

    Public Channels
    • +
  • + + # user-generated-metadata + +
    • +

    Private Channels

    Private Channels

    Private Channels

    Private Channels

    Private Channels

    Private Channels

    Private Channels

      @@ -128,7 +152,7 @@

      Group Direct Messages

      - +
      @@ -154,7 +178,7 @@

      Group Direct Messages

      - +
      @@ -474,6 +498,226 @@

      Group Direct Messages

      + + +
      +
      + + + + +
      + +
      Ewan Lord + (ewanlord@gmail.com) +
      +
      2024-01-31 05:28:42
       +
      +

      @Ewan Lord has joined the channel

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Michael Robinson + (michael.robinson@astronomer.io) +
      +
      2024-02-08 09:52:58
       +
      +

      There are two openings at Collibra: https://openlineage.slack.com/archives/C01CK9T7HKR/p1707402341155039

      +
      + + +
      + + + } + + Sheeri Cabral + (https://openlineage.slack.com/team/U0323HG8C8H) +
      + + + +
      +
      + <p>HIRING! Hello OpenLineage community members! Collibra is hiring for 2 Software Engineers (one regular, one senior) in Czech Republic, for our Lineage ETL team. These jobs are open because one engineer is leaving to hike the Pacific Coast Trail, and another is backfill (an engineer left a different lineage team and an engineer from ETL team is moving to that team).</p> + +<p>I am the Product Manager for this team, so you’d be working with me daily. Ask me anything (in a thread here or a DM). Creating an OpenLineage consumer is our next project starting later this month :D</p> + +<p><a href="https://www.linkedin.com/feed/update/urn:li:activity:7161361969311084544/">https://www.linkedin.com/feed/update/urn:li:activity:7161361969311084544/</a></p> +
      + + + + + + +
      Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1707402341155039
      + + +
      + + Thread in Slack Conversation +
      + + +
      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
      +
      2024-02-08 10:00:37
       +
      +

      @Sheeri Cabral (Collibra) has joined the channel

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Sheeri Cabral (Collibra) + (sheeri.cabral@collibra.com) +
      +
      2024-02-08 10:02:10
       +
      +

      (this is in addition to 2 positions in our Data Office - using Airflow among other tools. See https://www.collibra.com/us/en/company/careers#find-jobs for all the listings. They’re all in Czech Republic for now, unfortunately. We have to look for 6 months in Czech Republic before we are allowed to open it up to other places)

      +
      +
      collibra.com
      + + + +
      Opportunities Through Data
      +
      + Searching for a career in data? Change how organizations use data so our customers can change the world. Explore Collibra opportunities. +
      + + + + + + + + +
      Original URL: https://www.collibra.com/us/en/company/careers#find-jobs
      + + + +
      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Suraj Gupta + (suraj.gupta@atlan.com) +
      +
      2024-02-08 11:08:50
       +
      +

      @Suraj Gupta has joined the channel

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Matthew Paras + (matthewparas2020@u.northwestern.edu) +
      +
      2024-02-08 15:28:33
       +
      +

      @Matthew Paras has joined the channel

      + + + +
      +
      +
      +
      + + +
      diff --git a/channel/london-meetup/index.html b/channel/london-meetup/index.html index 9734290..c233fce 100644 --- a/channel/london-meetup/index.html +++ b/channel/london-meetup/index.html @@ -46,6 +46,18 @@

      Public Channels

      +
    • + + # github-discussions + +
      • + +
    • + + # github-notifications + +
      • +
    • # gx-integration @@ -94,6 +106,12 @@

      Public Channels
      • +
    • + + # spark-support-multiple-scala-versions + +
      • +
    • # spec-compliance @@ -106,6 +124,12 @@

      Public Channels
      • +
    • + + # user-generated-metadata + +
      • +

    Private Channels

    Private Channels

    Private Channels

    Private Channels

    Private Channels

    Private Channels

      diff --git a/channel/spark-support-multiple-scala-versions/index.html b/channel/spark-support-multiple-scala-versions/index.html new file mode 100644 index 0000000..3329ce6 --- /dev/null +++ b/channel/spark-support-multiple-scala-versions/index.html @@ -0,0 +1,24482 @@ + + + + + + Slack Export - #spark-support-multiple-scala-versions + + + + + +
      + + + +
      + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2023-12-11 11:30:58
       +
      +

      @Damien Hawes has joined the channel

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2023-12-11 11:31:09
       +
      +

      @Paweł Leszczyński has joined the channel

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Mattia Bertorello + (mattia.bertorello@booking.com) +
      +
      2023-12-11 11:31:09
       +
      +

      @Mattia Bertorello has joined the channel

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2023-12-11 11:32:59
       +
      +

      @Paweł Leszczyński - meet @Mattia Bertorello - one of my team members. He will be helping us to support Scala 2.13 with the Spark integration. For clarity, I am going to share the messages we exchanged earlier today in order to get him up to speed.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2023-12-11 11:36:15
       +
      +

      > Damien Hawes: Yeah - the cross-building of the spark integration isn't fun. I'm doing a toy project at the moment, to understand how to do it with Gradle. So far I can get a regular project to publish to my local maven repository with Scala 2.12 and Scala 2.13 +> Paweł Leszczyński yeah, i think we should have extra scalaVersion param as we have sparkVersion and publish two artifacts with scala suffix at the end +> Damien Hawes: Yup. The standard Scala suffixes. Though, I'm trying to wrap my head around how we currently get the entire integration to "come together" so to speak. I do have a concern though: if we publish two artifacts, what happens when someone runs a Spark 3.1 pipeline for example which only supports Scala 2.12 but we have Scala 2.13 code in there as well because of the way we "combine" all the sub-projects into 1 artifact +> Paweł Leszczyński: in circle ci there is a matrix of params and we should be able somehow to prevent publishing spark2 + scala2.13 thats' the version matrix file https://github.com/OpenLineage/OpenLineage/blob/main/.circleci/workflows/openlineage-spark.yml +> Damien Hawes So far I have this in my settings.gradle.kts of my toy project: +```rootProject.name = "openlineage-spark"

      + +

      val scalaBinaryVersion = settings.extra["scalaVersion"] + .toString() + .split(".") + .take(2) + .joinToString(".")

      + +

      include("openlineage-spark-core")

      + +

      if (scalaBinaryVersion == "2.12") { + include("openlineage-spark-spark30") + include("openlineage-spark-spark31") + include("openlineage-spark-spark32") + include("openlineage-spark-spark33") + include("openlineage-spark-spark34") + include("openlineage-spark-spark35") +}

      + +

      if (scalaBinaryVersion == "2.13") { + include("openlineage-spark-spark32") + include("openlineage-spark-spark32") + include("openlineage-spark-spark33") + include("openlineage-spark-spark34") + include("openlineage-spark-spark35") +} +&gt; Damien Hawes: Then in my gradle.properties I have this: +# This is used as the default Scala version

      + +

      Suggested versions of Scala to pick from are: 2.11.12, 2.12.18, 2.13.12

      + +

      scalaVersion=2.12.18

      + +

      This is used as the default Spark version

      + +

      sparkVersion=3.2.1```

      +
      + + + + +
      <https://github.com/OpenLineage/OpenLineage/blob/main/.circleci/workflows/openlineage-spark.yml | openlineage-spark.yml>
      +
      + <pre><code>workflows: + openlineage-spark: + jobs: + - build-integration-spark: + matrix: + parameters: + spark-version: [ '2.4.6', '3.2.4', '3.3.3', '3.4.1', '3.5.0' ] + requires: + - build-integration-sql-java + - build-client-java + - integration-test-databricks-integration-spark: + context: integration-tests + matrix: + parameters: + spark-version: [ '3.4.1' ] + requires: + - build-integration-spark + - integration-test-integration-spark: + context: integration-tests + matrix: + parameters: + spark-version: [ '2.4.6', '3.2.4', '3.3.3', '3.4.1', '3.5.0' ] + requires: + - build-integration-spark + - workflow_complete: + requires: + - integration-test-integration-spark +</code></pre> +
      + + + + + + + +
      + + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
      + + +
      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2023-12-12 02:52:22
       +
      +

      @Maciej Obuchowski has joined the channel

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2023-12-12 02:52:39
       +
      +

      👋

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2023-12-12 08:47:23
       +
      +

      Hello. I've been trying to balance of developer experience versus publishing experience versus user experience.

      + +

      There are several approaches we could take.

      + +

      Approach #1

      + +

      Produce and publish an artefact per version of Spark x Scala. Given the table below, this would mean Spark 3.2.x, 3.3.x, 3.4.x, and 3.5.x would see two artefacts produced, something like io.openlineage:openlineage_spark_3.2_2.12:$VERSION & io.openlineage:openlineage_spark_3.2_2.13:$VERSION

      + +
      | Spark Version | Scala 2.10 | Scala 2.11 | Scala 2.12 | Scala 2.13 |
++---------------+------------+------------+------------+------------+
+| 2.2.x         |     ✓      |     ✓      |            |            |
++---------------+------------+------------+------------+------------+
+| 2.3.x         |            |     ✓      |            |            |
++---------------+------------+------------+------------+------------+
+| 2.4.x         |            |     ✓      |     ✓      |            |
++---------------+------------+------------+------------+------------+
+| 3.0.x         |            |            |     ✓      |            |
++---------------+------------+------------+------------+------------+
+| 3.1.x         |            |            |     ✓      |            |
++---------------+------------+------------+------------+------------+
+| 3.2.x         |            |            |     ✓      |     ✓      |
++---------------+------------+------------+------------+------------+
+| 3.3.x         |            |            |     ✓      |     ✓      |
++---------------+------------+------------+------------+------------+
+| 3.4.x         |            |            |     ✓      |     ✓      |
++---------------+------------+------------+------------+------------+
+| 3.5.x         |            |            |     ✓      |     ✓      |
++---------------+------------+------------+------------+------------
+
      + +

      Benefits:

      + +
      1. Very clear boundaries of the Spark and Scala version the artefact supports.
      2. We probably will not run into Scala binary incompatibility issues.

      3. We don't include code from other versions of Spark into the published JAR. +Drawbacks:

      4. The developer experience becomes a bit more difficult, as it becomes necessary to edit the root gradle.properties file to select the correct versions.

      5. We need to add integration tests.
      6. We need to conditionally compile things.
      7. There are more artefacts to publish.
      8. We have to educate users about this, that they need to carefully select which one to go for. +Approach #2
        9. +
      + +

      Produce an artefact per Scala major version. This means, something like: Scala 2.12 would include support for Spark 2.4, Spark 3.0, Spark 3.1, Spark 3.2, Spark 3.3, Spark 3.4, and Spark 3.5.

      + +

      Benefits:

      + +
      1. Two artefacts are published

      2. The developer experienced is kept roughly the same, activating Scala 2.13 would deactivate 3.1 and older from being considered in the Gradle build. +Drawbacks:

      3. We may end up including Scala 2.12 code with Scala 2.13 artefacts and vice versa.

      4. We need to add integration tests.
      5. We need to conditionally compile things.
      6. More artefacts to publish. +

        5. We have to educate users about this, that they need to carefully select which one to go for.

        7. +
      + +

      Now there is another elephant in the room, switching to Scala 2.13 is not an easy feat. According to @Mattia Bertorello it breaks some code, where exactly, I'm not so sure. Perhaps he can shed light on this when he has the time.

      + +

      What do you folks think? @Paweł Leszczyński @Maciej Obuchowski

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2023-12-12 08:54:16
       +
      +

      *Thread Reply:* I would definitely gravitate for option 2, given that we already have infrastructure build to publish one artifact for all Spark versions we support for a single Scala version.

      + +

      Could you elaborate on this, as I'm not sure why would it happen +> We may end up including Scala 2.12 code with Scala 2.13 artefacts and vice versa. +Also: +> More artefacts to publish. +Seems like less, only openlineage-spark_2.12 and openlineage_spark_2.13 - and possibly later versions of Scala, rather than full Spark x Scala matrix 🙂

      + +

      > According to @Mattia Bertorello it breaks some code, where exactly, I'm not so sure. Perhaps he can shed light on this when he has the time. +Agreed, this is a challenge. My first idea would be to have just two different classes conditionally compiled, similar to what we're doing now with particular Spark versions - although the problem might be that we'll have way too many internal subprojects... not an easy challenge and probably requires research - if it's just few classes then probably okay, if we need to have two versions of pretty much every class it's a no go.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2023-12-12 09:17:22
       +
      +

      *Thread Reply:* Yeah, I know the Spline folks did some magic with runtime code generation. But I think the amount of code they generated was small, and used for json serialization purposes.

      + +

      > Could you elaborate on this, as I'm not sure why would it happen +>> We may end up including Scala 2.12 code with Scala 2.13 artefacts and vice versa. +> +When I first starting playing around with cross-building my toy project, I noticed that Gradle was building all the projects, i.e., there was a JAR being generated for each when I ran ./gradlew clean build even if I ran ./gradlew clean build -PscalaVersion=2.12.

      + +

      However, I have made some changes to the settings.gradle.kts file (the one you see earlier in the channel), and it seems to be working as expected, i.e., running the same command as specified above, deactivates the unused projects. Though, IntelliJ hates this, and red squiggly lines are displayed on those projects.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2023-12-12 09:24:01
       +
      +

      *Thread Reply:* Oh right, now I remember the other concern I had.

      + +

      core is compiled against a specific version of Spark, for the sake of development. The concern I have, is if that interface (or the objects it relies on) changes. So far, we haven't seen it yet - thankfully.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Mattia Bertorello + (mattia.bertorello@booking.com) +
      +
      2023-12-12 10:20:50
       +
      +

      *Thread Reply:* Hi @Maciej Obuchowski +I did some work to make it compile with the Scala 2.13 version and see how many changes are necessary. Until now, this version still has some issues and is not compiling.

      + +

      Then, I considered using this library scala-collection-compat to allow the cross-compilation, but I still need to test it. +Most of the changes done in scala 2.13 that impact OpenLineage are: +• JavaConverters were removed +• Use the immutable data type from the collection library +• Some types of issues with the Seq Builder +My next steps are to try to compile these changes in 2.12 and fix the issues that are still there in the 2.13

      + +

      Here is my code, and it is still in the WIP +https://github.com/OpenLineage/OpenLineage/compare/main...mattiabertorello:OpenLineage:feature/support_scala_2_13?expand=1

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2023-12-12 10:25:14
       +
      +

      *Thread Reply:* > core is compiled against a specific version of Spark, for the sake of development. The concern I have, is if that interface (or the objects it relies on) changes. So far, we haven't seen it yet - thankfully. +With every new Spark version we have to do some things anyway to make everything work, as we basically see by having all those directories with version-specific code 🙂 +We'd see that in integration test code - and I think Spark now is vary to remove core stuff everyone is using.

      + +

      @Mattia Bertorello nice work 🙂 +Would it be able to put this behind some wrapper library, to be code-compatible with 2.12 code?

      + +

      We might still need to release versions compiled for specific version, but sounds like those changes would not require source code changes if we can isolate those.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Mattia Bertorello + (mattia.bertorello@booking.com) +
      +
      2023-12-12 10:32:30
       +
      +

      *Thread Reply:* > Would it be able to put this behind some wrapper library, to be code-compatible with 2.12 code? +Maybe use the already exist class ScalaConversionUtils everywhere to make it easy to change. I will also check for the other changes if a wrapper class is necessary

      + + + +
      + 🙌 Paweł Leszczyński, Maciej Obuchowski +
      + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2023-12-12 11:03:11
       +
      +

      *Thread Reply:* Yeah. would be perfect if we could have ScalaConversionUtils that calls either ScalaConversionUtils_2_12 or ScalaConversionUtils_2_13 methods and stands as the only place in the whole codebase to be aware of which scala version is used.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2023-12-12 11:05:31
       +
      +

      *Thread Reply:* perhaps we could add empty Seq to ScalaConversionUtils as well

      + + + +
      + 👍 Mattia Bertorello +
      + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2023-12-12 10:46:28
       +
      +

      @Maciej Obuchowski - how much hate would I get, if I switched from Groovy DSL to Kotlin DSL?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2023-12-12 11:08:52
       +
      +

      *Thread Reply:* you would only get love 😄

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2023-12-12 11:09:45
       +
      +

      *Thread Reply:* I definitely like Kotlin one more as well, but Groovy was from the beginning and never felt the urge to make the switch myself 🙂

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2023-12-12 11:20:41
       +
      +

      *Thread Reply:* The only reason why I'm feeling the urge to change it, is because I did all this exploratory work in Kotlin, and I am loathe to change it to Groovy.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2023-12-12 11:30:10
       +
      +

      *Thread Reply:* Go for it - one thing I'd ask is to make change in a separate PR

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2023-12-12 11:30:20
       +
      +

      *Thread Reply:* Aye

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2023-12-12 11:35:13
       +
      +

      *Thread Reply:* The plan I have is 4-phased approach.

      + +
      1. Add Scala 2.13 build support (1st PR), but still have the project publishing with Scala 2.12.
      2. Have @Mattia Bertorello use the above to build the project locally against Scala 2.12 and Scala 2.13 (2nd PR)
      3. Modify the project to build and publish Scala 2.12 and Scala 2.13 artefacts (3rd PR)
      4. (Optionally) Migrate to Kotlin DSL +I don't want to convert my stuff to Groovy, but honestly, it'll be the smallest amount of changes, to what promises to be a fairly big PR (#1). Hence, I'll leave the Kotlin DSL migration to last.
        5. +
      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2023-12-12 11:46:08
       +
      +

      *Thread Reply:* might be worthy to do Kotlin first then?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2023-12-12 11:46:29
       +
      +

      *Thread Reply:* I mean, convert all the existing Groovy stuff and then just add on top of that

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2023-12-12 11:46:41
       +
      +

      *Thread Reply:* But if you think it's easier to do it this way, then sure go for it

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2023-12-12 11:47:33
       +
      +

      *Thread Reply:* Yeah - though I'm thinking in terms of not conflicting with Mattia too much, as he will have a ton of merge conflicts to resolve.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2023-12-12 11:47:45
       +
      +

      *Thread Reply:* Though, maybe it is best to get the pain out of the way sooner.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2023-12-12 11:07:11
       +
      +

      btw guys, I don't know if you read my letter to santa clause, but more collaboration on spark code was definitely on the list 😉 thank you for coming and I am super happy to see you working on this 🎅

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2023-12-15 10:40:39
       +
      +

      An update @Maciej Obuchowski @Paweł Leszczyński - the effort to support Scala 2.13 isn't small. Holy hell.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2023-12-15 10:41:32
       +
      +

      *Thread Reply:* The challenging bit is the coupling of the spark32, spark33, spark34 , and spark35 modules to the shared and spark3 modules.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2023-12-15 10:42:56
       +
      +

      *Thread Reply:* spark3 cannot be compiled with Scala 2.13, because, well, no spark dependencies were ever released for it with Scala 2.13

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2023-12-18 01:59:48
       +
      +

      *Thread Reply:* would it help if we upgraded sparkVersion in spark3 to 3.2 for Scala 2.13? 3.2 should be able to support 2.13

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Mattia Bertorello + (mattia.bertorello@booking.com) +
      +
      2023-12-21 05:54:03
       +
      +

      *Thread Reply:* Hi Pawel,

      + +

      I upgraded spark3 project to 3.2 version and also the shared project. Now they can support scala 2.12 and 2.13

      + +

      If you want to have a look at the changes, there are a lot https://github.com/OpenLineage/OpenLineage/compare/main...mattiabertorello:feature/support_scala_2_13?expand=1

      + +

      A summary of what I have done is: +• Use the ScalaConversionUtils everywhere to hide the conversion that changes between Scala 2.12 and 2.13. And add some methods to avoid the use of inner classes like Seq$ and Map$ outside the class. +• Upgrade to Spark 3.2 the spark3 project by temporarily copying the spark32 . On this part I would like your suggestion on how to handle it. +• Upgrade shared project spark version from 2.4.8 to 3.2.2, same of spark3 and spark32 +• Upgrade net.snowflake:spark-snowflake from 2.9.3 to 2.11.0 to support Scala 2.13 +• Change from com.databricks:dbutils-api to com.databricks:databricks-dbutils-scala to support Scala 2.13 +• Upgrade com.google.cloud.spark:spark-bigquery-with-dependencies from 0.26.0 to 0.29.0 because the new version supports Scala 2.13 +• In Gradle files support an easy change between Scala versions with a property file. +• In the Gradle app project and in every project, support a dynamic Scala version through a variable. +Now, I need some help understanding how to move on to make the contribution and know if this error exists before. +Scala module 2.12.3 requires Jackson Databind version >= 2.12.0 and < 2.13.0 +Is it normal or not? When I run the tests, that also happens when I run gradle build on main.

      + +

      Then now, I need to understand what are the next steps: +• Which tests do I still need to do to say that Scala 2.13 is working? If I can run the CI, it would be helpful. +• How can I start opening PRs to avoid merging everything in a single PR? What about start opening a PR to use the ScalaConversionUtils everywhere? +• Then, the difficult parts: continue working on the Damien side to make Gradle generate multiple jars, each one with each Scala version. In general I still need clarification on how to do it. I'm happy to hear some suggestions 🙂 +Thanks!

      + + + +
      + 👀 Paweł Leszczyński +
      + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2023-12-21 08:20:27
       +
      +

      *Thread Reply:* Hi @Mattia Bertorello,

      + +

      Great news. I've clicked "draft PR" #2329 just to see if ./gradlew clean build works fine for different Spark versions. We can close this PR later. Some of the integration tests require extra credentials (like access to big query), so they need to be triggered by us. Just to mention, so that you don't worry once they fail. CI starts automatically on new commit into the PR.

      + +

      I really like your approach and I think we can already go with the separate PRs and steps: (1) use ScalaConversionUtils (2) make scala version parameterized and stored in gradle.properties although still equal to 2.12

      + +

      Scala module 2.12.3 requires Jackson Databind version &gt;= 2.12.0 and &lt; 2.13.0 -> not sure of this. we do use jackson and databind in openlineage-java + and shadow them within the package. However, we're using 2.15.3 which is quite recent. That's the recent build on main in CI -> https://app.circleci.com/pipelines/github/OpenLineage/OpenLineage/8702/workflows/f1aed08c-8333-4460-a08b-1ad746cb046b can you find the same log there?

      + +

      Can we keep building shared conditionally and use spark 2.4.8 for scala 2.12 while 3.2 for scala 2.13. that's an extra complexity which, on the other hand, assures the content of the submodule can be really shared among different spark versions. although a single jar is published at the end, this makes sure our code compiles with different versions. i think the same, as with shared , can be done with spark3.

      + +

      The scope of this work is quite big, so I would be happy to work iteratively on the PRs and merge them even though still working on final solution.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2023-12-21 12:55:30
       +
      +

      *Thread Reply:* > Upgrade shared project spark version from 2.4.8 to 3.2.2, same of spark3 and spark32 +This effectively means moving minimum supported Spark version to 3.2.2, right?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Mattia Bertorello + (mattia.bertorello@booking.com) +
      +
      2023-12-22 11:48:19
       +
      +

      *Thread Reply:* I still need to check if the 2.4.8 version still works since I didn't touch it. but I ran some test to avoid changing the version in the shared project and I get this error. +```io.openlineage.spark.agent.lifecycle.LibraryTest

      + +

      Test testRdd(Path, SparkSession) FAILED (2.3s)

      + +

      java.lang.NoSuchMethodError: org.apache.spark.rdd.RDD.dependencies()Lscala/collection/Seq; + at io.openlineage.spark.agent.lifecycle.Rdds.flattenRDDs(Rdds.java:34)`` +Only when I ran the test with the scala2.13after update to3.2+all the spark3 and 3x projects otherwise the same code run with no issues with scala2.12` .

      + +

      I still need to understand why is happing. I know that It's not great ditching the support for the 2.4 version

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Mattia Bertorello + (mattia.bertorello@booking.com) +
      +
      2024-01-09 10:47:54
       +
      +

      *Thread Reply:* Hi team!

      + +

      Could you have a look to these PRs? +https://github.com/OpenLineage/OpenLineage/pull/2357/ +https://github.com/OpenLineage/OpenLineage/pull/2359 +https://github.com/OpenLineage/OpenLineage/pull/2360

      + +

      They are part of the effort to support Scala 2.13; for the moment, none of them changes any logic, just some code refactoring and a library update. +We're still working on the Gradle side, which has some challenges.

      + +

      Thanks!

      + + + +
      + 👀 Maciej Obuchowski +
      + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Vladimir Fedotov + (vladimir.fedotov@booking.com) +
      +
      2023-12-18 04:45:08
       +
      +

      @Vladimir Fedotov has joined the channel

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Harel Shein + (harel.shein@gmail.com) +
      +
      2023-12-19 18:54:20
       +
      +

      @Harel Shein has joined the channel

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Teemu Mattelmäki + (teemu.mattelmaki@teradata.com) +
      +
      2024-01-05 05:34:59
       +
      +

      @Teemu Mattelmäki has joined the channel

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Teemu Mattelmäki + (teemu.mattelmaki@teradata.com) +
      +
      2024-01-05 05:34:59
       +
      +

      Hello! I tried to find if OL Spark column level lineage is dependent on Databricks implementation. According to this link it feels that the answer is is YES. https://openlineage.io/docs/integrations/spark/quickstart_databricks Could you confirm that dependency?

      +
      +
      openlineage.io
      + + + +
      Quickstart with Databricks | OpenLineage
      +
      + OpenLineage's Spark Integration can be installed on Databricks leveraging init scripts. Please note, Databricks on Google Cloud does not currently support the DBFS CLI, so the proposed solution will not work on Google Cloud until that feature is enabled. +
      + + + + + + +
      Original URL: https://openlineage.io/docs/integrations/spark/quickstart_databricks
      + + + +
      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-01-05 08:00:03
       +
      +

      *Thread Reply:* What do you mean by dependent on Databricks implementation?

      + +

      Databricks provides their own, closed source versions of some LogicalPlan for which visitors generally need to be implemented by reflection or by checking that we have the necessary classes, like here - but that does not mean integration for OSS Spark depends on some Databricks dependency

      +
      + + + + +
      <https://github.com/OpenLineage/OpenLineage/blob/31f8ce588526e9c7c4bc7d849699cb7ce2969c8f/integration/spark/spark3/src/main/java/io/openlineage/spark3/agent/utils/DatasetVersionDatasetFacetUtils.java | DatasetVersionDatasetFacetUtils.java>
      +
      + <pre><code> .loadClass("org.apache.spark.sql.delta.files.TahoeLogFileIndex"); +</code></pre> +
      + + + + + + + +
      + + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
      + + +
      + + + +
      + 🙌 Paweł Leszczyński, Teemu Mattelmäki, Harel Shein +
      + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Teemu Mattelmäki + (teemu.mattelmaki@teradata.com) +
      +
      2024-01-05 08:59:18
       +
      +

      *Thread Reply:* Hello @Maciej Obuchowski! I have a customer that has on-prem. Hadoop environment with Spark & Hive. We want to establish column level lineage on that environment. We would need to install OpenLineage (OL) Spark Agent. Does OpenLineage support this type of environment or does it only support Spark on Databricks?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-01-05 09:16:36
       +
      +

      *Thread Reply:* > Does OpenLineage support this type of environment +yes, you can try it in local notebook for example 🙂

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-01-05 09:17:04
       +
      +

      *Thread Reply:* https://openlineage.io/docs/integrations/spark/quickstart_local

      +
      +
      openlineage.io
      + + + +
      Quickstart with Jupyter | OpenLineage
      +
      + Trying out the Spark integration is super easy if you already have Docker Desktop and git installed. +
      + + + + + + +
      Original URL: https://openlineage.io/docs/integrations/spark/quickstart_local
      + + + +
      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Teemu Mattelmäki + (teemu.mattelmaki@teradata.com) +
      +
      2024-01-06 03:22:44
       +
      +

      *Thread Reply:* Thanks! We'll try this option.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-10 05:16:48
       +
      +

      @Maciej Obuchowski @Paweł Leszczyński - Happy New Year. Just a note, we stumbled into a bit of roadblock with testing of the changes for Scala 2.13. Notably: Bitnami only publishes Spark images for Scala 2.12. This means the CI/CD pipeline will fail under Scala 2.13 conditions.

      + +

      For example:

      + +

      ```SparkContainerIntegrationTest > testCreateTable() STANDARD_ERROR

      + +
      24/01/10 09:54:01 ERROR Utils: uncaught error in thread spark-listener-group-shared, stopping SparkContext
+java.lang.NoSuchMethodError: 'scala.collection.immutable.Seq org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.map(scala.Function1)'
+    at io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.lambda$buildInputDatasets$6(OpenLineageRunEventBuilder.java:342)
+    at java.base/java.util.Optional.map(Optional.java:260)
+    at io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.buildInputDatasets(OpenLineageRunEventBuilder.java:340)
+    at io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.populateRun(OpenLineageRunEventBuilder.java:296)
+    at io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.buildRun(OpenLineageRunEventBuilder.java:280)
+    at io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.buildRun(OpenLineageRunEventBuilder.java:229)
+    at io.openlineage.spark.agent.lifecycle.SparkSQLExecutionContext.start(SparkSQLExecutionContext.java:87)
+    at io.openlineage.spark.agent.OpenLineageSparkListener.lambda$sparkSQLExecStart$0(OpenLineageSparkListener.java:91)
+    at java.base/java.util.Optional.ifPresent(Optional.java:178)
+    at io.openlineage.spark.agent.OpenLineageSparkListener.sparkSQLExecStart(OpenLineageSparkListener.java:91)
+    at io.openlineage.spark.agent.OpenLineageSparkListener.onOtherEvent(OpenLineageSparkListener.java:82)
+24/01/10 09:54:01 INFO SparkContext: SparkContext is stopping with exitCode 0.
+
      + +

      ```

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-10 05:17:46
       +
      +

      *Thread Reply:* This is because of a breaking change between Scala 2.12 and Scala 2.13 in the Scala Collections API.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-10 05:18:22
       +
      +

      *Thread Reply:* In Scala 2.12, Seq is an alias for scala.collection.Seq and in 2.13, Seq is an alias for scala.collection.immutable.Seq

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-10 05:23:59
       +
      +

      *Thread Reply:* The question becomes, does OL (as an org), have a way to overcome this challenge? i.e., custom docker images?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-01-10 05:59:13
       +
      +

      *Thread Reply:* even official Spark images don't have 2.13 version 😞 https://hub.docker.com/_/spark

      +
      +
      hub.docker.com
      + + + +
      spark - Official Image | Docker Hub
      +
      + Apache Spark - A unified analytics engine for large-scale data processing +
      + + + + + + +
      Original URL: https://hub.docker.com/_/spark
      + + + +
      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-01-10 06:06:20
       +
      +

      *Thread Reply:* as for Seq issue, maybe the solution would be to have something similar to the visitor factories we have: +provide two factories, make integration choose at runtime which one to use, and initialize Seq there?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Mattia Bertorello + (mattia.bertorello@booking.com) +
      +
      2024-01-10 06:24:12
       +
      +

      *Thread Reply:* It seems we need to build a Docker image for Spark Scala 2.13

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-01-10 06:43:04
       +
      +

      *Thread Reply:* there are even some examples but they mostly focus on pure Spark, not python one https://github.com/xebia-functional/spark-scala3-example/blob/main/docker/Dockerfile

      +
      + + + + +
      <https://github.com/xebia-functional/spark-scala3-example/blob/main/docker/Dockerfile | Dockerfile>
      +
      + <pre><code># This Dockerfile is taken from &lt;https://github.com/mvillarrealb/docker-spark-cluster/blob/master/docker-compose.yml&gt; +# and updated to install the Scala 2.13 version of Spark 3.2.0 + +FROM openjdk:11.0.13-jre-slim-buster as builder + +# Add Dependencies for PySpark +RUN apt-get update &amp;&amp; apt-get install -y curl vim wget software-properties-common ssh net-tools ca-certificates python3 python3-pip python3-numpy python3-matplotlib python3-scipy python3-pandas python3-simpy + +RUN update-alternatives --install "/usr/bin/python" "python" "$(which python3)" 1 + +# Fix the value of PYTHONHASHSEED +# Note: this is needed when you use Python 3.3 or greater +ENV SPARK_VERSION=3.2.0 \ +SCALA_VERSION=2.13 \ +HADOOP_VERSION=3.2 \ +SPARK_HOME=/opt/spark \ +PYTHONHASHSEED=1 + +RUN wget --no-verbose -O apache-spark.tgz "&lt;https://archive.apache.org/dist/spark/spark-${SPARK_VERSION}/spark-${SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}-scala${SCALA_VERSION}.tgz&gt;" \ +&amp;&amp; mkdir -p /opt/spark \ +&amp;&amp; tar -xf apache-spark.tgz -C /opt/spark --strip-components=1 \ +&amp;&amp; rm apache-spark.tgz + + +FROM builder as apache-spark + +WORKDIR /opt/spark + +ENV SPARK_MASTER_PORT=7077 \ +SPARK_MASTER_WEBUI_PORT=8080 \ +SPARK_LOG_DIR=/opt/spark/logs \ +SPARK_MASTER_LOG=/opt/spark/logs/spark-master.out \ +SPARK_WORKER_LOG=/opt/spark/logs/spark-worker.out \ +SPARK_WORKER_WEBUI_PORT=8080 \ +SPARK_WORKER_PORT=7000 + +EXPOSE 8080 7077 7000 + +RUN mkdir -p $SPARK_LOG_DIR &amp;&amp; \ +touch $SPARK_MASTER_LOG &amp;&amp; \ +touch $SPARK_WORKER_LOG &amp;&amp; \ +ln -sf /dev/stdout $SPARK_MASTER_LOG &amp;&amp; \ +ln -sf /dev/stdout $SPARK_WORKER_LOG + +COPY start-spark.sh / + +CMD ["/bin/bash", "/start-spark.sh"] + +</code></pre> +
      + + + + + + + +
      + + <a href="https://github.com/xebia-functional/spark-scala3-example">xebia-functional/spark-scala3-example</a> +
      + + +
      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-01-10 06:43:27
       +
      +

      *Thread Reply:* I think people who run PySpark generally don't care about scala version 🤔

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-10 06:43:39
       +
      +

      *Thread Reply:* Indeed.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-10 06:44:50
       +
      +

      *Thread Reply:* The problem (for us) is that Scala 2.12's Jackson module has a security vulnerability in it, that was flagged. So our Data Platform team(s) had to mitigate. The mitigation strategy involved migrating to Scala 2.13.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-01-10 06:53:04
       +
      +

      *Thread Reply:* looking at https://github.com/bitnami/containers/blob/main/bitnami/spark/3.5/debian-11/Dockerfile there does not seem to be very simple way to just switch it to 2.13

      +
      + + + + +
      <https://github.com/bitnami/containers/blob/main/bitnami/spark/3.5/debian-11/Dockerfile | Dockerfile>
      +
      + <pre><code># Copyright VMware, Inc. +# SPDX-License-Identifier: APACHE-2.0 + +FROM &lt;a href="http://docker.io/bitnami/minideb:bullseye"&gt;docker.io/bitnami/minideb:bullseye&lt;/a&gt; + +ARG JAVA_EXTRA_SECURITY_DIR="/bitnami/java/extra-security" +ARG TARGETARCH + +LABEL com.vmware.cp.artifact.flavor="sha256:1e1b4657a77f0d47e9220f0c37b9bf7802581b93214fff7d1bd2364c8bf22e8e" \ + org.opencontainers.image.base.name="&lt;a href="http://docker.io/bitnami/minideb:bullseye"&gt;docker.io/bitnami/minideb:bullseye&lt;/a&gt;" \ + org.opencontainers.image.created="2023-12-16T16:24:11Z" \ + org.opencontainers.image.description="Application packaged by VMware, Inc" \ + org.opencontainers.image.licenses="Apache-2.0" \ + org.opencontainers.image.ref.name="3.5.0-debian-11-r17" \ + org.opencontainers.image.title="spark" \ + org.opencontainers.image.vendor="VMware, Inc." \ + org.opencontainers.image.version="3.5.0" + +ENV HOME="/" \ + OS_ARCH="${TARGETARCH:-amd64}" \ + OS_FLAVOUR="debian-11" \ + OS_NAME="linux" \ + PATH="/opt/bitnami/python/bin:/opt/bitnami/java/bin:/opt/bitnami/spark/bin:/opt/bitnami/spark/sbin:$PATH" + +COPY prebuildfs / +SHELL ["/bin/bash", "-o", "errexit", "-o", "nounset", "-o", "pipefail", "-c"] +# Install required system packages and dependencies +RUN install_packages ca-certificates curl libbz2-1.0 libcom-err2 libcrypt1 libffi7 libgcc-s1 libgssapi-krb5-2 libk5crypto3 libkeyutils1 libkrb5-3 libkrb5support0 liblzma5 libncursesw6 libnsl2 libreadline8 libsqlite3-0 libssl1.1 libstdc++6 libtinfo6 libtirpc3 procps zlib1g +RUN mkdir -p /tmp/bitnami/pkg/cache/ ; cd /tmp/bitnami/pkg/cache/ ; \ + COMPONENTS=( \ + "python-3.11.7-0-linux-${OS_ARCH}-debian-11" \ + "java-17.0.9-11-6-linux-${OS_ARCH}-debian-11" \ + "spark-3.5.0-6-linux-${OS_ARCH}-debian-11" \ + ) ; \ + for COMPONENT in "${COMPONENTS[@]}"; do \ + if [ ! -f "${COMPONENT}.tar.gz" ]; then \ + curl -SsLf "&lt;https://downloads.bitnami.com/files/stacksmith/${COMPONENT}.tar.gz&gt;" -O ; \ + curl -SsLf "&lt;https://downloads.bitnami.com/files/stacksmith/${COMPONENT}.tar.gz.sha256&gt;" -O ; \ + fi ; \ + sha256sum -c "${COMPONENT}.tar.gz.sha256" ; \ + tar -zxf "${COMPONENT}.tar.gz" -C /opt/bitnami --strip-components=2 --no-same-owner --wildcards '**/files' ; \ + rm -rf "${COMPONENT}".tar.gz{,.sha256} ; \ + done +RUN apt-get autoremove --purge -y curl &amp;&amp; \ + apt-get update &amp;&amp; apt-get upgrade -y &amp;&amp; \ + apt-get clean &amp;&amp; rm -rf /var/lib/apt/lists /var/cache/apt/archives +RUN chmod g+rwX /opt/bitnami +RUN mkdir /.local &amp;&amp; chmod g+rwX /.local + +COPY rootfs / +RUN /opt/bitnami/scripts/spark/postunpack.sh +RUN /opt/bitnami/scripts/java/postunpack.sh +ENV APP_VERSION="3.5.0" \ + BITNAMI_APP_NAME="spark" \ + JAVA_HOME="/opt/bitnami/java" \ + LD_LIBRARY_PATH="/opt/bitnami/python/lib:/opt/bitnami/spark/venv/lib/python3.8/site-packages/numpy.libs:$LD_LIBRARY_PATH" \ + LIBNSS_WRAPPER_PATH="/opt/bitnami/common/lib/libnss_wrapper.so" \ + NSS_WRAPPER_GROUP="/opt/bitnami/spark/tmp/nss_group" \ + NSS_WRAPPER_PASSWD="/opt/bitnami/spark/tmp/nss_passwd" \ + PYTHONPATH="/opt/bitnami/spark/python/:$PYTHONPATH" \ + SPARK_HOME="/opt/bitnami/spark" \ + SPARK_USER="spark" + +WORKDIR /opt/bitnami/spark +USER 1001 +ENTRYPOINT [ "/opt/bitnami/scripts/spark/entrypoint.sh" ] +CMD [ "/opt/bitnami/scripts/spark/run.sh" ] + +</code></pre> +
      + + + + + + + +
      + + <a href="https://github.com/bitnami/containers">bitnami/containers</a> +
      + + +
      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-10 06:54:57
       +
      +

      *Thread Reply:* Nope. You have to download the Spark binaries compiled with 2.13, and build an entirely new docker image

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-10 06:55:20
       +
      +

      *Thread Reply:* or at least, download the binaries and overwrite everything at runtime

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-01-10 06:56:38
       +
      +

      *Thread Reply:* tbh I can't even find where they specify Scala 2.12

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-10 06:56:54
       +
      +

      *Thread Reply:* They don't. It's the default version for Spark 3.x+

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-10 06:57:12
       +
      +

      *Thread Reply:* You have to explicitly select Scala 2.13 version from Spark's website

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-01-10 06:57:14
       +
      +

      *Thread Reply:* maybe doing it from scratch is only option? but I don't know if it's that easy; I feel if it was easy to build and maintain there would be existing image

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-01-11 02:34:39
       +
      +

      *Thread Reply:* This may be silly, but... what if we just turned off docker tests for scala 2.13? There are a lot of non-docker integration tests and I don't feel we can fix the whole world.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-01-11 05:23:35
       +
      +

      *Thread Reply:* It will be very easy to break it later then, even if we test it manually at the start 🙂

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Mattia Bertorello + (mattia.bertorello@booking.com) +
      +
      2024-01-11 05:25:50
       +
      +

      *Thread Reply:* What about using the custom docker image suggested here only for scala 2.13 https://openlineage.slack.com/archives/C06A9EAUWUQ/p1704886984670229?thread_ts=1704881808.866719&cid=C06A9EAUWUQ +and built at run time, no super efficient but better than nothing 🙂 +new ImageFromDockerfile() + .withFileFromPath("Dockerfile", Paths.get("./spark/docker/scala213/Dockerfile")) + .withBuildArg("SPARK_VERSION", System.getProperty("spark.version")))

      +
      + + +
      + + + } + + Maciej Obuchowski + (https://openlineage.slack.com/team/U01RA9B5GG2) +
      + + + +
      +
      + <p>there are even some examples but they mostly focus on pure Spark, not python one <a href="https://github.com/xebia-functional/spark-scala3-example/blob/main/docker/Dockerfile">https://github.com/xebia-functional/spark-scala3-example/blob/main/docker/Dockerfile</a></p> +
      + + + + + + +
      Original URL: https://openlineage.slack.com/archives/C06A9EAUWUQ/p1704886984670229?thread_ts=1704881808.866719&amp;cid=C06A9EAUWUQ
      + + +
      + + Thread in Slack Conversation +
      + + +
      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-01-11 05:27:04
       +
      +

      *Thread Reply:* Yeah might be good for now. I think we should see how long it takes

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Mattia Bertorello + (mattia.bertorello@booking.com) +
      +
      2024-01-11 05:29:29
       +
      +

      *Thread Reply:* For the moment we run it in local to test the changes. I will open a PR when we have all the stuff running.

      + + + +
      + 👍 Maciej Obuchowski +
      + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-10 07:11:15
       +
      +

      @Maciej Obuchowski - starting a new thread here:

      + +

      Regarding the factories comment

      + +
      +

      as for Seq issue, maybe the solution would be to have something similar to the visitor factories we have: + provide two factories, make integration choose at runtime which one to use, and initialize Seq there? + I'm unsure about your intended use of factories and their placement in the call stack. To illustrate the issue at hand, I developed a simple project. Here's an overview:

      +
      + +

      app project: +```public class Application { + public static void main(String[] args) { + Demonstration demonstration = new Demonstration(); + java.util.List items = demonstration.items(); + items.forEach(System.out::println); + } +}

      + +

      public class Demonstration { + public java.util.List items() { + return convert(Provider.getStrings()); + }

      + +
      public java.util.List<String> convert(scala.collection.Seq<String> strings) {
+    return scala.collection.JavaConverters.seqAsJavaList(strings);
+}
+
+public java.util.List<String> convert(scala.collection.immutable.Seq<String> strings) {
+    return scala.collection.JavaConverters.seqAsJavaList(strings);
+}
+
      + +

      } +**thing_2.12 & thing_2.13 projects** +object Provider { + def getStrings(): Seq[String] = Seq("a", "b", "c") +}``` +Explanation:

      + +

      • Compilation Against Scala 2.12 (thing2.12): When compiling the 'app' project against Scala 2.12, the Java compiler recognizes that it should use the method java.util.List<String> convert(scala.collection.Seq<String> strings). +• Compilation Against Scala 2.13 (thing2.13): Conversely, when compiling against Scala 2.13, the compiler selects java.util.List<String> convert(scala.collection.immutable.Seq<String> strings) for dispatch. +Runtime Behavior:

      + +

      • Matching Versions: Running the 'app' code compiled against thing_2.12 with the thing_2.12 jar in the classpath functions correctly. The same is true when pairing app compiled against thing_2.13 with the thing_2.13 jar. +• Mismatched Versions: Issues arise when there's a mismatch in versions. For instance, using the app compiled against thing_2.12 but providing the thing_2.13 jar in the classpath leads to problems, and vice versa. This occurs because the JVM expects the method matching the compilation version, which it fails to find. +Expanding on the Mismatched Versions topic:

      + +

      Compile-Time Binding in Java:

      + +

      • n Java, method overloading resolution is a compile-time decision. The compiler determines which method to call based on the method signatures and the types of the arguments as they are known at compile time. +• In our case, the convert method is overloaded: one version takes a scala.collection.Seq<String>, and the other takes a scala.collection.immutable.Seq<String>. +• When you compile against Scala 2.12, scala.collection.Seq is the prevalent sequence type. The Java compiler binds calls to convert to the method accepting scala.collection.Seq. +• Conversely, compiling against Scala 2.13, where scala.collection.immutable.Seq is the base type, leads the Java compiler to bind to the method accepting scala.collection.immutable.Seq. +Runtime Behavior and Classpath:

      + +

      • At runtime, the Java Virtual Machine (JVM) expects to find the same class types and method signatures that were present during compilation. +• If the runtime environment (classpaths and libraries) matches the compile-time environment, the application runs smoothly. +• However, if there's a mismatch – for instance, the application is compiled against Scala 2.12 but run with Scala 2.13 in the classpath – the JVM may not find the expected method signatures. +• This mismatch can lead to a NoSuchMethodError or similar runtime exceptions. The JVM looks for convert(scala.collection.Seq<String>) but only finds convert(scala.collection.immutable.Seq<String>), or vice versa.

      + + + +
      + 👀 Maciej Obuchowski +
      + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-01-10 07:32:35
       +
      +

      *Thread Reply:* Thanks for the explanation!

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-01-10 07:32:48
       +
      +

      *Thread Reply:* Yeah, I thought only initialization is an issue 🙂

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-01-10 07:40:02
       +
      +

      *Thread Reply:* I can't think of better solution than reflection-based one right now

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-01-10 07:58:02
       +
      +

      *Thread Reply:* @Damien Hawes Would something like this work? +We have interface: +public interface SeqConverter{ + <T> scala.collection.Seq<T> convert(scala.collection.Seq<T> items); + <T> scala.collection.Seq<T> convert(scala.collection.immutable.Seq<T> items); +} +and implementations for 2.12 and 2.13 within separate modules compiled with respective Scala versions - just as we compile modules with respective Spark versions now, example: +```public class Seq212Converter implements SeqConcerter { + public scala.collection.Seq convert(scala.collection.Seq items) { + return items; + }

      + +

      public scala.collection.Seq convert(scala.collection.immutable.Seq items) { + // Whatever that conversion would look like + return scala.collection.immutable.Seq$(items); + } +} +then naively call Seq methods: + public static Object callWithSeq(Object clazz, String method, scala.collection.Seq seq) { + String Scala212Converter = + "io.openlineage.spark.agent.scala212.Seq212Converter";

      + +
      String Scala213Converter =
+  "io.openlineage.spark.agent.scala213.Scala213Converter";
+
+if ( scala.util.Properties$.MODULE$.versionNumberString().startsWith("2.12")) {
+  SeqConverter converter = Class.forName(Scala212Converter).newInstance();
+  return MethodUtils.invokeMethod(clazz, method, converter(seq));
+} else {
+  SeqConverter converter = Class.forName(Scala213Converter).newInstance();
+  return MethodUtils.invokeMethod(clazz, method, converter(seq));
+}
+
      + +

      }``` +The last part is definitely crude, as it allows you to just call a method with singular Seq argument, but it's just for sake of this discussion 🙂

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-01-10 07:59:32
       +
      +

      *Thread Reply:* @Paweł Leszczyński when you get back take a look at this thread 🙂

      + + + +
      + 👀 Paweł Leszczyński +
      + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-01-10 08:02:31
       +
      +

      *Thread Reply:* Not sure if what I wrote would work with something like this: https://github.com/OpenLineage/OpenLineage/blob/31f8ce588526e9c7c4bc7d849699cb7ce2[…]cle/plan/column/visitors/IcebergMergeIntoDependencyVisitor.java

      +
      + + + + +
      <https://github.com/OpenLineage/OpenLineage/blob/31f8ce588526e9c7c4bc7d849699cb7ce2969c8f/integration/spark/spark3/src/main/java/io/openlineage/spark3/agent/lifecycle/plan/column/visitors/IcebergMergeIntoDependencyVisitor.java | IcebergMergeIntoDependencyVisitor.java>
      +
      + <pre><code> boolean isNewerImplementation = +</code></pre> +
      + + + + + + + +
      + + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
      + + +
      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-01-10 08:02:43
       +
      +

      *Thread Reply:* Seq&lt;Seq&lt;Seq&lt;Expression&gt;&gt;&gt; matched = + (Seq&lt;Seq&lt;Seq&lt;Expression&gt;&gt;&gt;) mergeRows.getMethod("matchedOutputs").invoke(node); + Seq&lt;Seq&lt;Expression&gt;&gt; notMatched = + (Seq&lt;Seq&lt;Expression&gt;&gt;) mergeRows.getMethod("notMatchedOutputs").invoke(node); +🤔

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-10 10:14:27
       +
      +

      *Thread Reply:* It could work. However, something to keep in mind: we have to eliminate all references to scala.collection.Seq in import statements and fully-qualified types. Basically it becomes a case of, "Immediately convert sequence to java.util.List".

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-10 10:14:50
       +
      +

      *Thread Reply:* Otherwise we will face a combination of runtime and compile time errors.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-10 10:15:29
       +
      +

      *Thread Reply:* That code you linked, is one such place where we will need to change it - somehow.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-10 10:15:42
       +
      +

      *Thread Reply:* That, or we need to port that code to Scala.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-10 10:16:05
       +
      +

      *Thread Reply:* So that we don't need to constantly do a dance between Scala and Java collections.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-01-10 11:37:24
       +
      +

      *Thread Reply:* We might go with writing common module in Scala though. It would be really good if we could do that incrementally. Even better if we could have single module for both 2.12 and 2.13

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-01-10 11:39:17
       +
      +

      *Thread Reply:* We use a lot of Seq and I doubt it's the only place we'll have problems

      + +
      +
      Screenshot 2024-01-10 at 17.38.27.png
      + + + + + + + + +
      + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-01-11 02:42:37
       +
      +

      *Thread Reply:* Would using this https://github.com/scala/scala-collection-compat help?

      +
      + + + + +
      scala/scala-collection-compat
      +
      + makes some Scala 2.13 APIs (primarily collections, also some others) available on 2.11 and 2.12, to aid cross-building +
      + +
      +
      Stars
      + 194 +
      + +
      +
      Language
      + Scala +
      + + + + + + + + +
      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-11 03:55:06
       +
      +

      *Thread Reply:* AS far as I could tell, no. That adds methods to the APIs

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-17 06:50:17
       +
      +

      @Maciej Obuchowski @Paweł Leszczyński - spark3 what is its purpose? Is it meant to be common code for all Apache Spark 3.x.y variants?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-01-17 06:50:36
       +
      +

      *Thread Reply:* yes

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-17 06:51:18
       +
      +

      *Thread Reply:* I'm asking, because I noticed this situation, spark3 is compiled using Spark 3.1.x, but something like JdbcHandler doesn't exist in Spark 3.0.x

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-01-17 06:52:56
       +
      +

      *Thread Reply:* if the classes using JdbcHandler aren't loaded, this shouldn't break the integration. However, we don't test our package with 3.0.x

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-18 09:50:41
       +
      +

      @Maciej Obuchowski @Paweł Leszczyński - right, first prep PR for support Scala 2.13 is up. I'm breaking my changes up into smaller PRs to make then more consumable.

      + +

      You'll find it here:

      + +

      https://github.com/OpenLineage/OpenLineage/pull/2376

      + + + +
      + 👀 Paweł Leszczyński, Maciej Obuchowski +
      + +
      + 🔥 Maciej Obuchowski, Harel Shein +
      + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-18 10:13:37
       +
      +

      *Thread Reply:* Ah, right. I can't run the integration tests.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-18 10:14:00
       +
      +

      *Thread Reply:* My heart sank a bit, when i saw the integration tests fail

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-18 10:14:21
       +
      +

      *Thread Reply:* I was like, "How?! I didn't touch the spark code!"

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-01-18 10:21:57
       +
      +

      *Thread Reply:* pushed it 🙂

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-18 10:28:51
       +
      +

      *Thread Reply:* Sorry, pushed what?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-01-18 10:32:44
       +
      +

      *Thread Reply:* We have a script to push external contributions to some origin branch, GH CI does apparently look at commit IDs instead of associating CI runs with branches so after this finishes: https://app.circleci.com/pipelines/github/OpenLineage/OpenLineage/8924/workflows/efdcc509-cc79-4158-a925-50a59733d446 it will be shown as CI results for your branch

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-01-18 10:33:12
       +
      +

      *Thread Reply:* Anyway, we have different solution to this problem soon: https://github.com/OpenLineage/OpenLineage/pull/2374

      +
      + + + + +
      #2374 ci: approve integration tests
      +
      + <p><strong>Problem</strong></p> + +<p>Currently, there's no option to conveniently let new committers to either run integration tests (Airflow integration case) or use OL credentials to external services needed to run integration tests successfully (Spark integration case). The only solution we've had so far was <a href="https://github.com/jklukas/git-push-fork-to-upstream-branch">https://github.com/jklukas/git-push-fork-to-upstream-branch</a> but this was far from good.</p> + +<p><strong>Solution</strong></p> + +<p>• Add approval steps before integration tests. +• Remove approval steps <strong>if pipeline runs from someone with write access to the repo</strong></p> + +<p>Several additional notes:</p> + +<p>• <strong>Most important</strong> we should enable <code>Pass secrets to builds from forked pull requests</code> after merging this PR (solves Spark integration problem). +• Side-effect of changing <code>determine_changed_module</code> is a huge speedup (from ~1 min to literally seconds with added new step). +• I forced (with code change in circleCI config) to not remove approval steps just for test. You can check the result <a href="https://app.circleci.com/pipelines/gh/OpenLineage/OpenLineage/8912/workflows/14897b21-7ae6-4726-9070-a73b6f0f23c3">here</a>. Screenshot attached with task on hold waiting for approval:<br /> + <a href="https://private-user-images.githubusercontent.com/3889552/297518241-5a12a1ef-ebc2-43ed-ba84-054fe98c32f6.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MDU1OTIyOTQsIm5iZiI6MTcwNTU5MTk5NCwicGF0aCI6Ii8zODg5NTUyLzI5NzUxODI0MS01YTEyYTFlZi1lYmMyLTQzZWQtYmE4NC0wNTRmZTk4YzMyZjYucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDExOCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDAxMThUMTUzMzE0WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9OGQ1YzZmMDY0MmZjZTFmNzM4M2FjMTg3MjA4NmIyYjAxOTU2NjBhNjgzZTYwZGUwZDZkZjY1MDMzYjJkMDE2YiZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.oeNpY_GTDfTHBqV44fWvWjuO6E8BtHl-sVEto7p0rak">image</a></p> + +<p><strong>One-line summary:</strong></p> + +<p>Introduce approvals to integration tests in CI.</p> + +<p><strong>Checklist</strong></p> + +<p>☑︎ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☑︎ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☑︎ Your changes are accompanied by tests (<em>if relevant</em>) +☑︎ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☐ You've updated any relevant documentation (<em>if relevant</em>) +☑︎ Your comment includes a one-liner for the changelog about the specific purpose of the change (<em>if necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>) +☐ You've added a <a href="https://github.com/OpenLineage/OpenLineage/tree/main/.github/header_templates.md">header</a> to source files (<em>if relevant</em>)</p> + +<hr /> + +<p>SPDX-License-Identifier: Apache-2.0<br /> +Copyright 2018-2023 contributors to the OpenLineage project</p> +
      + +
      +
      Labels
      + ci, tests +
      + +
      +
      Comments
      + 3 +
      + + + + + + + +
      + + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
      + + +
      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-18 10:53:47
       +
      +

      *Thread Reply:* Perfect

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-18 10:54:02
       +
      +

      *Thread Reply:* I see the tests went through

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-01-19 02:20:39
       +
      +

      *Thread Reply:* It looks good to me and I approved PR. I would leave it unmerged for a moment to let Maciej review it if he likes to. We can merge it later this day.

      + +

      BTW that's a really great improvement to the codebase @Damien Hawes. Thanks for preparing this.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-01-19 05:33:20
       +
      +

      *Thread Reply:* Great PR!

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-01-19 05:34:44
       +
      +

      *Thread Reply:* I've :gh_merged: it

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-19 05:39:05
       +
      +

      *Thread Reply:* Thanks!

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Arnab Bhattacharyya + (arnab.bhattacharyya@booking.com) +
      +
      2024-01-19 04:37:18
       +
      +

      @Arnab Bhattacharyya has joined the channel

      + + + +
      + 👋 Damien Hawes, Maciej Obuchowski, Paweł Leszczyński +
      + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-19 05:30:38
       +
      +

      @Maciej Obuchowski @Paweł Leszczyński - right the next PR is up for review: https://github.com/OpenLineage/OpenLineage/pull/2377

      + +

      It builds off the previous one, so we see a lot more changes that what is strictly necessary. Once the previous one is merged, I'll rebase off of main.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-19 06:17:32
       +
      +

      *Thread Reply:* @Maciej Obuchowski - I see that the CI is failing because of environment variables not being set. Is there any way to retry the jobs to see if this is transient?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-19 06:18:09
       +
      +

      *Thread Reply:* ```ERROR: Missing environment variable {i}

      + +

      Exited with code exit status 1```

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-01-19 06:36:20
       +
      +

      *Thread Reply:* @Jakub Dardziński Will your PR 2374 fix this as well? +https://app.circleci.com/pipelines/github/OpenLineage/OpenLineage/8936/workflows/d164feb2-219e-4f2a-8d8c-066d54e37efc/jobs/164588

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-01-19 06:38:09
       +
      +

      *Thread Reply:* https://app.circleci.com/pipelines/github/OpenLineage/OpenLineage/8937/workflows/1b945af1-8353-4de7-8509-e0d1d714637a running here for now

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-19 07:25:30
       +
      +

      *Thread Reply:* Awesome

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-19 07:25:34
       +
      +

      *Thread Reply:* Seems like it went through

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-01-19 09:56:03
       +
      +

      *Thread Reply:* :gh_merged:

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-19 10:05:19
       +
      +

      *Thread Reply:* Awesome!

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-22 05:20:34
       +
      +

      @Paweł Leszczyński - thanks for re-running the CI/CD

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-22 06:04:42
       +
      +

      Thanks! @Paweł Leszczyński

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-22 06:53:30
       +
      +

      @Paweł Leszczyński @Maciej Obuchowski - this one is ready: https://github.com/OpenLineage/OpenLineage/pull/2384

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-22 10:24:31
       +
      +

      *Thread Reply:* @Paweł Leszczyński - responded to your comments.

      + + + +
      + 👀 Paweł Leszczyński +
      + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-22 06:53:53
       +
      +

      Oh, I see the integration tests are failing with that environment variable thing

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-22 07:41:39
       +
      +

      Done- the above PR is ready for review

      + + + +
      + 👍 Maciej Obuchowski +
      + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-22 12:40:45
       +
      +

      Any idea why the CI/CD pipelines keep failing with that "No environment variable" issue?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Harel Shein + (harel.shein@gmail.com) +
      +
      2024-01-22 13:13:41
       +
      +

      *Thread Reply:* hmmm... these come from the integration-tests context. which are available for all members of OpenLineage org. so perhaps you need to be a member for your build to have access to that context? +@Jakub Dardziński might know more here

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Harel Shein + (harel.shein@gmail.com) +
      +
      2024-01-22 13:14:12
       +
      +

      *Thread Reply:* example here: https://app.circleci.com/pipelines/github/OpenLineage/OpenLineage/9011/workflows/00f378bb-b10c-4338-94ff-1e4ec8ee70e8/jobs/166105

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Harel Shein + (harel.shein@gmail.com) +
      +
      2024-01-22 13:14:47
       +
      +

      *Thread Reply:* I know we didn't merge your PR yet @Jakub Dardziński

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Jakub Dardziński + (jakub.dardzinski@getindata.com) +
      +
      2024-01-22 13:14:48
       +
      +

      *Thread Reply:* well, so for that one I have this PR: https://github.com/OpenLineage/OpenLineage/pull/2374

      + +

      it is indeed an issue that should be resolved with that

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Harel Shein + (harel.shein@gmail.com) +
      +
      2024-01-22 13:15:22
       +
      +

      *Thread Reply:* ok, great. let's test this out after this is merged then 🙂

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Jakub Dardziński + (jakub.dardzinski@getindata.com) +
      +
      2024-01-22 13:15:40
       +
      +

      *Thread Reply:* I was about to merge it today, forgot to do that after rebasing :face_palm:

      + + + +
      + 😅 Harel Shein, Paweł Leszczyński +
      + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Jakub Dardziński + (jakub.dardzinski@getindata.com) +
      +
      2024-01-22 13:13:45
       +
      +

      @Jakub Dardziński has joined the channel

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-01-23 03:38:09
       +
      +

      @Damien Hawes https://github.com/OpenLineage/OpenLineage/pull/2384#discussion_r1462874134 -> please let me know what do you think about that?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-23 03:40:03
       +
      +

      *Thread Reply:* The problem exists in the MatchesMapRecursively class. It happens when k == "schema" . The code expects the type of target.get(k) to be a subtype of java.util.Map, however it is actually a subtype of java.util.ArrayList.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-23 03:40:33
       +
      +

      *Thread Reply:* public static Predicate&lt;Map&lt;String, Object&gt;&gt; predicate( + Map&lt;String, Object&gt; target, Set&lt;String&gt; omittedKeys) { + return (map) -&gt; { + if (!map.keySet().containsAll(target.keySet())) { + log.error("Object keys {} does not match target keys {}", map.keySet(), target.keySet()); + return false; + } + for (String k : target.keySet()) { + if (omittedKeys.contains(k)) { + continue; + } + Object val = map.get(k); + boolean eq; + if (val instanceof Map) { + eq = + MatchesMapRecursively.predicate((Map&lt;String, Object&gt;) target.get(k), omittedKeys) + .test((Map&lt;String, Object&gt;) val); + } else if (val instanceof List) { + eq = + MatchesMapRecursively.predicate((List&lt;Object&gt;) target.get(k), omittedKeys) + .test((List&lt;Object&gt;) val); + } else if (PRODUCER_WITH_UNDERSCORE.equals(k) || PRODUCER.equals(k)) { + eq = Versions.OPEN_LINEAGE_PRODUCER_URI.toString().equals(val); + } else if (val == null) { + eq = true; + } else { + eq = val.equals(target.get(k)); + } + if (!eq) { + log.error( + "For key {} - passed object {} does not match target object {}", + k, + map.get(k), + target.get(k)); + return false; + } + } + return true; + }; + }

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-23 03:41:33
       +
      +

      *Thread Reply:*

      + +
      +
      image.png
      + + + + + + + + +
      + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-01-23 03:48:00
       +
      +

      *Thread Reply:* it looks like, after changing bigquery connector version, some of the properties of BigQueryRelation changed their type. However, the test relies on serialized json https://github.com/OpenLineage/OpenLineage/blob/1.8.0/integration/spark/shared/src/test/resources/test_data/serde/bigqueryrelation-node.json

      +
      + + + + +
      <https://github.com/OpenLineage/OpenLineage/blob/1.8.0/integration/spark/shared/src/test/resources/test_data/serde/bigqueryrelation-node.json | bigqueryrelation-node.json>
      +
      + <pre><code>{ + "relation":{ + "tableId": { + "dataset":"dataset", + "table":"test" + }, + "tableName":"dataset.test" + }, + "output":[ + { + "name":"name", + "dataType":{ + "ordering":{} + }, + "nullable":false, + "metadata":{ + "map":{} + }, + "qualifier":[], + "origin":{}, + "deterministic":true, + "resolved":true + } + ], + "isStreaming":false, + "schema":[ + { + "name": "name", + "nullable": false + } + ], + "allAttributes":{ + "attrs":[] + }, + "resolved":true, + "attributeMap":{}, + "traceEnabled":false, + "canonicalizedPlan":false +} +</code></pre> +
      + + + + + + + +
      + + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
      + + +
      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-23 03:48:34
       +
      +

      *Thread Reply:* So schema was always a list?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-01-23 03:48:35
       +
      +

      *Thread Reply:* and this json may contain somewhere a list which is expected to be a map within new library

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-23 03:48:48
       +
      +

      *Thread Reply:* it failed on the key "schema"

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-23 03:49:49
       +
      +

      *Thread Reply:* Though, I just fixed the test.

      + + + +
      + 🙌 Paweł Leszczyński +
      + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-23 03:50:09
       +
      +

      *Thread Reply:* Object val = map.get(k); + Object targetVal = target.get(k); + boolean eq; + if (val instanceof Map) { + if (targetVal instanceof Map) { + Predicate&lt;Map&lt;String, Object&gt;&gt; predicate = predicate((Map&lt;String, Object&gt;) targetVal, omittedKeys); + return predicate.test((Map&lt;String, Object&gt;) val); + } else if (targetVal instanceof List) { + Predicate&lt;List&lt;Object&gt;&gt; predicate = predicate((List&lt;Object&gt;) targetVal, omittedKeys); + return predicate.test((List&lt;Object&gt;) val); + } else { + eq = false; + } + }

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-23 03:50:44
       +
      +

      *Thread Reply:* For Scala 2.12 at least

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-01-23 03:50:53
       +
      +

      *Thread Reply:* great

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-23 03:56:33
       +
      +

      *Thread Reply:* OK. Here's a mess-with-your-head moment.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-23 03:56:45
       +
      +

      *Thread Reply:* The above change works.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-23 03:57:02
       +
      +

      *Thread Reply:* If you collapse it, into eq = predicate(...).test(...);

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-23 03:57:05
       +
      +

      *Thread Reply:* the test fails

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-23 03:59:14
       +
      +

      *Thread Reply:* Ah

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-23 03:59:16
       +
      +

      *Thread Reply:* Nevermind

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-23 03:59:21
       +
      +

      *Thread Reply:* I see I have changed the behaviour of the test

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-01-23 05:07:17
       +
      +

      *Thread Reply:* in case you find it difficult or time consuming, you can remove this test i think. i don't consider this test in that form important

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-23 05:08:14
       +
      +

      *Thread Reply:* It's just a bit tough, because I don't know what the bigquery stuff is supposed to output.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-23 05:08:28
       +
      +

      *Thread Reply:* So I don't know what the expected behaviour is.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-01-23 05:08:46
       +
      +

      *Thread Reply:* i would try removing schema from json file and if it does not help we can remove the test

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-23 05:08:56
       +
      +

      *Thread Reply:* I tried a different trick

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-23 05:09:00
       +
      +

      *Thread Reply:* Namely

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-23 05:09:03
       +
      +

      *Thread Reply:* Flatten the dictionary

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-23 05:09:13
       +
      +

      *Thread Reply:* Yeah ...

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-23 05:09:28
       +
      +

      *Thread Reply:* The actual results have a lot more properties than the expected

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-23 05:10:00
       +
      +

      *Thread Reply:*

      + +
      +
      Untitled
      + + + + + + +
      + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-23 05:21:38
       +
      +

      *Thread Reply:* @Paweł Leszczyński - I think a better approach would be:

      + +
      1. Keep the test disabled
      2. Explain why it is disabled (more than what I've done so far)
      3. Create an issue to fix the test, so it isn't lost to time.
        4. +
      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-23 05:22:09
       +
      +

      *Thread Reply:* What do you think?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-01-23 05:25:45
       +
      +

      *Thread Reply:* > It's just a bit tough, because I don't know what the bigquery stuff is supposed to output. +Last time I've bumped connector version I just serialized the bigquery node and partially replaced what was there previously... I think the test is bad conceptually because we depend on some internal structure of outside connector that we don't really care about

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-01-23 05:26:13
       +
      +

      *Thread Reply:* I think we should remove it, and if we want a test to keep serialization stable we could try to serialize some fake node that we ourselves provide

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-23 05:26:18
       +
      +

      *Thread Reply:* Yeah, that's what I found I was doing with my flattener

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-01-23 05:26:23
       +
      +

      *Thread Reply:* but I think this is outside of the scope of this PR 🙂

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-23 05:26:29
       +
      +

      *Thread Reply:* I was just adding an ever increasing amount of keys to omit

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-23 05:26:39
       +
      +

      *Thread Reply:* and I just thought to myself, "If I am doing this, I'm doing something wrong"

      + + + +
      + 😄 Maciej Obuchowski +
      + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-23 05:27:19
       +
      +

      *Thread Reply:* OK.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-23 05:27:25
       +
      +

      *Thread Reply:* I'll remove the test then.

      + + + +
      + 👍 Maciej Obuchowski +
      + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-23 05:28:59
       +
      +

      *Thread Reply:* Just to clarify, you're arguing that the whole test of the LogicalPlanSerializer is flawed?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-23 05:29:23
       +
      +

      *Thread Reply:* i.e., we're serialising spark internal structures that could change without warning

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-01-23 05:31:25
       +
      +

      *Thread Reply:* I think tests like testSerializeSlicesExcessivePayload are good - we provide our own implementation of LogicalPlan and we test the structure

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-01-23 05:32:06
       +
      +

      *Thread Reply:* Even something like testSerializeInsertIntoHadoopPlan is less problematic as the InsertIntoHadoopFsRelationCommand is way more stable

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-01-23 05:32:13
       +
      +

      *Thread Reply:* but BQ connector isn't...

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-23 05:55:20
       +
      +

      *Thread Reply:* OK

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-23 05:55:25
       +
      +

      *Thread Reply:* Changes pushed to all branches

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-23 06:29:55
       +
      +

      *Thread Reply:* 🎉 'shared' was merged, thanks @Paweł Leszczyński

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-01-23 06:30:16
       +
      +

      *Thread Reply:* :medal: awesome

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-01-23 06:30:26
       +
      +

      *Thread Reply:* could u rebase other one on the list?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-01-23 06:30:39
       +
      +

      *Thread Reply:* 2385 has some conflict

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-23 06:30:39
       +
      +

      *Thread Reply:* It's odd that there are conflicts

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-23 06:30:50
       +
      +

      *Thread Reply:* Because I had telescoping branches

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-23 06:30:56
       +
      +

      *Thread Reply:* I created spark3 from shared

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-23 06:31:00
       +
      +

      *Thread Reply:* spark31 from spark3, etc

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-23 06:31:10
       +
      +

      *Thread Reply:* But sure, I'll rebase

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-01-23 06:37:58
       +
      +

      *Thread Reply:* what if you rebased spark35 '2390' ?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-01-23 06:38:09
       +
      +

      *Thread Reply:* perhaps it would be doable to merge it in one-go

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-23 06:38:40
       +
      +

      *Thread Reply:* That last one is 2391

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-23 06:38:44
       +
      +

      *Thread Reply:* the 'spark2'

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-23 06:38:59
       +
      +

      *Thread Reply:* I'm seeing a failure on spark32

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-23 06:39:13
       +
      +

      *Thread Reply:* Something to do with JAckson

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-23 06:39:33
       +
      +

      *Thread Reply:* java.lang.ExceptionInInitializerError + at io.openlineage.spark.agent.lifecycle.LibraryTest.testRdd(LibraryTest.java:112) + Caused by: com.fasterxml.jackson.databind.JsonMappingException: Scala module 2.13.2 requires Jackson Databind version &gt;= 2.13.0 and &lt; 2.14.0 - Found jackson-databind version 2.15.3 + at io.openlineage.spark.agent.lifecycle.LibraryTest.testRdd(LibraryTest.java:112)

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-01-23 06:39:41
       +
      +

      *Thread Reply:* ok, i approved 2385 and triggered integration tests on that

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-01-23 06:40:38
       +
      +

      *Thread Reply:* jackson things are always a problem

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-01-23 06:41:01
       +
      +

      *Thread Reply:* we do have jackson packaged and shaded within openlineage-java

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-23 06:41:12
       +
      +

      *Thread Reply:* Yeah - I'm trying to bisect where exactly things went wrong

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-23 06:41:27
       +
      +

      *Thread Reply:* i.e., what changes did I make in spark32, that cause these jackson errors

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-23 06:42:21
       +
      +

      *Thread Reply:* And I see 'spark31' is failing during integrationTests when run under spark 3.3.3

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-01-23 06:42:44
       +
      +

      *Thread Reply:* sometimes rerun from failed simply helps

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-23 06:43:07
       +
      +

      *Thread Reply:* I can't rerun

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-01-23 06:43:20
       +
      +

      *Thread Reply:* integration tests depend on external resources like s3 buckets / big query tables and if you run multiple instances they can fail

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-01-23 06:43:51
       +
      +

      *Thread Reply:* i can rerun it but anyway we will have to rebase each PR one by one and run tests again

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-23 06:44:04
       +
      +

      *Thread Reply:* Indeed.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-23 06:50:19
       +
      +

      *Thread Reply:* Ah, I see the approval step now.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-23 06:56:29
       +
      +

      *Thread Reply:* > we do have jackson packaged and shaded within openlineage-java +Hmm, when I dump the dependencies, I see raw Jackson dependencies. Not shaded ones.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-23 06:57:41
       +
      +

      *Thread Reply:* (The 2.13.2 migration is because I force Jackson to 2.13.2)

      + +
      +
      Untitled
      + + + + + + +
      + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-01-23 07:09:11
       +
      +

      *Thread Reply:* sry, we relocate jackson within openlineage-spark https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/build.gradle#L153

      +
      + + + + +
      <https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/build.gradle | build.gradle>
      +
      + <pre><code> relocate 'com.fasterxml.jackson', 'io.openlineage.spark.shaded.com.fasterxml.jackson' +</code></pre> +
      + + + + + + + +
      + + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
      + + +
      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-01-23 07:18:56
       +
      +

      *Thread Reply:* To expand on it more, there are two jacksons - one is version provided/required by particular Spark version, and other one used by our integration to serialize OpenLineage events. The second one needs to be shaded

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-23 07:19:56
       +
      +

      *Thread Reply:* Hmm - it might be worth creating a module that has jackson shaded already

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-23 07:20:01
       +
      +

      *Thread Reply:* ah

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-23 07:20:05
       +
      +

      *Thread Reply:* but that won't work

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-23 07:20:11
       +
      +

      *Thread Reply:* because of the openlineage-java thing

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-01-23 07:20:27
       +
      +

      *Thread Reply:* https://github.com/OpenLineage/OpenLineage/pull/2386 -> @Damien Hawes could u rebase it?

      +
      + + + + +
      #2386 [SPARK] Conform the 'spark31' module with the 'shared' and 'spark3'
      +
      + <p><strong>Summary: Conform the 'spark31' module with the rest of the migrated modules</strong> +<strong>Problem</strong></p> + +<p>Spark 3.2.x, 3.3.x, 3.4.x, and 3.5.x are compiled using Scala 2.12 and Scala 2.13. Due to a change in the Scala Collections API in Scala 2.13, NoSuchMethodErrors are thrown when running the openlineage-spack connector in an Apache Spark runtime when the runtime was compiled using Scala 2.13.</p> + +<p>Relates to: <a href="https://github.com/OpenLineage/OpenLineage/issues/2303">#2303</a></p> + +<p><strong>Solution</strong></p> + +<p>This PR is the <strong>6th</strong> of several PRs to support producing Scala 2.12 and Scala 2.13 variants of the OpenLineage Spark integration.</p> + +<p>In this PR, we conform the spark31 module with the shared and spark3 modules</p> + +<p><strong>Checklist</strong></p> + +<p>☑︎ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☑︎ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☑︎ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☑︎ Your comment includes a one-liner for the changelog about the specific purpose of the change (<em>if necessary</em>) +☑︎ You've added a <a href="https://github.com/OpenLineage/OpenLineage/tree/main/.github/header_templates.md">header</a> to source files (<em>if relevant</em>)</p> + +<hr /> + +<p>SPDX-License-Identifier: Apache-2.0<br /> +Copyright 2018-2023 contributors to the OpenLineage project</p> +
      + +
      +
      Labels
      + documentation, integration/spark +
      + + + + + + + +
      + + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
      + + +
      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-23 07:21:31
       +
      +

      *Thread Reply:* Done

      + + + +
      + 👍 Paweł Leszczyński +
      + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-01-23 07:21:38
       +
      +

      *Thread Reply:* i think there was some jackson related hack that is necessary for serializing logical plan bcz we need to serialize logical plan's with jackson of the spark

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-23 07:21:55
       +
      +

      *Thread Reply:* Yeah, the LogicalPlanSerializer

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-23 07:22:01
       +
      +

      *Thread Reply:* does some reflection things

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-01-23 07:22:51
       +
      +

      *Thread Reply:* if we want to serialize plan node in a given scala version, we need to run it with jackson of the same scala version

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-01-23 07:23:18
       +
      +

      *Thread Reply:* but jackson attached to spark is missing jackson-databind (if i remember it right)

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-23 05:58:31
       +
      +

      @Jakub Dardziński - ~how does the CI/CD pipeline now work? I see that the "run steps" aren't present when I push. Is this intended?~ Nevermind, it seems it just took a while before Circle CI spun them up and GitHub showed them as running.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Jakub Dardziński + (jakub.dardzinski@getindata.com) +
      +
      2024-01-23 06:10:11
       +
      +

      *Thread Reply:* the change is that now integration tests from forked repos require approval

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-23 09:59:37
       +
      +

      I don't understand. I ran the failing integration test of spark31 locally, it failed. I ran it again, it passed.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-01-23 10:09:45
       +
      +

      *Thread Reply:* it's possible that spark has already cleared the QueryExecution from the executionIdToQueryExecution in the SQLExecution class if the job runs very fast

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-01-23 10:10:02
       +
      +

      *Thread Reply:* (if the reason for failure was that START event was not send)

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-23 10:17:52
       +
      +

      *Thread Reply:* Any idea how I debug the integration tests via intellij?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-23 10:18:07
       +
      +

      *Thread Reply:* Each time I try and run the ColumnLineageIntegrationTest it only runs the unit tests.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-23 10:18:28
       +
      +

      *Thread Reply:* I've told intellij to use the integrationTest task, but it's like, "nope"

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-23 10:18:44
       +
      +

      *Thread Reply:* and just runs the unit tests any way

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-01-23 10:18:58
       +
      +

      *Thread Reply:* Can you check if it's integrationTest in run configurations?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-23 10:19:22
       +
      +

      *Thread Reply:* This is what its running:

      + +

      :app:integrationTest --tests "io.openlineage.spark.agent.ColumnLineageIntegrationTest" -Pspark.version=3.3.1

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-23 09:59:46
       +
      +

      Send help.

      + + + +
      + 😂 Harel Shein, Paweł Leszczyński, Maciej Obuchowski +
      + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-23 10:08:04
       +
      +

      OK - I found one:

      + +
      all > io.openlineage.spark.agent > ColumnLineageIntegrationTest
+columnLevelLineageSingleDestinationTest(
+
      + + + +
      +
      +
      +
      + + + + + + + +
      +
      + + + + +
      + +
      jayant joshi + (itsjayantjoshi@gmail.com) +
      +
      2024-01-24 01:10:44
       +
      +

      @jayant joshi has joined the channel

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-01-24 02:41:46
       +
      +

      @Damien Hawes https://github.com/OpenLineage/OpenLineage/pull/2386/files#r1464456420

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-24 03:18:25
       +
      +

      *Thread Reply:* Hmm

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-24 03:18:30
       +
      +

      *Thread Reply:* Good hypothesis

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-24 03:36:38
       +
      +

      *Thread Reply:* If I run ./gradlew app:test -Pspark.version=3.3.1

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-24 03:36:52
       +
      +

      *Thread Reply:* I get this failure:

      + +

      Execution failed for task ':app:compileJava'. +&gt; Could not resolve all files for configuration ':app:compileClasspath'. + &gt; Could not resolve com.fasterxml.jackson:jackson_bom:2.13.4.1. + Required by: + project :app &gt; org.apache.spark:spark_core_2.12:3.3.1 &gt; com.fasterxml.jackson.core:jackson_databind:2.13.4.1 + &gt; Could not resolve com.fasterxml.jackson:jackson_bom:2.13.4.1. + &gt; Could not parse POM <https://astronomer.jfrog.io/artifactory/maven-public-libs-snapshot/com/fasterxml/jackson/jackson-bom/2.13.4.1/jackson-bom-2.13.4.1.pom> + &gt; Already seen doctype. + &gt; Could not resolve com.fasterxml.jackson:jackson_bom:2.13.4.1. + Required by: + project :app &gt; org.apache.spark:spark_core_2.12:3.3.1 &gt; org.apache.avro:avro:1.11.0 &gt; com.fasterxml.jackson.core:jackson_core:2.13.4 + project :app &gt; org.apache.spark:spark_core_2.12:3.3.1 &gt; org.apache.spark:spark_kvstore_2.12:3.3.1 &gt; com.fasterxml.jackson.core:jackson_annotations:2.13.4 + &gt; Could not resolve com.fasterxml.jackson:jackson_bom:2.13.4.1. + &gt; Could not parse POM <https://astronomer.jfrog.io/artifactory/maven-public-libs-snapshot/com/fasterxml/jackson/jackson-bom/2.13.4.1/jackson-bom-2.13.4.1.pom> + &gt; Already seen doctype.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-24 03:37:42
       +
      +

      *Thread Reply:* spark-core requires 2.13.4.1

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-24 03:38:06
       +
      +

      *Thread Reply:* Suggestion: bump the spark versions in the CI/CD configuration to the latest patch versions

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-01-24 03:40:37
       +
      +

      *Thread Reply:* https://issues.apache.org/jira/browse/SPARK-40886 +https://spark.apache.org/releases/spark-release-3-3-2.html

      + +

      something was changed in 3.3.2

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-01-24 03:40:45
       +
      +

      *Thread Reply:* why do we have 3.3.1 ?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-24 03:41:23
       +
      +

      *Thread Reply:* We might not

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-24 03:41:28
       +
      +

      *Thread Reply:* I had a look at the circle ci

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-24 03:41:31
       +
      +

      *Thread Reply:* and I saw 3.3.3

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-24 03:41:39
       +
      +

      *Thread Reply:* but I could have sworn I saw 3.3.1 in one of the tests

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-01-24 03:42:35
       +
      +

      *Thread Reply:* 3.3.1 is in spark33

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-01-24 03:42:55
       +
      +

      *Thread Reply:* and the module has some tests

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-24 03:49:28
       +
      +

      *Thread Reply:* OK

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-24 03:49:33
       +
      +

      *Thread Reply:* I'm going to push to spark35

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-24 03:49:40
       +
      +

      *Thread Reply:* with changes to circle ci

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-24 03:49:51
       +
      +

      *Thread Reply:* and also updating the gradle.properties for all branches that came before it

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-24 03:50:22
       +
      +

      *Thread Reply:* along with removing the jackson override in app/build.gradle

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-24 03:50:41
       +
      +

      *Thread Reply:* and if that passes, we can merge spark35 in, and take everything else with. Does that sound OK, @Paweł Leszczyński?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-01-24 03:50:55
       +
      +

      *Thread Reply:* yes

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-24 03:55:31
       +
      +

      *Thread Reply:* OK

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-24 03:55:34
       +
      +

      *Thread Reply:* Pushed changes for 35

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-24 04:25:03
       +
      +

      *Thread Reply:* Seems bitnami doesn't produce a 2.4.8 image

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-24 04:25:07
       +
      +

      *Thread Reply:* Last is 2.4.6

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-24 04:25:12
       +
      +

      *Thread Reply:* Reverting to 2.4.6

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-24 04:43:10
       +
      +

      *Thread Reply:* @Paweł Leszczyński - could you approve the CI/CD pipeline for the spark integration test tasks?

      + + + +
      + 👀 Paweł Leszczyński +
      + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-24 04:43:16
       +
      +

      *Thread Reply:* https://github.com/OpenLineage/OpenLineage/pull/2390

      +
      + + + + +
      #2390 [SPARK] Migrated 'spark35' module to produce Scala 2.12 and Scala 2.13 variants
      +
      + <p><strong>Summary: Migrated 'spark35' module to produce Scala 2.12 and Scala 2.13 variants</strong> +<strong>Problem</strong></p> + +<p>Spark 3.2.x, 3.3.x, 3.4.x, and 3.5.x are compiled using Scala 2.12 and Scala 2.13. Due to a change in the Scala Collections API in Scala 2.13, NoSuchMethodErrors are thrown when running the openlineage-spack connector in an Apache Spark runtime when the runtime was compiled using Scala 2.13.</p> + +<p>Relates to: <a href="https://github.com/OpenLineage/OpenLineage/issues/2303">#2303</a></p> + +<p><strong>Solution</strong></p> + +<p>This PR is the <strong>10th</strong> of several PRs to support producing Scala 2.12 and Scala 2.13 variants of the OpenLineage Spark integration.</p> + +<p>In this PR, we migrate the 'spark35' module to use the refactored Gradle plugins.</p> + +<p>The module now produces 3 JARS; one default one, one compiled using the Scala 2.12 variant of Apache Spark, and one compiled using the Scala 2.13 variant of Apache Spark.</p> + +<p><strong>Checklist</strong></p> + +<p>☑︎ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☑︎ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☑︎ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☑︎ Your comment includes a one-liner for the changelog about the specific purpose of the change (<em>if necessary</em>) +☑︎ You've added a <a href="https://github.com/OpenLineage/OpenLineage/tree/main/.github/header_templates.md">header</a> to source files (<em>if relevant</em>)</p> + +<hr /> + +<p>SPDX-License-Identifier: Apache-2.0<br /> +Copyright 2018-2023 contributors to the OpenLineage project</p> +
      + +
      +
      Labels
      + documentation, integration/spark, ci, tests +
      + + + + + + + +
      + + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
      + + +
      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-24 04:59:41
       +
      +

      *Thread Reply:* > GoogleCloudIntegrationTest

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-01-24 05:00:23
       +
      +

      *Thread Reply:* i know this

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-01-24 05:00:45
       +
      +

      *Thread Reply:* it's @Maciej Obuchowski stuff :face_palm:

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-01-24 05:02:24
       +
      +

      *Thread Reply:* let me work first on upgrading spark branches to latest patch versions

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-01-24 05:02:32
       +
      +

      *Thread Reply:* will come back to you

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-01-24 05:47:38
       +
      +

      *Thread Reply:* :gh_merged:

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-01-24 05:55:47
       +
      +

      @Damien Hawes are the PRs for spark31-34 valid anymore? can you rebase spark2?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-24 06:46:52
       +
      +

      *Thread Reply:* No - those PRs can be closed, because spark35 built on top of 34, which built on top of 33, etc.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-24 06:48:41
       +
      +

      *Thread Reply:* Rebasing spark2 now

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-24 06:50:49
       +
      +

      *Thread Reply:* OK. PRs are closed, with a link to 2390

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-24 06:56:14
       +
      +

      *Thread Reply:* @Paweł Leszczyński - spark2 has been rebased and pushed.

      + + + +
      + 👍 Paweł Leszczyński +
      + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-01-24 07:09:11
       +
      +

      *Thread Reply:* &gt; Task :app:compileTestJava FAILED

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-24 07:09:29
       +
      +

      *Thread Reply:* 🤨

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-01-24 07:09:36
       +
      +

      *Thread Reply:* https://app.circleci.com/pipelines/github/OpenLineage/OpenLineage/9067/workflows/97c9d796-f67d-47ee-8adc-f353306ae353/jobs/167350

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-01-24 07:09:46
       +
      +

      *Thread Reply:* i expected it to fail but it's kind of early

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-24 07:10:47
       +
      +

      *Thread Reply:* Running it locally.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-24 07:11:03
       +
      +

      *Thread Reply:* Interesting.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-24 07:11:32
       +
      +

      *Thread Reply:* > Task :spark2:compileTestJava FAILED +OptimizedCreateHiveTableAsSelectCommandVisitorTest.java:36: error: cannot find symbol +import org.apache.spark.sql.hive.execution.OptimizedCreateHiveTableAsSelectCommand; + ^ + symbol: class OptimizedCreateHiveTableAsSelectCommand + location: package org.apache.spark.sql.hive.execution +1 error

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-01-24 07:11:52
       +
      +

      *Thread Reply:* perhaps it's in 2.4.8

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-01-24 07:11:57
       +
      +

      *Thread Reply:* was this changed?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-24 07:12:10
       +
      +

      *Thread Reply:* I reverted back to 2.4.6

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-24 07:12:15
       +
      +

      *Thread Reply:* To match the spark image

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-24 07:12:21
       +
      +

      *Thread Reply:* It was always 2.4.6

      + + + +
      + 👍 Paweł Leszczyński +
      + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-01-24 07:13:17
       +
      +

      *Thread Reply:* wasn't it compiled with 2.4.8?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-01-24 07:13:36
       +
      +

      *Thread Reply:* sparkVersion = '2.4.8' it was

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-24 07:13:46
       +
      +

      *Thread Reply:* lol

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-24 07:13:53
       +
      +

      *Thread Reply:* But integration test sits at 2.4.6

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-24 07:13:54
       +
      +

      *Thread Reply:* Nice

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-01-24 07:14:00
       +
      +

      *Thread Reply:* we compiled it with 2.4.8 but run integration tests based on 2.4.6

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-24 07:14:22
       +
      +

      *Thread Reply:* Re-running with 2.4.8

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-24 07:15:07
       +
      +

      *Thread Reply:* Yeah

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-24 07:15:13
       +
      +

      *Thread Reply:* Things going through with 2.4.8

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-24 07:15:17
       +
      +

      *Thread Reply:* Look at that

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-24 07:15:28
       +
      +

      *Thread Reply:* Apache Spark introducing new features in patch versions

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-24 07:20:15
       +
      +

      *Thread Reply:* @Paweł Leszczyński - pushed the changes, along with changes to the CI process

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-24 07:20:21
       +
      +

      *Thread Reply:* Gradle will now use plain logs

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-24 07:20:30
       +
      +

      *Thread Reply:* Which means it should be easier to see things

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-24 07:20:44
       +
      +

      *Thread Reply:* i.e., I found it difficult to figure out which test is failing

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-24 07:20:48
       +
      +

      *Thread Reply:* Unless it was near the bottom

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-24 07:41:35
       +
      +

      *Thread Reply:* OK - I need to step out for a bit

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-24 07:41:43
       +
      +

      *Thread Reply:* I'll handle this issue when I am back

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-24 07:41:51
       +
      +

      *Thread Reply:* app:test fails under spark 2.4.6

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-01-24 08:40:30
       +
      +

      *Thread Reply:* OptimizedCreateHiveTableAsSelectCommand +was added in 2.4.7 or 2.4.8 AFAIK

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-01-24 08:43:31
       +
      +

      *Thread Reply:* > we compiled it with 2.4.8 but run integration tests based on 2.4.6 +The reason was lack of Docker image for 2.4.8, right?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-24 10:51:26
       +
      +

      *Thread Reply:* Yes

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-24 11:48:14
       +
      +

      *Thread Reply:* OK - I've managed to fix the issue with app:compileTestJava

      + + + +
      + ❤️ Maciej Obuchowski, Paweł Leszczyński +
      + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-24 11:48:22
       +
      +

      *Thread Reply:* and the subsequent failures that came afterwards.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-24 12:34:26
       +
      +

      *Thread Reply:* @Paweł Leszczyński - if you're able to, can you approve the integration tests for: https://github.com/OpenLineage/OpenLineage/pull/2391

      +
      + + + + +
      #2391 [SPARK] Conform the 'spark2' module to the new build process
      +
      + <p><strong>Summary: Conform the 'spark32' module with the rest of the migrated modules</strong> +<strong>Problem</strong></p> + +<p>Spark 3.2.x, 3.3.x, 3.4.x, and 3.5.x are compiled using Scala 2.12 and Scala 2.13. Due to a change in the Scala Collections API in Scala 2.13, NoSuchMethodErrors are thrown when running the openlineage-spack connector in an Apache Spark runtime when the runtime was compiled using Scala 2.13.</p> + +<p>Relates to: <a href="https://github.com/OpenLineage/OpenLineage/issues/2303">#2303</a></p> + +<p><strong>Solution</strong></p> + +<p>This PR is the <strong>11th</strong> of several PRs to support producing Scala 2.12 and Scala 2.13 variants of the OpenLineage Spark integration.</p> + +<p>In this PR, we conform the spark2 module with the other migrated modules</p> + +<p><strong>Checklist</strong></p> + +<p>☑︎ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☑︎ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☑︎ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☑︎ Your comment includes a one-liner for the changelog about the specific purpose of the change (<em>if necessary</em>) +☑︎ You've added a <a href="https://github.com/OpenLineage/OpenLineage/tree/main/.github/header_templates.md">header</a> to source files (<em>if relevant</em>)</p> + +<hr /> + +<p>SPDX-License-Identifier: Apache-2.0<br /> +Copyright 2018-2023 contributors to the OpenLineage project</p> +
      + +
      +
      Labels
      + documentation, integration/spark, ci +
      + +
      +
      Comments
      + 2 +
      + + + + + + + +
      + + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
      + + +
      + + + +
      + 👀 Paweł Leszczyński +
      + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-01-25 02:49:37
       +
      +

      *Thread Reply:* there is still one test for 2.4.8 spark_kafka that is not passing +Caused by: java.lang.NoSuchMethodError: org.apache.spark.internal.Logging.$init$(Lorg/apache/spark/internal/Logging;)V

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-25 08:46:50
       +
      +

      *Thread Reply:* Yeah

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-25 08:47:13
       +
      +

      *Thread Reply:* with @Mattia Bertorello’s help, we discovered the source of the problem

      + + + +
      + 🚀 Mattia Bertorello +
      + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-25 08:47:26
       +
      +

      *Thread Reply:* It's a Scala binary problem.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-25 08:47:55
       +
      +

      *Thread Reply:* Bitnami's image is Scala 2.11 for 2.4.6

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-25 08:49:05
       +
      +

      *Thread Reply:* We are pushing org.apache.spark:spark_sql_kafka_0_10_2.12:2.4.6 to the image

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-01-25 08:52:55
       +
      +

      *Thread Reply:* can we use Scala 2.11 for 2.4? I think most people use 2.11 for it

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-01-25 08:53:19
       +
      +

      *Thread Reply:* and I would really try to not push anything new there 🙂 if you use 2.4 your priority should be to upgrade

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-25 08:56:48
       +
      +

      *Thread Reply:* Yeah - I reverted back to 2.11

      + + + +
      + 👍 Maciej Obuchowski +
      + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-25 10:15:38
       +
      +

      *Thread Reply:* OK. Pushed the changes to the branch

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-01-25 10:37:10
       +
      +

      *Thread Reply:* it failed 😞

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-25 11:56:08
       +
      +

      *Thread Reply:* OK - https://github.com/OpenLineage/OpenLineage/pull/2391

      + +

      Is waiting at the integration test phase

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-01-25 12:11:11
       +
      +

      *Thread Reply:* approved 🙂

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-25 12:15:50
       +
      +

      *Thread Reply:* Btw the airflow integration test is waiting for approval.

      + + + +
      + 👀 Paweł Leszczyński +
      + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-01-26 02:45:21
       +
      +

      *Thread Reply:* :gh_merged:

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-01-25 12:11:44
       +
      +

      @Damien Hawes is this last required PR? We're thinking about releasing soon to get this feature out fast

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-25 12:13:11
       +
      +

      *Thread Reply:* Nope.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-25 12:13:20
       +
      +

      *Thread Reply:* There are a few more things to do. The 'app' module for instance needs to be migrated and that's a tough one, because it has integration tests that need to be 'ported'. For instance, Scala 2.13 variants of the integration tests.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-25 12:15:28
       +
      +

      *Thread Reply:* I don't feel comfortable saying, "Hey, let's release 2.12 and 2.13" without those checks and balances being in place.

      + + + +
      + 👍 Maciej Obuchowski +
      + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-26 05:52:16
       +
      +

      @Paweł Leszczyński @Maciej Obuchowski - is it possible for OpenLineage to use the GitHub container registry, to host custom docker images, i.e., the Scala 2.13 variants of Apache Spark.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-01-26 06:00:04
       +
      +

      *Thread Reply:* I think the limits are very low for free accounts? https://docs.github.com/en/get-started/learning-about-github/githubs-plans#github-free-for-organizations

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-01-26 06:00:21
       +
      +

      *Thread Reply:* I've used quay.io for free hosting of docker images

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-26 06:22:00
       +
      +

      *Thread Reply:* > • 500 MB GitHub Packages storage +Wow.

      + +

      1 spark image will blow that storage limit out of the water.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-26 06:22:42
       +
      +

      *Thread Reply:* Trust me, I know. I've built several Spark images before. They come in at like 1 GB - 1.5 GB per image.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-01-26 06:29:37
       +
      +

      *Thread Reply:* they give 50gb in github enterprise. this would not make any sense...

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-01-26 06:34:03
       +
      +

      *Thread Reply:* https://github.com/features/packages#pricing

      + +
      +
      Screenshot 2024-01-26 at 12.33.24.png
      + + + + + + + + +
      + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-26 06:34:45
       +
      +

      *Thread Reply:* Aye - but what's the limit on the size of the packages? 🙂

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-01-26 06:38:17
       +
      +

      *Thread Reply:* we might be able to use Google's container registry too https://cloud.google.com/artifact-registry

      +
      +
      Google Cloud
      + + + +
      Artifact Registry  |  Google Cloud
      +
      + The next generation of Container Registry. Store, manage, and secure your build artifacts. +
      + + + + + + + + +
      Original URL: https://cloud.google.com/artifact-registry
      + + + +
      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-01-26 06:38:48
       +
      +

      *Thread Reply:* although I'm not sure our account has the permissions

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-01-29 05:24:48
       +
      +

      @Paweł Leszczyński @Maciej Obuchowski - what's the possibility of temporarily dropping support for Spark 2.4.x? Reasoning:

      + +

      We run integration tests against 2.4.x, 3.2.x, 3.3.x, 3.4.x, and 3.5.x. Of them 2.4 is the only one that doesn't support Scala 2.13.

      + +

      It really complicates the build process of 'app'.

      + +

      Furthermore, 'shared' is compiled using Spark 3.2.4, and it is the first version to support Scala 2.13.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-01-29 05:42:49
       +
      +

      *Thread Reply:* I don't think it's possible for quite a bit of time still. There are people on, like EMR 5.X versions which uses 2.4.8 and is still supported, that are using OL

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-01-29 05:43:34
       +
      +

      *Thread Reply:* I wish we could tho 🙂

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Mattia Bertorello + (mattia.bertorello@booking.com) +
      +
      2024-01-30 10:06:21
       +
      +

      Hi team, I opened a PR about start moving gradually the code of so called vendor like Snowflake, BigQuery, Iceberg, Delta, Databricks in to a separate project and using the service loader to load the objects base on the runtime availability of the vendor itself. +I started with the easy one Snowflake and I already have a branch for Iceberg that is the one brings some issues during the Scala 2.13 support work.

      + +

      So now I find out a similar effort to separate this code is in progress in another PR #2272 +@Maciej Obuchowski @Paweł Leszczyński Do you know an ETA for this PR #2272 been release and if has exactly the same scope of mine? +In general my idea was not changing too much the code to reduce too big PR and moving incrementally.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Mattia Bertorello + (mattia.bertorello@booking.com) +
      +
      2024-01-30 10:06:57
       +
      +

      *Thread Reply:* Also I think there is some conflict with the @Damien Hawes works there.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-01-30 12:18:02
       +
      +

      *Thread Reply:* I haven't fully reviewed your work, and I think your solution is really nice, but I think there are some significant differences between your approach @Mattia Bertorello and @Paweł Leszczyński’s one.

      + +

      First, is that Paweł's interface is segregated from rest of OpenLineage-Spark integration, and OpenLineage-Spark depends on it. Second, is that we're trying to move the responsibility of maintaining lineage from the external DataSource implementations to those implementations - and we're getting good responses from for example BigQuery connector maintainer: https://github.com/OpenLineage/OpenLineage/issues/2349

      + +

      I guess on the other way, Paweł's approach does not really solve your issue which is Scala compatibility and ease of compilation.

      + +

      So I think we want to have both Paweł's approach as a public API, and your solution as a stop-gap to support the Scala 2.13 support as well? I guess to make it more precise - I would prefer to have Paweł's solution as a public API, while your as internal, not advertised one - I would prefer vendors to implement support in their own code, rather than provide us with implementation that would make us responsible for maintaining it.

      + +

      What do you think?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Mattia Bertorello + (mattia.bertorello@booking.com) +
      +
      2024-01-31 08:21:37
       +
      +

      *Thread Reply:* > I would prefer vendors to implement support in their own code, rather than provide us with implementation that would make us responsible for maintaining it. +It makes sense to move the responsibility to them and the work in Paweł's PR; It's great to reach this goal.

      + +

      Mine was more for the short term, and yes, it can be considered a first step towards that goal. In this way, we will already have all the code from the vendors separated into projects, and the only work is to adapt to the new interface without changing the core.

      + +

      btw could you trigger the integration tests? Thanks! +https://app.circleci.com/pipelines/github/OpenLineage/OpenLineage/9118/workflows/cfb67fa1-db0e-4043-a73e-14de40968534

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-01-31 10:15:07
       +
      +

      *Thread Reply:* Added a few comments to PR 🙂

      + + + +
      + :gratitude_thank_you: Mattia Bertorello +
      + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Mattia Bertorello + (mattia.bertorello@booking.com) +
      +
      2024-01-31 10:17:49
       +
      +

      *Thread Reply:* Let me work on them in these days 🙂

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Julien Le Dem + (julien@apache.org) +
      +
      2024-01-30 12:11:38
       +
      +

      @Julien Le Dem has joined the channel

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      tati + (tatiana.alchueyr@astronomer.io) +
      +
      2024-01-30 12:13:13
       +
      +

      @tati has joined the channel

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-02 08:39:32
       +
      +

      @Paweł Leszczyński @Maciej Obuchowski - just curious, how certain are we that Spark 2 classes (CreateTableLikeCommand) are never created in Spark 3.2.x - Spark 3.5.x applications?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-02 08:43:12
       +
      +

      *Thread Reply:* there's class io.openlineage.spark3.agent.lifecycle.plan.CreateTableLikeCommandVisitor in spark3

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-02 08:44:52
       +
      +

      *Thread Reply:* Hmmm - so if the listener gets a CreateTableLikeCommand how does it know which CreateTableLikeCommandVisitor to delegate to?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-02 08:44:54
       +
      +

      *Thread Reply:* Oh wait.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-02 08:44:56
       +
      +

      *Thread Reply:* The factories.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-02-02 08:47:54
       +
      +

      *Thread Reply:* 🙂

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-05 05:26:43
       +
      +

      @Paweł Leszczyński - with the PySpark integration tests, does the presence of "RUNNING" events cause the assertions to fail or not?

      + +

      I'm encountering a failure in the Kafka read write test, and I see additional running events, which makes me wonder what is being asserted for, except for the certain properties.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-05 05:27:29
       +
      +

      *Thread Reply:* in most cases, we don't emit nor assert running events

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-05 05:27:56
       +
      +

      *Thread Reply:* it makes more sense to check running for streaming jobs

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-05 05:29:46
       +
      +

      *Thread Reply:* what's the test you have problem with?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-05 05:39:36
       +
      +

      *Thread Reply:* SparkContainerIntegrationTest#testPysparkKafkaReadWrite

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-05 05:39:49
       +
      +

      *Thread Reply:* I removed the facets, to make the JSONs smaller

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-05 05:40:14
       +
      +

      *Thread Reply:* and I see that the run events are missing their input and output datasets.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-05 06:28:31
       +
      +

      *Thread Reply:* Finally. I see the exception

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-05 06:30:58
       +
      +

      *Thread Reply:* @Paweł Leszczyński have a look:

      + +

      https://gist.github.com/d-m-h/95a8005063478c4844abd78f4eaf4e47

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-05 06:50:34
       +
      +

      *Thread Reply:* why doesn't this go into +if (KafkaRelationVisitor.isKafkaSource(command.dataSource())) +condtition?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-05 06:51:18
       +
      +

      *Thread Reply:* I'm not sure. I tested with the original bitnami spark 3.2.4 scala 2.12

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-05 06:51:22
       +
      +

      *Thread Reply:* and I get the same error

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-05 06:53:19
       +
      +

      *Thread Reply:* I see that KafkaRelationVisitor is in the uber jar

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-05 06:53:28
       +
      +

      *Thread Reply:* jar tvf openlineage-spark-app_2.13-1.9.0-SNAPSHOT.jar | grep Kafka + 11382 Mon Feb 05 11:44:56 CET 2024 io/openlineage/spark/agent/lifecycle/plan/KafkaRelationVisitor.class + 1739 Mon Jan 22 16:48:56 CET 2024 io/openlineage/client/transports/KafkaConfig.class + 1626 Mon Jan 22 16:48:56 CET 2024 io/openlineage/client/transports/KafkaTransportBuilder.class + 3594 Mon Jan 22 16:48:56 CET 2024 io/openlineage/client/transports/KafkaTransport.class

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-05 07:05:30
       +
      +

      *Thread Reply:* Found the reason:

      + +

      public static boolean hasKafkaClasses() { + log.debug("Checking for Kafka classes"); + try { + KafkaRelationVisitor.class + .getClassLoader() + .loadClass("org.apache.spark.sql.kafka010.KafkaSourceProvider"); + log.debug("Kafka classes found"); + return true; + } catch (Exception e) { + log.debug("Kafka classes not found"); + return false; + } +} +2024-02-05 12:04:11 DEBUG io.openlineage.spark.agent.lifecycle.plan.KafkaRelationVisitor: Checking for Kafka classes +2024-02-05 12:04:11 DEBUG io.openlineage.spark.agent.lifecycle.plan.KafkaRelationVisitor: Kafka classes not found

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-05 07:06:20
       +
      +

      *Thread Reply:* It seems when the Kafka jars are loaded via --packages - we can't find the class.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-05 07:14:50
       +
      +

      *Thread Reply:* OK. I changed the check to this:

      + +

      Thread.currentThread() + .getContextClassLoader() + .loadClass("org.apache.spark.sql.kafka010.KafkaSourceProvider");

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-05 07:15:32
       +
      +

      *Thread Reply:* I see this:

      + +

      2024-02-05 12:14:19 DEBUG io.openlineage.spark.agent.lifecycle.plan.KafkaRelationVisitor: Checking for Kafka classes +2024-02-05 12:14:19 DEBUG io.openlineage.spark.agent.lifecycle.plan.KafkaRelationVisitor: Kafka classes found +2024-02-05 12:14:19 DEBUG io.openlineage.spark.agent.lifecycle.plan.KafkaRelationVisitor: Checking for Kafka classes +2024-02-05 12:14:19 DEBUG io.openlineage.spark.agent.lifecycle.plan.KafkaRelationVisitor: Kafka classes found

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-05 07:16:07
       +
      +

      *Thread Reply:* However:

      + +

      java.lang.NoClassDefFoundError: org/apache/spark/sql/kafka010/KafkaSourceProvider + at io.openlineage.spark.agent.lifecycle.plan.KafkaRelationVisitor.isKafkaSource(KafkaRelationVisitor.java:72) +java.lang.NoClassDefFoundError: org/apache/spark/sql/kafka010/KafkaSourceProvider + at io.openlineage.spark.agent.lifecycle.plan.KafkaRelationVisitor.isKafkaSource(KafkaRelationVisitor.java:72) + at io.openlineage.spark.agent.lifecycle.plan.SaveIntoDataSourceCommandVisitor.apply(SaveIntoDataSourceCommandVisitor.java:97) + at io.openlineage.spark.agent.lifecycle.plan.SaveIntoDataSourceCommandVisitor.apply(SaveIntoDataSourceCommandVisitor.java:46)

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-05 07:17:11
       +
      +

      *Thread Reply:* recalling folks from microsoft had similar problem long time ago here -> https://github.com/OpenLineage/OpenLineage/blob/bba64fbddbea3557fc2342a3d2fbd4194b[…]penlineage/spark/agent/lifecycle/plan/KustoRelationVisitor.java

      +
      + + + + +
      <https://github.com/OpenLineage/OpenLineage/blob/bba64fbddbea3557fc2342a3d2fbd4194b508fc3/integration/spark/shared/src/main/java/io/openlineage/spark/agent/lifecycle/plan/KustoRelationVisitor.java | KustoRelationVisitor.java>
      +
      + <pre><code>public class KustoRelationVisitor&lt;D extends OpenLineage.Dataset&gt; +</code></pre> +
      + + + + + + + +
      + + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
      + + +
      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-05 07:18:11
       +
      +

      *Thread Reply:* lmao

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-05 07:18:15
       +
      +

      *Thread Reply:* I was about to do just that

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-05 07:20:07
       +
      +

      *Thread Reply:* Just for curiosity's sake:

      + +

      public static boolean isKafkaSource(CreatableRelationProvider provider) { + log.debug("Checking if provider is KafkaSourceProvider"); + if (!hasKafkaClasses()) { + return false; + } + log.debug("Checking if the provider is a KafkaSourceProvider. {}", provider.getClass().getCanonicalName()); + boolean check = provider instanceof KafkaSourceProvider; + log.debug("Is the provider a KafkaSourceProvider? {}", check); + return check; + }

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-05 07:20:31
       +
      +

      *Thread Reply:* 2024-02-05 12:19:33 DEBUG io.openlineage.spark.agent.lifecycle.plan.KafkaRelationVisitor: Checking if provider is KafkaSourceProvider +2024-02-05 12:19:33 DEBUG io.openlineage.spark.agent.lifecycle.plan.KafkaRelationVisitor: Checking for Kafka classes +2024-02-05 12:19:33 DEBUG io.openlineage.spark.agent.lifecycle.plan.KafkaRelationVisitor: Kafka classes found +2024-02-05 12:19:33 DEBUG io.openlineage.spark.agent.lifecycle.plan.KafkaRelationVisitor: Checking if the provider is a KafkaSourceProvider. org.apache.spark.sql.kafka010.KafkaSourceProvider +2024-02-05 12:19:33 INFO io.openlineage.spark.agent.util.PlanUtils: apply method failed with +java.lang.NoClassDefFoundError: org/apache/spark/sql/kafka010/KafkaSourceProvider

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-05 07:20:31
       +
      +

      *Thread Reply:* Wild

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-05 07:20:43
       +
      +

      *Thread Reply:* The class says that it is that instance

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-05 07:20:55
       +
      +

      *Thread Reply:* but the instance check says "no, I'm not dealing with you"

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-02-05 07:37:09
       +
      +

      *Thread Reply:* Different classloaders?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-05 07:37:18
       +
      +

      *Thread Reply:* Yup

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-05 07:37:32
       +
      +

      *Thread Reply:* The main class loader loads the openlineage-spark jar

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-05 07:37:48
       +
      +

      *Thread Reply:* and another class loader loads the kafka jars

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-05 07:38:13
       +
      +

      *Thread Reply:* Thread.getCurrentThread().getContextClassLoader().loadClass(<name>) does the trick

      + + + +
      + 👍 Maciej Obuchowski +
      + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-02-05 07:39:17
       +
      +

      *Thread Reply:* maybe we can generalize the solution? +ClassloaderUtils.loadClass +that tries to load it from both classloaders?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-02-05 07:39:42
       +
      +

      *Thread Reply:* The idea is to not make next person debug the same issue with another connector 🙂

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-05 11:11:56
       +
      +

      *Thread Reply:* Well - that was annoying.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-05 11:12:05
       +
      +

      *Thread Reply:* But, it seems like the kafka tests pass now

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-02-05 11:27:26
       +
      +

      @Mattia Bertorello it's failing on spotless 🙂

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Mattia Bertorello + (mattia.bertorello@booking.com) +
      +
      2024-02-05 11:29:15
       +
      +

      *Thread Reply:* Yeah sorry, I apply some IntelliJ suggestions 😅

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Mattia Bertorello + (mattia.bertorello@booking.com) +
      +
      2024-02-05 11:33:43
       +
      +

      *Thread Reply:* Regarding the ServiceLoader problem, I implemented a static version with a classic class loader. +My main concern is that we cannot ensure loose coupling between the vendor projects and the app/spark3 projects if we put the class name inside those. In this case, we need to be aware of the implementations.

      + +

      Here the code +```public interface Vendors { + List<String> VENDORS = + Arrays.asList("io.openlineage.spark.agent.vendor.snowflake.SnowflakeVendor");

      + +

      static Vendors getVendors() { + ClassLoader cl = Thread.currentThread().getContextClassLoader();

      + +
      List&lt;Vendor&gt; vendors =
+    VENDORS.stream()
+        .map(
+            vendorClassName -&gt; {
+              try {
+                Class&lt;?&gt; vendor = cl.loadClass(vendorClassName);
+                return (Vendor) vendor.newInstance();
+              } catch (ClassNotFoundException
+                  | InstantiationException
+                  | IllegalAccessException e) {
+                return null;
+              }
+            })
+        .filter(Objects::nonNull)
+        .filter(Vendor::isVendorAvailable)
+        .collect(Collectors.toList());```
+
      + + + +
      + 👍 Maciej Obuchowski +
      + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Mattia Bertorello + (mattia.bertorello@booking.com) +
      +
      2024-02-06 06:48:04
       +
      +

      *Thread Reply:* @Paweł Leszczyński Do you think this version is better? +I can merge it so we can unblock the PR and maybe merge it. + @Maciej Obuchowski do you think your comments are resolved, or is there something else to do?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-02-06 06:54:17
       +
      +

      *Thread Reply:* I think mine are resolved 🙂

      + + + +
      + :gratitude_thank_you: Mattia Bertorello +
      + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-06 07:27:40
       +
      +

      *Thread Reply:* I lost current status of ServiceLoader discussion. I found thumbs-up in github under my comment and then some doubts about using class.forName . I got confused which is more recent and which direction is preferred to go? I am open for discussion. @Mattia Bertorello @Maciej Obuchowski

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Mattia Bertorello + (mattia.bertorello@booking.com) +
      +
      2024-02-06 07:30:50
       +
      +

      *Thread Reply:* No problem, having the ServiceLoader brings more loose coupling between the app project and others. But if you already found users that struggle with creating a uber-jar that keep/merge the META-inf files could be better to scarify the loose coupling.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-06 07:34:19
       +
      +

      *Thread Reply:* https://github.com/OpenLineage/OpenLineage/issues/1860 -> this is an example of issue we have in our backlog. there are at least 3 different people involved so I assumed it's not that uncommon

      +
      + + + + +
      #1860 io.openlineage.client.OpenLineageClientException: io.openlineage.spark.shaded.com.fasterxml.jackson.databind.JsonMappingException: Failed to find TransportBuilder
      +
      + <p>On Version <code>0.25.0</code> when running <code>OpenLineage</code> with Spark, we are getting:</p> + +<pre><code>io.openlineage.client.OpenLineageClientException: io.openlineage.spark.shaded.com.fasterxml.jackson.databind.JsonMappingException: Failed to find TransportBuilder (through reference chain: io.openlineage.client.OpenLineageYaml["transport"]) + at io.openlineage.client.OpenLineageClientUtils.loadOpenLineageYaml(OpenLineageClientUtils.java:140) + at io.openlineage.spark.agent.ArgumentParser.extractOpenlineageConfFromSparkConf(ArgumentParser.java:114) + at io.openlineage.spark.agent.ArgumentParser.parse(ArgumentParser.java:78) + at io.openlineage.spark.agent.OpenLineageSparkListener.initializeContextFactoryIfNotInitialized(OpenLineageSparkListener.java:277) + at io.openlineage.spark.agent.OpenLineageSparkListener.onApplicationStart(OpenLineageSparkListener.java:267) + at org.apache.spark.scheduler.SparkListenerBus$class.doPostEvent(SparkListenerBus.scala:55) + at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) + at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) + at org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:91) + at &lt;a href="http://org.apache.spark.scheduler.AsyncEventQueue.org"&gt;org.apache.spark.scheduler.AsyncEventQueue.org&lt;/a&gt;$apache$spark$scheduler$AsyncEventQueue$$super$postToAll(AsyncEventQueue.scala:92) + at org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply$mcJ$sp(AsyncEventQueue.scala:92) + at org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply(AsyncEventQueue.scala:87) + at org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply(AsyncEventQueue.scala:87) + at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58) + at &lt;a href="http://org.apache.spark.scheduler.AsyncEventQueue.org"&gt;org.apache.spark.scheduler.AsyncEventQueue.org&lt;/a&gt;$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:87) + at org.apache.spark.scheduler.AsyncEventQueue$$anon$1$$anonfun$run$1.apply$mcV$sp(AsyncEventQueue.scala:83) + at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1347) + at org.apache.spark.scheduler.AsyncEventQueue$$anon$1.run(AsyncEventQueue.scala:82) +Caused by: io.openlineage.spark.shaded.com.fasterxml.jackson.databind.JsonMappingException: Failed to find TransportBuilder (through reference chain: io.openlineage.client.OpenLineageYaml["transport"]) + at io.openlineage.spark.shaded.com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:392) + at io.openlineage.spark.shaded.com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:351) + at io.openlineage.spark.shaded.com.fasterxml.jackson.databind.deser.BeanDeserializerBase.wrapAndThrow(BeanDeserializerBase.java:1821) + at io.openlineage.spark.shaded.com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:316) + at io.openlineage.spark.shaded.com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:177) + at io.openlineage.spark.shaded.com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:323) + at io.openlineage.spark.shaded.com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4674) + at io.openlineage.spark.shaded.com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3666) + at io.openlineage.client.OpenLineageClientUtils.loadOpenLineageYaml(OpenLineageClientUtils.java:138) + ... 17 more +Caused by: java.lang.IllegalArgumentException: Failed to find TransportBuilder + at io.openlineage.client.transports.TransportResolver.lambda$getTransportBuilder$3(TransportResolver.java:38) + at java.util.Optional.orElseThrow(Optional.java:290) + at io.openlineage.client.transports.TransportResolver.getTransportBuilder(TransportResolver.java:37) + at io.openlineage.client.transports.TransportResolver.resolveTransportConfigByType(TransportResolver.java:16) + at io.openlineage.client.transports.TransportConfigTypeIdResolver.typeFromId(TransportConfigTypeIdResolver.java:35) + at io.openlineage.spark.shaded.com.fasterxml.jackson.databind.jsontype.impl.TypeDeserializerBase._findDeserializer(TypeDeserializerBase.java:159) + at io.openlineage.spark.shaded.com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:125) + at io.openlineage.spark.shaded.com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:110) + at io.openlineage.spark.shaded.com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType(AbstractDeserializer.java:263) + at io.openlineage.spark.shaded.com.fasterxml.jackson.databind.deser.impl.FieldProperty.deserializeAndSet(FieldProperty.java:147) + at io.openlineage.spark.shaded.com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:314) + ... 22 more +</code></pre> + +<p>We see that <code>OpenLineage</code> added support for <em>custom transport types</em> in <code>0.24.0</code> We're not sure if it's related to that. We checked with version <code>0.23.0</code> and we don't see the exception in that version.</p> + +<p>We are using <code>OpenLineage</code> with Spark using <code>spark-submit</code>, dependency: <code>io.openlineage:openlineage_spark:0.23.0</code></p> +
      + +
      +
      Assignees
      + <a href="https://github.com/pawel-big-lebowski">@pawel-big-lebowski</a> +
      + +
      +
      Labels
      + bug, integration/spark +
      + + + + + + + +
      + + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
      + + +
      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Mattia Bertorello + (mattia.bertorello@booking.com) +
      +
      2024-02-06 07:37:35
       +
      +

      *Thread Reply:* ok, makes sense. +So I can commit the change above that doesn't use the service loader and at the same time doesn't need the compile code be aware of the implementation since it happen at runtime and only through the interface. +I can add this issues in the comments to justify the solution.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Mattia Bertorello + (mattia.bertorello@booking.com) +
      +
      2024-02-06 07:41:07
       +
      +

      *Thread Reply:* BTW I'm working on having JobTypeJobFacet in Spark and then in DBT integration. Have you worked on this before?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-06 07:46:21
       +
      +

      *Thread Reply:* I worked on that for Flink

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Mattia Bertorello + (mattia.bertorello@booking.com) +
      +
      2024-02-06 07:47:42
       +
      +

      *Thread Reply:* Yeah I'm checking that work to replicate on the spark integration 🙂

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-06 07:48:15
       +
      +

      *Thread Reply:* https://github.com/OpenLineage/OpenLineage/pull/2241/files#diff-040d58841330856813d9683119f75b3d455ccc03a9e7b1a10b54966b2d029b99

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-06 07:48:33
       +
      +

      *Thread Reply:* it's not that super easy to determine if a job is batch or streaming

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Mattia Bertorello + (mattia.bertorello@booking.com) +
      +
      2024-02-06 07:48:51
       +
      +

      *Thread Reply:* Do we need also in spark?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-06 07:49:05
       +
      +

      *Thread Reply:* it's possible to have streaming job in spark

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-06 07:49:38
       +
      +

      *Thread Reply:* and processingType property is required within the facet

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-02-06 08:03:30
       +
      +

      *Thread Reply:* Don't streaming jobs in Spark have different logical plans?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Mattia Bertorello + (mattia.bertorello@booking.com) +
      +
      2024-02-06 11:28:49
       +
      +

      *Thread Reply:* It seems there is a isStreaming() method in the LogicalPlan +https://github.com/OpenLineage/OpenLineage/pull/2410 +Let me know what do you think 🙂

      +
      + + + + +
      #2410 JobTypeJobFacet added and used in Spark
      +
      + <p><strong>Problem</strong></p> + +<p>Currently, a lineage backend has difficulty distinguishing between job types, Spark, DBT, and Airflow and needs regex on the <code>_producer</code> to get this information.<br /> +We need to add the <code>JobType</code> to make it more clear.</p> + +<p><a href="https://openlineage.io/docs/spec/facets/job-facets/job-type">https://openlineage.io/docs/spec/facets/job-facets/job-type</a></p> + +<p><strong>Solution</strong></p> + +<p>• Introduce new facet, +• Include new facet within Spark integration.</p> + +<blockquote> + <p><strong>Note:</strong> All schema changes require discussion. Please <a href="https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue">link the issue</a> for context.</p> +</blockquote> + +<p>☐ Your change modifies the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json">core</a> OpenLineage model +☐ Your change modifies one or more OpenLineage <a href="https://github.com/OpenLineage/OpenLineage/tree/main/spec/facets">facets</a></p> + +<p>If you're contributing a new integration, please specify the scope of the integration and how/where it has been tested (e.g., Apache Spark integration supports <code>S3</code> and <code>GCS</code> filesystem operations, tested with AWS EMR).</p> + +<p><strong>One-line summary:</strong> +<strong>Checklist</strong></p> + +<p>☐ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☐ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☐ Your changes are accompanied by tests (<em>if relevant</em>) +☐ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☐ You've updated any relevant documentation (<em>if relevant</em>) +☐ Your comment includes a one-liner for the changelog about the specific purpose of the change (<em>if necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>) +☐ You've added a <a href="https://github.com/OpenLineage/OpenLineage/tree/main/.github/header_templates.md">header</a> to source files (<em>if relevant</em>)</p> + +<hr /> + +<p>SPDX-License-Identifier: Apache-2.0<br /> +Copyright 2018-2023 contributors to the OpenLineage project</p> +
      + +
      +
      Labels
      + integration/spark +
      + + + + + + + +
      + + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
      + + +
      + + + +
      + 👍 Paweł Leszczyński, Maciej Obuchowski +
      + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-06 11:29:24
       +
      +

      *Thread Reply:* is the extra param in buildRun necessary?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-06 11:29:39
       +
      +

      *Thread Reply:* why wouldn't this match in JobBuilder?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Mattia Bertorello + (mattia.bertorello@booking.com) +
      +
      2024-02-06 11:31:15
       +
      +

      *Thread Reply:* Not really I can use use the jobBuilder and then extract the facets from there. +I started with that implementation but then it wasn't so clean so I changed my mind

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Mattia Bertorello + (mattia.bertorello@booking.com) +
      +
      2024-02-06 11:32:41
       +
      +

      *Thread Reply:* But no problem to implement in that way. +Because I need to build the jobBuilder to extract the facets that is not super great 😅 because then the code use the jobBuilder to overwrite the facets

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-06 11:34:27
       +
      +

      *Thread Reply:* yeah, looks like the problem of nested builders

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-06 11:39:50
       +
      +

      *Thread Reply:* thinking loadly - could we pass jobFacetsBuilder as a param to avoid adding each facet as param in future?

      + + + +
      + 👍 Mattia Bertorello +
      + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Mattia Bertorello + (mattia.bertorello@booking.com) +
      +
      2024-02-06 11:59:14
       +
      +

      *Thread Reply:* Done

      + + + +
      + 🎉 Paweł Leszczyński, Maciej Obuchowski +
      + +
      + 🙌 Paweł Leszczyński, Maciej Obuchowski +
      + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-06 12:05:23
       +
      +

      *Thread Reply:* could you add a line into changelog?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Mattia Bertorello + (mattia.bertorello@booking.com) +
      +
      2024-02-06 12:06:31
       +
      +

      *Thread Reply:* Sure, but tomorrow, I'm heading home 🙂

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-06 12:06:49
       +
      +

      *Thread Reply:* sure, giving an approval with a comment to fill changelog

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Mattia Bertorello + (mattia.bertorello@booking.com) +
      +
      2024-02-07 04:41:07
       +
      +

      *Thread Reply:* Done

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-07 02:18:41
       +
      +

      @Mattia Bertorello PR 2405 is failin on +VendorsTest &gt; testGetVendors() FAILED + org.opentest4j.AssertionFailedError at VendorsTest.java:60

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Mattia Bertorello + (mattia.bertorello@booking.com) +
      +
      2024-02-07 04:40:48
       +
      +

      *Thread Reply:* Sorry, I forgot to fix the tests

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-07 05:39:15
       +
      +

      *Thread Reply:* :gh_merged:

      + + + +
      + :gratitude_thank_you: Mattia Bertorello +
      + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-07 03:46:11
       +
      +

      @Paweł Leszczyński @Maciej Obuchowski - when you run

      + +

      './gradlwe :app:test' --tests "io.openlineage.spark.agent.lifecycle.SparkSQLExecutionContextTest" -Pspark.version=3.2.4

      + +

      Do these tests have a InvocationTargetException which is caused by a JsonMappingException which is caused by a StackOverflowError?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-07 03:58:25
       +
      +

      *Thread Reply:* I have this reference chain

      + +
      +
      Untitled
      + + + + + + +
      + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-02-07 05:07:04
       +
      +

      *Thread Reply:* I got BUILD SUCCESSFUL in 55s on latest main

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-07 05:10:59
       +
      +

      *Thread Reply:* build successful on my side, did you try rebuilding and republishing to local openlineage-java ?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-07 08:00:19
       +
      +

      @Damien Hawes @Mattia Bertorello How far you from the ultimate goal? Will you need to rewrite all the extensions to vendors or snowflake only? Are you going to release multiple artifacts at the end per scala 2.12 and 2.13. Feeling kind of lost where are we at the moment.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-07 08:07:42
       +
      +

      *Thread Reply:* The idea is that there will be two artefacts.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-07 08:07:57
       +
      +

      *Thread Reply:* I'm working on the integration tests at the moment.

      + + + +
      + 🙏 Maciej Obuchowski +
      + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Mattia Bertorello + (mattia.bertorello@booking.com) +
      +
      2024-02-07 08:21:23
       +
      +

      *Thread Reply:* I will continue working on the vendor extraction, now the next one is Iceberg. +And not related to that I need to add the job type to Airflow and DBT.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-07 08:24:15
       +
      +

      *Thread Reply:* what's the problem with iceberg? i thought is published for both: scala 2.12 and 2.13

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Mattia Bertorello + (mattia.bertorello@booking.com) +
      +
      2024-02-07 09:45:40
       +
      +

      *Thread Reply:* It will make it easier to support multiple versions of the same vendor, in this case, Iceberg, if there are breaking changes, including supporting the old version more easily without too much reflection, like if the code is mixed with the core spark code.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-08 03:59:11
       +
      +

      @Paweł Leszczyński - https://github.com/OpenLineage/OpenLineage/pull/2413

      + + + +
      + 👀 Paweł Leszczyński +
      + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-02-08 06:48:15
       +
      +

      *Thread Reply:* that's... not a small pr 🙂

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-08 06:48:37
       +
      +

      *Thread Reply:* Thankfully, it's just "add new stuff"

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-08 06:49:10
       +
      +

      *Thread Reply:* It could be split into 3

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-08 06:49:38
       +
      +

      *Thread Reply:* 1. Modification of the build.gradle files and the common config plugin

      + +
      1. Creation of Docker plugin
      2. Creation of Spark Builds plugin
        3. +
      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-08 07:12:48
       +
      +

      *Thread Reply:* OK - splitting it up.

      + +

      1st one:

      + +

      Docker build plugin: https://github.com/OpenLineage/OpenLineage/pull/2414

      + +

      (365 lines)

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-08 07:22:03
       +
      +

      *Thread Reply:* 2nd one: https://github.com/OpenLineage/OpenLineage/pull/2415

      + +

      @Maciej Obuchowski @Paweł Leszczyński

      + +

      74 additions, 62 removals

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-02-08 07:24:20
       +
      +

      *Thread Reply:* 👍

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-08 11:06:58
       +
      +

      *Thread Reply:* 3rd one:

      + +

      https://github.com/OpenLineage/OpenLineage/pull/2416

      + +

      (The big one)

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-08 08:39:09
       +
      +

      @Mattia Bertorello @Damien Hawes we are thinking about the OL release in next few days. Just wondering if we should wait for your PRs, or shall we release without waiting for this. Would love to hear your opinion on that. Keep in mind, a voting for an extra release can be initiated anytime you like in case you didn't want to wait the whole month.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Mattia Bertorello + (mattia.bertorello@booking.com) +
      +
      2024-02-08 08:48:44
       +
      +

      *Thread Reply:* Thanks for letting us know. I would like to implement the JobType in Airflow; can you wait until Monday at the end of the day?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-08 09:01:33
       +
      +

      *Thread Reply:* From my side

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-08 09:01:36
       +
      +

      *Thread Reply:* Don't wait for me.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Mattia Bertorello + (mattia.bertorello@booking.com) +
      +
      2024-02-08 12:58:55
       +
      +

      *Thread Reply:* @Maciej Obuchowski +I did the PR in the Airflow repository. +https://github.com/apache/airflow/pull/37255 +and fix the other one +https://github.com/OpenLineage/OpenLineage/pull/2412/

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-09 07:25:45
       +
      +

      *Thread Reply:* Thx for info

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Mattia Bertorello + (mattia.bertorello@booking.com) +
      +
      2024-02-13 07:11:13
       +
      +

      *Thread Reply:* Do you think we can merge it #2412 before the release? Is there some missing? @Maciej Obuchowski

      +
      + + + + +
      #2412 [AIRFLOW] Add support for JobTypeJobFacet properties.
      +
      + <p><strong>Problem</strong></p> + +<p>Currently, a lineage backend has difficulty distinguishing between job types, DBT, and Airflow and needs regex on the _producer to get this information.<br /> +We need to add the JobType to make it more clear.</p> + +<p><a href="https://openlineage.io/docs/spec/facets/job-facets/job-type">https://openlineage.io/docs/spec/facets/job-facets/job-type</a></p> + +<p><strong>Solution</strong></p> + +<p>Include new facet within Airflow integration.</p> + +<blockquote> + <p><strong>Note:</strong> All schema changes require discussion. Please <a href="https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue">link the issue</a> for context.</p> +</blockquote> + +<p>☐ Your change modifies the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json">core</a> OpenLineage model +☐ Your change modifies one or more OpenLineage <a href="https://github.com/OpenLineage/OpenLineage/tree/main/spec/facets">facets</a></p> + +<p>If you're contributing a new integration, please specify the scope of the integration and how/where it has been tested (e.g., Apache Spark integration supports <code>S3</code> and <code>GCS</code> filesystem operations, tested with AWS EMR).</p> + +<p><strong>One-line summary:</strong> +<strong>Checklist</strong></p> + +<p>☐ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☐ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☐ Your changes are accompanied by tests (<em>if relevant</em>) +☐ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☐ You've updated any relevant documentation (<em>if relevant</em>) +☐ Your comment includes a one-liner for the changelog about the specific purpose of the change (<em>if necessary</em>) +☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>) +☐ You've added a <a href="https://github.com/OpenLineage/OpenLineage/tree/main/.github/header_templates.md">header</a> to source files (<em>if relevant</em>)</p> + +<hr /> + +<p>SPDX-License-Identifier: Apache-2.0<br /> +Copyright 2018-2023 contributors to the OpenLineage project</p> +
      + +
      +
      Labels
      + documentation, integration/airflow +
      + +
      +
      Comments
      + 1 +
      + + + + + + + +
      + + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
      + + +
      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-09 07:27:34
       +
      +

      @Damien Hawes I merged 2413 and approved https://github.com/OpenLineage/OpenLineage/pull/2415 could u rebase it and resolve conflict?

      +
      + + + + +
      #2415 [SPARK] Refactor the CommonConfig Gradle plugin
      +
      + <p><strong>Summary</strong></p> + +<p>This PR introduces an important refactor of the CommonConfig plugin to better align with Gradle best practices.</p> + +<p><strong>Problem</strong></p> + +<p>The CommonConfig plugin was a "strong" opinion plugin. It applied plugins to projects, whether they needed it or not.</p> + +<p><strong>Solution</strong> +<strong>CommonConfig Plugin Refactor</strong></p> + +<p>• <strong>Best Practices</strong>: Modified to react to the application of plugins (Java library, PMD, Lombok, Spotless) rather than applying them directly, introducing a paradigm shift towards increased adherence to Gradle best practices. +• <strong>Build Scripts Adjustment</strong>: Ensured build integrity by explicitly applying the necessary plugins within each build script, following the refactor of the CommonConfig plugin.</p> + +<p><strong>Checklist</strong></p> + +<p>☑︎ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☑︎ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☑︎ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained +☑︎ You've added a <a href="https://github.com/OpenLineage/OpenLineage/tree/main/.github/header_templates.md">header</a> to source files (<em>if relevant</em>)</p> + +<hr /> + +<p>SPDX-License-Identifier: Apache-2.0<br /> +Copyright 2018-2023 contributors to the OpenLineage project</p> +
      + +
      +
      Labels
      + integration/spark +
      + + + + + + + +
      + + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
      + + +
      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-12 08:07:29
       +
      +

      @Paweł Leszczyński - this is ready

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-13 08:11:43
       +
      +

      *Thread Reply:* I can't find SparkIcebergIntegrationTest being run in integration-test-integration-spark-scala-2_12-3.5.0 -> https://app.circleci.com/pipelines/github/OpenLineage/OpenLineage/9275/workflows/78f45083-84f5-4379-b475-96ccececbbe5/jobs/173596

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-13 08:12:12
       +
      +

      *Thread Reply:* 3.5.0 never had iceberg though

      + + + +
      + 👍 Paweł Leszczyński +
      + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-13 08:12:43
       +
      +

      *Thread Reply:*

      + +
      +
      image.png
      + + + + + + + + +
      + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-13 08:13:03
       +
      +

      *Thread Reply:* integration tests run in 3mins30secs which is like way faster than it used to be

      + + + +
      +
      +
      +
      + + + + + +
      + +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-13 08:13:15
       +
      +

      *Thread Reply:* i cannot find it for 3.4.2 as well

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-13 08:13:22
       +
      +

      *Thread Reply:* hmm

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-13 08:17:17
       +
      +

      *Thread Reply:* but it's the same on recent main I think. This is main build from 6 days ago https://app.circleci.com/pipelines/github/OpenLineage/OpenLineage/9160/workflows/33a4d308-d0e6-4d75-a06b-7d8ef89bb1fe and SparkIcebergIntegrationTest is present there

      + +
      +
      Screenshot 2024-02-13 at 14.16.20.png
      + + + + + + + + +
      + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-13 08:17:49
       +
      +

      *Thread Reply:* and execution time are above 10 mins

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-13 08:32:10
       +
      +

      *Thread Reply:* I'll check, might be the logic for determining it is faulty

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 02:36:38
       +
      +

      *Thread Reply:* @Paweł Leszczyński - any ideas why the Delta and Iceberg tests will execute and pass on my local device, but in the CI they fail?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 02:37:00
       +
      +

      *Thread Reply:* https://app.circleci.com/pipelines/github/OpenLineage/OpenLineage/9292/workflows/30ba0bdc-a831-4431-b023-d35379137a61/jobs/174557/tests

      + +

      I cannot replicate these failures locally

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 02:37:18
       +
      +

      *Thread Reply:*

      + +
      +
      image.png
      + + + + + + + + +
      + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-14 02:38:30
       +
      +

      *Thread Reply:* in one of the previous PRs some Docker timeout got shortened I think.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-14 02:38:53
       +
      +

      *Thread Reply:* My guess is that local environment is a way faster

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 02:39:09
       +
      +

      *Thread Reply:* Yeah - it's hard without seeing the STDOUT logs

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 02:39:22
       +
      +

      *Thread Reply:* Is there a way to see them, besides scrolling through the crazy logs?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 02:39:39
       +
      +

      *Thread Reply:* like when you get the test report locally, I can see STDOUT and STDERR

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 02:39:58
       +
      +

      *Thread Reply:* Otherwise, I have to go log diving 😂

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-14 02:41:32
       +
      +

      *Thread Reply:* i think you can also ssh into circleCI and run single test

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-14 02:45:05
       +
      +

      *Thread Reply:* i also search the logs for () FAILED which brings closer to real error but it's kind of log diving

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 03:05:54
       +
      +

      *Thread Reply:* Then there is this one:

      + +

      testExternalRDDWithS3Bucket(SparkSession)

      + +

      ava.lang.UnsupportedClassVersionError: com/sun/istack/Pool has been compiled by a more recent version of the Java Runtime (class file version 55.0), this version of the Java Runtime only recognizes class file versions up to 52.0 + at java.lang.ClassLoader.defineClass1(Native Method) + at java.lang.ClassLoader.defineClass(ClassLoader.java:756) + at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) + at java.net.URLClassLoader.defineClass(URLClassLoader.java:473) + at java.net.URLClassLoader.access$100(URLClassLoader.java:74) + at java.net.URLClassLoader$1.run(URLClassLoader.java:369) + at java.net.URLClassLoader$1.run(URLClassLoader.java:363) + at java.security.AccessController.doPrivileged(Native Method) + at java.net.URLClassLoader.findClass(URLClassLoader.java:362) + at java.lang.ClassLoader.loadClass(ClassLoader.java:418) + at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) + at java.lang.ClassLoader.loadClass(ClassLoader.java:351) + at com.sun.xml.bind.v2.runtime.JAXBContextImpl$JAXBContextBuilder.build(JAXBContextImpl.java:1126) + at com.sun.xml.bind.v2.ContextFactory.createContext(ContextFactory.java:135) + at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) + at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) + at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) + at java.lang.reflect.Method.invoke(Method.java:498) + at javax.xml.bind.ContextFinder.newInstance(ContextFinder.java:247) + at javax.xml.bind.ContextFinder.newInstance(ContextFinder.java:234) + at javax.xml.bind.ContextFinder.find(ContextFinder.java:441) + at javax.xml.bind.JAXBContext.newInstance(JAXBContext.java:641) + at javax.xml.bind.JAXBContext.newInstance(JAXBContext.java:584) + at com.amazonaws.util.Base64.&lt;clinit&gt;(Base64.java:52) + at com.amazonaws.util.BinaryUtils.toBase64(BinaryUtils.java:59) + at com.amazonaws.services.s3.AmazonS3Client.populateRequestHeaderWithMd5(AmazonS3Client.java:4547) + at com.amazonaws.services.s3.AmazonS3Client.deleteObjects(AmazonS3Client.java:2325) + at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$deleteObjects$16(S3AFileSystem.java:2785) +It blows my mind that the Amazon stuff uses Java 8, but they declare a dependency that is compiled with Java 11.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 03:42:10
       +
      +

      *Thread Reply:* Yeah - I don't know. I increased the timeout to 60 seconds and the CI still complains

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 03:51:50
       +
      +

      *Thread Reply:* Any ideas @Paweł Leszczyński?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 03:52:01
       +
      +

      *Thread Reply:* I'm at quite the loss

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 03:59:32
       +
      +

      *Thread Reply:* The common theme amongst the failing tests, is that they all spin up local spark sessions.

      + + + +
      + 👀 Paweł Leszczyński +
      + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-14 07:06:47
       +
      +

      *Thread Reply:* there is no events emitted nor error message, which is totally weird and can be mockserver related. I saw you changed mockserver ports to be above 10000. perhaps it's not allowed to use such high ports in circle ci. if a default http transport timeout is higher than timeout for mockserver assertion, this can look like that -> no error and assertion failed

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 07:07:28
       +
      +

      *Thread Reply:* Yeah, there were two tests that had mock server ports set to 1081

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 07:07:35
       +
      +

      *Thread Reply:* and they were running into each other

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 07:07:45
       +
      +

      *Thread Reply:* so I was like, "Let's make them random"

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-14 07:07:59
       +
      +

      *Thread Reply:* could u set it for delta to static value as before and see if they fail again?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 07:09:42
       +
      +

      *Thread Reply:* sure

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 07:39:46
       +
      +

      *Thread Reply:* OK

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 07:39:54
       +
      +

      *Thread Reply:* Switching back to hard coded ports

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 07:40:31
       +
      +

      *Thread Reply:* Made the 3.5.0 pipeline pass everything except the io.openlineage.spark.agent.SparkGenericIntegrationTest and the test that tries to use S3

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 07:40:50
       +
      +

      *Thread Reply:* The former failed, once again, with the assertion error

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 07:40:56
       +
      +

      *Thread Reply:* the latter, about the java version error

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-14 08:16:44
       +
      +

      *Thread Reply:* @Maciej Obuchowski SparkGenericIntegrationTest is your recent test. You call getEventsEmitted and wait only as long as list is non-empty. So if you expect two events, you don't wait for the second one. Am I right?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 08:26:08
       +
      +

      *Thread Reply:* TIL: If you relocate org.slf4j (which app was previously doing) - you will not see the logs of the shaded jar.

      + + + +
      + 😨 Paweł Leszczyński +
      + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-02-14 08:31:02
       +
      +

      *Thread Reply:* yes, that's why I always comment that and @Paweł Leszczyński always asks why I've done it during review if I forget to revert it 🙂

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-02-14 08:31:46
       +
      +

      *Thread Reply:* @Paweł Leszczyński I do verifyEvents before getEventsEmitted, that should take care of getting all the events I care about

      + + + +
      + 🙌 Paweł Leszczyński +
      + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 08:37:02
       +
      +

      *Thread Reply:* Hmm - did I find an undocumented "feature"?

      + +

      Basically, if the OpenLineage Java client detects that you're trying to post to /api/v1/namespaces/<namespace> it will inject the namespace into the run event under $.job.namespace if $.job.namespace is equal to default?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 10:04:59
       +
      +

      *Thread Reply:* I'm seeing this in the logs for the integration test:

      + +

      io.openlineage.client.OpenLineageClientException: io.openlineage.spark.shaded.org.apache.http.conn.HttpHostConnectException: Connect to localhost:1081 [localhost/127.0.0.1] failed: Connection refused (Connection refused) + at io.openlineage.client.transports.HttpTransport.emit(HttpTransport.java:128) ~[openlineage-spark-agent_2.13-1.9.0-SNAPSHOT-shadow.jar:1.9.0-SNAPSHOT] + at io.openlineage.client.transports.HttpTransport.emit(HttpTransport.java:111) ~[openlineage-spark-agent_2.13-1.9.0-SNAPSHOT-shadow.jar:1.9.0-SNAPSHOT] + at io.openlineage.client.OpenLineageClient.emit(OpenLineageClient.java:46) ~[openlineage-spark-agent_2.13-1.9.0-SNAPSHOT-shadow.jar:1.9.0-SNAPSHOT] + at io.openlineage.spark.agent.EventEmitter.emit(EventEmitter.java:69) ~[openlineage-spark-agent_2.13-1.9.0-SNAPSHOT-shadow.jar:?] + at io.openlineage.spark.agent.OpenLineageSparkListener.emitApplicationEvent(OpenLineageSparkListener.java:321) ~[openlineage-spark-agent_2.13-1.9.0-SNAPSHOT-shadow.jar:?] + at io.openlineage.spark.agent.OpenLineageSparkListener.emitApplicationEndEvent(OpenLineageSparkListener.java:329) ~[openlineage-spark-agent_2.13-1.9.0-SNAPSHOT-shadow.jar:?] + at io.openlineage.spark.agent.OpenLineageSparkListener.onApplicationEnd(OpenLineageSparkListener.java:229) ~[openlineage-spark-agent_2.13-1.9.0-SNAPSHOT-shadow.jar:?] + at org.apache.spark.scheduler.SparkListenerBus.doPostEvent(SparkListenerBus.scala:57) ~[spark-core_2.13-3.2.4.jar:3.2.4] + at org.apache.spark.scheduler.SparkListenerBus.doPostEvent$(SparkListenerBus.scala:28) ~[spark-core_2.13-3.2.4.jar:3.2.4] + at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) ~[spark-core_2.13-3.2.4.jar:3.2.4] + at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) ~[spark-core_2.13-3.2.4.jar:3.2.4] + at org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:117) ~[spark-core_2.13-3.2.4.jar:3.2.4] + at org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:101) ~[spark-core_2.13-3.2.4.jar:3.2.4] + at org.apache.spark.scheduler.AsyncEventQueue.super$postToAll(AsyncEventQueue.scala:105) ~[spark-core_2.13-3.2.4.jar:3.2.4] + at org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:105) ~[spark-core_2.13-3.2.4.jar:3.2.4] + at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.scala:17) ~[iceberg-spark-runtime-3.2_2.13-0.14.0.jar:?] + at scala.util.DynamicVariable.withValue(DynamicVariable.scala:59) ~[iceberg-spark-runtime-3.2_2.13-0.14.0.jar:?] + at <a href="http://org.apache.spark.scheduler.AsyncEventQueue.org">org.apache.spark.scheduler.AsyncEventQueue.org</a>$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:100) ~[spark-core_2.13-3.2.4.jar:3.2.4] + at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:96) ~[spark-core_2.13-3.2.4.jar:3.2.4] + at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1433) [spark-core_2.13-3.2.4.jar:3.2.4] + at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:96) [spark-core_2.13-3.2.4.jar:3.2.4] + Caused by: io.openlineage.spark.shaded.org.apache.http.conn.HttpHostConnectException: Connect to localhost:1081 [localhost/127.0.0.1] failed: Connection refused (Connection refused) + at io.openlineage.spark.shaded.org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) ~[openlineage-spark-agent_2.13-1.9.0-SNAPSHOT-shadow.jar:1.9.0-SNAPSHOT] + at io.openlineage.spark.shaded.org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[openlineage-spark-agent_2.13-1.9.0-SNAPSHOT-shadow.jar:1.9.0-SNAPSHOT] + at io.openlineage.spark.shaded.org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[openlineage-spark-agent_2.13-1.9.0-SNAPSHOT-shadow.jar:1.9.0-SNAPSHOT] + at io.openlineage.spark.shaded.org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[openlineage-spark-agent_2.13-1.9.0-SNAPSHOT-shadow.jar:1.9.0-SNAPSHOT] + at io.openlineage.spark.shaded.org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[openlineage-spark-agent_2.13-1.9.0-SNAPSHOT-shadow.jar:1.9.0-SNAPSHOT] + at io.openlineage.spark.shaded.org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[openlineage-spark-agent_2.13-1.9.0-SNAPSHOT-shadow.jar:1.9.0-SNAPSHOT] + at io.openlineage.spark.shaded.org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[openlineage-spark-agent_2.13-1.9.0-SNAPSHOT-shadow.jar:1.9.0-SNAPSHOT] + at io.openlineage.spark.shaded.org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[openlineage-spark-agent_2.13-1.9.0-SNAPSHOT-shadow.jar:1.9.0-SNAPSHOT] + at io.openlineage.spark.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[openlineage-spark-agent_2.13-1.9.0-SNAPSHOT-shadow.jar:1.9.0-SNAPSHOT] + at io.openlineage.spark.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[openlineage-spark-agent_2.13-1.9.0-SNAPSHOT-shadow.jar:1.9.0-SNAPSHOT] + at io.openlineage.client.transports.HttpTransport.emit(HttpTransport.java:123) ~[openlineage-spark-agent_2.13-1.9.0-SNAPSHOT-shadow.jar:1.9.0-SNAPSHOT] + ... 20 more

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 10:06:27
       +
      +

      *Thread Reply:* I'm honestly wondering if this is the source of my troubles with the integration tests failing. i.e., the mockserver just isn't binding to the port, or the port isn't accepting connections, or something.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 10:07:02
       +
      +

      *Thread Reply:* That, for example, comes from the Google Cloud Integration test

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-02-14 10:34:34
       +
      +

      *Thread Reply:* > Basically, if the OpenLineage Java client detects that you're trying to post to /api/v1/namespaces/&lt;namespace&gt; +I don't think it sends to .../namespaces anywhere by default, the canonical endpoint is api/v1/lineage ? +> I'm honestly wondering if this is the source of my troubles with the integration tests failing. i.e., the mockserver just isn't binding to the port, or the port isn't accepting connections, or something. +Or the tests run before mockserver started accepting connections?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-02-14 10:36:26
       +
      +

      *Thread Reply:* In Docker tests, we have a long timeout on the container +static MockServerContainer makeMockServerContainer(Network network) { + return new MockServerContainer(MOCKSERVER_IMAGE) + .withNetwork(network) + .withNetworkAliases("openlineageclient") + .withStartupTimeout(Duration.of(2, ChronoUnit.MINUTES)); + }

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-02-14 10:36:58
       +
      +

      *Thread Reply:* but I don't think there's anything like that on local tests? +mockServer = ClientAndServer.startClientAndServer(configuration, MOCKSERVER_PORT); + mockServer + .when(request("/api/v1/lineage")) + .respond(org.mockserver.model.HttpResponse.response().withStatusCode(201));

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 11:31:58
       +
      +

      *Thread Reply:* @Maciej Obuchowski - possible, possible. Nice thought.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-02-14 12:07:00
       +
      +

      *Thread Reply:* Ran your branch locally and I'm getting the same errors

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 12:07:31
       +
      +

      *Thread Reply:* What OS and arch are you using?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 12:09:48
       +
      +

      *Thread Reply:* I'm using an Apple M1 with MacOS Sonoma

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-02-14 12:13:02
       +
      +

      *Thread Reply:* pretty much the same, M1 Pro with Sonoma

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 12:13:25
       +
      +

      *Thread Reply:* 🤨

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 12:16:19
       +
      +

      *Thread Reply:* What JDK are you running the project with?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-02-14 12:18:47
       +
      +

      *Thread Reply:* Azul Zulu 8 Arm build

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 12:18:58
       +
      +

      *Thread Reply:* ¯_(ツ)_/¯

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 12:19:01
       +
      +

      *Thread Reply:* Same.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 12:19:19
       +
      +

      *Thread Reply:* I guess I just have some magic on my machine

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-02-14 12:19:44
       +
      +

      *Thread Reply:* +➜ code java -version +openjdk version "1.8.0_312" +OpenJDK Runtime Environment (Zulu 8.58.0.13-CA-macos-aarch64) (build 1.8.0_312-b07) +OpenJDK 64-Bit Server VM (Zulu 8.58.0.13-CA-macos-aarch64) (build 25.312-b07, mixed mode)

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-02-14 12:19:59
       +
      +

      *Thread Reply:* I'll try to dig in more

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 12:20:18
       +
      +

      *Thread Reply:* I have 392, but I don't think it makes much of a difference

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 12:21:25
       +
      +

      *Thread Reply:* Also, I have access to a centos box running on x86_64, which I am running the tests with.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 12:21:35
       +
      +

      *Thread Reply:* To see if I can replicate the failures there

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 12:26:59
       +
      +

      *Thread Reply:* 2024-02-14 18:26:32 ERROR io.openlineage.spark.agent.EventEmitter - Could not emit lineage w/ exception + io.openlineage.client.OpenLineageClientException: io.openlineage.spark.shaded.org.apache.http.conn.HttpHostConnectException: Connect to localhost:1081 [localhost/127.0.0.1] failed: Connection refused (Connection refused) + at io.openlineage.client.transports.HttpTransport.emit(HttpTransport.java:128) ~[openlineage-spark-agent_2.13-1.9.0-SNAPSHOT-shadow.jar:1.9.0-SNAPSHOT] + at io.openlineage.client.transports.HttpTransport.emit(HttpTransport.java:111) ~[openlineage-spark-agent_2.13-1.9.0-SNAPSHOT-shadow.jar:1.9.0-SNAPSHOT] + at io.openlineage.client.OpenLineageClient.emit(OpenLineageClient.java:46) ~[openlineage-spark-agent_2.13-1.9.0-SNAPSHOT-shadow.jar:1.9.0-SNAPSHOT] + at io.openlineage.spark.agent.EventEmitter.emit(EventEmitter.java:69) ~[openlineage-spark-agent_2.13-1.9.0-SNAPSHOT-shadow.jar:?] + at io.openlineage.spark.agent.lifecycle.SparkSQLExecutionContext.end(SparkSQLExecutionContext.java:141) ~[openlineage-spark-agent_2.13-1.9.0-SNAPSHOT-shadow.jar:?] + at io.openlineage.spark.agent.OpenLineageSparkListener.sparkSQLExecEnd(OpenLineageSparkListener.java:100) ~[openlineage-spark-agent_2.13-1.9.0-SNAPSHOT-shadow.jar:?] + at io.openlineage.spark.agent.OpenLineageSparkListener.onOtherEvent(OpenLineageSparkListener.java:86) ~[openlineage-spark-agent_2.13-1.9.0-SNAPSHOT-shadow.jar:?] + at org.apache.spark.scheduler.SparkListenerBus.doPostEvent(SparkListenerBus.scala:100) ~[spark-core_2.13-3.4.2.jar:3.4.2] + at org.apache.spark.scheduler.SparkListenerBus.doPostEvent$(SparkListenerBus.scala:28) ~[spark-core_2.13-3.4.2.jar:3.4.2] + at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) ~[spark-core_2.13-3.4.2.jar:3.4.2] + at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) ~[spark-core_2.13-3.4.2.jar:3.4.2] + at org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:117) ~[spark-core_2.13-3.4.2.jar:3.4.2] + at org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:101) ~[spark-core_2.13-3.4.2.jar:3.4.2] + at org.apache.spark.scheduler.AsyncEventQueue.super$postToAll(AsyncEventQueue.scala:105) ~[spark-core_2.13-3.4.2.jar:3.4.2] + at org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:105) ~[spark-core_2.13-3.4.2.jar:3.4.2] + at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.scala:17) ~[scala-library-2.13.11.jar:?] + at scala.util.DynamicVariable.withValue(DynamicVariable.scala:59) ~[scala-library-2.13.11.jar:?] + at <a href="http://org.apache.spark.scheduler.AsyncEventQueue.org">org.apache.spark.scheduler.AsyncEventQueue.org</a>$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:100) ~[spark-core_2.13-3.4.2.jar:3.4.2] + at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:96) ~[spark-core_2.13-3.4.2.jar:3.4.2] + at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1471) [spark-core_2.13-3.4.2.jar:3.4.2] + at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:96) [spark-core_2.13-3.4.2.jar:3.4.2] + Caused by: io.openlineage.spark.shaded.org.apache.http.conn.HttpHostConnectException: Connect to localhost:1081 [localhost/127.0.0.1] failed: Connection refused (Connection refused) +Got it on the linux box

      + + + +
      + 🙂 Maciej Obuchowski +
      + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 12:29:04
       +
      +

      *Thread Reply:* ```parkDeltaIntegrationTest STANDARD_OUT + 2024-02-14 18:26:07 WARN org.apache.kafka.clients.NetworkClient - [AdminClient clientId=adminclient-1] Connection to node 1 (localhost/127.0.0.1:32801) could not be established. Broker may not be available. + 2024-02-14 18:26:08 INFO io.openlineage.spark.agent.SparkDeltaIntegrationTest - Waiting for mock server to start on port 1082

      + +

      SparkDeltaIntegrationTest > testMergeInto() STANDARDOUT + 2024-02-14 18:26:08 ERROR io.openlineage.spark.agent.EventEmitter - Could not emit lineage w/ exception + io.openlineage.client.OpenLineageClientException: io.openlineage.spark.shaded.org.apache.http.conn.HttpHostConnectException: Connect to localhost:1081 [localhost/127.0.0.1] failed: Connection refused (Connection refused) + at io.openlineage.client.transports.HttpTransport.emit(HttpTransport.java:128) ~[openlineage-spark-agent2.13-1.9.0-SNAPSHOT-shadow.jar:1.9.0-SNAPSHOT] + at io.openlineage.client.transports.HttpTransport.emit(HttpTransport.java:111) ~[openlineage-spark-agent2.13-1.9.0-SNAPSHOT-shadow.jar:1.9.0-SNAPSHOT] + at io.openlineage.client.OpenLineageClient.emit(OpenLineageClient.java:46) ~[openlineage-spark-agent2.13-1.9.0-SNAPSHOT-shadow.jar:1.9.0-SNAPSHOT] + at io.openlineage.spark.agent.EventEmitter.emit(EventEmitter.java:69) ~[openlineage-spark-agent2.13-1.9.0-SNAPSHOT-shadow.jar:?] + at io.openlineage.spark.agent.OpenLineageSparkListener.emitApplicationEvent(OpenLineageSparkListener.java:321) ~[openlineage-spark-agent2.13-1.9.0-SNAPSHOT-shadow.jar:?] + at io.openlineage.spark.agent.OpenLineageSparkListener.emitApplicationStartEvent(OpenLineageSparkListener.java:325) ~[openlineage-spark-agent2.13-1.9.0-SNAPSHOT-shadow.jar:?] + at io.openlineage.spark.agent.OpenLineageSparkListener.onApplicationStart(OpenLineageSparkListener.java:247) ~[openlineage-spark-agent2.13-1.9.0-SNAPSHOT-shadow.jar:?] + at org.apache.spark.scheduler.SparkListenerBus.doPostEvent(SparkListenerBus.scala:55) ~[spark-core2.13-3.4.2.jar:3.4.2] + at org.apache.spark.scheduler.SparkListenerBus.doPostEvent$(SparkListenerBus.scala:28) ~[spark-core2.13-3.4.2.jar:3.4.2] + at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) ~[spark-core2.13-3.4.2.jar:3.4.2] + at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) ~[spark-core2.13-3.4.2.jar:3.4.2] + at org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:117) ~[spark-core2.13-3.4.2.jar:3.4.2] + at org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:101) ~[spark-core2.13-3.4.2.jar:3.4.2] + at org.apache.spark.scheduler.AsyncEventQueue.super$postToAll(AsyncEventQueue.scala:105) ~[spark-core2.13-3.4.2.jar:3.4.2] + at org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:105) ~[spark-core2.13-3.4.2.jar:3.4.2] + at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.scala:17) ~[scala-library-2.13.11.jar:?] + at scala.util.DynamicVariable.withValue(DynamicVariable.scala:59) ~[scala-library-2.13.11.jar:?] + at org.apache.spark.scheduler.AsyncEventQueue.org$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:100) ~[spark-core2.13-3.4.2.jar:3.4.2] + at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:96) ~[spark-core2.13-3.4.2.jar:3.4.2] + at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1471) [spark-core2.13-3.4.2.jar:3.4.2] + at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:96) [spark-core2.13-3.4.2.jar:3.4.2]```

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 12:29:20
       +
      +

      *Thread Reply:* So, the fact that the test proceeds, means that the mock server has started.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 12:29:49
       +
      +

      *Thread Reply:* Because I have this

      + +

      ``` @Override + public void beforeAll(ExtensionContext context) throws Exception { + int port = basePort.getAndIncrement(); + Configuration configuration = new Configuration(); + configuration.logLevel(Level.ERROR); + clientAndServer = ClientAndServer.startClientAndServer(configuration, port); + clientAndServer + .when(request("/api/v1/lineage")) + .respond(HttpResponse.response().withStatusCode(201));

      + +
      if (context.getTestClass().isPresent()) {
+  Logger logger = LoggerFactory.getLogger(context.getTestClass().get());
+  Awaitility.await("wait-for-mock-server-start-up")
+      .atMost(Duration.ofMinutes(2))
+      .pollInterval(Duration.ofSeconds(1))
+      .until(
+          () -&gt; {
+            <a href="http://logger.info">logger.info</a>(
+                "Waiting for mock server to start on port {}", clientAndServer.getPort());
+            return clientAndServer.isRunning();
+          });
+}
+
      + +

      }```

      + + + +
      + 👍 Maciej Obuchowski +
      + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 12:30:34
       +
      +

      *Thread Reply:* So there's another reason why the connection is being refused.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 12:35:35
       +
      +

      *Thread Reply:* https://github.com/mock-server/mockserver/issues/498

      +
      + + + + +
      #498 Mocking request fails with 'Connection refused'
      +
      + <p>I try to mock an external request that fails with the following stacktrace</p> + +<pre><code>org.mockserver.client.netty.SocketConnectionException: Unable to connect to socket localhost/127.0.0.1:1080 + at org.mockserver.client.netty.NettyHttpClient.sendRequest(NettyHttpClient.java:97) + at org.mockserver.client.MockServerClient.sendRequest(MockServerClient.java:137) + at org.mockserver.client.MockServerClient.sendExpectation(MockServerClient.java:514) + at org.mockserver.client.ForwardChainExpectation.respond(ForwardChainExpectation.java:32) + at com....mock() + at java.lang.Thread.run(Thread.java:748) +Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:1080 + at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) + at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) + at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:325) + at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) + at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:633) + at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) + at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) + at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) + at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886) + at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) + ... 1 common frames omitted +Caused by: java.net.ConnectException: Connection refused + ... 11 common frames omitted +</code></pre> + +<p>The mockserver will be used as a proxy (by setting system properties 'http.proxyHost' and 'http.proxyPort') and should answer the requests to external resources that have been mocked but forward everything else.</p> + +<p>For me it seems that the mockserver is not so good in dealing with concurrent requests?! Normally we use concurrent JUnit test execution (on class and suite level) and I had to revert the external resource tests to become sequentially executed because of similar issues as mentioned above.</p> + +<p>Is it possible to enable debug log to get more insights to what happen within the mockserver.</p> +
      + +
      +
      Assignees
      + <a href="https://github.com/jamesdbloom">@jamesdbloom</a> +
      + +
      +
      Labels
      + need_more_information +
      + + + + + + + +
      + + <a href="https://github.com/mock-server/mockserver">mock-server/mockserver</a> +
      + + +
      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 12:40:27
       +
      +

      *Thread Reply:* What also has me confused, is that I see a lot of Kafka Admin Client related configurations.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 12:40:40
       +
      +

      *Thread Reply:* Being unable to connect to a broker. Which is super strange.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-02-14 12:42:12
       +
      +

      *Thread Reply:* I would try to see if this happens on the containers test in particular

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 12:44:00
       +
      +

      *Thread Reply:* With the container test, I see that some of the tests are showing 404's

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 12:44:09
       +
      +

      *Thread Reply:* But, they pass.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 12:44:13
       +
      +

      *Thread Reply:* Which is strange, imo.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 13:43:23
       +
      +

      *Thread Reply:* Further digging:

      + +

      If I run one test class in isolation, the tests go through.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 13:43:39
       +
      +

      *Thread Reply:* However, if I run multiple tests that require mock server, bad things happen.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 13:43:58
       +
      +

      *Thread Reply:* i.e., the connection refused errors appear

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-14 13:44:18
       +
      +

      *Thread Reply:* is this related to a shared mock server or shared spark session?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 13:44:47
       +
      +

      *Thread Reply:* Are you asking about shared within the test class, or shared across test classes?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-14 13:45:01
       +
      +

      *Thread Reply:* shared across test classes

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 13:45:16
       +
      +

      *Thread Reply:* In that case, neither the spark session nor the mock server are shared.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 13:45:59
       +
      +

      *Thread Reply:* For example, SparkDeltaIntegrationTest and SparkIcebergIntergrationTest

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 13:46:07
       +
      +

      *Thread Reply:* They each get their own servers and sessions

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 13:46:19
       +
      +

      *Thread Reply:* However, if you run them in isolation, they will individually pass

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 13:47:03
       +
      +

      *Thread Reply:* But, if you run them in parallel or sequentially (but in the same JVM process), the one that executes after will fail.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-14 13:52:46
       +
      +

      *Thread Reply:* that's weird. They can still share derby location or /tmp/whatver where data is stored

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 13:57:28
       +
      +

      *Thread Reply:* For example: ./gradlew integrationTest -x test -x buildDockerImage -Pscala.binary.version=2.13 -Pspark.version=3.4.2 --tests SparkDeltaIntegrationTest --tests SparkGenericIntegrationTest --tests SparkIcebergIntegrationTest

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 13:57:58
       +
      +

      *Thread Reply:* I started seeing the connection refused error, around the 2nd last test of the SparkDeltaIntegrationTest

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 13:58:18
       +
      +

      *Thread Reply:* which makes me wonder if MockServer only wants 1 instance per JVM process

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 13:58:31
       +
      +

      *Thread Reply:* Or are we hitting heapsize limits

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 13:58:49
       +
      +

      *Thread Reply:* i.e., the JVM is running out of resources and that's why we're seeing connection refused

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-14 13:59:28
       +
      +

      *Thread Reply:* mockserver is way to heavy for what it's doing. that's why maciej trully hates it

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-14 14:00:45
       +
      +

      *Thread Reply:* you can reduce paralellism to 1 and see if tests suceed. this will take ages but at least we would this

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 14:01:00
       +
      +

      *Thread Reply:* I think I already did that

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 14:01:46
       +
      +

      *Thread Reply:* Oh no

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 14:01:50
       +
      +

      *Thread Reply:* I removed the setting

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-14 14:02:43
       +
      +

      *Thread Reply:* which one?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 14:02:57
       +
      +

      *Thread Reply:* Fork every and max parallel

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 14:05:34
       +
      +

      *Thread Reply:* Rnning the test with jvmArgs("-Xmx4096m", "-XX:MaxPermSize=512m") now

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 14:07:14
       +
      +

      *Thread Reply:* Nope.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 14:07:17
       +
      +

      *Thread Reply:* That did nothing.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 14:14:06
       +
      +

      *Thread Reply:* OK. So I did something completely the opposite

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 14:14:11
       +
      +

      *Thread Reply:* I increased the parallelism

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 14:14:23
       +
      +

      *Thread Reply:* I said, max parallel forks = 5

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 14:14:30
       +
      +

      *Thread Reply:* I had to change port assignment

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-14 14:14:38
       +
      +

      *Thread Reply:* i run the command locally and have a lot of io.openlineage.client.OpenLineageClientException: <a href="http://io.openlineage.spark.shaded.org">io.openlineage.spark.shaded.org</a>.apache.http.conn.HttpHostConnectException: Connect to localhost:1081 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 14:14:48
       +
      +

      *Thread Reply:* Yup

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-14 14:14:52
       +
      +

      *Thread Reply:* sth is stopping mockserver while other test tries to connect it

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 14:15:09
       +
      +

      *Thread Reply:* I think MockServer isn't cleaning up after itself properly

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 14:15:38
       +
      +

      *Thread Reply:* SUCCESS: Executed 26 tests in 1m 31s

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 14:18:26
       +
      +

      *Thread Reply:* With:

      + +

      tasks.register("integrationTest", Test.class) { + group = "verification" + dependsOn(integrationTestDependencies) + testClassesDirs = testSourceSet.output.classesDirs + classpath = files(tasks.shadowJar.outputs.files.singleFile, sourceSets.test.runtimeClasspath) + useJUnitPlatform { + includeTags("integration-test") + excludeTags("databricks") + logger.warn("[IntegrationTest] hasDeltaDependencies: ${hasDeltaDependencies(spark, scala)}") + if (!hasDeltaDependencies(spark, scala)) { + logger.warn("[IntegrationTest] Excluding delta tests") + excludeTags("delta") + } + logger.warn("[IntegrationTest] hasIcebergDependencies: ${hasIcebergDependencies(spark, scala)}") + if (!hasIcebergDependencies(spark, scala)) { + logger.warn("[IntegrationTest] Excluding iceberg tests") + excludeTags("iceberg") + } + } + systemProperties.put("test.results.dir", buildDirectory.dir("test-results/${name}/o").get().asFile.absolutePath) + setForkEvery(1) + maxParallelForks = 5 + setMaxHeapSize("1024m") +}

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 14:18:55
       +
      +

      *Thread Reply:* I get this:

      + +

      SUCCESS: Executed 26 tests in 1m 32s

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-14 14:21:00
       +
      +

      *Thread Reply:* I am also seeing java.lang.ClassCastException: org.apache.spark.sql.catalyst.analysis.ResolvedIdentifier cannot be cast to org.apache.spark.sql.catalyst.analysis.ResolvedTable which don't make tests fail

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 14:21:18
       +
      +

      *Thread Reply:* Yeah - that's something to do with Delta casting

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 14:21:39
       +
      +

      *Thread Reply:* The code is trying to cast table to an identifier

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 14:21:51
       +
      +

      *Thread Reply:* (I didn't touch that)

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-14 14:29:11
       +
      +

      *Thread Reply:* hmyy, i thought resetting mockServer also clears the expectations

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-14 14:29:50
       +
      +

      *Thread Reply:* and this

      + +
              .when(request("/api/v1/lineage"))
+        .respond(HttpResponse.response().withStatusCode(201))
+
      + +

      is also an expectation

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 14:38:46
       +
      +

      *Thread Reply:* If it was an expectation

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 14:38:54
       +
      +

      *Thread Reply:* wouldn't it clear after the first test?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-14 14:40:33
       +
      +

      *Thread Reply:* ``` 2024-02-14 20:38:02 INFO io.openlineage.spark.agent.SparkDeltaIntegrationTest - Stopping mockserver on port 1923

      + +

      2024-02-14 20:38:02 WARN org.apache.spark.sql.SparkSession - An existing Spark session exists as the active or default session. + This probably means another suite leaked it. Attempting to stop it before continuing. + This existing Spark session was created at:

      + +

      org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:1035) + io.openlineage.spark.agent.SparkDeltaIntegrationTest.beforeEach(SparkDeltaIntegrationTest.java:111) + sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) + sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) + sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) + java.lang.reflect.Method.invoke(Method.java:498) + org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:728) + org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60) + org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131) + org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156) + org.junit.jupiter.engine.extension.TimeoutExtension.interceptLifecycleMethod(TimeoutExtension.java:128) + org.junit.jupiter.engine.extension.TimeoutExtension.interceptBeforeEachMethod(TimeoutExtension.java:78) + org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103) + org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93) + org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106) + org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64) + org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45) + org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37) + org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:92) + org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:86)

      + +

      2024-02-14 20:38:02 INFO io.openlineage.spark.agent.SparkDeltaIntegrationTest - Stopping mockserver on port 1923

      + +

      SparkGenericIntegrationTest STANDARD_OUT + 2024-02-14 20:38:03 INFO io.openlineage.spark.agent.SparkGenericIntegrationTest - Waiting for mock server to start on port 1930

      + +

      SparkGenericIntegrationTest > sparkEmitsApplicationLevelEvents() STANDARDOUT + 2024-02-14 20:38:03 ERROR io.openlineage.spark.agent.EventEmitter - Could not emit lineage w/ exception + io.openlineage.client.OpenLineageClientException: io.openlineage.spark.shaded.org.apache.http.NoHttpResponseException: localhost:1923 failed to respond + at io.openlineage.client.transports.HttpTransport.emit(HttpTransport.java:128) ~[openlineage-spark-agent2.13-1.9.0-SNAPSHOT-shadow.jar:1.9.0-SNAPSHOT] + at io.openlineage.client.transports.HttpTransport.emit(HttpTransport.java:111) ~[openlineage-spark-agent2.13-1.9.0-SNAPSHOT-shadow.jar:1.9.0-SNAPSHOT] + at io.openlineage.client.OpenLineageClient.emit(OpenLineageClient.java:60) ~[openlineage-spark-agent2.13-1.9.0-SNAPSHOT-shadow.jar:1.9.0-SNAPSHOT] + at io.openlineage.spark.agent.EventEmitter.emit(EventEmitter.java:69) ~[openlineage-spark-agent2.13-1.9.0-SNAPSHOT-shadow.jar:?] + at io.openlineage.spark.agent.OpenLineageSparkListener.emitApplicationEvent(OpenLineageSparkListener.java:321) ~[openlineage-spark-agent2.13-1.9.0-SNAPSHOT-shadow.jar:?] + at io.openlineage.spark.agent.OpenLineageSparkListener.emitApplicationStartEvent(OpenLineageSparkListener.java:325) ~[openlineage-spark-agent2.13-1.9.0-SNAPSHOT-shadow.jar:?] + at io.openlineage.spark.agent.OpenLineageSparkListener.onApplicationStart(OpenLineageSparkListener.java:247) ~[openlineage-spark-agent2.13-1.9.0-SNAPSHOT-shadow.jar:?] + at org.apache.spark.scheduler.SparkListenerBus.doPostEvent(SparkListenerBus.scala:55) ~[spark-core2.13-3.4.2.jar:3.4.2] + at org.apache.spark.scheduler.SparkListenerBus.doPostEvent$(SparkListenerBus.scala:28) ~[spark-core2.13-3.4.2.jar:3.4.2] + at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) ~[spark-core2.13-3.4.2.jar:3.4.2] + at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) ~[spark-core2.13-3.4.2.jar:3.4.2] + at org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:117) ~[spark-core2.13-3.4.2.jar:3.4.2] + at org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:101) ~[spark-core2.13-3.4.2.jar:3.4.2] + at org.apache.spark.scheduler.AsyncEventQueue.super$postToAll(AsyncEventQueue.scala:105) ~[spark-core2.13-3.4.2.jar:3.4.2] + at org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:105) ~[spark-core2.13-3.4.2.jar:3.4.2] + at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.scala:17) ~[scala-library-2.13.11.jar:?] + at scala.util.DynamicVariable.withValue(DynamicVariable.scala:59) ~[scala-library-2.13.11.jar:?] + at org.apache.spark.scheduler.AsyncEventQueue.org$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:100) ~[spark-core2.13-3.4.2.jar:3.4.2] + at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:96) ~[spark-core2.13-3.4.2.jar:3.4.2] + at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1471) [spark-core2.13-3.4.2.jar:3.4.2] + at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:96) [spark-core2.13-3.4.2.jar:3.4.2] + Caused by: io.openlineage.spark.shaded.org.apache.http.NoHttpResponseException: localhost:1923 failed to respond``` +I added log to notify mockServer is stopped. I see it being stopped and then something tries to connect it.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 14:42:01
       +
      +

      *Thread Reply:* Hmm - I find that really strange, that SparkGenericIntegrationTest, is trying to connect to SparkDeltaIntegrationTestt's mock server

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 14:42:20
       +
      +

      *Thread Reply:* It should have a new mock server injected

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 14:42:39
       +
      +

      *Thread Reply:* Unless that atomic integer isn't working as expected

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-14 14:42:43
       +
      +

      *Thread Reply:* first, I changed atomicInteger to random but this did not help

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 14:42:52
       +
      +

      *Thread Reply:* Yeah, I did the same.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 14:43:18
       +
      +

      *Thread Reply:* Does this mean, MockServer is keeping state across tests?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 14:45:26
       +
      +

      *Thread Reply:* ```State can be cleared from MockServer selectively:

      + +

      use type to select which type of state to clear, supported values are: all, log, expectations +use a request matcher to clear matching items +use an expectation id to clear matching items```

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 14:46:06
       +
      +

      *Thread Reply:* new MockServerClient("localhost", 1080).clear( + request(), + ClearType.LOG +);

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 15:01:07
       +
      +

      *Thread Reply:* So

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 15:01:17
       +
      +

      *Thread Reply:* Using the official MockServerExtension

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 15:01:24
       +
      +

      *Thread Reply:* Results in the same behaviour.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 15:05:44
       +
      +

      *Thread Reply:* Yeah - I see it.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 15:06:04
       +
      +

      *Thread Reply:* SparkDeltaIntegrationTest completes, and then SparkGenericIntegrationTest starts

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 15:06:24
       +
      +

      *Thread Reply:* SparkGenericIntegrationTest is trying to connect to the same port as SparkDeltaIntegrationTest

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 15:14:11
       +
      +

      *Thread Reply:* OK. Using a single JVM process

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 15:14:20
       +
      +

      *Thread Reply:* I have managed to get the tests to run

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 15:14:32
       +
      +

      *Thread Reply:* without them failing due to connection refused

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 15:14:53
       +
      +

      *Thread Reply:* however, they are failing due to request matching

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 15:15:04
       +
      +

      *Thread Reply:* I really don't think mockserver is thread safe

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 15:15:51
       +
      +

      *Thread Reply:* This is what I've done to the tests:

      + +

      ```import org.mockserver.junit.jupiter.MockServerExtension; +import org.mockserver.junit.jupiter.MockServerSettings;

      + +

      @ExtendWith(MockServerExtension.class) +@MockServerSettings(ports = {SparkIcebergIntegrationTest.MOCKSERVERPORT}, perTestSuite = true) +public class SparkIcebergIntegrationTest { + public static final int MOCKSERVERPORT = 1084;

      + +

      public SparkIcebergIntegrationTest(ClientAndServer mockServer) { + this.mockServer = mockServer; + this.mockServer + .when(request("/api/v1/lineage")) + .respond(org.mockserver.model.HttpResponse.response().withStatusCode(201)); + }

      + +

      public void beforeEach() { + mockServer.clear(request(), ClearType.LOG);

      + +

      // stuff +} +}```

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 15:19:43
       +
      +

      *Thread Reply:* Pretty crazy though, that if you fork. Things work.

      + + + +
      + 🙌 Paweł Leszczyński +
      + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 15:25:13
       +
      +

      *Thread Reply:* Btw, to get the official extension:

      + +

      testImplementation("org.mock-server:mockserver_netty:5.14.0:shaded") { + exclude group: 'com.google.guava', module: 'guava' + exclude group: 'com.fasterxml.jackson.core' + exclude group: 'com.fasterxml.jackson.datatype' + exclude group: 'com.fasterxml.jackson.dataformat' + exclude group: 'org.mock-server.mockserver-client-java' + } + testImplementation("org.mock-server:mockserver_junit_jupiter_no_dependencies:5.14.0")

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 16:31:31
       +
      +

      *Thread Reply:* OK. I am happy to say that we've gotten the failures down to the same tests across all variants.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 16:32:20
       +
      +

      *Thread Reply:* ```io.openlineage.spark.agent.GoogleCloudIntegrationTest +> testRddWriteToBucket() +> testReadAndWriteFromBigquery()

      + +

      io.openlineage.spark.agent.lifecycle.SparkReadWriteIntegTest +> testExternalRDDWithS3Bucket(SparkSession)```

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 16:32:37
       +
      +

      *Thread Reply:* I will say, the last one (the class version error) is the one that I'm unsure how to solve.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 16:33:34
       +
      +

      *Thread Reply:* Perhaps something is bringing in the JDK 11 variant? Maybe we try and exclude the variant from the classpath?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-14 17:40:19
       +
      +

      *Thread Reply:* org.glassfish.jaxb:jaxb-runtime - yeah ... this one is the reason for the failure with the S3 test.

      + +

      I removed spark-mllib during the migration, because I couldn't find any references to it. AS it turns out, the Google BigQuery stuff requires it. So I added it back, without excluding glassfish.

      + + + +
      + 👍 Paweł Leszczyński +
      + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-15 03:23:38
       +
      +

      *Thread Reply:* I think I got It 😉

      + +

      https://github.com/OpenLineage/OpenLineage/pull/2432/files#diff-5dc301670bf7398239b365b562a80e759ab1bb4f0e3f3d39617b0adedc061eedL106

      + +

      Why are you removing OpenLineageSparkListener setting? This made the GoogleCloudIntegrationTest succeed on my end.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 03:41:41
       +
      +

      *Thread Reply:* Ah

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 03:41:47
       +
      +

      *Thread Reply:* Good one

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 03:42:01
       +
      +

      *Thread Reply:* I remember I was rearranging the args, to make sure the OL stuff was grouped together.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 03:48:59
       +
      +

      *Thread Reply:* Thanks for your help @Paweł Leszczyński

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 04:37:14
       +
      +

      *Thread Reply:* Hmm - testExternalRddWithS3Bucket fails with:

      + +

      Caused by: java.io.FileNotFoundException: No such file or directory: <s3a://openlineage-test/rdd_a_3.3.4/_temporary/0/task_202402150933081796269000648724168_0000_m_000001/part-00001-b5762fba-cb9a-401d-9f14-73cda8c6706c-c000.snappy.parquet>

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-15 04:46:34
       +
      +

      *Thread Reply:* 2.12 integration tests are failing because of 404 on mockserver

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 04:47:03
       +
      +

      *Thread Reply:* I just realised why the bigquery tests are failing

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 04:47:16
       +
      +

      *Thread Reply:* Not because of the code, but because the fixtures are no longer valid

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 04:47:21
       +
      +

      *Thread Reply:* err

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 04:47:22
       +
      +

      *Thread Reply:* the events

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 04:47:43
       +
      +

      *Thread Reply:* "name": "openlineage-ci.airflow_integration.{spark_version}_target" must be changed to "name": "openlineage-ci.airflow_integration.{spark_version}_{scala}_target"

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 04:47:49
       +
      +

      *Thread Reply:* For all bigquery events

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 04:48:05
       +
      +

      *Thread Reply:* Probably for the MetaStore2 and MetaStore3 events

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 04:52:44
       +
      +

      *Thread Reply:* Regarding the 404:

      + +

      https://app.circleci.com/pipelines/github/OpenLineage/OpenLineage/9322/workflows/d03fc2fb-9a74-446d-83e8-cc69fb798dde/jobs/175615/tests

      + +

      This one didn't experience those failures, but it is 3.5.0, so it may not have the same tests running

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 04:53:17
       +
      +

      *Thread Reply:* https://app.circleci.com/pipelines/github/OpenLineage/OpenLineage/9322/workflows/d03fc2fb-9a74-446d-83e8-cc69fb798dde/jobs/175611/tests

      + +

      Same with this one. It is only the bigquery test failing.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 04:53:41
       +
      +

      *Thread Reply:* https://app.circleci.com/pipelines/github/OpenLineage/OpenLineage/9322/workflows/d03fc2fb-9a74-446d-83e8-cc69fb798dde/jobs/175614/tests

      + +

      This one is different though

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-15 06:42:47
       +
      +

      *Thread Reply:* pls run spotless, so we could see what's still failing

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 06:59:43
       +
      +

      *Thread Reply:* Yeah

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 06:59:45
       +
      +

      *Thread Reply:* Just did it

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-15 07:00:29
       +
      +

      *Thread Reply:* I am off next week and would love to merge it this week 😉

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 07:00:43
       +
      +

      *Thread Reply:* That makes two of us

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-15 08:00:13
       +
      +

      *Thread Reply:* I need to recall what has Metastore test to do with google storage.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-15 08:00:21
       +
      +

      *Thread Reply:* That's the failing test for now

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 08:00:29
       +
      +

      *Thread Reply:* It uses it for storage of the files

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-15 08:00:54
       +
      +

      *Thread Reply:* does it expect some content of the storage initially?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-15 08:01:07
       +
      +

      *Thread Reply:* i don't get why it's suceeding for 3.5.0

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 08:02:17
       +
      +

      *Thread Reply:* It succeeds for 3.4.2 and 3.5.0.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 08:02:37
       +
      +

      *Thread Reply:* And the 2.12 variant of 3.3.4

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-15 08:09:10
       +
      +

      *Thread Reply:* logs are also flooded with io.openlineage.client.OpenLineageClientException: code: 404, response:

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 08:12:09
       +
      +

      *Thread Reply:* THat's for the container tests

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 08:12:11
       +
      +

      *Thread Reply:* afaik

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 08:12:24
       +
      +

      *Thread Reply:* And I see that I haven't changed the mockserver container at all

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 08:12:31
       +
      +

      *Thread Reply:* (compared with main)

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-02-15 08:12:40
       +
      +

      *Thread Reply:* if 2.12 tests and 2.13 refer to the same warehouse path maybe they overwrite each other's data?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 08:12:52
       +
      +

      *Thread Reply:* I've changed that

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-02-15 08:12:53
       +
      +

      *Thread Reply:* ➜ ~ gcloud storage ls <gs://openlineage-ci-testing/warehouse> +<gs://openlineage-ci-testing/warehouse/3.1.3/> +<gs://openlineage-ci-testing/warehouse/3.2.2/> +<gs://openlineage-ci-testing/warehouse/3.2.4/> +<gs://openlineage-ci-testing/warehouse/3.3.0/> +<gs://openlineage-ci-testing/warehouse/3.3.1/> +<gs://openlineage-ci-testing/warehouse/3.3.2/> +<gs://openlineage-ci-testing/warehouse/3.3.3/> +<gs://openlineage-ci-testing/warehouse/3.3.4/> +<gs://openlineage-ci-testing/warehouse/3.4.0/> +<gs://openlineage-ci-testing/warehouse/3.4.1/> +<gs://openlineage-ci-testing/warehouse/3.4.2/> +<gs://openlineage-ci-testing/warehouse/3.5.0/> +<gs://openlineage-ci-testing/warehouse/no_iceberg_test.db/> +<gs://openlineage-ci-testing/warehouse/spark-3_2_4/> +<gs://openlineage-ci-testing/warehouse/spark-3_3_4/> +<gs://openlineage-ci-testing/warehouse/spark-3_4_2/> +<gs://openlineage-ci-testing/warehouse/spark-3_5_0/> +I don't think your tests created any new paths there

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-02-15 08:13:22
       +
      +

      *Thread Reply:* also the errors indicate that +org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 1 times, most recent failure: Lost task 0.0 in stage 1.0 (TID 2) (ip-10-0-109-117.ec2.internal executor driver): org.apache.iceberg.exceptions.RuntimeIOException: Failed to get status for file: <gs://openlineage-ci-testing/warehouse/3.3.4/hive3/test/data/00000-0-b84bfcf2-3c44-47c6-9da9-c6e420db237c-00001.parquet>

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-02-15 08:13:29
       +
      +

      *Thread Reply:* there's no scala version in the path

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 08:13:54
       +
      +

      *Thread Reply:* java.io.IOException: Path: //openlineage-ci-testing/warehouse/spark-3_3_4/scala-2_13/hive3/test_3_3_4_2_13/data is a directory, which is not supported by the record reader when `mapreduce.input.fileinputformat.input.dir.recursive` is false.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 08:14:18
       +
      +

      *Thread Reply:* org.apache.iceberg.exceptions.NotFoundException: Failed to open input stream for file: <gs://openlineage-ci-testing/warehouse/spark-3_3_4/scala-2_13/hive3/test_3_3_4_2_13/metadata/00001-2862ac34-7a06-4b6a-8c13-b07446745975.metadata.json>

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-02-15 08:15:25
       +
      +

      *Thread Reply:* ah it's has the underscores now

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-02-15 08:15:27
       +
      +

      *Thread Reply:* ➜ ~ gcloud storage ls <gs://openlineage-ci-testing/warehouse/spark-3_3_4/> +<gs://openlineage-ci-testing/warehouse/spark-3_3_4/scala-2_12/> +<gs://openlineage-ci-testing/warehouse/spark-3_3_4/scala-2_13/>

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 08:16:30
       +
      +

      *Thread Reply:* Yeah - I created my own GCP project

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-02-15 08:16:32
       +
      +

      *Thread Reply:* I see, I was looking at the earlier failures

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 08:16:34
       +
      +

      *Thread Reply:* Just to test this locally

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-15 08:21:39
       +
      +

      *Thread Reply:* https://github.com/OpenLineage/OpenLineage/pull/2432/files#diff-137aa17091138b69681510e13e3b7d66aa9c9c7c81fe8fe13f09f0de76448dd5R48 -> this is using same port as mockserver container is using

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-15 08:22:14
       +
      +

      *Thread Reply:* I looked at todays ci build on main and there are no 404 errors

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-02-15 08:24:17
       +
      +

      *Thread Reply:* Caused by: java.io.FileNotFoundException: Item not found: 'gs://****************************-testing/warehouse/spark-3_3_4/scala-2_13/hive3/test_3_3_4_2_13/metadata/00001-0906d19b-1c0b-4e62-9cd5-b35a5410b0d8.metadata.json'. Note, it is possible that the live version is still available but the requested generation is deleted. +this sounds to me like there is some concurrency

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-15 08:24:39
       +
      +

      *Thread Reply:* it gets retried 4 times

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-15 08:24:45
       +
      +

      *Thread Reply:* so chances are low

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-15 08:24:55
       +
      +

      *Thread Reply:* to me it looks like something is not able to clear the path

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-15 08:25:08
       +
      +

      *Thread Reply:* but why would it clear properly for other versions

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-02-15 08:25:20
       +
      +

      *Thread Reply:* yeah so maybe it's red herring, it might fail at +2024-02-15 12:23:38 ERROR EventEmitter - Could not emit lineage w/ exception + io.openlineage.client.OpenLineageClientException: code: 404, response: + at io.openlineage.client.transports.HttpTransport.throwOnHttpError(HttpTransport.java:150) ~[openlineage-spark-agent_2.13-1.9.0-SNAPSHOT-shadow.jar:1.9.0-SNAPSHOT] +and then there is something leftover?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-15 08:25:49
       +
      +

      *Thread Reply:* to me 404 are results of two tests using the same mockserver and resetting it

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-02-15 08:26:24
       +
      +

      *Thread Reply:* yes, it might fail on that, then it retries and then fails earlier because of storage leftovers?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-02-15 08:26:51
       +
      +

      *Thread Reply:* all the more reasons to get rid of mockserver 🙂

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-02-15 08:26:57
       +
      +

      *Thread Reply:* one day 🙂

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-15 08:28:35
       +
      +

      *Thread Reply:* this would make sense assuming retry wouldn't do @BeforeAll nor @AfterAll when failing

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 08:28:41
       +
      +

      *Thread Reply:* Dude ... I wrote a "FileOutputExtension" to replace MockServer

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 08:28:45
       +
      +

      *Thread Reply:* and then I deleted it

      + + + +
      + 😅 Paweł Leszczyński +
      + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 08:29:21
       +
      +

      *Thread Reply:* I regret it now.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-02-15 08:29:22
       +
      +

      *Thread Reply:* I started something few weeks ago https://github.com/OpenLineage/OpenLineage/commit/604f35ee66c164550b2b08c87a043d5a0beed62d

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-15 08:30:20
       +
      +

      *Thread Reply:* @Damien Hawes could we change port of ColumnLineageIntegrationTest from 1080 to something else

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-15 08:30:21
       +
      +

      *Thread Reply:* ?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 08:30:26
       +
      +

      *Thread Reply:* Sure

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 08:30:39
       +
      +

      *Thread Reply:* 1090 sound good?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-15 08:31:00
       +
      +

      *Thread Reply:* yes, as mockserver container is not using this

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 08:31:45
       +
      +

      *Thread Reply:* @Maciej Obuchowski - yeah, I was using the file transport and then used ObjectMapper + JSONAssert from org.skyscreamer to perform a weak match on the expected and actual json

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 08:31:50
       +
      +

      *Thread Reply:* It worked really well

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 08:31:52
       +
      +

      *Thread Reply:* but I was like

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 08:31:56
       +
      +

      *Thread Reply:* "Too much change"

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 08:31:59
       +
      +

      *Thread Reply:* so I deleted it

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-15 08:36:01
       +
      +

      *Thread Reply:* what a pity @Maciej Obuchowski;)

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-02-15 08:36:39
       +
      +

      *Thread Reply:* yeah would be best as a separate PR 🙂

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-02-15 08:36:52
       +
      +

      *Thread Reply:* maybe randomize the mockserver port?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-02-15 08:37:04
       +
      +

      *Thread Reply:* 1/65536 is pretty low chance of collision

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 08:37:13
       +
      +

      *Thread Reply:* I did that before

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 08:37:17
       +
      +

      *Thread Reply:* I deleted that code

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 08:37:22
       +
      +

      *Thread Reply:* BEcause turns out

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 08:37:29
       +
      +

      *Thread Reply:* mockserver hates running in the same thread

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 08:37:32
       +
      +

      *Thread Reply:* across multiple tests

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 08:37:39
       +
      +

      *Thread Reply:* same JVM**

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-15 08:37:56
       +
      +

      *Thread Reply:* one day we will rewrite it to rust, scala, github actions, non-mocksercer whatever but for now let's just juggle mockserver ports to make s.it work 😉

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 08:38:57
       +
      +

      *Thread Reply:* Scala 🤮

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 08:39:00
       +
      +

      *Thread Reply:* After this

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 08:39:08
       +
      +

      *Thread Reply:* I never want to touch Scala again

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-02-15 08:39:27
       +
      +

      *Thread Reply:* I actually wish we've started all of this in Scala, we'd not have to do any of it

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-02-15 08:40:04
       +
      +

      *Thread Reply:* But even like 2 years ago we've thought it's too much work, and now we have 5x more code and related problems

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-02-15 08:40:13
       +
      +

      *Thread Reply:* Sunken cost fallacy

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 08:40:18
       +
      +

      *Thread Reply:* Yup

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 09:01:37
       +
      +

      *Thread Reply:* WARNING: An illegal reflective access operation has occurred + WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/opt/bitnami/spark/jars/spark-unsafe_2.12-3.2.4.jar) to constructor java.nio.DirectByteBuffer(long,int) + WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform + WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations + WARNING: All illegal access operations will be denied in a future release +Hmm, has bitnami upgraded the JDK on their images?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 09:08:35
       +
      +

      *Thread Reply:* @Paweł Leszczyński - when I run the column lineage test, in isolation, on my local

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 09:08:50
       +
      +

      *Thread Reply:* 2024-02-15 15:07:56 WARN NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable +2024-02-15 15:07:58 ERROR EventEmitter - Could not emit lineage w/ exception +io.openlineage.client.OpenLineageClientException: code: 404, response: + at io.openlineage.client.transports.HttpTransport.throwOnHttpError(HttpTransport.java:150) ~[openlineage-spark-agent_2.12-1.9.0-SNAPSHOT-shadow.jar:1.9.0-SNAPSHOT] + at io.openlineage.client.transports.HttpTransport.emit(HttpTransport.java:124) ~[openlineage-spark-agent_2.12-1.9.0-SNAPSHOT-shadow.jar:1.9.0-SNAPSHOT] + at io.openlineage.client.transports.HttpTransport.emit(HttpTransport.java:111) ~[openlineage-spark-agent_2.12-1.9.0-SNAPSHOT-shadow.jar:1.9.0-SNAPSHOT] + at io.openlineage.client.OpenLineageClient.emit(OpenLineageClient.java:46) ~[openlineage-spark-agent_2.12-1.9.0-SNAPSHOT-shadow.jar:1.9.0-SNAPSHOT] + at io.openlineage.spark.agent.EventEmitter.emit(EventEmitter.java:69) ~[openlineage-spark-agent_2.12-1.9.0-SNAPSHOT-shadow.jar:?]

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 09:09:12
       +
      +

      *Thread Reply:*

      + +
      +
      image.png
      + + + + + + + + +
      + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 09:12:36
       +
      +

      *Thread Reply:* I wonder

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 09:12:38
       +
      +

      *Thread Reply:* I wonder

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-02-15 09:13:30
       +
      +

      *Thread Reply:* The illegal reflective stuff should not actually break anything right now afaik

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 09:13:57
       +
      +

      *Thread Reply:* @Paweł Leszczyński @Maciej Obuchowski

      + +
              .when(request("/api/v1/lineage"))
+        .respond(org.mockserver.model.HttpResponse.response().withStatusCode(201))
+
      + +

      However, when I examine debug logs and the code I see:

      + +

      2024-02-15 15:11:12 INFO IcebergHandler - {spark.openlineage.transport.url=<http://localhost:1090/api/v1/namespaces/default>, .... +.config( + "spark.openlineage.transport.url", + "<http://localhost>:" + MOCKSERVER_PORT + "/api/v1/namespaces/default")

      + + + +
      + 👀 Maciej Obuchowski +
      + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-02-15 09:16:01
       +
      +

      *Thread Reply:* Ah, so you think it emits to the wrong port?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 09:16:11
       +
      +

      *Thread Reply:* No. To the wrong resource

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 09:16:18
       +
      +

      *Thread Reply:* The error is 404

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-02-15 09:16:31
       +
      +

      *Thread Reply:* Oh yes, why it is namespaces endpoint?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 09:16:36
       +
      +

      *Thread Reply:* I don't know

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 09:16:44
       +
      +

      *Thread Reply:* I don't think it was me

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-02-15 09:16:55
       +
      +

      *Thread Reply:* Yeah, just wondering

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 09:20:17
       +
      +

      *Thread Reply:* I know

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 09:20:20
       +
      +

      *Thread Reply:* I figured it out

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-15 09:20:25
       +
      +

      *Thread Reply:* ?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 09:20:32
       +
      +

      *Thread Reply:* ``` public ColumnLineageIntegrationTest(ClientAndServer mockServer) { + this.mockServer = mockServer; + mockServer + .when(request("/api/v1/lineage")) + .respond(org.mockserver.model.HttpResponse.response().withStatusCode(201)); + }

      + +

      @SneakyThrows + @BeforeAll + public static void setup() { + METASTORECONTAINER.start(); + mappedPort = METASTORECONTAINER.getMappedPort(MetastoreTestUtils.POSTGRESPORT); + spark = getSparkSession(); + Arrays.asList("v2source1", "v2source_2") + .forEach(e -> spark.sql("drop table if exists " + e)); + getIcebergTable(spark, 1); + getIcebergTable(spark, 2); + databaseUrl = String.format("jdbc::%s/%s", mappedPort, database); + }```

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 09:20:46
       +
      +

      *Thread Reply:* The 404's are coming from the @BeforeAll lifecycle method

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 09:20:51
       +
      +

      *Thread Reply:* MockServer hasn't been created or configured by then. MockServer is instance scoped, but the method is static scope.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 09:23:54
       +
      +

      *Thread Reply:* So it's possible that via the extension, MockServer is instantiated and is listening on that port, but no expectations have been configured.

      + + + +
      + 😬 Maciej Obuchowski +
      + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 09:24:23
       +
      +

      *Thread Reply:* So MockServer is like, "404 everything"

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-15 09:24:35
       +
      +

      *Thread Reply:* yes, that's possible

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 11:32:11
       +
      +

      *Thread Reply:* Oh - regarding the Scala conversation. It would require a change to the architecture of the solution.

      + +

      We'd probably have to follow the route of the spline folks. Namely: 1 artifact per spark x scala version. In our case, we'd have to consider producing a Scala 2.11 build just for 2.4.8.

      + +

      (I know this entire conversation is entirely hypothetical)

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-15 11:33:19
       +
      +

      *Thread Reply:* 1. SparkContainerIntegrationTest is throwing 404s.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-02-15 14:39:36
       +
      +

      *Thread Reply:* I think any large effort like this would be more likely to target moving some of the code to Spark itself, or to enhance it's APIs to make developing this kind of integration easier

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-16 04:37:44
       +
      +

      *Thread Reply:* More thoughts @Paweł Leszczyński @Maciej Obuchowski:

      + +
      1. Spark 3.5.0 doesn't fail on the Metastore tests, because it doesn't have iceberg, despite the fact that in metastore 2 we say no iceberg and metastore 3 we say yes to iceberg
        2. +
      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-16 04:38:42
       +
      +

      *Thread Reply:* But I'm still at a loss for this: +java.io.IOException: Path: //openlineage-ci-testing/warehouse/spark-3_3_4/scala-2_12/hive3/test_3_3_4_2_12/data is a directory, which is not supported by the record reader when `mapreduce.input.fileinputformat.input.dir.recursive` is false.

      + + + +
      + 👀 Maciej Obuchowski, Paweł Leszczyński +
      + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-16 04:56:58
       +
      +

      *Thread Reply:* Oh wait, I see metastore 3 has an @Tag("iceberg") on its unit test

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-16 04:56:59
       +
      +

      *Thread Reply:* nevermind

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-16 04:58:32
       +
      +

      *Thread Reply:* Spark 3.5.0 won't run it

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-02-16 05:02:20
       +
      +

      *Thread Reply:* java.io.IOException: Path: //openlineage-ci-testing/warehouse/spark-3_3_4/scala-2_12/hive3/test_3_3_4_2_12/data is a directory, which is not supported by the record reader when `mapreduce.input.fileinputformat.input.dir.recursive` is false. +that confuses me too

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-16 05:02:35
       +
      +

      *Thread Reply:* Yup

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-16 05:11:32
       +
      +

      *Thread Reply:* Wait

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-16 05:11:39
       +
      +

      *Thread Reply:* What's the point of the Metastore tests?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-16 05:11:57
       +
      +

      *Thread Reply:* What are we testing there?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-16 05:12:30
       +
      +

      *Thread Reply:* interaction with datasets registered and accessed through Hive Metastore

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-16 05:12:35
       +
      +

      *Thread Reply:* I mean, we're not evening loading the OpenLineageSparkListener

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-16 05:13:10
       +
      +

      *Thread Reply:* I mean, this is what we're testing:

      + +

      void IcebergTablesTest() { + executeSql("create database if not exists %s", database); + executeSql("drop table if exists %s.%s", database, table); + executeSql( + "create external table %s.%s (id int, value string) USING iceberg location '%s'", + database, table, MetastoreTestUtils.getTableLocation(database, table)); + executeSql("insert into table %s.%s VALUES (1, 'value1'), (2, 'value2')", database, table); + Dataset&lt;Row&gt; rowDataset = executeSql(String.format("select ** from %s.%s", database, table)); + List&lt;Row&gt; rows = rowDataset.collectAsList(); + assertThat(rows).hasSize(2); + assertThat(rows.get(0).get(0)).isEqualTo(1); + }

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-16 05:13:58
       +
      +

      *Thread Reply:* This is what the Spark Conf looks like:

      + +

      ``` public static SparkConf getCommonSparkConf( + String appName, String metastoreName, int mappedPort, Boolean isIceberg) { + SparkConf conf = + new SparkConf() + .setAppName(appName + VERSION) + .setMaster("local[**]") + .set("spark.driver.host", LOCALIP) + .set("org.jpox.autoCreateSchema", "true") + .set( + "javax.jdo.option.ConnectionURL", + String.format("jdbc::%d/%s", mappedPort, metastoreName)) + .set("javax.jdo.option.ConnectionDriverName", "org.postgresql.Driver") + .set("javax.jdo.option.ConnectionUserName", "hiveuser") + .set("javax.jdo.option.ConnectionPassword", "password") + .set("spark.sql.warehouse.dir", BASEPATH) + .set("spark.hadoop.google.cloud.project.id", GOOGLESAPROPERTIES.get("projectid")) + .set("spark.hadoop.google.cloud.auth.service.account.enable", "true") + .set( + "fs.gs.auth.service.account.private.key.id", + GOOGLESAPROPERTIES.get("privatekeyid")) + .set("fs.gs.auth.service.account.private.key", GOOGLESAPROPERTIES.get("privatekey")) + .set("fs.gs.auth.service.account.email", GOOGLESAPROPERTIES.get("client_email")) + .set("fs.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem") + .set("fs.AbstractFileSystem.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS");

      + +
      if (isIceberg) {
+  conf.set("spark.sql.catalog.spark_catalog", "org.apache.iceberg.spark.SparkCatalog")
+      .set("spark.sql.catalog.spark_catalog.type", "hive")
+      .set(
+          "spark.sql.extensions",
+          "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions");
+}
+return conf;
+
      + +

      }```

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-16 05:14:40
       +
      +

      *Thread Reply:* ^ that's on main

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-16 05:14:57
       +
      +

      *Thread Reply:* So we're testing whether the hive metastore works on GCS?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-16 05:15:35
       +
      +

      *Thread Reply:* Which confuses me, because shouldn't that test be done in Hive's test suite and not ours?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-02-16 05:15:52
       +
      +

      *Thread Reply:* ... 👀

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-16 05:18:40
       +
      +

      *Thread Reply:* Unless there's a good reason for these tests, that don't actually test our code, I'm going to disable the tests. Do you agree @Paweł Leszczyński @Maciej Obuchowski?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-02-16 05:23:40
       +
      +

      *Thread Reply:* I feel this is unfinished feature

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-02-16 05:27:29
       +
      +

      *Thread Reply:* I feel like we can disable it

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-02-16 05:27:41
       +
      +

      *Thread Reply:* but still don't understand why it fails, maybe it will always fail in 2.13?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-16 05:45:22
       +
      +

      *Thread Reply:* If I test it locally, under 3.2.4 and 2.13, it passes.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-16 06:03:56
       +
      +

      *Thread Reply:* this looks like unfinished feature with a first PR to introduce spark+real hive metastore environment and other PRs never happened later. i don't know why it's not clearing the path. this can be related to a changed hadoop client dependencies. I think I don't mind turning test off and creating a separate issue to fix it later.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-16 06:44:47
       +
      +

      *Thread Reply:* And there we go

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-16 06:44:54
       +
      +

      *Thread Reply:* All integration tests for Spark pass

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-16 07:05:01
       +
      +

      *Thread Reply:* :gh_merged:

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-16 07:05:16
       +
      +

      *Thread Reply:* Wait

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-16 07:05:20
       +
      +

      *Thread Reply:* with 65 commits squashed?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-16 07:05:40
       +
      +

      *Thread Reply:* I was just about to squash them

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-16 07:06:16
       +
      +

      *Thread Reply:* Including commits like, "Hello Spotless my old friend"

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-16 07:06:19
       +
      +

      *Thread Reply:* xD

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-16 07:06:33
       +
      +

      *Thread Reply:* sry 😉

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-02-16 07:06:41
       +
      +

      *Thread Reply:* We have squash on merge set up

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-16 07:07:01
       +
      +

      *Thread Reply:* Aye, but I didn't want this:

      + +
      +
      image.png
      + + + + + + + + +
      + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-16 07:07:07
       +
      +

      *Thread Reply:* I wanted to summarise my changes

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-16 07:07:08
       +
      +

      *Thread Reply:* 🙂

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-02-16 07:07:37
       +
      +

      *Thread Reply:* Let's fix this by adding changelog entry?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-16 07:13:11
       +
      +

      *Thread Reply:* Good point

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-16 07:18:41
       +
      +

      *Thread Reply:* btw @Paweł Leszczyński - I had an idea of how to reduce the duplication in the CI config

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-16 07:18:49
       +
      +

      *Thread Reply:* we can use YAML references/pointers

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-16 07:19:10
       +
      +

      *Thread Reply:* And add a parameter for scala binary version

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-16 07:19:14
       +
      +

      *Thread Reply:* I liked the idea of mocksever extension

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-16 07:19:46
       +
      +

      *Thread Reply:* it got reverted, perhaps this was too many improvements at once but this was cool

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-16 07:19:58
       +
      +

      *Thread Reply:* I can reintroduce it, because they publish their own extension

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-16 07:20:05
       +
      +

      *Thread Reply:* which I used, after deleting my own

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-16 07:20:22
       +
      +

      *Thread Reply:* But the tests do need to be adjusted though

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-16 07:20:46
       +
      +

      *Thread Reply:* so that we don't rely on anything in static space

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-16 07:28:04
       +
      +

      *Thread Reply:* My desired solution would be to write events to file with file transport and then verify events from that with no need for extra servers and ports. Maciej would prefer some lightweight server container to have a generic solution unrelated if it is java or python.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-16 07:28:36
       +
      +

      *Thread Reply:* Yeah - I want to go the file solution as well

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-16 07:28:38
       +
      +

      *Thread Reply:* let's see who will be the first 😉

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-16 07:28:41
       +
      +

      *Thread Reply:* It's less resource consumption

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-16 07:28:55
       +
      +

      *Thread Reply:* It took me an hour to write it.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-16 07:29:03
       +
      +

      *Thread Reply:* Including the verification

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-16 07:29:29
       +
      +

      *Thread Reply:* does it require to modify all the existing tests?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-16 07:29:44
       +
      +

      *Thread Reply:* It basically looked the exact same as the current mock server verifyEvents method

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-16 07:29:56
       +
      +

      *Thread Reply:* The only difference was instead of passing in a mock server instance

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-16 07:30:11
       +
      +

      *Thread Reply:* I passed in the file location (java.nio.file.Path)

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-16 07:30:36
       +
      +

      *Thread Reply:* > does it require to modify all the existing tests? +Regardless of which ever approach you take, you will have to modify the existing tests.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-16 07:31:25
       +
      +

      *Thread Reply:* yes, but do we need to modify assertion files with expected OL events

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-16 07:31:26
       +
      +

      *Thread Reply:* If I were to redo it though, I would write it as an extension

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-16 07:31:39
       +
      +

      *Thread Reply:* You mean with replacements?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-16 07:31:47
       +
      +

      *Thread Reply:* Because if yes, I already did that 😓

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-16 07:32:19
       +
      +

      *Thread Reply:* https://github.com/OpenLineage/OpenLineage/tree/main/integration/spark/app/integrations/container

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-16 07:32:37
       +
      +

      *Thread Reply:* can we reuse those files?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-16 07:32:44
       +
      +

      *Thread Reply:* Yes

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-16 07:32:58
       +
      +

      *Thread Reply:* As I said, the only thing that changed in the method

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-16 07:33:04
       +
      +

      *Thread Reply:* was the type of the first parameter

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-16 07:33:15
       +
      +

      *Thread Reply:* it sounds cool to me

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-16 07:33:18
       +
      +

      *Thread Reply:* ClientAndServer -> java.nio.file.Path

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-16 07:33:42
       +
      +

      *Thread Reply:* Yeah but I deleted the code, because I was like "Too many changes"

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-16 07:33:50
       +
      +

      *Thread Reply:* but it won't be hard to add it back

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-16 07:33:57
       +
      +

      *Thread Reply:* I think

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-16 07:33:58
       +
      +

      *Thread Reply:* I wouldn't miss mockserver at all 😉

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-16 07:34:08
       +
      +

      *Thread Reply:* unless I didn't commit it

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-16 07:40:15
       +
      +

      *Thread Reply:* Yup

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-16 07:40:18
       +
      +

      *Thread Reply:* I committed it

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-16 07:40:21
       +
      +

      *Thread Reply:* managed to recover it

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-16 07:41:21
       +
      +

      *Thread Reply:* This is what I had @Paweł Leszczyński

      + +

      ``` private static final ObjectReader READER = new ObjectMapper().findAndRegisterModules().reader();

      + +

      static void verifyEvents(Path filePath, String... eventFiles) { + verifyEvents(filePath, ignored -> true, Collections.emptyMap(), eventFiles); + }

      + +

      static void verifyEvents(Path filePath, Predicate<JsonNode> eventFilter, String... eventFiles) { + verifyEvents(filePath, eventFilter, Collections.emptyMap(), eventFiles); + }

      + +

      static void verifyEvents(Path filePath, Map<String, String> replacements, String... eventFiles) { + verifyEvents(filePath, ignored -> true, replacements, eventFiles); + }

      + +

      @SneakyThrows + static List<OpenLineage.RunEvent> getEmittedEvents(Path filePath) { + List<String> lines = Files.readAllLines(filePath); + return lines.stream() + .map(OpenLineageClientUtils::runEventFromJson) + .collect(Collectors.toList()); + }

      + +

      @SneakyThrows + static void verifyEvents( + Path filePath, + Predicate<JsonNode> eventFilter, + Map<String, String> replacements, + String... expectedEventFileNames) { + Path eventsDir = Paths.get("integrations/container/");

      + +
      final List&lt;JsonNode&gt; actualJsonObjects =
+    Files.readAllLines(filePath).stream()
+        .map(MockServerUtils::sneakyRead)
+        .filter(eventFilter)
+        .collect(Collectors.toList());
+
+final List&lt;JsonNode&gt; expectedJsonObjects =
+    Arrays.stream(expectedEventFileNames)
+        .map(eventsDir::resolve)
+        .map(MockServerUtils::sneakyRead)
+        .map(json -&gt; doReplace(json, replacements))
+        .map(MockServerUtils::sneakyRead)
+        .collect(Collectors.toList());
+
+IntStream.range(0, expectedJsonObjects.size())
+    .forEach(
+        index -&gt; {
+          JsonNode actualJson = actualJsonObjects.get(index);
+          JsonNode expectedJson = expectedJsonObjects.get(index);
+          try {
+            <a href="http://log.info">log.info</a>("Comparing {} to {}", expectedJson, actualJson);
+            JSONAssert.assertEquals(expectedJson.toString(), actualJson.toString(), false);
+          } catch (JSONException e) {
+            Assertions.fail("Failed to match JSON", e);
+          }
+        });
+
      + +

      }

      + +

      @SneakyThrows + private static JsonNode sneakyRead(String json) { + return MockServerUtils.READER.readTree(json); + }

      + +

      @SneakyThrows + private static String sneakyRead(Path path) { + return new String(Files.readAllBytes(path)); + }

      + +

      private static String doReplace(String s, Map<String, String> replacements) { + String result = s; + for (Map.Entry<String, String> entry : replacements.entrySet()) { + result = result.replace(entry.getKey(), entry.getValue()); + } + return result; + }```

      + + + +
      + ❤️ Paweł Leszczyński +
      + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-16 07:43:30
       +
      +

      *Thread Reply:* Why was the predicate needed? Because MockServer can match certain events, for example, we only asked it to match on start and complete events, however the listener would publish running events.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-16 07:44:05
       +
      +

      *Thread Reply:* So the predicate can be used to omit running events before the comparison takes place

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-02-16 07:52:28
       +
      +

      *Thread Reply:* you need to make sure you're not comparing stuff like timestamps or UUIDs that are generated for each run

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-16 07:52:39
       +
      +

      *Thread Reply:* Aye

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-16 07:52:52
       +
      +

      *Thread Reply:* That's this line:

      + +

      JSONAssert.assertEquals(expectedJson.toString(), actualJson.toString(), false);

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-02-16 07:52:55
       +
      +

      *Thread Reply:* On Airflow side we have solution with jinja templates, that's why I wanted to reuse that code

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-16 07:53:01
       +
      +

      *Thread Reply:* The "false" means, don't do strict matching

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-12 08:07:29
       +
      +

      https://github.com/OpenLineage/OpenLineage/pull/2419

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Mattia Bertorello + (mattia.bertorello@booking.com) +
      +
      2024-02-13 08:10:19
       +
      +

      Hey team! +There is a high probability that we would need an expansion to the Kafka transport to support IAM authentication in MSK.

      + +

      First, can I put it in the open lineage repository, or will we create an internal transport? +Then, do you prefer some additional functionality at the already existing Kafka transport or a new one that extends the Kafka one? In the second one, we can separate the dependencies.

      + +

      https://github.com/aws/aws-msk-iam-sasl-signer-python

      +
      + + + + +
      aws/aws-msk-iam-sasl-signer-python
      +
      + +
      + +
      +
      Stars
      + 16 +
      + +
      +
      Language
      + Python +
      + + + + + + + + +
      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-02-13 08:33:22
       +
      +

      *Thread Reply:* We'd prefer to not put additional dependencies as large as +import boto3 +import botocore.session +to generic client that everyone using OpenLineage will download

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Mattia Bertorello + (mattia.bertorello@booking.com) +
      +
      2024-02-13 08:35:18
       +
      +

      *Thread Reply:* So what about a openlineage-python[msk] ?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Jakub Dardziński + (jakub.dardzinski@getindata.com) +
      +
      2024-02-13 09:01:45
       +
      +

      *Thread Reply:* I’m ok with adding it as extra dependency with extending the existing Kafka transport

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-02-13 10:26:35
       +
      +

      *Thread Reply:* I'm okay with openlineage-python[msk] too

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Mattia Bertorello + (mattia.bertorello@booking.com) +
      +
      2024-02-13 10:28:04
       +
      +

      *Thread Reply:* Ok perfect.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Julien Le Dem + (julien@apache.org) +
      +
      2024-02-13 19:44:13
       +
      +

      *Thread Reply:* +1

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-19 04:08:30
       +
      +

      @Paweł Leszczyński @Maciej Obuchowski - can I get this one reviewed: https://github.com/OpenLineage/OpenLineage/pull/2446

      +
      + + + + +
      #2446 [SPARK]Added the capability to publish Scala 2.12 and 2.13 variants of `openlineage-spark`
      +
      + <p><strong>Problem</strong></p> + +<p>Recent changes to the Spark integration have resulted us in being able to compile and test with Scala 2.12 and Scala 2.13.</p> + +<p>Now, we need to publish things.</p> + +<p><strong>Solution</strong></p> + +<p>Here, we add a new publish step to the <code>release-integration-spark</code> step.</p> + +<p>First, we publish with <code>-Pscala.binary.version=2.12</code> and then <code>-Pscala.binary.version=2.13</code>.</p> + +<p>We also modify the <code>integration/spark/build.gradle</code> file to source the correct variants of the modules.</p> + +<p>We also updated the docs.</p> + +<p><strong>One-line summary: Added the capability to publish Scala 2.12 and 2.13 variants of <code>openlineage-spark</code></strong> +<strong>Checklist</strong></p> + +<p>☑︎ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work +☑︎ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> +☑︎ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained</p> + +<hr /> + +<p>SPDX-License-Identifier: Apache-2.0<br /> +Copyright 2018-2023 contributors to the OpenLineage project</p> +
      + +
      +
      Labels
      + documentation, integration/spark, ci +
      + +
      +
      Assignees
      + <a href="https://github.com/d-m-h">@d-m-h</a> +
      + + + + + + + +
      + + <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a> +
      + + +
      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-02-19 05:39:54
       +
      +

      *Thread Reply:* I will do it. @Paweł Leszczyński is on vacation this week, so expect slower response time unfortunately

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-19 05:40:10
       +
      +

      *Thread Reply:* Ah right, I remember now.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-19 08:11:52
       +
      +

      *Thread Reply:* https://github.com/OpenLineage/docs/pull/286

      +
      + + + + +
      #286 Updated the docs regarding the OpenLineage Spark integration
      +
      + <p>As the title suggests, this clarifies the new approach that we're taking.</p> + +<p>This is because of <a href="https://github.com/OpenLineage/OpenLineage/pull/2446">OpenLineage/OpenLineage#2446</a></p> +
      + +
      +
      Assignees
      + <a href="https://github.com/d-m-h">@d-m-h</a> +
      + + + + + + + +
      + + <a href="https://github.com/OpenLineage/docs">OpenLineage/docs</a> +
      + + +
      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-02-19 09:46:55
       +
      +

      @Damien Hawes @Mattia Bertorello is this ⬆️ and Iceberg PR enough for 2.13 support?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Damien Hawes + (damien.hawes@booking.com) +
      +
      2024-02-19 09:57:40
       +
      +

      *Thread Reply:* I believe so

      + + + +
      + 👍 Maciej Obuchowski +
      + +
      + 🎉 tati +
      + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Mattia Bertorello + (mattia.bertorello@booking.com) +
      +
      2024-02-20 09:57:55
       +
      +

      Hey! I would like to discuss this solution about DBT wrapper https://github.com/OpenLineage/OpenLineage/pull/2449 +A stakeholder is concern about a possibile failure that can occur when emit the event from DBT event if the job is successful.

      + +

      Do you think we can make of catching any exception a default one or we need to put a "feature flag" with an environment variable?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-02-20 10:18:44
       +
      +

      *Thread Reply:* I think we can make it default

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-02-20 10:18:56
       +
      +

      *Thread Reply:* It's not "breaking change", if anything it makes it more resilient

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-02-20 10:19:28
       +
      +

      *Thread Reply:* I would change the +<a href="http://logger.info">logger.info</a>(f"Emitted {len(events) + 2} openlineage events") +to accurately count emitted events and display failed ones too

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Mattia Bertorello + (mattia.bertorello@booking.com) +
      +
      2024-02-20 10:20:07
       +
      +

      *Thread Reply:* Nice catch

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Mattia Bertorello + (mattia.bertorello@booking.com) +
      +
      2024-02-20 10:20:45
       +
      +

      *Thread Reply:* Obviously, in this case, it could happen that the job is successful, and there is no lineage.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Mattia Bertorello + (mattia.bertorello@booking.com) +
      +
      2024-02-20 10:24:46
       +
      +

      *Thread Reply:* Done, I've done a emitted_events counter

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Mattia Bertorello + (mattia.bertorello@booking.com) +
      +
      2024-02-23 04:39:20
       +
      +

      *Thread Reply:* Hi Maciej, +Could we merge this PR #2449 before release the 1.9.0? +Do I still need to work on something?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-02-23 04:46:39
       +
      +

      *Thread Reply:* probably yes, sorry for not merging it earlier

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-02-23 09:13:07
       +
      +

      *Thread Reply:* @Mattia Bertorello it's merged 🙂

      + + + +
      + :gratitude_thank_you: Mattia Bertorello +
      + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Mattia Bertorello + (mattia.bertorello@booking.com) +
      +
      2024-02-23 09:38:42
       +
      +

      *Thread Reply:* Thanks 🙏

      + + + +
      +
      +
      +
      + + + +
      +
      + + + + \ No newline at end of file diff --git a/channel/spec-compliance/index.html b/channel/spec-compliance/index.html index 482b4f5..c69d24c 100644 --- a/channel/spec-compliance/index.html +++ b/channel/spec-compliance/index.html @@ -46,6 +46,18 @@

      Public Channels

      +
    • + + # github-discussions + +
      • + +
    • + + # github-notifications + +
      • +
    • # gx-integration @@ -94,6 +106,12 @@

      Public Channels
      • +
    • + + # spark-support-multiple-scala-versions + +
      • +
    • # spec-compliance @@ -106,6 +124,12 @@

      Public Channels
      • +
    • + + # user-generated-metadata + +
      • +

    Private Channels

    Private Channels

      @@ -158,7 +182,7 @@

      Group Direct Messages

      - +
      @@ -404,7 +428,7 @@

      Group Direct Messages

      - +
      diff --git a/channel/user-generated-metadata/index.html b/channel/user-generated-metadata/index.html new file mode 100644 index 0000000..afa1629 --- /dev/null +++ b/channel/user-generated-metadata/index.html @@ -0,0 +1,713 @@ + + + + + + Slack Export - #user-generated-metadata + + + + + +
      + + + +
      + + + +
      +
      + + + + +
      + +
      Julien Le Dem + (julien@apache.org) +
      +
      2021-05-21 18:36:08
       +
      +

      @Julien Le Dem has joined the channel

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Stephen Pimentel + (stephenpiment@gmail.com) +
      +
      2021-05-24 16:57:29
       +
      +

      @Stephen Pimentel has joined the channel

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Laurent Paris + (laurent@datakin.com) +
      +
      2021-05-24 20:11:54
       +
      +

      @Laurent Paris has joined the channel

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Willy Lulciuc + (willy@datakin.com) +
      +
      2021-05-25 19:35:12
       +
      +

      @Willy Lulciuc has joined the channel

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Mario Measic + (mario.measic.gavran@gmail.com) +
      +
      2021-05-26 06:28:57
       +
      +

      @Mario Measic has joined the channel

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Kevin Mellott + (kevin.r.mellott@gmail.com) +
      +
      2021-05-28 00:15:20
       +
      +

      @Kevin Mellott has joined the channel

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Pedram + (pedram@hightouch.io) +
      +
      2021-06-01 17:49:18
       +
      +

      @Pedram has joined the channel

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Kedar Rajwade + (kedar@cloudzealous.com) +
      +
      2021-06-08 10:01:02
       +
      +

      @Kedar Rajwade has joined the channel

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Arthur Wiedmer + (awiedmer@apple.com) +
      +
      2021-06-08 12:31:45
       +
      +

      @Arthur Wiedmer has joined the channel

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Supratim Mukherjee + (smukh.work@gmail.com) +
      +
      2021-06-17 11:50:16
       +
      +

      @Supratim Mukherjee has joined the channel

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maayan Salom + (maayan172s@gmail.com) +
      +
      2021-07-11 05:10:12
       +
      +

      @Maayan Salom has joined the channel

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Rifa Achrinza + (rifa@achrinza.com) +
      +
      2021-07-18 00:06:15
       +
      +

      @Rifa Achrinza has joined the channel

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      William Angel + (buildingwill@gmail.com) +
      +
      2021-07-20 14:55:54
       +
      +

      @William Angel has joined the channel

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Joe Regensburger + (jregensburger@immuta.com) +
      +
      2021-08-04 13:33:49
       +
      +

      @Joe Regensburger has joined the channel

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Luke Smith + (luke.smith@kinandcarta.com) +
      +
      2021-08-20 13:51:36
       +
      +

      @Luke Smith has joined the channel

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      marko + (marko.kristian.helin@gmail.com) +
      +
      2021-08-24 09:53:27
       +
      +

      @marko has joined the channel

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Jason Ma + (jason.ma@alation.com) +
      +
      2021-08-30 16:32:41
       +
      +

      @Jason Ma has joined the channel

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Gibson Wasukira + (gwasukira@gmail.com) +
      +
      2021-09-01 10:12:54
       +
      +

      @Gibson Wasukira has joined the channel

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Yuki Tannai + (tannai-yuki@dmm.com) +
      +
      2021-09-08 09:41:12
       +
      +

      @Yuki Tannai has joined the channel

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      davzucky + (davzucky@hotmail.com) +
      +
      2021-09-17 02:20:49
       +
      +

      @davzucky has joined the channel

      + + + +
      +
      +
      +
      + + + +
      +
      + + + + \ No newline at end of file diff --git a/index.html b/index.html index d5a5c9f..bcc1320 100644 --- a/index.html +++ b/index.html @@ -46,6 +46,18 @@

      Public Channels

      +
    • + + # github-discussions + +
      • + +
    • + + # github-notifications + +
      • +
    • # gx-integration @@ -94,6 +106,12 @@

      Public Channels
      • +
    • + + # spark-support-multiple-scala-versions + +
      • +
    • # spec-compliance @@ -106,6 +124,12 @@

      Public Channels
      • +
    • + + # user-generated-metadata + +
      • +

    Private Channels

      @@ -1162,7 +1186,7 @@

      Group Direct Messages

      #1 initial spec proposal
      - <p>The goal is to specify the initial model (just plain lineage) that will be extended with various facets. <br /> + <p>The goal is to specify the initial model (just plain lineage) that will be extended with various facets.<br /> It does not intend to restrict to HTTP. Those same PUT calls without output can be translated to any async protocol</p> <p>Signed-off-by: Julien Le Dem <a href="mailto:julien@apache.org">julien@apache.org</a></p> @@ -1337,7 +1361,7 @@

      Group Direct Messages

      #1 initial spec proposal
      - <p>The goal is to specify the initial model (just plain lineage) that will be extended with various facets. <br /> + <p>The goal is to specify the initial model (just plain lineage) that will be extended with various facets.<br /> It does not intend to restrict to HTTP. Those same PUT calls without output can be translated to any async protocol</p> <p>Signed-off-by: Julien Le Dem <a href="mailto:julien@apache.org">julien@apache.org</a></p> @@ -1551,7 +1575,7 @@

      Group Direct Messages

      #1 initial spec proposal
      - <p>The goal is to specify the initial model (just plain lineage) that will be extended with various facets. <br /> + <p>The goal is to specify the initial model (just plain lineage) that will be extended with various facets.<br /> It does not intend to restrict to HTTP. Those same PUT calls without output can be translated to any async protocol</p> <p>Signed-off-by: Julien Le Dem <a href="mailto:julien@apache.org">julien@apache.org</a></p> @@ -1644,7 +1668,7 @@

      Group Direct Messages

      #1 initial spec proposal
      - <p>The goal is to specify the initial model (just plain lineage) that will be extended with various facets. <br /> + <p>The goal is to specify the initial model (just plain lineage) that will be extended with various facets.<br /> It does not intend to restrict to HTTP. Those same PUT calls without output can be translated to any async protocol</p> <p>Signed-off-by: Julien Le Dem <a href="mailto:julien@apache.org">julien@apache.org</a></p> @@ -3095,11 +3119,11 @@

      Group Direct Messages

      #15 [PROPOSAL] Move the OpenLineage spec to JonSchema (from OpenAPI)
      - <p><strong>Purpose:</strong> <br /> -OpenAPI and JsonSchema are mostly the same model with a goal of OpenAPI to be a strict superset of JSONSchema in the near future. <br /> + <p><strong>Purpose:</strong><br /> +OpenAPI and JsonSchema are mostly the same model with a goal of OpenAPI to be a strict superset of JSONSchema in the near future.<br /> The goal is to make OpenLineage independent from the underlying protocol. OpenAPI assumes a REST endpoint which is not our intent here. This would still make possible to define an OpenLineage HTTP endpoint while decoupling the model and make it more obvious that it can be used with Kafka for example.</p> -<p><strong>Proposed implementation</strong> <br /> +<p><strong>Proposed implementation</strong><br /> Convert the OpenAPI spec to a json schema.</p>
      @@ -3168,7 +3192,7 @@

      Group Direct Messages

      #17 adding Json-Schema for OpenLineage
      - <p>resolves <a href="https://github.com/OpenLineage/OpenLineage/issues/15">#15</a> <br /> + <p>resolves <a href="https://github.com/OpenLineage/OpenLineage/issues/15">#15</a><br /> Signed-off-by: Julien Le Dem <a href="mailto:julien@apache.org">julien@apache.org</a></p>
      @@ -3465,7 +3489,7 @@

      Group Direct Messages

      #17 adding Json-Schema for OpenLineage
      - <p>resolves <a href="https://github.com/OpenLineage/OpenLineage/issues/15">#15</a> <br /> + <p>resolves <a href="https://github.com/OpenLineage/OpenLineage/issues/15">#15</a><br /> Signed-off-by: Julien Le Dem <a href="mailto:julien@apache.org">julien@apache.org</a></p>
      @@ -3689,12 +3713,12 @@

      Group Direct Messages

      #20 [PROPOSAL] Add InputFacets and OutputFacets
      - <p><strong>Purpose:</strong> <br /> -We currently have JobFacets for everything that does not change every run (example: the current version in source control like the git sha). We have RunFacets for everything that changes every Run (example: the schedule time). <br /> -We are missing the same notion for input and output datasets. <br /> + <p><strong>Purpose:</strong><br /> +We currently have JobFacets for everything that does not change every run (example: the current version in source control like the git sha). We have RunFacets for everything that changes every Run (example: the schedule time).<br /> +We are missing the same notion for input and output datasets.<br /> Currently we only have Dataset facets for things that don't change every run (example: Schema). I propose to add Input and Output facets to capture things that change in datasets every run (ex: number of row produced, etc)</p> -<p><strong>Proposed implementation</strong> <br /> +<p><strong>Proposed implementation</strong><br /> Add InputFacets and OutputFacets in the core model. Directly in the Input or Output field.</p> <pre><code>"inputs": { @@ -3789,12 +3813,12 @@

      Group Direct Messages

      #20 [PROPOSAL] Add InputFacets and OutputFacets
      - <p><strong>Purpose:</strong> <br /> -We currently have JobFacets for everything that does not change every run (example: the current version in source control like the git sha). We have RunFacets for everything that changes every Run (example: the schedule time). <br /> -We are missing the same notion for input and output datasets. <br /> + <p><strong>Purpose:</strong><br /> +We currently have JobFacets for everything that does not change every run (example: the current version in source control like the git sha). We have RunFacets for everything that changes every Run (example: the schedule time).<br /> +We are missing the same notion for input and output datasets.<br /> Currently we only have Dataset facets for things that don't change every run (example: Schema). I propose to add Input and Output facets to capture things that change in datasets every run (ex: number of row produced, etc)</p> -<p><strong>Proposed implementation</strong> <br /> +<p><strong>Proposed implementation</strong><br /> Add InputFacets and OutputFacets in the core model. Directly in the Input or Output field.</p> <pre><code>"inputs": { @@ -3884,7 +3908,7 @@

      Group Direct Messages

      #23 add input and output facets
      - <p>This PR resolves <a href="https://github.com/OpenLineage/OpenLineage/issues/20">#20</a> <br /> + <p>This PR resolves <a href="https://github.com/OpenLineage/OpenLineage/issues/20">#20</a><br /> Signed-off-by: Julien Le Dem <a href="mailto:julien@apache.org">julien@apache.org</a></p>
      @@ -6561,8 +6585,8 @@

      Group Direct Messages

      <p><strong>Naming</strong></p> -<p>We define the unique name strategy per datasource to ensure it is followed uniformly independently from the type of job consuming and producing data. <br /> -The namespace for a dataset is the unique name for its datasource. <br /> +<p>We define the unique name strategy per datasource to ensure it is followed uniformly independently from the type of job consuming and producing data.<br /> +The namespace for a dataset is the unique name for its datasource.<br /> For jobs it is the scheduler.</p> <p><strong>Datasets:</strong></p> @@ -6621,9 +6645,9 @@

      Group Direct Messages

      <p><strong>BigQuery</strong></p> -<p>See: <br /> -<a href="https://cloud.google.com/resource-manager/docs/creating-managing-projects|Creating and managing projects | Resource Manager Documentation">https://cloud.google.com/resource-manager/docs/creating-managing-projects|Creating and managing projects | Resource Manager Documentation</a> <br /> -<a href="https://cloud.google.com/bigquery/docs/datasets-intro|Introduction to datasets | BigQuery">https://cloud.google.com/bigquery/docs/datasets-intro|Introduction to datasets | BigQuery</a> <br /> +<p>See:<br /> +<a href="https://cloud.google.com/resource-manager/docs/creating-managing-projects|Creating and managing projects | Resource Manager Documentation">https://cloud.google.com/resource-manager/docs/creating-managing-projects|Creating and managing projects | Resource Manager Documentation</a><br /> +<a href="https://cloud.google.com/bigquery/docs/datasets-intro|Introduction to datasets | BigQuery">https://cloud.google.com/bigquery/docs/datasets-intro|Introduction to datasets | BigQuery</a><br /> <a href="https://cloud.google.com/bigquery/docs/tables-intro|Introduction to tables | BigQuery">https://cloud.google.com/bigquery/docs/tables-intro|Introduction to tables | BigQuery</a></p> <p>Datasource hierarchy:</p> @@ -6757,14 +6781,14 @@

      Group Direct Messages

      <p><strong>Contributing to OpenLineage</strong></p> -<p>This project welcomes contributors from any organization or background, provided they are <br /> -willing to follow the simple processes outlined below, as well as adhere to the <br /> +<p>This project welcomes contributors from any organization or background, provided they are<br /> +willing to follow the simple processes outlined below, as well as adhere to the<br /> &lt;CODE<em>OF</em>CONDUCT.md|Code of Conduct&gt;.</p> <p><strong>Joining the community</strong></p> -<p>The community collaborates primarily through <code>GitHub</code> and the instance messaging tool, <code>Slack</code>. <br /> -There is also a mailing list. <br /> +<p>The community collaborates primarily through <code>GitHub</code> and the instance messaging tool, <code>Slack</code>.<br /> +There is also a mailing list.<br /> See how to join <a href="https://github.com/OpenLineage/OpenLineage#community">here</a></p> <p><strong>Reporting an Issue</strong></p> @@ -6778,10 +6802,10 @@

      Group Direct Messages

      <p>Before sending a Pull Request with significant changes, please use the <a href="https://github.com/OpenLineage/OpenLineage/issues">issue tracker</a> to discuss the potential improvements you want to make.</p> -<p>OpenLineage uses <a href="https://help.github.com/articles/about-collaborative-development-models/">GitHub's fork and pull model</a> <br /> +<p>OpenLineage uses <a href="https://help.github.com/articles/about-collaborative-development-models/">GitHub's fork and pull model</a><br /> to create a contribution.</p> -<p>Every contribution is signed to say that the contributor has the rights to make the contribution and <br /> +<p>Every contribution is signed to say that the contributor has the rights to make the contribution and<br /> agrees with the &lt;why-the-dco.md|Developer Certificate of Origin (DCO)&gt;</p> <p><strong>Proposing changes</strong></p> @@ -7648,14 +7672,14 @@

      Group Direct Messages

      <p><strong>Contributing to OpenLineage</strong></p> -<p>This project welcomes contributors from any organization or background, provided they are <br /> -willing to follow the simple processes outlined below, as well as adhere to the <br /> +<p>This project welcomes contributors from any organization or background, provided they are<br /> +willing to follow the simple processes outlined below, as well as adhere to the<br /> &lt;CODE<em>OF</em>CONDUCT.md|Code of Conduct&gt;.</p> <p><strong>Joining the community</strong></p> -<p>The community collaborates primarily through <code>GitHub</code> and the instance messaging tool, <code>Slack</code>. <br /> -There is also a mailing list. <br /> +<p>The community collaborates primarily through <code>GitHub</code> and the instance messaging tool, <code>Slack</code>.<br /> +There is also a mailing list.<br /> See how to join <a href="https://github.com/OpenLineage/OpenLineage#community">here</a></p> <p><strong>Reporting an Issue</strong></p> @@ -7669,10 +7693,10 @@

      Group Direct Messages

      <p>Before sending a Pull Request with significant changes, please use the <a href="https://github.com/OpenLineage/OpenLineage/issues">issue tracker</a> to discuss the potential improvements you want to make.</p> -<p>OpenLineage uses <a href="https://help.github.com/articles/about-collaborative-development-models/">GitHub's fork and pull model</a> <br /> +<p>OpenLineage uses <a href="https://help.github.com/articles/about-collaborative-development-models/">GitHub's fork and pull model</a><br /> to create a contribution.</p> -<p>Every contribution is signed to say that the contributor has the rights to make the contribution and <br /> +<p>Every contribution is signed to say that the contributor has the rights to make the contribution and<br /> agrees with the &lt;why-the-dco.md|Developer Certificate of Origin (DCO)&gt;</p> <p><strong>Proposing changes</strong></p> @@ -8623,8 +8647,8 @@

      Group Direct Messages

      <https://github.com/OpenLineage/OpenLineage/issues/63|#63 [PROPOSAL] Define the OpenLineage spec versioning mechanism>
      - <p><strong>Purpose:</strong> <br /> -We need to define how the OpenLineage spec gets versioned and published. <br /> + <p><strong>Purpose:</strong><br /> +We need to define how the OpenLineage spec gets versioned and published.<br /> Some requirements:</p> <p>• The OpenLineage spec and related libraries are in the OpenLineage repo. @@ -8633,9 +8657,9 @@

      Group Direct Messages

      • We want to version the OpenLineage spec independently of the api spec. • The mechanism to version and publish the OpenLineage core spec, should apply to publishing custom facets.</p> -<p><strong>Proposed implementation</strong> <br /> -I have started a google doc to make it easier to collaborate: <br /> -<a href="https://docs.google.com/document/d/1inhmb90SB62VyYf8nkkpjBDPpyr2wxkIXQ_AFIQiF9I/edit">https://docs.google.com/document/d/1inhmb90SB62VyYf8nkkpjBDPpyr2wxkIXQ_AFIQiF9I/edit</a> <br /> +<p><strong>Proposed implementation</strong><br /> +I have started a google doc to make it easier to collaborate:<br /> +<a href="https://docs.google.com/document/d/1inhmb90SB62VyYf8nkkpjBDPpyr2wxkIXQ_AFIQiF9I/edit">https://docs.google.com/document/d/1inhmb90SB62VyYf8nkkpjBDPpyr2wxkIXQ_AFIQiF9I/edit</a><br /> please comment in the doc.</p>
      @@ -9936,7 +9960,7 @@

      Supress success

      <https://github.com/OpenLineage/OpenLineage/issues/84|#84 Finalize the OpenLineage mission statement>
      - <p>It is part of our technical charter. <br /> + <p>It is part of our technical charter.<br /> Currently we have: "to enable the industry at-large to collect lineage metadata consistently and comprehensively across complex pipelines, creating a deeper understanding of data."</p> <p>We'd want to capture that this is specifically operational lineage, or lineage and metadata collected as the job is running.</p> @@ -10022,8 +10046,8 @@

      Supress success

      <https://github.com/OpenLineage/OpenLineage/issues/63|#63 [PROPOSAL] Define the OpenLineage spec versioning mechanism>
      - <p><strong>Purpose:</strong> <br /> -We need to define how the OpenLineage spec gets versioned and published. <br /> + <p><strong>Purpose:</strong><br /> +We need to define how the OpenLineage spec gets versioned and published.<br /> Some requirements:</p> <p>• The OpenLineage spec and related libraries are in the OpenLineage repo. @@ -10034,34 +10058,34 @@

      Supress success

      <p><strong>Proposed implementation</strong></p> -<p>• The spec defines it’s current version using the “$id” field: <br /> - See: <br /> - <a href="https://json-schema.org/draft/2020-12/json-schema-core.html#rfc.section.8.2.1">json schema core doc</a> <br /> - <a href="https://json-schema.org/draft/2019-09/schema">Json schema spec $id</a> <a href="https://github.com/json-schema-org/json-schema-spec/blob/draft-next/meta/core.json">also on github</a> <br /> - Example: <br /> - "$id": "<a href="https://openlineage.io/spec/0.1.0/OpenLineage.json">https://openlineage.io/spec/0.1.0/OpenLineage.json</a>" <br /> - The URL in $id is resolvable and returns that version of the spec. <br /> +<p>• The spec defines it’s current version using the “$id” field:<br /> + See:<br /> + <a href="https://json-schema.org/draft/2020-12/json-schema-core.html#rfc.section.8.2.1">json schema core doc</a><br /> + <a href="https://json-schema.org/draft/2019-09/schema">Json schema spec $id</a> <a href="https://github.com/json-schema-org/json-schema-spec/blob/draft-next/meta/core.json">also on github</a><br /> + Example:<br /> + "$id": "<a href="https://openlineage.io/spec/0.1.0/OpenLineage.json">https://openlineage.io/spec/0.1.0/OpenLineage.json</a>"<br /> + The URL in $id is resolvable and returns that version of the spec.<br /> We use github pages to publish the spec to <a href="http://openlineage.io">openlineage.io</a> -• The $id urls uses a SEMVER compliant version, following the <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver/">SCHEMAVER semantics</a> <br /> +• The $id urls uses a SEMVER compliant version, following the <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver/">SCHEMAVER semantics</a><br /> MODEL-REVISION-ADDITION • MODEL when you make a breaking schema change which will prevent interaction with any historical data • REVISION when you introduce a schema change which may prevent interaction with some historical data • ADDITION when you make a schema change that is compatible with all historical data • CI verification that: - • the version changes when the spec changed: <br /> + • the version changes when the spec changed:<br /> When resolving “$id”, the build fails if the spec is not exactly the same. - • The version does not change when the spec does not change. <br /> + • The version does not change when the spec does not change.<br /> TODO: define a mechanism to verify that the current version of the spec is not already published with a different. Possibly hash based. • Libraries are generating event with current version • Make sure the spec is backward compatible (only add optional fields) and consistent with the versioning semantics -• git pre commit: <br /> +• git pre commit:<br /> Increments the versions automatically when the spec changes. -• CI publish (on tag): <br /> - The spec gets published automatically to the $id URL on <a href="http://OpenLineage.io">OpenLineage.io</a> <br /> - Check the $id field has the right domain prefix <br /> +• CI publish (on tag):<br /> + The spec gets published automatically to the $id URL on <a href="http://OpenLineage.io">OpenLineage.io</a><br /> + Check the $id field has the right domain prefix<br /> Publish current OpenLineage.json to that URL</p> -<p>The discussion is available on a google doc: <br /> +<p>The discussion is available on a google doc:<br /> <a href="https://docs.google.com/document/d/1inhmb90SB62VyYf8nkkpjBDPpyr2wxkIXQ_AFIQiF9I/edit">https://docs.google.com/document/d/1inhmb90SB62VyYf8nkkpjBDPpyr2wxkIXQ_AFIQiF9I/edit</a></p>
      @@ -10178,10 +10202,10 @@

      Supress success

      <https://github.com/OpenLineage/OpenLineage/issues/35|#35 [PROPOSAL] Dataset version facet>
      - <p><strong>Purpose:</strong> <br /> + <p><strong>Purpose:</strong><br /> Many databases have explicit versioning in place for datasets. It is highly desirable to track the specific version of the dataset if it is known at consumption/generation time. Rather than relying on the service to correctly tie job inputs to specific versions of a dataset, the client should be allowed to report that information if it is available.</p> -<p><strong>Proposed implementation</strong> <br /> +<p><strong>Proposed implementation</strong><br /> A dataset version facet may be sufficient to report dataset version information, if it exists. Custom facets can be used to include any database specific version information that may exist, while the minimum required fields should include a versionName string field.</p>
      @@ -10553,8 +10577,8 @@

      Supress success

      <https://github.com/OpenLineage/OpenLineage/issues/63|#63 [PROPOSAL] Define the OpenLineage spec versioning mechanism>
      - <p><strong>Purpose:</strong> <br /> -We need to define how the OpenLineage spec gets versioned and published. <br /> + <p><strong>Purpose:</strong><br /> +We need to define how the OpenLineage spec gets versioned and published.<br /> Some requirements:</p> <p>• The OpenLineage spec and related libraries are in the OpenLineage repo. @@ -10570,24 +10594,24 @@

      Supress success

      • <a href="https://json-schema.org/draft/2020-12/json-schema-core.html#rfc.section.8.2.1">json schema core doc</a> • <a href="https://json-schema.org/draft/2019-09/schema">Json schema spec $id</a> • <a href="https://github.com/json-schema-org/json-schema-spec/blob/draft-next/meta/core.json">also on github</a> - • Example: <br /> - <code>"$id": "&lt;https://openlineage.io/spec/0.1.0/OpenLineage.json&gt;"</code> <br /> - The URL in $id is resolvable and returns that version of the spec. <br /> + • Example:<br /> + <code>"$id": "&lt;https://openlineage.io/spec/0.1.0/OpenLineage.json&gt;"</code><br /> + The URL in $id is resolvable and returns that version of the spec.<br /> We use github pages to publish the spec to <a href="http://openlineage.io">openlineage.io</a> -• The $id urls uses a SEMVER compliant version, following the <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver/">SCHEMAVER semantics</a> <br /> +• The $id urls uses a SEMVER compliant version, following the <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver/">SCHEMAVER semantics</a><br /> MODEL-REVISION-ADDITION • MODEL when you make a breaking schema change which will prevent interaction with any historical data • REVISION when you introduce a schema change which may prevent interaction with some historical data • ADDITION when you make a schema change that is compatible with all historical data • The spec in /spec/OpenLineage.json might be using a prerelease version (ex: 0.1.0-rc0) • CI verifies that: - • the version changes when the spec changed: <br /> + • the version changes when the spec changed:<br /> When resolving “$id”, the build fails if the spec is not exactly the same. - • The version does not change when the spec does not change. <br /> + • The version does not change when the spec does not change.<br /> We can verify that the current version of the spec is not already published with a different version. • Libraries are generating event with current version • Make sure the spec is backward compatible (only add optional fields) and consistent with the versioning semantics -• git pre commit: <br /> +• git pre commit:<br /> Increments the versions automatically when the spec changes. • spec publication: • update the $id to the correct value @@ -10595,7 +10619,7 @@

      Supress success

      • The spec is published automatically to the $id URL on <a href="http://OpenLineage.io">OpenLineage.io</a> • Validation that the $id field has the right domain prefix</p> -<p>The discussion is available on a google doc: <br /> +<p>The discussion is available on a google doc:<br /> <a href="https://docs.google.com/document/d/1inhmb90SB62VyYf8nkkpjBDPpyr2wxkIXQ_AFIQiF9I/edit">https://docs.google.com/document/d/1inhmb90SB62VyYf8nkkpjBDPpyr2wxkIXQ_AFIQiF9I/edit</a></p>
      @@ -10656,7 +10680,7 @@

      Supress success

      <https://github.com/OpenLineage/OpenLineage/issues/84|#84 Finalize the OpenLineage mission statement>
      - <p>The mission statement is part of our technical charter. <br /> + <p>The mission statement is part of our technical charter.<br /> We started from:</p> <p>&gt; "The mission of the Project is to enable the industry at-large to collect lineage metadata consistently and comprehensively across complex pipelines, creating a deeper understanding of data."</p> @@ -12017,11 +12041,11 @@

      Supress success

      <https://github.com/OpenLineage/OpenLineage/issues/148|#148 [PROPOSAL] column level lineage facet>
      - <p><strong>Purpose:</strong> <br /> + <p><strong>Purpose:</strong><br /> For transformations like SQL queries (but also in other cases), we can extract column level lineage. This allows answering questions like ‘which root input columns are used to construct column x?’. We need to be able to capture this information in the OpenLineage model</p> -<p><strong>Proposed implementation</strong> <br /> -We propose to introduce a new dataset facet: <br /> +<p><strong>Proposed implementation</strong><br /> +We propose to introduce a new dataset facet:<br /> example:</p> <pre><code>{ @@ -12254,8 +12278,8 @@

      Supress success

      <https://github.com/OpenLineage/OpenLineage/issues/153|#153 [PROPOSAL] Version facets independently of the core spec>
      - <p><strong>Purpose:</strong> <br /> -The OpenLineage core spec is versioned using a version URL in the <code>$id</code> field: <br /> + <p><strong>Purpose:</strong><br /> +The OpenLineage core spec is versioned using a version URL in the <code>$id</code> field:<br /> See:</p> <p>• <a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/Versioning.md">https://github.com/OpenLineage/OpenLineage/blob/main/spec/Versioning.md</a> @@ -12268,8 +12292,8 @@

      Supress success

      <p>Currently all the core facets are also defined in that same core spec.</p> -<p>This means that the core spec gets a new version whenever a facet is changed along with every other facet. <br /> -We'd rather have the core spec and each facet versioned independently so that their version changes only when they change. <br /> +<p>This means that the core spec gets a new version whenever a facet is changed along with every other facet.<br /> +We'd rather have the core spec and each facet versioned independently so that their version changes only when they change.<br /> This matches much better the use cases for using the version number.</p> <p>For example:</p> @@ -12284,13 +12308,13 @@

      Supress success

      <p>This leaves the core spec to only contain the <code>LineageEvent</code> and related core entities (LineageEvent, Run, Job, Dataset) but none of the core facets.</p> -<p>See for example: <br /> -<a href="https://github.com/OpenLineage/OpenLineage/blob/789b55fc6e6e07d0aedbe130f775721829e83618/spec/OpenLineage.json">https://github.com/OpenLineage/OpenLineage/blob/789b55fc6e6e07d0aedbe130f775721829e83618/spec/OpenLineage.json</a> <br /> -This would be the spec that a consumer would implement to accept any facet version. <br /> -This would ensure that the core spec version changes rarely, only when we add fields to the core model. <br /> +<p>See for example:<br /> +<a href="https://github.com/OpenLineage/OpenLineage/blob/789b55fc6e6e07d0aedbe130f775721829e83618/spec/OpenLineage.json">https://github.com/OpenLineage/OpenLineage/blob/789b55fc6e6e07d0aedbe130f775721829e83618/spec/OpenLineage.json</a><br /> +This would be the spec that a consumer would implement to accept any facet version.<br /> +This would ensure that the core spec version changes rarely, only when we add fields to the core model.<br /> Each facet would be versioned independently following the same versioning mechanism. Each facet version will change if and only if this particular facet schema changes. It also makes core facets and custom facets work exactly the same, they are just maintained in different repositories.</p> -<p>Example: <br /> +<p>Example:<br /> <a href="https://openlineage.io/spec/1-0-0/DatasetSchemaFacet.json">https://openlineage.io/spec/1-0-0/DatasetSchemaFacet.json</a></p> <pre><code>{ @@ -12675,8 +12699,8 @@

      Supress success

      <https://github.com/OpenLineage/OpenLineage/pull/204|#204 Introduce UnknownFacet for spark>
      - <p>PR for <a href="https://github.com/OpenLineage/OpenLineage/issues/175">Issue:</a> <br /> -Adding new custom facet for spark with name"spark_unknown". <br /> + <p>PR for <a href="https://github.com/OpenLineage/OpenLineage/issues/175">Issue:</a><br /> +Adding new custom facet for spark with name"spark_unknown".<br /> Holds information about unvisited nodes from spark plan.</p>
      @@ -15058,7 +15082,7 @@

      Supress success

      </code></pre> -<p>Do i do something wrong or is spark 3 simply not yet supported? <br /> +<p>Do i do something wrong or is spark 3 simply not yet supported?<br /> Thanks</p>
      @@ -15115,7 +15139,7 @@

      Supress success

      <p>Hi Team</p> -<p>I have tried to collect lineage on spark 2.4 with openlineage 0.2.2. <br /> +<p>I have tried to collect lineage on spark 2.4 with openlineage 0.2.2.<br /> Here is my pom.xml;</p> <pre><code>&amp;lt;project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" @@ -19314,9 +19338,9 @@

      Supress success

      <p><strong>BigQuery</strong></p> -<p>See: <br /> -<a href="https://cloud.google.com/resource-manager/docs/creating-managing-projects|Creating and managing projects | Resource Manager Documentation">https://cloud.google.com/resource-manager/docs/creating-managing-projects|Creating and managing projects | Resource Manager Documentation</a> <br /> -<a href="https://cloud.google.com/bigquery/docs/datasets-intro|Introduction to datasets | BigQuery">https://cloud.google.com/bigquery/docs/datasets-intro|Introduction to datasets | BigQuery</a> <br /> +<p>See:<br /> +<a href="https://cloud.google.com/resource-manager/docs/creating-managing-projects|Creating and managing projects | Resource Manager Documentation">https://cloud.google.com/resource-manager/docs/creating-managing-projects|Creating and managing projects | Resource Manager Documentation</a><br /> +<a href="https://cloud.google.com/bigquery/docs/datasets-intro|Introduction to datasets | BigQuery">https://cloud.google.com/bigquery/docs/datasets-intro|Introduction to datasets | BigQuery</a><br /> <a href="https://cloud.google.com/bigquery/docs/tables-intro|Introduction to tables | BigQuery">https://cloud.google.com/bigquery/docs/tables-intro|Introduction to tables | BigQuery</a></p> <p>Datasource hierarchy:</p> @@ -19390,13 +19414,13 @@

      Supress success

      <p><strong>Jobs</strong> <strong>Context</strong></p> -<p>A <code>Job</code> is a recurring data transformation with Inputs and outputs. Each execution is captured as a <code>Run</code> with corresponding metadata. <br /> -A <code>Run</code> event identifies the <code>Job</code> it is an instance of by providing the job’s unique identifier. <br /> +<p>A <code>Job</code> is a recurring data transformation with Inputs and outputs. Each execution is captured as a <code>Run</code> with corresponding metadata.<br /> +A <code>Run</code> event identifies the <code>Job</code> it is an instance of by providing the job’s unique identifier.<br /> The <code>Job</code> identifier is composed of a <code>Namespace</code> and a <code>Name</code>. The job name is unique within that namespace.</p> <p>The core property we want to identify about a <code>Job</code> is how it changes over time. Different schedules of the same logic applied to different datasets (possibly with different parameters) are different jobs. The notion of a <code>job</code> is tied to a recurring schedule with specific inputs and outputs. It could be an incremental update or a full reprocess or even a streaming job.</p> -<p>If the same code artifact (for example a spark jar or a templated SQL query) is used in the context of different schedules with different input or outputs then they are different jobs. <br /> +<p>If the same code artifact (for example a spark jar or a templated SQL query) is used in the context of different schedules with different input or outputs then they are different jobs.<br /> We are interested first in how they affect the datasets they produce.</p> <p><strong>Job Namespace and constructing job names</strong></p> @@ -19417,12 +19441,12 @@

      Supress success

      <p><strong>Parent job run: a nested hierarchy of Jobs</strong></p> -<p>It is often the case that jobs are part of a nested hierarchy. <br /> +<p>It is often the case that jobs are part of a nested hierarchy.<br /> For example an Airflow DAG contains tasks. An instance of the DAG is finished when all of the tasks are finished. Similarly a Spark job can spawn multiple actions each of them running independently. Additionally, a Spark job can be launched by an Airflow task within a DAG.</p> <p>Since what we care about is identifying the job as rooted in a recurring schedule, we want to capture that connection and make sure that we treat the same application logic triggered at different schedules as different jobs. For example: if an Airflow DAG runs individual tasks per partition (for example market segments) using the same underlying job logic, they will be tracked as separate jobs.</p> -<p>To capture this, a run event provides <a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json#L282-L331">a <code>ParentRun</code> facet</a>, referring to the parent <code>Job</code> and <code>Run</code>. This allows tracking a recurring job from the root of the schedule it is running for. <br /> +<p>To capture this, a run event provides <a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json#L282-L331">a <code>ParentRun</code> facet</a>, referring to the parent <code>Job</code> and <code>Run</code>. This allows tracking a recurring job from the root of the schedule it is running for.<br /> If there's a parent job, we use the same namespace, otherwise it is provided by configuration.</p> <p>Example:</p> @@ -20007,11 +20031,11 @@

      Supress success

      <https://github.com/OpenLineage/OpenLineage/issues/148|#148 [PROPOSAL] column level lineage facet>
      - <p><strong>Purpose:</strong> <br /> + <p><strong>Purpose:</strong><br /> For transformations like SQL queries (but also in other cases), we can extract column level lineage. This allows answering questions like ‘which root input columns are used to construct column x?’. We need to be able to capture this information in the OpenLineage model</p> -<p><strong>Proposed implementation</strong> <br /> -We propose to introduce a new dataset facet: <br /> +<p><strong>Proposed implementation</strong><br /> +We propose to introduce a new dataset facet:<br /> example:</p> <pre><code>{ @@ -20603,7 +20627,7 @@

      Supress success

      <https://github.com/OpenLineage/OpenLineage/issues/318|#318 Add Redshift support to dbt integration>
      - <p>As of <code>dbt-ol==0.2.2</code>, it is not possible to generate lineage if the target DWH is Redshift. <br /> + <p>As of <code>dbt-ol==0.2.2</code>, it is not possible to generate lineage if the target DWH is Redshift.<br /> Try running <code>dbt-ol</code> and you get the following error:</p> <pre><code>Traceback (most recent call last): @@ -23192,7 +23216,7 @@

      Supress success

      <https://github.com/OpenLineage/OpenLineage/pull/322|#322 dbt: support manifest and run-results v3>
      - <p>There are no major differences between v2 and v3 metadata, at least those impacting information that we're collecting. <br /> + <p>There are no major differences between v2 and v3 metadata, at least those impacting information that we're collecting.<br /> This PR makes <code>dbt-ol</code> treat v3 metadata the same way we treated v2.</p> <p>Signed-off-by: Maciej Obuchowski <a href="mailto:maciej.obuchowski@getindata.com">maciej.obuchowski@getindata.com</a></p> @@ -23500,7 +23524,7 @@

      Supress success

      <p>I.e</p> -<p>As a user conducting testing of a data set I wish to DELETE it and then start a new <br /> +<p>As a user conducting testing of a data set I wish to DELETE it and then start a new<br /> As a user creating a job I linked the wrong datasets, I wish to the DELETE the job</p> <p>Ideally for:</p> @@ -23645,15 +23669,15 @@

      Supress success

      <https://github.com/OpenLineage/OpenLineage/issues/323|#323 [PROPOSAL] Ability to annotate datasets with metadata without a job run>
      - <p><strong>Purpose:</strong> <br /> -We need a way to make observations about a Dataset current state outside of a run reading or writing the data. <br /> + <p><strong>Purpose:</strong><br /> +We need a way to make observations about a Dataset current state outside of a run reading or writing the data.<br /> For example, this could be:</p> <p>• collecting schema information about a table from a warehouse • informing that a table has been deleted • etc</p> -<p><strong>Proposed implementation</strong> <br /> +<p><strong>Proposed implementation</strong><br /> Adding Dataset event types that are not lineage events. This would be the Dataset entity with its facets: </p> <p><a href="https://github.com/OpenLineage/OpenLineage/blob/3a1ccbd854bbf202bbe6437bf81786cb01ed4d01/spec/OpenLineage.json#L178-L203">OpenLineage/spec/OpenLineage.json</a></p> @@ -24745,11 +24769,11 @@

      Supress success

      <https://github.com/OpenLineage/OpenLineage/issues/148|#148 [PROPOSAL] column level lineage facet>
      - <p><strong>Purpose:</strong> <br /> + <p><strong>Purpose:</strong><br /> For transformations like SQL queries (but also in other cases), we can extract column level lineage. This allows answering questions like ‘which root input columns are used to construct column x?’. We need to be able to capture this information in the OpenLineage model</p> -<p><strong>Proposed implementation</strong> <br /> -We propose to introduce a new dataset facet: <br /> +<p><strong>Proposed implementation</strong><br /> +We propose to introduce a new dataset facet:<br /> example:</p> <pre><code>{ @@ -25685,7 +25709,7 @@

      Supress success

      <https://github.com/OpenLineage/OpenLineage/issues/336|#336 [PROPOSAL] Add a design doc process for proposals>
      - <p><strong>Purpose:</strong> <br /> + <p><strong>Purpose:</strong><br /> Add a design doc section for proposals. The process would be the following:</p> <p>• open an issue with the "PROPOSAL" template @@ -26558,8 +26582,8 @@

      Supress success

      <https://github.com/OpenLineage/OpenLineage/issues/376|#376 [INTEGRATION][dbt] support dbt built>
      - <p>dbt build is a superset of the dbt run and dbt test commands. <br /> -We should apply the same logic there. <br /> + <p>dbt build is a superset of the dbt run and dbt test commands.<br /> +We should apply the same logic there.<br /> <a href="https://docs.getdbt.com/reference/commands/build">https://docs.getdbt.com/reference/commands/build</a></p>
      @@ -26614,7 +26638,7 @@

      Supress success

      <https://github.com/OpenLineage/OpenLineage/pull/383|#383 dbt: use column description from metadata.json even if we have catalog>
      - <p>Column descriptions are properly set in <code>metadata.json</code> file. Fetch it from there, even if we have <code>catalog.json</code>. <br /> + <p>Column descriptions are properly set in <code>metadata.json</code> file. Fetch it from there, even if we have <code>catalog.json</code>.<br /> Also, remove empty column descriptions from being checked in most tests.</p> <p>Signed-off-by: Maciej Obuchowski <a href="mailto:maciej.obuchowski@getindata.com">maciej.obuchowski@getindata.com</a></p> @@ -26679,7 +26703,7 @@

      Supress success

      <https://github.com/OpenLineage/OpenLineage/issues/336|#336 [PROPOSAL] Add a design doc process for proposals>
      - <p><strong>Purpose:</strong> <br /> + <p><strong>Purpose:</strong><br /> Add a design doc section for proposals. The process would be the following:</p> <p>• open an issue with the "PROPOSAL" template @@ -27707,7 +27731,7 @@

      Supress success

      So in this case in my template I am adding:

      -

      ```env:
      +

      ```env:
      ADDITIONALPYTHONDEPS: "openpyxl==3.0.3 smartopen==2.0.0 apache-airflow-providers-http apache-airflow-providers-cncf-kubernetes apache-airflow-providers-amazon openlineage-airflow" OPENLINEAGEURL: https://marquez-internal-eks.eu-west-1.dev.hbi.systems OPENLINEAGENAMESPACE: dnsairflow @@ -30620,9 +30644,9 @@

      Supress success

      <p><strong>BigQuery</strong></p> -<p>See: <br /> -<a href="https://cloud.google.com/resource-manager/docs/creating-managing-projects|Creating and managing projects | Resource Manager Documentation">https://cloud.google.com/resource-manager/docs/creating-managing-projects|Creating and managing projects | Resource Manager Documentation</a> <br /> -<a href="https://cloud.google.com/bigquery/docs/datasets-intro|Introduction to datasets | BigQuery">https://cloud.google.com/bigquery/docs/datasets-intro|Introduction to datasets | BigQuery</a> <br /> +<p>See:<br /> +<a href="https://cloud.google.com/resource-manager/docs/creating-managing-projects|Creating and managing projects | Resource Manager Documentation">https://cloud.google.com/resource-manager/docs/creating-managing-projects|Creating and managing projects | Resource Manager Documentation</a><br /> +<a href="https://cloud.google.com/bigquery/docs/datasets-intro|Introduction to datasets | BigQuery">https://cloud.google.com/bigquery/docs/datasets-intro|Introduction to datasets | BigQuery</a><br /> <a href="https://cloud.google.com/bigquery/docs/tables-intro|Introduction to tables | BigQuery">https://cloud.google.com/bigquery/docs/tables-intro|Introduction to tables | BigQuery</a></p> <p>Datasource hierarchy:</p> @@ -30711,13 +30735,13 @@

      Supress success

      <p><strong>Jobs</strong> <strong>Context</strong></p> -<p>A <code>Job</code> is a recurring data transformation with Inputs and outputs. Each execution is captured as a <code>Run</code> with corresponding metadata. <br /> -A <code>Run</code> event identifies the <code>Job</code> it is an instance of by providing the job’s unique identifier. <br /> +<p>A <code>Job</code> is a recurring data transformation with Inputs and outputs. Each execution is captured as a <code>Run</code> with corresponding metadata.<br /> +A <code>Run</code> event identifies the <code>Job</code> it is an instance of by providing the job’s unique identifier.<br /> The <code>Job</code> identifier is composed of a <code>Namespace</code> and a <code>Name</code>. The job name is unique within that namespace.</p> <p>The core property we want to identify about a <code>Job</code> is how it changes over time. Different schedules of the same logic applied to different datasets (possibly with different parameters) are different jobs. The notion of a <code>job</code> is tied to a recurring schedule with specific inputs and outputs. It could be an incremental update or a full reprocess or even a streaming job.</p> -<p>If the same code artifact (for example a spark jar or a templated SQL query) is used in the context of different schedules with different input or outputs then they are different jobs. <br /> +<p>If the same code artifact (for example a spark jar or a templated SQL query) is used in the context of different schedules with different input or outputs then they are different jobs.<br /> We are interested first in how they affect the datasets they produce.</p> <p><strong>Job Namespace and constructing job names</strong></p> @@ -30738,12 +30762,12 @@

      Supress success

      <p><strong>Parent job run: a nested hierarchy of Jobs</strong></p> -<p>It is often the case that jobs are part of a nested hierarchy. <br /> +<p>It is often the case that jobs are part of a nested hierarchy.<br /> For example an Airflow DAG contains tasks. An instance of the DAG is finished when all of the tasks are finished. Similarly a Spark job can spawn multiple actions each of them running independently. Additionally, a Spark job can be launched by an Airflow task within a DAG.</p> <p>Since what we care about is identifying the job as rooted in a recurring schedule, we want to capture that connection and make sure that we treat the same application logic triggered at different schedules as different jobs. For example: if an Airflow DAG runs individual tasks per partition (for example market segments) using the same underlying job logic, they will be tracked as separate jobs.</p> -<p>To capture this, a run event provides <a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json#L282-L331">a <code>ParentRun</code> facet</a>, referring to the parent <code>Job</code> and <code>Run</code>. This allows tracking a recurring job from the root of the schedule it is running for. <br /> +<p>To capture this, a run event provides <a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json#L282-L331">a <code>ParentRun</code> facet</a>, referring to the parent <code>Job</code> and <code>Run</code>. This allows tracking a recurring job from the root of the schedule it is running for.<br /> If there's a parent job, we use the same namespace, otherwise it is provided by configuration.</p> <p>Example:</p> @@ -31202,7 +31226,7 @@

      Supress success

      *Thread Reply:* This is an example Lineage event JSON I am sending.

      -
      Example lineage event
      +
      Example lineage event
      @@ -32473,7 +32497,7 @@

      Supress success

      <p><strong>Summary of Changes</strong></p> -<p>Databulder extractor to extract lineage information from OpenLineage format <br /> +<p>Databulder extractor to extract lineage information from OpenLineage format<br /> <a href="https://github.com/OpenLineage/OpenLineage">https://github.com/OpenLineage/OpenLineage</a></p> <p><strong>Tests</strong> @@ -37425,9 +37449,9 @@

      Supress success

      <p><strong>OpenLineage Spec</strong> <strong>Specification</strong></p> -<p>The specification for OpenLineage is formalized as a JsonSchema &lt;OpenLineage.json|OpenLineage.json&gt;. <br /> -An OpenAPI spec is also provided for HTTP based implementations: &lt;OpenLineage.yml|OpenLineage.yml&gt; <br /> -The documentation is published at: <a href="https://openlineage.github.io/">https://openlineage.github.io/</a> <br /> +<p>The specification for OpenLineage is formalized as a JsonSchema &lt;OpenLineage.json|OpenLineage.json&gt;.<br /> +An OpenAPI spec is also provided for HTTP based implementations: &lt;OpenLineage.yml|OpenLineage.yml&gt;<br /> +The documentation is published at: <a href="https://openlineage.github.io/">https://openlineage.github.io/</a><br /> It allows extensions to the spec using <code>Custom Facets</code> as described in this document.</p> <p><strong>Core concepts</strong> @@ -37441,7 +37465,7 @@

      Supress success

      • <strong>Run</strong>: An instance of a running job with a start and completion (or failure) time. A run is identified by a globally unique ID relative to its job definition. A run ID <strong>must</strong> be an <a href="https://datatracker.ietf.org/doc/html/rfc4122">UUID</a>. • <strong>Facet</strong>: A piece of metadata attached to one of the entities defined above.</p> -<p>example: <br /> +<p>example:<br /> Here is an example of a simple start run event not adding any facet information:</p> <pre><code>{ @@ -37474,10 +37498,10 @@

      Supress success

      <p><strong>Lifecycle</strong></p> -<p>The OpenLineage API defines events to capture the lifecycle of a <em>Run</em> for a given <em>Job</em>. <br /> -When a <em>job</em> is being <em>run</em>, we capture metadata by sending run events when the state of the job transitions to a different state. <br /> -We might observe different aspects of the job run at different stages. This means that different metadata might be collected in each event during the lifecycle of a run. <br /> -All metadata is additive. for example, if more inputs or outputs are detected as the job is running we might send additional events specifically for those datasets without re-emitting previously observed inputs or outputs. <br /> +<p>The OpenLineage API defines events to capture the lifecycle of a <em>Run</em> for a given <em>Job</em>.<br /> +When a <em>job</em> is being <em>run</em>, we capture metadata by sending run events when the state of the job transitions to a different state.<br /> +We might observe different aspects of the job run at different stages. This means that different metadata might be collected in each event during the lifecycle of a run.<br /> +All metadata is additive. for example, if more inputs or outputs are detected as the job is running we might send additional events specifically for those datasets without re-emitting previously observed inputs or outputs.<br /> Example:</p> <p>• When the run starts, we collect the following Metadata: @@ -37504,7 +37528,7 @@

      Supress success

      <p>A facet is an atomic piece of metadata identified by its name. This means that emitting a new facet with the same name for the same entity replaces the previous facet instance for that entity entirely). It is defined as a JSON object that can be either part of the spec or custom facets defined in a different project.</p> -<p>Custom facets must use a distinct prefix named after the project defining them to avoid collision with standard facets defined in the &lt;OpenLineage.json|OpenLineage.json&gt; spec. <br /> +<p>Custom facets must use a distinct prefix named after the project defining them to avoid collision with standard facets defined in the &lt;OpenLineage.json|OpenLineage.json&gt; spec.<br /> They have a _schemaURL field pointing to the corresponding version of the facet schema (as a JSONPointer: <a href="https://swagger.io/docs/specification/using-ref/">$ref URL location</a> ).</p> <p>Example: <a href="https://raw.githubusercontent.com/OpenLineage/OpenLineage/main/spec/OpenLineage.json#/definitions/MyCustomJobFacet">https://raw.githubusercontent.com/OpenLineage/OpenLineage/main/spec/OpenLineage.json#/definitions/MyCustomJobFacet</a></p> @@ -37515,8 +37539,8 @@

      Supress success

      <p><strong>Custom Facet Naming</strong></p> -<p>Naming of custom facets should follow pattern <code>{prefix}{name}{entity}Facet</code> PascalCased. <br /> -Prefix must be distinct identifier named after the project defining them to avoid colision with standard facets defined in the &lt;OpenLineage.json|OpenLineage.json&gt; spec. <br /> +<p>Naming of custom facets should follow pattern <code>{prefix}{name}{entity}Facet</code> PascalCased.<br /> +Prefix must be distinct identifier named after the project defining them to avoid colision with standard facets defined in the &lt;OpenLineage.json|OpenLineage.json&gt; spec.<br /> Entity is the core entity for which the facet is attached.</p> <p>When attached to core entity, the key should follow pattern <code>{prefix}_{name}</code>, where both prefix and name follow snakeCase pattern.</p> @@ -39212,7 +39236,7 @@

      Supress success

      <p>Generic implementation of EnvironmentFacet gives the ability to add environmental details. Since EnvironmentFacet produces HashMap it can handle properties dynamically.</p> -<p>In the current implementation added ability to collect properties for Databricks environment including mount point details. <br /> +<p>In the current implementation added ability to collect properties for Databricks environment including mount point details.<br /> Below are some examples:</p> <p>• "orgId" @@ -39577,7 +39601,7 @@

      Supress success

      <p>Generic implementation of EnvironmentFacet gives the ability to add environmental details. Since EnvironmentFacet produces HashMap it can handle properties dynamically.</p> -<p>In the current implementation added ability to collect properties for Databricks environment including mount point details. <br /> +<p>In the current implementation added ability to collect properties for Databricks environment including mount point details.<br /> Below are some examples:</p> <p>• "orgId" @@ -39779,7 +39803,7 @@

      Supress success

      I've attached the logs and a screenshot of what I'm seeing the Spark UI. If you had a chance to take a look, it's a bit verbose but I'd appreciate a second pair of eyes on my analysis. Hopefully I got something wrong 😅

      -
      log4j-SYNAPSE2-SUBSET.txt
      +
      log4j-SYNAPSE2-SUBSET.txt
      @@ -40572,7 +40596,7 @@

      Supress success

      *Thread Reply:* This is the one I wrote:

      -
      http_to_bigquery.py
      +
      http_to_bigquery.py
      @@ -44175,10 +44199,10 @@

      Supress success

      <p><strong>Solution</strong></p> -<p>We've added the <code>Created by Run</code> section on the dataset page as well as the dataset version list view. Also updated is syntax highlighting support for <code>sql</code> contexts and the empty states for context that is missing for us. <br /> -<a href="https://user-images.githubusercontent.com/7514204/160910381-644773d7-6ed6-499d-ad96-2ebdde14495b.png">image</a> <br /> -<a href="https://user-images.githubusercontent.com/7514204/160910466-5c62380f-0ad1-4e14-8292-c5c0b98b5ccb.png">image</a> <br /> -<a href="https://user-images.githubusercontent.com/7514204/160910773-4fbc6329-5ec3-483e-bf82-9269c0223906.png">image</a> <br /> +<p>We've added the <code>Created by Run</code> section on the dataset page as well as the dataset version list view. Also updated is syntax highlighting support for <code>sql</code> contexts and the empty states for context that is missing for us.<br /> +<a href="https://user-images.githubusercontent.com/7514204/160910381-644773d7-6ed6-499d-ad96-2ebdde14495b.png">image</a><br /> +<a href="https://user-images.githubusercontent.com/7514204/160910466-5c62380f-0ad1-4e14-8292-c5c0b98b5ccb.png">image</a><br /> +<a href="https://user-images.githubusercontent.com/7514204/160910773-4fbc6329-5ec3-483e-bf82-9269c0223906.png">image</a><br /> <a href="https://user-images.githubusercontent.com/7514204/160917522-b6a6c52f-e2bb-4590-85c1-6f6ed46b4eb7.png">image</a></p> <p>&gt; <strong>Note:</strong> All database schema changes require discussion. Please <a href="https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue">link the issue</a> for context.</p> @@ -49800,7 +49824,7 @@

      logger = logging.getLogger(name)

      Any thoughts?

      -
      serial_job.py
      +
      serial_job.py
      @@ -49809,7 +49833,7 @@

      logger = logging.getLogger(name)

      -
      repos.py
      +
      repos.py
      @@ -52481,7 +52505,7 @@

      logger = logging.getLogger(name)

      <p>Issue reported on Slack: <a href="https://openlineage.slack.com/archives/C01CK9T7HKR/p1653624210694359?thread_ts=1651498343.959749&amp;cid=C01CK9T7HKR|https://openlineage.slack.com/archives/C01CK9T7HKR/p1653624210694359?thread_ts=1651498343.959749&amp;cid=C01CK9T7HKR">https://openlineage.slack.com/archives/C01CK9T7HKR/p1653624210694359?thread<em>ts=1651498343.959749&amp;cid=C01CK9T7HKR|https://openlineage.slack.com/archives/C01CK9T7HKR/p1653624210694359?thread</em>ts=1651498343.959749&amp;cid=C01CK9T7HKR</a></p> -<p>Following up on this as I encounter the same issue with the Openlineage Databricks integration. This issue seems quite malicious as it crashes the Spark Context and requires a restart. <br /> +<p>Following up on this as I encounter the same issue with the Openlineage Databricks integration. This issue seems quite malicious as it crashes the Spark Context and requires a restart.<br /> I have marquez running on AWS EKS; I’m using Openlineage 0.8.2 on Databricks 10.4 (Spark 3.2.1) and my Spark config looks like this:</p> <pre><code>spark.openlineage.host &lt;https://internal-xxxxxxxxxxxxxxxxxxxxxxx-xxxxxxxxxxx.us-east-1.elb.amazonaws.com/&gt; @@ -52498,8 +52522,8 @@

      logger = logging.getLogger(name)

      </code></pre> -<p>After doing this a few times I get The spark context has stopped and the driver is restarting. Your notebook will be automatically reattached. <br /> -<a href="https://pastebin.com/9NbEjnGL">stderr shows a bunch of things</a>. log4j shows the same as for Kostikey: ERROR EventEmitter: [...] Unable to serialize logical plan due to: Infinite recursion (StackOverflowError) <br /> +<p>After doing this a few times I get The spark context has stopped and the driver is restarting. Your notebook will be automatically reattached.<br /> +<a href="https://pastebin.com/9NbEjnGL">stderr shows a bunch of things</a>. log4j shows the same as for Kostikey: ERROR EventEmitter: [...] Unable to serialize logical plan due to: Infinite recursion (StackOverflowError)<br /> I have one more piece of information which I can’t make much sense of, but hopefully someone else can; if I include the port in the host, I can very reliably crash the Spark Context on the first attempt. So:</p> <pre><code>&lt;https://internal-xxxxxxxxxxxxxxxxxxxxxxx-xxxxxxxxxxx.us-east-1.elb.amazonaws.com/&gt; &amp;lt;- crashes after a couple of attempts, sometimes it takes me a while to reproduce it while repeatedly reading/writing the same datasets @@ -52763,8 +52787,8 @@

      logger = logging.getLogger(name)

      <https://github.com/MarquezProject/marquez/issues/1736|#1736 Ability to DELETE jobs and datasets>
      - <p>When jobs or datasets are deleted (or renamed) we need a way to tell Marquez that they don't exist anymore so that they don't clutter the lineage view. <br /> -It would make sense to have <code>DELETE</code> support on <code>/api/v1/namespaces/{ns}/datasets/{name}</code> and <code>/api/v1/namespaces/{ns}/jobs/{name}</code> resources. <br /> + <p>When jobs or datasets are deleted (or renamed) we need a way to tell Marquez that they don't exist anymore so that they don't clutter the lineage view.<br /> +It would make sense to have <code>DELETE</code> support on <code>/api/v1/namespaces/{ns}/datasets/{name}</code> and <code>/api/v1/namespaces/{ns}/jobs/{name}</code> resources.<br /> This is a soft delete in the internal marquez model. It creates a new <code>deleted</code> version. The resources will not show up in the "current" lineage view in <code>GET /api/v1/lineage</code></p>
      @@ -54982,7 +55006,7 @@

      [2022-05-19, 15:24:50 UTC] {taskinstance.py:1357} INFO -

      <p>I see a change in the metadata collected from Airflow jobs which I think was introduced with the combination of Airflow 2.3/OpenLineage 0.8.1. There's an <code>airflow_version</code> facet that contains an <code>operator</code> attribute that is added by the integration.</p> -<p>In Airflow 2.2 for a <code>PostgresOperator</code> that attribute would have had a value: <code>airflow.providers.postgres.operators.postgres.PostgresOperator</code> but now in Airflow 2.3 that operator is tracked as: <code>airflow.models.taskinstance.TaskInstance</code> <br /> +<p>In Airflow 2.2 for a <code>PostgresOperator</code> that attribute would have had a value: <code>airflow.providers.postgres.operators.postgres.PostgresOperator</code> but now in Airflow 2.3 that operator is tracked as: <code>airflow.models.taskinstance.TaskInstance</code><br /> ( fwiw there's also a <code>taskInfo</code> attribute in there containing a json string which itself has a <code>operator</code> that is still set to <code>PostgresOperator</code> )</p>
      @@ -58405,15 +58429,15 @@

      [2022-05-19, 15:24:50 UTC] {taskinstance.py:1357} INFO -

      <https://github.com/OpenLineage/OpenLineage/issues/323|#323 [PROPOSAL] Ability to annotate datasets with metadata without a job run>
      - <p><strong>Purpose:</strong> <br /> -We need a way to make observations about a Dataset current state outside of a run reading or writing the data. <br /> + <p><strong>Purpose:</strong><br /> +We need a way to make observations about a Dataset current state outside of a run reading or writing the data.<br /> For example, this could be:</p> <p>• collecting schema information about a table from a warehouse • informing that a table has been deleted • etc</p> -<p><strong>Proposed implementation</strong> <br /> +<p><strong>Proposed implementation</strong><br /> Adding Dataset event types that are not lineage events. This would be the Dataset entity with its facets: </p> <p><a href="https://github.com/OpenLineage/OpenLineage/blob/3a1ccbd854bbf202bbe6437bf81786cb01ed4d01/spec/OpenLineage.json#L178-L203">OpenLineage/spec/OpenLineage.json</a></p> @@ -59427,7 +59451,7 @@

      [2022-05-19, 15:24:50 UTC] {taskinstance.py:1357} INFO -

      <p><strong>Problem</strong></p> -<p>Currently, in Airflow integration, lineage is only captured via a few extractors which greatly limits its use case. Airflow allows users to manually annotate lineage for the tasks using inlets and outlets. These can be a great fallback mechanism when extractors cannot automatically extract the input and output Datasets. <br /> +<p>Currently, in Airflow integration, lineage is only captured via a few extractors which greatly limits its use case. Airflow allows users to manually annotate lineage for the tasks using inlets and outlets. These can be a great fallback mechanism when extractors cannot automatically extract the input and output Datasets.<br /> The PR closes <a href="https://github.com/OpenLineage/OpenLineage/issues/384">#384</a></p> <p>Addresses: <a href="https://github.com/OpenLineage/OpenLineage/issues/384">#384</a></p> @@ -59598,7 +59622,7 @@

      [2022-05-19, 15:24:50 UTC] {taskinstance.py:1357} INFO -

      <p><strong>Problem</strong></p> -<p>Currently, in Airflow integration, lineage is only captured via a few extractors which greatly limits its use case. Airflow allows users to manually annotate lineage for the tasks using inlets and outlets. These can be a great fallback mechanism when extractors cannot automatically extract the input and output Datasets. <br /> +<p>Currently, in Airflow integration, lineage is only captured via a few extractors which greatly limits its use case. Airflow allows users to manually annotate lineage for the tasks using inlets and outlets. These can be a great fallback mechanism when extractors cannot automatically extract the input and output Datasets.<br /> The PR closes <a href="https://github.com/OpenLineage/OpenLineage/issues/384">#384</a></p> <p>Addresses: <a href="https://github.com/OpenLineage/OpenLineage/issues/384">#384</a></p> @@ -65625,7 +65649,7 @@

      SundayFunday

      ./gradlew :shared:spotlessApply &amp;&amp; ./gradlew :app:spotlessApply &amp;&amp; ./gradlew clean build test

      -
      testOutput.txt
      +
      testOutput.txt
      @@ -66014,7 +66038,7 @@

      SundayFunday

      <ol><li><code>java.lang.NoSuchMethodError: io.openlineage.client.OpenLineageClientUtils.newObjectMapper()Lcom/fasterxml/jackson/databind/ObjectMapper;</code></li><li>when ObjectMapper was changed there is issue with serialization, if test is run with <code>./gradlew app:test</code> the tests are passing but if it's run with <code>./gradlew app:build</code> the <code>OpenLineage.RunFacets.additionalProperties</code> is serialized as additional field and test is failing</li> </ol> -<p>EDIT: <br /> +<p>EDIT:<br /> It turns out there are also issues with Delta.</p> <p>Closes: potentially -&gt; <a href="https://openlineage.slack.com/archives/C01CK9T7HKR/p1658868064009839">https://openlineage.slack.com/archives/C01CK9T7HKR/p1658868064009839</a></p> @@ -66127,11 +66151,11 @@

      SundayFunday

      <https://github.com/OpenLineage/OpenLineage/issues/148|#148 [PROPOSAL] column level lineage facet>
      - <p><strong>Purpose:</strong> <br /> + <p><strong>Purpose:</strong><br /> For transformations like SQL queries (but also in other cases), we can extract column level lineage. This allows answering questions like ‘which root input columns are used to construct column x?’. We need to be able to capture this information in the OpenLineage model</p> -<p><strong>Proposed implementation</strong> <br /> -We propose to introduce a new dataset facet: <br /> +<p><strong>Proposed implementation</strong><br /> +We propose to introduce a new dataset facet:<br /> example:</p> <pre><code>{ @@ -68315,7 +68339,7 @@

      SundayFunday

      - +
      @@ -70193,7 +70217,7 @@

      SundayFunday

      - +
      @@ -70482,7 +70506,7 @@

      SundayFunday

      - +
      @@ -70823,7 +70847,7 @@

      SundayFunday

      - +
      @@ -73103,7 +73127,7 @@

      SundayFunday

      <p><strong>point-in-time upstream lineage</strong></p> -<p>The point-in-time upstream lineage leverages the run to dataset version relation to track back the lineage of a given dataset of job version. <br /> +<p>The point-in-time upstream lineage leverages the run to dataset version relation to track back the lineage of a given dataset of job version.<br /> Dataset version -&gt; run that produced it -&gt; consumed Dataset Versions.</p> <p><strong>Next Steps</strong></p> @@ -74261,7 +74285,7 @@

      SundayFunday

      <p><strong>point-in-time upstream lineage</strong></p> -<p>The point-in-time upstream lineage leverages the run to dataset version relation to track back the lineage of a given dataset of job version. <br /> +<p>The point-in-time upstream lineage leverages the run to dataset version relation to track back the lineage of a given dataset of job version.<br /> Dataset version -&gt; run that produced it -&gt; consumed Dataset Versions.</p> <p><strong>Next Steps</strong></p> @@ -75466,17 +75490,17 @@

      SundayFunday

      <p><code>pluggy</code> mechanism enables us to execute multiple, or none, listeners that implement particular extension point, so that users can use multiple listeners seamlessly.</p> -<p>In this PR, three such extension points are added. <br /> +<p>In this PR, three such extension points are added.<br /> When TaskInstance's state is changed to <code>RUNNING</code>, <code>on_task_instance_running</code> hook is called. On change to<code>SUCCESS</code> <code>on_task_instance_success</code> is called, similarly on <code>FAILED</code> <code>on_task_instance_failed</code> is called.</p> <p>Actual notification mechanism is be implemented using <a href="https://docs.sqlalchemy.org/en/13/orm/session_events.html#after-flush">SQLAlchemy’s events mechanism</a>. This ensures that plugins will get every change of state, regardless of where in the codebase it happened, and not require manual annotation of TI state changes across the codebase.</p> <p>To make sure that this change is not affecting performance, running this mechanism on scheduler is disabled by default. The SQLAlchemy event mechanism is also not affected by default - the event listener is only added if we have any plugin which actually provides any listener.</p> -<p>Design doc: <br /> +<p>Design doc:<br /> <a href="https://docs.google.com/document/d/1L3xfdlWVUrdnFXng1Di4nMQYQtzMfhvvWDR9K4wXnDU/edit?usp=sharing">https://docs.google.com/document/d/1L3xfdlWVUrdnFXng1Di4nMQYQtzMfhvvWDR9K4wXnDU/edit?usp=sharing</a></p> -<p>Background to this change: <br /> +<p>Background to this change:<br /> When discussing changing LineageBackend api for OpenLineage plugin, we came to consensus that the changes proposed there would be better suited to the new API <a href="https://github.com/apache/airflow/issues/17984">#17984</a></p> <p>related: <a href="https://github.com/apache/airflow/issues/17984">#17984</a></p> @@ -77872,7 +77896,7 @@

      SundayFunday

      - +
      @@ -78047,7 +78071,7 @@

      SundayFunday

      - +
      @@ -78574,7 +78598,7 @@

      SundayFunday

      <p>Generic implementation of EnvironmentFacet gives the ability to add environmental details. Since EnvironmentFacet produces HashMap it can handle properties dynamically.</p> -<p>In the current implementation added ability to collect properties for Databricks environment including mount point details. <br /> +<p>In the current implementation added ability to collect properties for Databricks environment including mount point details.<br /> Below are some examples:</p> <p>• "orgId" @@ -79768,7 +79792,7 @@

      SundayFunday

      - +
      @@ -79846,7 +79870,7 @@

      SundayFunday

      - +
      @@ -79945,7 +79969,7 @@

      SundayFunday

      - +
      @@ -79997,7 +80021,7 @@

      SundayFunday

      - +
      @@ -81458,9 +81482,9 @@

      MARQUEZAPIKEY=[YOURAPIKEY]

      #1419 [PROPOSAL] Add custom environment variables to the RunEvent
      - <p>Hi, <br /> -I would like to capture some custom environment variables and send it along with the column lineage to the server. This feature will be of great use as there is a potential use case to capture and persist some environment-specific metadata from spark clusters. <br /> -Wondering if this feature can be supported, or is on the roadmap potentially? Thanks a lot! <br /> + <p>Hi,<br /> +I would like to capture some custom environment variables and send it along with the column lineage to the server. This feature will be of great use as there is a potential use case to capture and persist some environment-specific metadata from spark clusters.<br /> +Wondering if this feature can be supported, or is on the roadmap potentially? Thanks a lot!<br /> cc: <a href="https://github.com/pawel-big-lebowski">@pawel-big-lebowski</a></p>
      @@ -81545,7 +81569,7 @@

      MARQUEZAPIKEY=[YOURAPIKEY]

      <hr /> -<p>SPDX-License-Identifier: Apache-2.0 <br /> +<p>SPDX-License-Identifier: Apache-2.0<br /> Copyright 2018-2023 contributors to the OpenLineage project</p>
      @@ -82508,7 +82532,7 @@

      MARQUEZAPIKEY=[YOURAPIKEY]

      - +
      @@ -82566,9 +82590,9 @@

      MARQUEZAPIKEY=[YOURAPIKEY]

      <p><strong>OpenLineage Spec</strong> <strong>Specification</strong></p> -<p>The specification for OpenLineage is formalized as a JsonSchema <OpenLineage.json|OpenLineage.json>. <br /> -An OpenAPI spec is also provided for HTTP based implementations: <OpenLineage.yml|OpenLineage.yml> <br /> -The documentation is published at: <a href="https://openlineage.github.io/">https://openlineage.github.io/</a> <br /> +<p>The specification for OpenLineage is formalized as a JsonSchema <OpenLineage.json|OpenLineage.json>.<br /> +An OpenAPI spec is also provided for HTTP based implementations: <OpenLineage.yml|OpenLineage.yml><br /> +The documentation is published at: <a href="https://openlineage.github.io/">https://openlineage.github.io/</a><br /> It allows extensions to the spec using <code>Custom Facets</code> as described in this document.</p> <p><strong>Core concepts</strong> @@ -82582,7 +82606,7 @@

      MARQUEZAPIKEY=[YOURAPIKEY]

      • <strong>Run</strong>: An instance of a running job with a start and completion (or failure) time. A run is identified by a globally unique ID relative to its job definition. A run ID <strong>must</strong> be an <a href="https://datatracker.ietf.org/doc/html/rfc4122">UUID</a>. • <strong>Facet</strong>: A piece of metadata attached to one of the entities defined above.</p> -<p>example: <br /> +<p>example:<br /> Here is an example of a simple start run event not adding any facet information:</p> <pre><code>{ @@ -82614,10 +82638,10 @@

      MARQUEZAPIKEY=[YOURAPIKEY]

      <p><strong>Lifecycle</strong></p> -<p>The OpenLineage API defines events to capture the lifecycle of a <em>Run</em> for a given <em>Job</em>. <br /> -When a <em>job</em> is being <em>run</em>, we capture metadata by sending run events when the state of the job transitions to a different state. <br /> -We might observe different aspects of the job run at different stages. This means that different metadata might be collected in each event during the lifecycle of a run. <br /> -All metadata is additive. for example, if more inputs or outputs are detected as the job is running we might send additional events specifically for those datasets without re-emitting previously observed inputs or outputs. <br /> +<p>The OpenLineage API defines events to capture the lifecycle of a <em>Run</em> for a given <em>Job</em>.<br /> +When a <em>job</em> is being <em>run</em>, we capture metadata by sending run events when the state of the job transitions to a different state.<br /> +We might observe different aspects of the job run at different stages. This means that different metadata might be collected in each event during the lifecycle of a run.<br /> +All metadata is additive. for example, if more inputs or outputs are detected as the job is running we might send additional events specifically for those datasets without re-emitting previously observed inputs or outputs.<br /> Example:</p> <p>• When the run starts, we collect the following Metadata: @@ -82644,7 +82668,7 @@

      MARQUEZAPIKEY=[YOURAPIKEY]

      <p>A facet is an atomic piece of metadata identified by its name. This means that emitting a new facet with the same name for the same entity replaces the previous facet instance for that entity entirely). It is defined as a JSON object that can be either part of the spec or custom facets defined in a different project.</p> -<p>Custom facets must use a distinct prefix named after the project defining them to avoid collision with standard facets defined in the <OpenLineage.json|OpenLineage.json> spec. <br /> +<p>Custom facets must use a distinct prefix named after the project defining them to avoid collision with standard facets defined in the <OpenLineage.json|OpenLineage.json> spec.<br /> They have a _schemaURL field pointing to the corresponding version of the facet schema (as a JSONPointer: <a href="https://swagger.io/docs/specification/using-ref/">$ref URL location</a> ).</p> <p>Example: <a href="https://raw.githubusercontent.com/OpenLineage/OpenLineage/main/spec/OpenLineage.json#/definitions/MyCustomJobFacet">https://raw.githubusercontent.com/OpenLineage/OpenLineage/main/spec/OpenLineage.json#/definitions/MyCustomJobFacet</a></p> @@ -82655,8 +82679,8 @@

      MARQUEZAPIKEY=[YOURAPIKEY]

      <p><strong>Custom Facet Naming</strong></p> -<p>Naming of custom facets should follow pattern <code>{prefix}{name}{entity}Facet</code> PascalCased. <br /> -Prefix must be distinct identifier named after the project defining them to avoid colision with standard facets defined in the <OpenLineage.json|OpenLineage.json> spec. <br /> +<p>Naming of custom facets should follow pattern <code>{prefix}{name}{entity}Facet</code> PascalCased.<br /> +Prefix must be distinct identifier named after the project defining them to avoid colision with standard facets defined in the <OpenLineage.json|OpenLineage.json> spec.<br /> Entity is the core entity for which the facet is attached.</p> <p>When attached to core entity, the key should follow pattern <code>{prefix}_{name}</code>, where both prefix and name follow snakeCase pattern.</p> @@ -83989,7 +84013,7 @@

      MARQUEZAPIKEY=[YOURAPIKEY]

      - +
      @@ -87244,14 +87268,14 @@

      MARQUEZAPIKEY=[YOURAPIKEY]

      <p>Hi,</p> -<p>First activity : Making HTTP Call to pull the lookup data and store it in datalake. <br /> -Second Activity : After the completion of first activity, invoking Azure databricks to use the lookup file and generate the output tables. What are all the steps should be follow to refer databricks generated tables facets as an output in the current activity &amp; input to the subsequent activities in the pipeline. <br /> -When I configure spark generated tables as output to the current activity the existing spark metadata is not showing up. <br /> -How can this be achievable. <br /> -Can someone please guide me on this. <br /> +<p>First activity : Making HTTP Call to pull the lookup data and store it in datalake.<br /> +Second Activity : After the completion of first activity, invoking Azure databricks to use the lookup file and generate the output tables. What are all the steps should be follow to refer databricks generated tables facets as an output in the current activity &amp; input to the subsequent activities in the pipeline.<br /> +When I configure spark generated tables as output to the current activity the existing spark metadata is not showing up.<br /> +How can this be achievable.<br /> +Can someone please guide me on this.<br /> Note: After the execution of each activity in ADF Pipeline I am sending start and complete/fail event lineage to Marquez.</p> -<p>Thanks, <br /> +<p>Thanks,<br /> Brahma</p>
      @@ -87482,7 +87506,7 @@

      MARQUEZAPIKEY=[YOURAPIKEY]

      Hi folks, I’m opening a vote to release OpenLineage 0.20.0, featuring: • Airflow: add new extractor for GCSToGCSOperator Adds a new extractor for this operator. -• Proxy: implement lineage event validator for client proxy
      +• Proxy: implement lineage event validator for client proxy
      Implements logic in the proxy (which is still in development) for validating and handling lineage events. • A fix of a breaking change in the common integration and other bug fixes in the DBT, Airflow, Spark, and SQL integrations and in the Java and Python clients. As per the policy here, three +1s from committers will authorize. Thanks in advance.

      @@ -87911,7 +87935,7 @@

      MARQUEZAPIKEY=[YOURAPIKEY]

      - +
      @@ -88732,7 +88756,7 @@

      MARQUEZAPIKEY=[YOURAPIKEY]

      <p><strong>Solution</strong></p> -<p>If a table is used with a database (ex: <code>catalog.my_db.my_table</code> or <code>my_db.my_table</code>), the field <code>table.schema</code> is the database. However it is <code>None</code>. <br /> +<p>If a table is used with a database (ex: <code>catalog.my_db.my_table</code> or <code>my_db.my_table</code>), the field <code>table.schema</code> is the database. However it is <code>None</code>.<br /> So we can use this field to get database, or use operator default database if <code>table.schema is None</code></p> <p>☐ Your change modifies the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json">core</a> OpenLineage model @@ -88751,7 +88775,7 @@

      MARQUEZAPIKEY=[YOURAPIKEY]

      <hr /> -<p>SPDX-License-Identifier: Apache-2.0 <br /> +<p>SPDX-License-Identifier: Apache-2.0<br /> Copyright 2018-2023 contributors to the OpenLineage project</p>
      @@ -91198,7 +91222,7 @@

      MARQUEZAPIKEY=[YOURAPIKEY]

      - +
      @@ -91224,7 +91248,7 @@

      MARQUEZAPIKEY=[YOURAPIKEY]

      - +
      @@ -91327,7 +91351,7 @@

      MARQUEZAPIKEY=[YOURAPIKEY]

      - +
      @@ -91357,7 +91381,7 @@

      MARQUEZAPIKEY=[YOURAPIKEY]

      - +
      @@ -91883,7 +91907,7 @@

      MARQUEZAPIKEY=[YOURAPIKEY]

      <hr /> -<p>SPDX-License-Identifier: Apache-2.0 <br /> +<p>SPDX-License-Identifier: Apache-2.0<br /> Copyright 2018-2023 contributors to the OpenLineage project</p>
      @@ -92192,7 +92216,7 @@

      MARQUEZAPIKEY=[YOURAPIKEY]

      - +
      @@ -93395,8 +93419,8 @@

      MARQUEZAPIKEY=[YOURAPIKEY]

      #1747 [PROPOSAL] Support for V2SessionCatalog
      - <p>Delta catalog uses V2SessionCatalog for some operations. All catalog calls except the delta lake ones are delegated to V2SessionCatalog. Currently, openlineage does not support this spark catalog. <br /> -Adding support for the V2SessionCatalog will be of great use as we are currently unable to emit metadata from some important pipelines which use this spark catalog, to our backend. <br /> + <p>Delta catalog uses V2SessionCatalog for some operations. All catalog calls except the delta lake ones are delegated to V2SessionCatalog. Currently, openlineage does not support this spark catalog.<br /> +Adding support for the V2SessionCatalog will be of great use as we are currently unable to emit metadata from some important pipelines which use this spark catalog, to our backend.<br /> Wondering if this feature can be supported, or is on the roadmap potentially? Thanks a lot!</p>
      @@ -94389,7 +94413,7 @@

      MARQUEZAPIKEY=[YOURAPIKEY]

      - +
      @@ -96557,11 +96581,11 @@

      MARQUEZAPIKEY=[YOURAPIKEY]

      #1784 Null Pointer Exception with OpenlineageSparkListener
      - <p>Hi everyone, <br /> -I am facing this null pointer exception in OpenlineageSparkListener for my job saving into delta table. <br /> + <p>Hi everyone,<br /> +I am facing this null pointer exception in OpenlineageSparkListener for my job saving into delta table.<br /> <code>ERROR AsyncEventQueue: Listener OpenLineageSparkListener threw an exception java.lang.NullPointerException java.base/java.util.concurrent.ConcurrentHashMap.putVal(Unknown Source) java.base/java.util.concurrent.ConcurrentHashMap.put(Unknown Source) io.openlineage.spark.agent.JobMetricsHolder.addMetrics(JobMetricsHolder.java:40) io.openlineage.spark.agent.OpenLineageSparkListener.onTaskEnd(OpenLineageSparkListener.java:179) org.apache.spark.scheduler.SparkListenerBus.doPostEvent(SparkListenerBus.scala:45) org.apache.spark.scheduler.SparkListenerBus.doPostEvent$(SparkListenerBus.scala:28) org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:117) org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:101) org.apache.spark.scheduler.AsyncEventQueue.super$postToAll(AsyncEventQueue.scala:105) org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:105) scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23) scala.util.DynamicVariable.withValue(DynamicVariable.scala:62) &lt;a href="http://org.apache.spark.scheduler.AsyncEventQueue.org"&gt;org.apache.spark.scheduler.AsyncEventQueue.org&lt;/a&gt;$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:100) org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:96) org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1381) org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:96)</code></p> -<p>The spark submit command used is: <br /> +<p>The spark submit command used is:<br /> <code>spark-submit --py-files /usr/local/lib/common_utils.zip,/usr/local/lib/team_utils.zip,/usr/local/lib/project_utils.zip --conf spark.executor.cores=16 --conf spark.hadoop.fs.s3a.connection.maximum=100 --conf spark.sql.shuffle.partitions=1000 --conf spark.speculation=true --conf spark.sql.adaptive.advisoryPartitionSizeInBytes=256MB --conf spark.hadoop.fs.s3a.multiobjectdelete.enable=false --conf spark.memory.fraction=0.7 --conf spark.kubernetes.executor.label.experiment=some_label --conf spark.kubernetes.executor.label.team=team_name --conf spark.driver.memory=26112m --conf &lt;a href="http://spark.kubernetes.executor.label.app.kubernetes.io/managed-by=pipeline_name"&gt;spark.kubernetes.executor.label.app.kubernetes.io/managed-by=pipeline_name&lt;/a&gt; --conf spark.kubernetes.executor.label.instance-type=4xlarge --conf spark.executor.instances=10 --conf spark.kubernetes.executor.label.env=prd --conf spark.kubernetes.executor.label.job-name=job_name --conf spark.kubernetes.executor.label.owner=owner --conf spark.kubernetes.executor.label.pipeline=pipeline --conf spark.kubernetes.executor.label.platform-name=platform_name --conf spark.speculation.multiplier=10 --conf spark.memory.storageFraction=0.4 --conf spark.driver.maxResultSize=26112m --conf spark.kubernetes.executor.request.cores=15000m --conf spark.speculation.interval=1s --conf spark.executor.memory=104g --conf spark.sql.catalogImplementation=hive --conf spark.eventLog.dir=file:///logs/spark-events --conf spark.hadoop.fs.s3a.threads.max=100 --conf spark.speculation.quantile=0.75 job.py</code></p> <p>Would greatly appreciate any help on this! Thanks!</p> @@ -96699,7 +96723,7 @@

      MARQUEZAPIKEY=[YOURAPIKEY]

      <hr /> -<p>SPDX-License-Identifier: Apache-2.0 <br /> +<p>SPDX-License-Identifier: Apache-2.0<br /> Copyright 2018-2023 contributors to the OpenLineage project</p>
      @@ -97992,7 +98016,7 @@

      MARQUEZAPIKEY=[YOURAPIKEY]

      <hr /> -<p>SPDX-License-Identifier: Apache-2.0 <br /> +<p>SPDX-License-Identifier: Apache-2.0<br /> Copyright 2018-2023 contributors to the OpenLineage project</p>
      @@ -100797,7 +100821,7 @@

      MARQUEZAPIKEY=[YOURAPIKEY]

      <hr /> -<p>SPDX-License-Identifier: Apache-2.0 <br /> +<p>SPDX-License-Identifier: Apache-2.0<br /> Copyright 2018-2023 contributors to the OpenLineage project</p>
      @@ -101121,7 +101145,7 @@

      MARQUEZAPIKEY=[YOURAPIKEY]

      .partitionBy(**partitions).option("replaceWhere", where_clause).save(path) </code></pre> -<p>I also noticed the same issue when using the <code>MERGE INTO</code> command from spark sql. <br /> +<p>I also noticed the same issue when using the <code>MERGE INTO</code> command from spark sql.<br /> Would it be possible to extend the support to these df operations. too please? Thanks!</p>
      @@ -101896,7 +101920,7 @@

      MARQUEZAPIKEY=[YOURAPIKEY]

      #1828 [Spark] duplicate events when for delta
      - <p>Each join query run on top of delta tables have two start and two complete events. Openlineage 0.22..00 <br /> + <p>Each join query run on top of delta tables have two start and two complete events. Openlineage 0.22..00<br /> <a href="https://openlineage.slack.com/archives/C01CK9T7HKR/p1683014620909079">https://openlineage.slack.com/archives/C01CK9T7HKR/p1683014620909079</a></p>
      @@ -102013,7 +102037,7 @@

      MARQUEZAPIKEY=[YOURAPIKEY]

      <hr /> -<p>SPDX-License-Identifier: Apache-2.0 <br /> +<p>SPDX-License-Identifier: Apache-2.0<br /> Copyright 2018-2023 contributors to the OpenLineage project</p>
      @@ -102630,77 +102654,77 @@

      MARQUEZAPIKEY=[YOURAPIKEY]

      #1821 Azure databricks use aggregate funcation can not buildColumnLineageDatasetFacet
      - <p>Enviroment: <br /> -databricks 12.1 (includes Apache Spark 3.3.1, Scala 2.12) <br /> -Openlineage 0.21.1 <br /> -sql: <br /> -INSERT INTO TABLE openliange<em>test <br /> -SELECT <br /> -union</em>id, <br /> -first(name) AS name <br /> -FROM tmp <br /> -GROUP BY union_id; <br /> -when we use 'GROUP BY' function, the program report err: <br /> -23/03/10 07:49:15 ERROR ColumnLevelLineageUtils: Error when invoking static method 'buildColumnLineageDatasetFacet' for Spark3 <br /> -java.lang.reflect.InvocationTargetException <br /> -at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) <br /> -at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) <br /> -at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) <br /> -at java.lang.reflect.Method.invoke(Method.java:498) <br /> -at io.openlineage.spark.agent.lifecycle.plan.column.ColumnLevelLineageUtils.buildColumnLineageDatasetFacet(ColumnLevelLineageUtils.java:35) <br /> -at io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.lambda$buildOutputDatasets$21(OpenLineageRunEventBuilder.java:424) <br /> -at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) <br /> -at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) <br /> -at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) <br /> -at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) <br /> -at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) <br /> -at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) <br /> -at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566) <br /> -at io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.buildOutputDatasets(OpenLineageRunEventBuilder.java:437) <br /> -at io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.populateRun(OpenLineageRunEventBuilder.java:296) <br /> -at io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.buildRun(OpenLineageRunEventBuilder.java:279) <br /> -at io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.buildRun(OpenLineageRunEventBuilder.java:231) <br /> -at io.openlineage.spark.agent.lifecycle.SparkSQLExecutionContext.end(SparkSQLExecutionContext.java:95) <br /> -at io.openlineage.spark.agent.OpenLineageSparkListener.sparkSQLExecEnd(OpenLineageSparkListener.java:98) <br /> -at io.openlineage.spark.agent.OpenLineageSparkListener.onOtherEvent(OpenLineageSparkListener.java:84) <br /> -at org.apache.spark.scheduler.SparkListenerBus.doPostEvent(SparkListenerBus.scala:102) <br /> -at org.apache.spark.scheduler.SparkListenerBus.doPostEvent$(SparkListenerBus.scala:28) <br /> -at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:42) <br /> -at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:42) <br /> -at org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:118) <br /> -at org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:102) <br /> -at org.apache.spark.scheduler.AsyncEventQueue.super$postToAll(AsyncEventQueue.scala:114) <br /> -at org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:114) <br /> -at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23) <br /> -at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62) <br /> -at <a href="http://org.apache.spark.scheduler.AsyncEventQueue.org">org.apache.spark.scheduler.AsyncEventQueue.org</a>$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:109) <br /> -at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:105) <br /> -at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1655) <br /> -at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:105) <br /> -Caused by: java.lang.NoSuchMethodError: org.apache.spark.sql.catalyst.expressions.aggregate.AggregateExpression.resultId()Lorg/apache/spark/sql/catalyst/expressions/ExprId; <br /> -at io.openlineage.spark3.agent.lifecycle.plan.column.ExpressionDependencyCollector.traverseExpression(ExpressionDependencyCollector.java:71) <br /> -at io.openlineage.spark3.agent.lifecycle.plan.column.ExpressionDependencyCollector.lambda$traverseExpression$4(ExpressionDependencyCollector.java:66) <br /> -at java.util.Iterator.forEachRemaining(Iterator.java:116) <br /> -at scala.collection.convert.Wrappers$IteratorWrapper.forEachRemaining(Wrappers.scala:31) <br /> -at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) <br /> -at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:647) <br /> -at io.openlineage.spark3.agent.lifecycle.plan.column.ExpressionDependencyCollector.traverseExpression(ExpressionDependencyCollector.java:66) <br /> -at io.openlineage.spark3.agent.lifecycle.plan.column.ExpressionDependencyCollector.lambda$null$2(ExpressionDependencyCollector.java:52) <br /> -at java.util.LinkedList$LLSpliterator.forEachRemaining(LinkedList.java:1235) <br /> -at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:647) <br /> -at io.openlineage.spark3.agent.lifecycle.plan.column.ExpressionDependencyCollector.lambda$collect$3(ExpressionDependencyCollector.java:52) <br /> -at org.apache.spark.sql.catalyst.trees.TreeNode.foreach(TreeNode.scala:292) <br /> -at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$foreach$1(TreeNode.scala:293) <br /> -at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$foreach$1$adapted(TreeNode.scala:293) <br /> -at scala.collection.Iterator.foreach(Iterator.scala:943) <br /> -at scala.collection.Iterator.foreach$(Iterator.scala:943) <br /> -at scala.collection.AbstractIterator.foreach(Iterator.scala:1431) <br /> -at scala.collection.IterableLike.foreach(IterableLike.scala:74) <br /> -at scala.collection.IterableLike.foreach$(IterableLike.scala:73) <br /> -at scala.collection.AbstractIterable.foreach(Iterable.scala:56) <br /> -at org.apache.spark.sql.catalyst.trees.TreeNode.foreach(TreeNode.scala:293) <br /> -at io.openlineage.spark3.agent.lifecycle.plan.column.ExpressionDependencyCollector.collect(ExpressionDependencyCollector.java:35) <br /> -at io.openlineage.spark3.agent.lifecycle.plan.column.ColumnLevelLineageUtils.buildColumnLineageDatasetFacet(ColumnLevelLineageUtils.java:32) <br /> + <p>Enviroment:<br /> +databricks 12.1 (includes Apache Spark 3.3.1, Scala 2.12)<br /> +Openlineage 0.21.1<br /> +sql:<br /> +INSERT INTO TABLE openliange<em>test<br /> +SELECT<br /> +union</em>id,<br /> +first(name) AS name<br /> +FROM tmp<br /> +GROUP BY union_id;<br /> +when we use 'GROUP BY' function, the program report err:<br /> +23/03/10 07:49:15 ERROR ColumnLevelLineageUtils: Error when invoking static method 'buildColumnLineageDatasetFacet' for Spark3<br /> +java.lang.reflect.InvocationTargetException<br /> +at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)<br /> +at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)<br /> +at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)<br /> +at java.lang.reflect.Method.invoke(Method.java:498)<br /> +at io.openlineage.spark.agent.lifecycle.plan.column.ColumnLevelLineageUtils.buildColumnLineageDatasetFacet(ColumnLevelLineageUtils.java:35)<br /> +at io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.lambda$buildOutputDatasets$21(OpenLineageRunEventBuilder.java:424)<br /> +at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)<br /> +at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)<br /> +at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)<br /> +at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)<br /> +at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)<br /> +at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)<br /> +at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566)<br /> +at io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.buildOutputDatasets(OpenLineageRunEventBuilder.java:437)<br /> +at io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.populateRun(OpenLineageRunEventBuilder.java:296)<br /> +at io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.buildRun(OpenLineageRunEventBuilder.java:279)<br /> +at io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.buildRun(OpenLineageRunEventBuilder.java:231)<br /> +at io.openlineage.spark.agent.lifecycle.SparkSQLExecutionContext.end(SparkSQLExecutionContext.java:95)<br /> +at io.openlineage.spark.agent.OpenLineageSparkListener.sparkSQLExecEnd(OpenLineageSparkListener.java:98)<br /> +at io.openlineage.spark.agent.OpenLineageSparkListener.onOtherEvent(OpenLineageSparkListener.java:84)<br /> +at org.apache.spark.scheduler.SparkListenerBus.doPostEvent(SparkListenerBus.scala:102)<br /> +at org.apache.spark.scheduler.SparkListenerBus.doPostEvent$(SparkListenerBus.scala:28)<br /> +at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:42)<br /> +at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:42)<br /> +at org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:118)<br /> +at org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:102)<br /> +at org.apache.spark.scheduler.AsyncEventQueue.super$postToAll(AsyncEventQueue.scala:114)<br /> +at org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:114)<br /> +at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23)<br /> +at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)<br /> +at <a href="http://org.apache.spark.scheduler.AsyncEventQueue.org">org.apache.spark.scheduler.AsyncEventQueue.org</a>$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:109)<br /> +at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:105)<br /> +at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1655)<br /> +at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:105)<br /> +Caused by: java.lang.NoSuchMethodError: org.apache.spark.sql.catalyst.expressions.aggregate.AggregateExpression.resultId()Lorg/apache/spark/sql/catalyst/expressions/ExprId;<br /> +at io.openlineage.spark3.agent.lifecycle.plan.column.ExpressionDependencyCollector.traverseExpression(ExpressionDependencyCollector.java:71)<br /> +at io.openlineage.spark3.agent.lifecycle.plan.column.ExpressionDependencyCollector.lambda$traverseExpression$4(ExpressionDependencyCollector.java:66)<br /> +at java.util.Iterator.forEachRemaining(Iterator.java:116)<br /> +at scala.collection.convert.Wrappers$IteratorWrapper.forEachRemaining(Wrappers.scala:31)<br /> +at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)<br /> +at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:647)<br /> +at io.openlineage.spark3.agent.lifecycle.plan.column.ExpressionDependencyCollector.traverseExpression(ExpressionDependencyCollector.java:66)<br /> +at io.openlineage.spark3.agent.lifecycle.plan.column.ExpressionDependencyCollector.lambda$null$2(ExpressionDependencyCollector.java:52)<br /> +at java.util.LinkedList$LLSpliterator.forEachRemaining(LinkedList.java:1235)<br /> +at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:647)<br /> +at io.openlineage.spark3.agent.lifecycle.plan.column.ExpressionDependencyCollector.lambda$collect$3(ExpressionDependencyCollector.java:52)<br /> +at org.apache.spark.sql.catalyst.trees.TreeNode.foreach(TreeNode.scala:292)<br /> +at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$foreach$1(TreeNode.scala:293)<br /> +at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$foreach$1$adapted(TreeNode.scala:293)<br /> +at scala.collection.Iterator.foreach(Iterator.scala:943)<br /> +at scala.collection.Iterator.foreach$(Iterator.scala:943)<br /> +at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)<br /> +at scala.collection.IterableLike.foreach(IterableLike.scala:74)<br /> +at scala.collection.IterableLike.foreach$(IterableLike.scala:73)<br /> +at scala.collection.AbstractIterable.foreach(Iterable.scala:56)<br /> +at org.apache.spark.sql.catalyst.trees.TreeNode.foreach(TreeNode.scala:293)<br /> +at io.openlineage.spark3.agent.lifecycle.plan.column.ExpressionDependencyCollector.collect(ExpressionDependencyCollector.java:35)<br /> +at io.openlineage.spark3.agent.lifecycle.plan.column.ColumnLevelLineageUtils.buildColumnLineageDatasetFacet(ColumnLevelLineageUtils.java:32)<br /> ... 34 more</p>
      @@ -102953,7 +102977,7 @@

      MARQUEZAPIKEY=[YOURAPIKEY]

      *Thread Reply:* Here is the code we use.

      -
      Azure SQL_DeltaLake.py
      +
      Azure SQL_DeltaLake.py
      @@ -103598,7 +103622,7 @@

      MARQUEZAPIKEY=[YOURAPIKEY]

      - +
      @@ -104069,7 +104093,7 @@

      MARQUEZAPIKEY=[YOURAPIKEY]

      @Paweł Leszczyński @Michael Robinson

      -
      merge-test.py
      +
      merge-test.py
      @@ -104693,7 +104717,7 @@

      MARQUEZAPIKEY=[YOURAPIKEY]

      - +
      @@ -104771,7 +104795,7 @@

      MARQUEZAPIKEY=[YOURAPIKEY]

      - +
      @@ -104798,7 +104822,7 @@

      MARQUEZAPIKEY=[YOURAPIKEY]

      - +
      @@ -105528,7 +105552,7 @@

      MARQUEZAPIKEY=[YOURAPIKEY]

      final_df.write.mode("overwrite").saveAsTable("openlineage_poc.productordertest2") </code></pre> -<p>it's not present anymore. <br /> +<p>it's not present anymore.<br /> <a href="https://github.com/OpenLineage/OpenLineage/files/11505766/lineage.txt">lineage.txt</a></p> <p>Requested on slack: <a href="https://openlineage.slack.com/archives/C01CK9T7HKR/p1684391393701839">https://openlineage.slack.com/archives/C01CK9T7HKR/p1684391393701839</a></p> @@ -105657,10 +105681,10 @@

      MARQUEZAPIKEY=[YOURAPIKEY]

      #1866 Reg: Lineage Representation With Available Datasets Generated by the Jobs
      - <p><strong>Purpose:</strong> <br /> + <p><strong>Purpose:</strong><br /> As part of the data ingestion pipeline process. process#1 generating the tables using spark. How can refer process#1 as an input for my process#2. When we specify them in lineage process by refering spark namespace where the datasets resides it's not showling accurately on Marquez UI.</p> -<p><strong>Proposed implementation</strong> <br /> +<p><strong>Proposed implementation</strong><br /> If N number of jobs generates the datasets under respective namespaces. We should refer the same in the lineage representation process as is insated of them craeting again and again under the job execution namespace.</p>
      @@ -107215,7 +107239,7 @@

      MARQUEZAPIKEY=[YOURAPIKEY]

      <hr /> -<p>SPDX-License-Identifier: Apache-2.0 <br /> +<p>SPDX-License-Identifier: Apache-2.0<br /> Copyright 2018-2023 contributors to the OpenLineage project</p>
      @@ -108159,7 +108183,7 @@

      MARQUEZAPIKEY=[YOURAPIKEY]

      @channel We released OpenLineage 0.27.2, including: Fixes: -• Python client: deprecate client.from_environment, do not skip loading config #1908 @Maciej Obuchowski
      +• Python client: deprecate client.from_environment, do not skip loading config #1908 @Maciej Obuchowski
      For the details, see: Release: https://github.com/OpenLineage/OpenLineage/releases/tag/0.27.2 Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md @@ -108662,7 +108686,7 @@

      MARQUEZAPIKEY=[YOURAPIKEY]

      .config('spark.extraListeners', 'io.openlineage.spark.agent.OpenLineageSparkListener') .config('spark.jars.packages', 'io.openlineage:openlineage_spark:0.12.0') .config('spark.openlineage.host', '<http://marquez-api:5000>') - .config('spark.openlineage.namespace', 'spark_integration')
      + .config('spark.openlineage.namespace', 'spark_integration')
      .getOrCreate())

      spark.sparkContext.setLogLevel("INFO")

      @@ -111306,7 +111330,7 @@

      MARQUEZAPIKEY=[YOURAPIKEY]

      <hr /> -<p>SPDX-License-Identifier: Apache-2.0 <br /> +<p>SPDX-License-Identifier: Apache-2.0<br /> Copyright 2018-2023 contributors to the Marquez project.</p>
      @@ -111512,7 +111536,7 @@

      MARQUEZAPIKEY=[YOURAPIKEY]

      *Thread Reply:* This is the event generated for above query.

      -
      log4j-active (11).txt
      +
      log4j-active (11).txt
      @@ -111583,7 +111607,7 @@

      MARQUEZAPIKEY=[YOURAPIKEY]

      this is event for view for which no lineage is being generated

      -
      view.txt
      +
      view.txt
      @@ -113528,11 +113552,11 @@

      MARQUEZAPIKEY=[YOURAPIKEY]

      #372 spark: Cover streaming LogicalPlans
      - <p>Cover implementations of LogicalPlan in <br /> -org.apache.spark.sql.execution.streaming.continuous <br /> -WriteToContinuousDataSource <br /> -org.apache.spark.sql.execution.streaming.sources <br /> -MemoryPlan <br /> + <p>Cover implementations of LogicalPlan in<br /> +org.apache.spark.sql.execution.streaming.continuous<br /> +WriteToContinuousDataSource<br /> +org.apache.spark.sql.execution.streaming.sources<br /> +MemoryPlan<br /> WriteToMicroBatchDataSource</p>
      @@ -113984,7 +114008,7 @@

      MARQUEZAPIKEY=[YOURAPIKEY]

      <hr /> -<p>SPDX-License-Identifier: Apache-2.0 <br /> +<p>SPDX-License-Identifier: Apache-2.0<br /> Copyright 2018-2023 contributors to the OpenLineage project</p>
      @@ -114207,7 +114231,7 @@

      MARQUEZAPIKEY=[YOURAPIKEY]

      - +
      @@ -114259,7 +114283,7 @@

      MARQUEZAPIKEY=[YOURAPIKEY]

      - +
      @@ -114502,7 +114526,7 @@

      MARQUEZAPIKEY=[YOURAPIKEY]

      <p>“job and dataset facets apply to that entity from that moment on until they are replaced by a new version of the same facet”</p> -<p>However this doesn’t allow deleting the facet altogether if the facet is no longer applicable. <br /> +<p>However this doesn’t allow deleting the facet altogether if the facet is no longer applicable.<br /> For example: the dataset no longer has an owner.</p> <p>This proposal adds a mechanism for deleting job and dataset facets.</p> @@ -114527,7 +114551,7 @@

      MARQUEZAPIKEY=[YOURAPIKEY]

      <hr /> -<p>SPDX-License-Identifier: Apache-2.0 <br /> +<p>SPDX-License-Identifier: Apache-2.0<br /> Copyright 2018-2023 contributors to the OpenLineage project</p>
      @@ -114622,7 +114646,7 @@

      MARQUEZAPIKEY=[YOURAPIKEY]

      <p><strong>Problem</strong></p> -<p>Static metadata events (known also as runless) where presented within in the proposal: <br /> +<p>Static metadata events (known also as runless) where presented within in the proposal:<br /> <a href="https://github.com/OpenLineage/OpenLineage/blob/main/proposals/1837/static_lineage.md">https://github.com/OpenLineage/OpenLineage/blob/main/proposals/1837/static_lineage.md</a></p> <p>and is being introduced to Openlineage specification within the PR: <a href="https://github.com/OpenLineage/OpenLineage/pull/1880">OpenLineage/OpenLineage#1880</a></p> @@ -115055,7 +115079,7 @@

      MARQUEZAPIKEY=[YOURAPIKEY]

      - +
      @@ -115732,7 +115756,7 @@

      MARQUEZAPIKEY=[YOURAPIKEY]

      <hr /> -<p>SPDX-License-Identifier: Apache-2.0 <br /> +<p>SPDX-License-Identifier: Apache-2.0<br /> Copyright 2018-2023 contributors to the OpenLineage project</p>
      @@ -116315,7 +116339,7 @@

      MARQUEZAPIKEY=[YOURAPIKEY]

      *Thread Reply:*

      -
      Untitled
      +
      Untitled
      @@ -122816,7 +122840,7 @@

      Marquez as an OpenLineage Client

      - +
      @@ -122868,7 +122892,7 @@

      Marquez as an OpenLineage Client

      - +
      @@ -122924,7 +122948,7 @@

      Marquez as an OpenLineage Client

      - +
      @@ -122976,7 +123000,7 @@

      Marquez as an OpenLineage Client

      - +
      @@ -123049,7 +123073,7 @@

      Marquez as an OpenLineage Client

      - +
      @@ -123173,7 +123197,7 @@

      Marquez as an OpenLineage Client

      - +
      @@ -123306,7 +123330,7 @@

      Marquez as an OpenLineage Client

      - +
      @@ -123607,7 +123631,7 @@

      Marquez as an OpenLineage Client

      - +
      @@ -123801,7 +123825,7 @@

      Marquez as an OpenLineage Client

      <hr /> -<p>SPDX-License-Identifier: Apache-2.0 <br /> +<p>SPDX-License-Identifier: Apache-2.0<br /> Copyright 2018-2023 contributors to the OpenLineage project</p>
      @@ -125373,7 +125397,7 @@

      csv_file = location.csv

      <hr /> -<p>SPDX-License-Identifier: Apache-2.0 <br /> +<p>SPDX-License-Identifier: Apache-2.0<br /> Copyright 2018-2023 contributors to the OpenLineage project</p>
      @@ -125724,7 +125748,7 @@

      csv_file = location.csv

      <hr /> -<p>SPDX-License-Identifier: Apache-2.0 <br /> +<p>SPDX-License-Identifier: Apache-2.0<br /> Copyright 2018-2023 contributors to the OpenLineage project</p>
      @@ -126085,6 +126109,641 @@

      csv_file = location.csv

      +
      +
      + + + + +
      + +
      savan + (SavanSharan_Navalgi@intuit.com) +
      +
      2024-02-13 05:27:36
       +
      +

      *Thread Reply:* i am getting this error while building the project. +tried a lot of things, any pointers or leads will be helpful? +i am using apple m1 max chip computer. thanks +> ------ Running smoke test ------ +> Exception in thread “main” java.lang.UnsatisfiedLinkError: /private/var/folders/zz/zyxvpxvq6csfxvnn0000000000000/T/native-lib4719585175993676348/libopenlineagesqljava.dylib: dlopen(/private/var/folders/zz/zyxvpxvq6csfxvnn0000000000000/T/native-lib4719585175993676348/libopenlineagesqljava.dylib, 0x0001): tried: ‘/private/var/folders/zz/zyxvpxvq6csfxvnn0000000000000/T/native-lib4719585175993676348/libopenlineagesqljava.dylib’ (mach-o file, but is an incompatible architecture (have ‘arm64’, need ‘x8664’)), ‘/System/Volumes/Preboot/Cryptexes/OS/private/var/folders/zz/zyxvpxvq6csfxvnn0000000000000/T/native-lib4719585175993676348/libopenlineagesqljava.dylib’ (no such file), ‘/private/var/folders/zz/zyxvpxvq6csfxvnn0000000000000/T/native-lib4719585175993676348/libopenlineagesqljava.dylib’ (mach-o file, but is an incompatible architecture (have ‘arm64’, need ‘x86_64’)) +> at java.base/java.lang.ClassLoader$NativeLibrary.load0(Native Method)

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      savan + (SavanSharan_Navalgi@intuit.com) +
      +
      2024-02-13 05:51:45
       +
      +

      *Thread Reply:* the build passes with out the smoke tests. +but the command you gave is throwing below error

      + +
      +

      (base) snavalgi@macos-PD7LVVY6MQ spark % ./gradlew -q :app:integrationTest --tests io.openlineage.spark.agent.SparkIcebergIntegrationTest.testDropTable

      + +

      FAILURE: Build failed with an exception.

      + +

      ** Where: + Build file ‘/Users/snavalgi/Documents/GitHub/OpenLineage/integration/spark/app/build.gradle’ line: 256

      + +

      ** What went wrong: + A problem occurred evaluating project ‘:app’.

      + +
      +

      Could not resolve all files for configuration ‘:app:spark2’. + Could not resolve io.openlineage:openlineagejava:1.9.0-SNAPSHOT. + Required by: + project :app > project :shared + Could not resolve io.openlineage:openlineagejava:1.9.0-SNAPSHOT. + Unable to load Maven meta-data from https://astronomer.jfrog.io/artifactory/maven-public-libs-snapshot/io/openlineage/openlineage-java/1.9.0-SNAPSHOT/maven-metadata.xml. + org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 10; DOCTYPE is disallowed when the feature “http://apache.org/xml/features/disallow-doctype-decl” set to true. + Could not resolve io.openlineage:openlineagesqljava:1.9.0-SNAPSHOT. + Required by: + project :app > project :shared + Could not resolve io.openlineage:openlineagesqljava:1.9.0-SNAPSHOT. + Unable to load Maven meta-data from https://astronomer.jfrog.io/artifactory/maven-public-libs-snapshot/io/openlineage/openlineage-sql-java/1.9.0-SNAPSHOT/maven-metadata.xml. + org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 10; DOCTYPE is disallowed when the feature “http://apache.org/xml/features/disallow-doctype-decl” set to true.

      +
      + +

      ** Try:

      + +
      +

      Run with --stacktrace option to get the stack trace. + Run with --info or --debug option to get more log output. + Run with --scan to get full insights. + Get more help at https://help.gradle.org.

      +
      + +

      BUILD FAILED in 10s

      +
      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      savan + (SavanSharan_Navalgi@intuit.com) +
      +
      2024-02-14 00:07:56
       +
      +

      *Thread Reply:* updated the correct error message

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-02-14 05:30:22
       +
      +

      *Thread Reply:* @savan you need to build openlineage-java and openlineage-sql-java libraries as described here: https://github.com/OpenLineage/OpenLineage/blob/73b4a3bcd84239e7baedd22b5294624623d6f3ad/integration/spark/README.md#preparation

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      savan + (SavanSharan_Navalgi@intuit.com) +
      +
      2024-02-14 06:52:31
       +
      +

      *Thread Reply:* @Maciej Obuchowski thanks for the response. +the issue was with java-8 architecture i had installed.

      + +

      i am able to compile, build and run the integration test now , with java 11 ( of appropriate arch)

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      savan + (SavanSharan_Navalgi@intuit.com) +
      +
      2024-02-14 08:12:02
       +
      +

      *Thread Reply:* was able running some(createtable) integration tests successfully. +but now +the marquez-api container is repeated crashing. any pointers?

      + +
      +

      marquez-api | [Too many errors, abort] + marquez-api | qemu: uncaught target signal 6 (Aborted) - core dumped + marquez-api | /usr/src/app/entrypoint.sh: line 19: 44 Aborted java ${JAVAOPTS} -jar marquez-**.jar server ${MARQUEZCONFIG} + marquez-api exited with code 134

      +
      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      savan + (SavanSharan_Navalgi@intuit.com) +
      +
      2024-02-14 08:15:37
       +
      +

      *Thread Reply:* the marquez-api docker image has this warning

      + +
      +

      AMD64, image may have poor performance or fail, if run via emulation

      +
      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Maciej Obuchowski + (maciej.obuchowski@getindata.com) +
      +
      2024-02-14 08:32:54
       +
      +

      *Thread Reply:* @Willy Lulciuc I think publishing arm64 image of Marquez would be a good idea

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Willy Lulciuc + (willy@datakin.com) +
      +
      2024-02-14 13:36:46
       +
      +

      *Thread Reply:* Yeah, supporting multi-architectural docker builds makes sense. Here’s an article outlining an approach https://www.padok.fr/en/blog/multi-architectures-docker-iot#architectures. @Maciej Obuchowski what’s is what you’re suggesting here?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      savan + (SavanSharan_Navalgi@intuit.com) +
      +
      2024-02-25 02:53:54
       +
      +

      *Thread Reply:* @Maciej Obuchowski @Paweł Leszczyński +i have verified the integration test for dropTestTable on my local. +it is working fine. ✅✅✅✅✅ +can you please trigger the CI for this PR? +and expedite the review and merge process +https://github.com/OpenLineage/OpenLineage/pull/2214

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-26 03:33:48
       +
      +

      *Thread Reply:* the test is still failin in CI -> https://app.circleci.com/pipelines/github/OpenLineage/OpenLineage/9488/workflows/f669d751-aa18-4735-a51f-7d647415fee8/jobs/181187

      + +

      io.openlineage.spark.agent.SparkContainerIntegrationTest testDropTable() FAILED (31.2s)

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      savan + (SavanSharan_Navalgi@intuit.com) +
      +
      2024-02-26 04:00:06
       +
      +

      *Thread Reply:* on my local i see that the test is passing. +let me update the branch and test again

      + +
      +
      Screenshot 2024-02-26 at 2.28.21 PM.png
      + + + + + + + + +
      + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      savan + (SavanSharan_Navalgi@intuit.com) +
      +
      2024-02-26 05:18:06
       +
      +

      *Thread Reply:* i have made a minor change . +can you please trigger the CI again @Paweł Leszczyński

      + + + +
      + 👀 Paweł Leszczyński +
      + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      savan + (SavanSharan_Navalgi@intuit.com) +
      +
      2024-02-26 05:27:14
       +
      +

      *Thread Reply:* the test is again passing on my local with latest code. but i notice the below error in the previous CI failure.

      + +

      the previous CI build was failing because +the actual START event for droptable in the CI had empty input and output. +> "eventType" : "START", +&gt; "inputs" : [ ], +&gt; "outputs" : [ ] +but on my local , +the START event for droptable has output populated as below. +> { +&gt; "eventType": "START", +&gt; "job": { +&gt; "namespace": "testDropTable" +&gt; }, +&gt; "inputs": [], +&gt; "outputs": [ +&gt; { +&gt; "namespace": "file", +&gt; "name": "/tmp/drop_test/drop_table_test", +&gt; "facets": { +&gt; "dataSource": { +&gt; "name": "file", +&gt; "uri": "file" +&gt; }, +&gt; "symlinks": { +&gt; "identifiers": [ +&gt; { +&gt; "namespace": "file:/tmp/drop_test", +&gt; "name": "default.drop_table_test", +&gt; "type": "TABLE" +&gt; } +&gt; ] +&gt; }, +&gt; "lifecycleStateChange": { +&gt; "lifecycleStateChange": "DROP" +&gt; } +&gt; } +&gt; } +&gt; ] +&gt; } +>

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-26 05:29:47
       +
      +

      *Thread Reply:* Please note that CI runs tests against several Spark versions. This can be configured with +-Pspark.version=3.4.2 +It's possible that your test passing for some versions while still failing for other ones.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      savan + (SavanSharan_Navalgi@intuit.com) +
      +
      2024-02-26 05:33:05
       +
      +

      *Thread Reply:* if CI is verifying against many spark version, +does that mean, +some spark version have empty output:[] +and some have populated output:[] +for the same START event of a droptable +?

      + +

      if so then how do we specify different START events respectively for those versions of spark? +is that possible?

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      Paweł Leszczyński + (pawel.leszczynski@getindata.com) +
      +
      2024-02-26 05:42:24
       +
      +

      *Thread Reply:* For complete event the assertion with empty inputs and outputs verifies only if an complete event was emitted. It would make sense for start to verify if this contains information about the deleted dataset. If it is missing for a single spark version, we should first try to understand why is this happening and if there is any workaround for this.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      savan + (SavanSharan_Navalgi@intuit.com) +
      +
      2024-02-26 05:43:43
       +
      +

      *Thread Reply:* yes makes sense. +can you please approve to run CI for integration test again?

      + +

      I really wanted to check if this build passes.

      + + + +
      +
      +
      +
      + + + + + +
      +
      + + + + +
      + +
      savan + (SavanSharan_Navalgi@intuit.com) +
      +
      2024-02-26 05:57:51
       +
      +

      *Thread Reply:* and for the spark version for which we are getting empty output[] in START event for droptable +should i open a new ticket on openlineage and report the issue?

      + + + +
      +
      +
      +
      + + + + +
      @@ -127372,7 +128031,7 @@

      csv_file = location.csv

      <p>• Short-term fix: Remove adaptive<em>spark</em>plan from the excludedNodes in DatabricksEventFilter. • Long-term resolution: Consider integrating the proposal from <a href="https://github.com/OpenLineage/OpenLineage/issues/2056">#2056</a> for a more universal filtering mechanism.</p> -<p>Environment: <br /> +<p>Environment:<br /> Databricks version: 12.2 LTS (includes Apache Spark 3.3.2, Scala 2.12)</p> <p>I hope this provides a clear picture of the issue at hand. We're eager to collaborate on resolving this.</p> @@ -127414,7 +128073,7 @@

      csv_file = location.csv

      - +
      @@ -128382,8 +129041,8 @@

      csv_file = location.csv

      <p>While implementing the current OpenLineage Spark integration in Databricks, we've made an interesting observation regarding event filtering and its behavior with narrow and wide transformations:</p> -<p>For a narrow transformation (without shuffle), the event is execute<em>insert</em>into<em>hadoop</em>fs<em>relation</em>command and it doesn't include adaptive<em>spark</em>plan. <br /> -For a wide transformation (with shuffle), the event is adaptive<em>spark</em>plan and doesn't have execute<em>insert</em>into<em>hadoop</em>fs<em>relation</em>command. <br /> +<p>For a narrow transformation (without shuffle), the event is execute<em>insert</em>into<em>hadoop</em>fs<em>relation</em>command and it doesn't include adaptive<em>spark</em>plan.<br /> +For a wide transformation (with shuffle), the event is adaptive<em>spark</em>plan and doesn't have execute<em>insert</em>into<em>hadoop</em>fs<em>relation</em>command.<br /> We've identified that the adaptive<em>spark</em>plan is actively filtered out in the DatabricksEventFilter class. This results in the event being completely ignored due to this line:</p> <p>Closes: <a href="https://github.com/OpenLineage/OpenLineage/issues/2058">#2058</a></p> @@ -128409,7 +129068,7 @@

      csv_file = location.csv

      <hr /> -<p>SPDX-License-Identifier: Apache-2.0 <br /> +<p>SPDX-License-Identifier: Apache-2.0<br /> Copyright 2018-2023 contributors to the OpenLineage project</p>
      @@ -130967,7 +131626,7 @@

      csv_file = location.csv

      - +
      @@ -130993,7 +131652,7 @@

      csv_file = location.csv

      - +
      @@ -132669,7 +133328,7 @@

      csv_file = location.csv

      <hr /> -<p>SPDX-License-Identifier: Apache-2.0 <br /> +<p>SPDX-License-Identifier: Apache-2.0<br /> Copyright 2018-2023 contributors to the OpenLineage project</p>
      @@ -134154,7 +134813,7 @@

      set the log level for the openlineage spark library

      <hr /> -<p>SPDX-License-Identifier: Apache-2.0 <br /> +<p>SPDX-License-Identifier: Apache-2.0<br /> Copyright 2018-2023 contributors to the OpenLineage project</p>
      @@ -134404,7 +135063,7 @@

      set the log level for the openlineage spark library

      - +
      @@ -134536,7 +135195,7 @@

      set the log level for the openlineage spark library

      - +
      @@ -135020,7 +135679,7 @@

      set the log level for the openlineage spark library

      *Thread Reply:* these are the worker pod logs…where there is no log of openlineageplugin

      -
      downloaded-logs-20230913-180017.json
      +
      downloaded-logs-20230913-180017.json
      @@ -136512,7 +137171,7 @@

      set the log level for the openlineage spark library

      -
      spark_events.txt
      +
      spark_events.txt
      @@ -136547,7 +137206,7 @@

      set the log level for the openlineage spark library

      *Thread Reply:*

      -
      etl-mysql.py
      +
      etl-mysql.py
      @@ -136832,7 +137491,7 @@

      set the log level for the openlineage spark library

      Comment on #2104 [Spark Integration] Receiving duplicate OpenLineage events for a Spark Job
      - <p>Hi <a href="https://github.com/pawel-big-lebowski">@pawel-big-lebowski</a> I understand your point. I've also come across cases where the duplicate events have the exact same payload with no difference in facets. I'm assuming there should be some diff in the payload for the backend to merge them. I've also seen a single Job emit 7 events and there is no fixed pattern for this. <br /> + <p>Hi <a href="https://github.com/pawel-big-lebowski">@pawel-big-lebowski</a> I understand your point. I've also come across cases where the duplicate events have the exact same payload with no difference in facets. I'm assuming there should be some diff in the payload for the backend to merge them. I've also seen a single Job emit 7 events and there is no fixed pattern for this.<br /> I also agree that there should be only a single START and a single COMPLETE event for any Job since this what the OL spec says.</p> <p>This Job emits 7 events (5 Start and 2 Complete):</p> @@ -137417,7 +138076,7 @@

      set the log level for the openlineage spark library

      <p><strong>Goal</strong></p> -<p>Allow third parties to register their implementations or custom extensions to make them easy to discover. <br /> +<p>Allow third parties to register their implementations or custom extensions to make them easy to discover.<br /> Shorten “Producer” and “schema url” values</p> <p><strong>Proposed implementation</strong></p> @@ -137968,7 +138627,7 @@

      set the log level for the openlineage spark library

      <hr /> -<p>SPDX-License-Identifier: Apache-2.0 <br /> +<p>SPDX-License-Identifier: Apache-2.0<br /> Copyright 2018-2023 contributors to the OpenLineage project</p>
      @@ -138677,7 +139336,7 @@

      set the log level for the openlineage spark library

      I am attaching the log4j, there is no openlineagecontext

      -
      log4j-active (3).txt
      +
      log4j-active (3).txt
      @@ -138774,7 +139433,7 @@

      set the log level for the openlineage spark library

      - +
      @@ -138830,7 +139489,7 @@

      set the log level for the openlineage spark library

      - +
      @@ -138884,7 +139543,7 @@

      set the log level for the openlineage spark library

      - +
      @@ -138936,7 +139595,7 @@

      set the log level for the openlineage spark library

      - +
      @@ -139014,7 +139673,7 @@

      set the log level for the openlineage spark library

      - +
      @@ -139412,7 +140071,7 @@

      set the log level for the openlineage spark library

      #2105 [PROPOSAL] Spark jobs should have a parent Job describing the Run State of Spark pipeline.
      - <p><strong>Purpose:</strong> <br /> + <p><strong>Purpose:</strong><br /> Currently there is no way to group spark jobs running within the same SparkContext since there is no concept of parent Jobs in Spark. In airflow we support Parent Runs and this way we can group Tasks together within a single DAG.</p> <p><strong>Use cases:</strong></p> @@ -139420,7 +140079,7 @@

      set the log level for the openlineage spark library

      <p>• For data catalog based OL backends, a user might want to search for Spark ETL pipelines first and then dive deeper into a specific pipeline to view the Spark Jobs. This is similar to how a user can search for airflow DAG assets and then look into the Tasks for that DAG. • Also this will help display operational metadata for Spark pipelines. For Airflow we have DAG run status (parent job status), but for Spark we don't have any such concept to display the completion state of a pipeline run. We can only track it at SparkJob level.</p> -<p><strong>Proposed implementation</strong> <br /> +<p><strong>Proposed implementation</strong><br /> This should be similar to the airflow implementation.</p>
      @@ -141338,7 +141997,7 @@

      set the log level for the openlineage spark library

      - +
      @@ -141584,7 +142243,7 @@

      set the log level for the openlineage spark library

      - +
      @@ -141636,7 +142295,7 @@

      set the log level for the openlineage spark library

      - +
      @@ -141662,7 +142321,7 @@

      set the log level for the openlineage spark library

      - +
      @@ -141716,7 +142375,7 @@

      set the log level for the openlineage spark library

      <hr /> -<p>SPDX-License-Identifier: Apache-2.0 <br /> +<p>SPDX-License-Identifier: Apache-2.0<br /> Copyright 2018-2023 contributors to the OpenLineage project</p>
      @@ -141792,7 +142451,7 @@

      set the log level for the openlineage spark library

      - +
      @@ -142244,7 +142903,7 @@

      set the log level for the openlineage spark library

      <hr /> -<p>SPDX-License-Identifier: Apache-2.0 <br /> +<p>SPDX-License-Identifier: Apache-2.0<br /> Copyright 2018-2023 contributors to the OpenLineage project</p>
      @@ -142510,7 +143169,7 @@

      set the log level for the openlineage spark library

      <hr /> -<p>SPDX-License-Identifier: Apache-2.0 <br /> +<p>SPDX-License-Identifier: Apache-2.0<br /> Copyright 2018-2023 contributors to the OpenLineage project</p>
      @@ -142608,7 +143267,7 @@

      set the log level for the openlineage spark library

      -
      cluster-logs
      +
      cluster-logs
      @@ -143418,7 +144077,7 @@

      set the log level for the openlineage spark library

      #2131 [Spark Databricks] NoSuchMethodError on ReplaceTableAsSelect
      - <p>More details here: <a href="https://github.com/OpenLineage/OpenLineage/issues/2121">#2121</a> <br /> + <p>More details here: <a href="https://github.com/OpenLineage/OpenLineage/issues/2121">#2121</a><br /> Looks like different class implementation on databricks platform.</p>
      @@ -144199,7 +144858,7 @@

      set the log level for the openlineage spark library

      *Thread Reply:* attaching the html

      -
      io.openlineage.spark.agent.lifecycle.plan.AlterTableAddPartitionCommandVisitorTest.html
      +
      io.openlineage.spark.agent.lifecycle.plan.AlterTableAddPartitionCommandVisitorTest.html
      @@ -147369,7 +148028,7 @@

      set the log level for the openlineage spark library

      <p>Hi all!</p> -<p>I really appreciate this repository and would love to use it for its data lineage aspect! <br /> +<p>I really appreciate this repository and would love to use it for its data lineage aspect!<br /> Last few days, I've been trying to get the <code>marquez api</code> Docker image up and running but whatever I try, it does not seem to work.</p> <p>It returns the following errors in Docker Desktop:</p> @@ -147380,7 +148039,7 @@

      set the log level for the openlineage spark library

      <p><code>ERROR [2023-04-03 22:44:31,217] org.apache.tomcat.jdbc.pool.ConnectionPool: Unable to create initial connections of pool. 2023-04-03 17:44:31 ! org.postgresql.util.PSQLException: FATAL: password authentication failed for user "marquez"</code></p> -<p>All the steps in the README.md file were used to configure the PostgreSQL database for Marquez and to set the correct environment variables. <br /> +<p>All the steps in the README.md file were used to configure the PostgreSQL database for Marquez and to set the correct environment variables.<br /> During my trouble shooting practices, I found a relatively similar issue on StackOverflow which refers to the danger in using the same ports for Postgres and Marquez but this did not help me yet in solving the issue (<a href="https://stackoverflow.com/questions/62115827/org-postgresql-util-psqlexception-fatal-password-authentication-failed-for-use">https://stackoverflow.com/questions/62115827/org-postgresql-util-psqlexception-fatal-password-authentication-failed-for-use</a>).</p> <p>Could you please help me out?</p> @@ -147934,7 +148593,7 @@

      set the log level for the openlineage spark library

      *Thread Reply:*

      -
      logs.txt
      +
      logs.txt
      @@ -149482,7 +150141,7 @@

      show data

      @Paweł Leszczyński I tested 1.5.0, it works great now, but the environment facets is gone in START... which I very much want it.. any thoughts?

      -
      data (10).json
      +
      data (10).json
      @@ -151344,7 +152003,7 @@

      show data

      @Paweł Leszczyński I went back to 1.4.1, output does show adls location. But environment facet is gone in 1.4.1. It shows up in 1.5.0 but namespace is back to dbfs....

      -
      data (11).json
      +
      data (11).json
      @@ -151999,7 +152658,7 @@

      show data

      - +
      @@ -152052,7 +152711,7 @@

      show data

      - +
      @@ -152827,15 +153486,11 @@

      show data

      like ( file_name, size, modification time, creation time )

      -
      image.png
      +
      - - - +
      @@ -153796,15 +154451,11 @@

      show data

      execute_spark_script(1, "/home/haneefa/airflow/dags/saved_files/")

      -
      Screenshot from 2023-11-28 11-41-50.png
      +
      - - - +
      @@ -154436,7 +155087,7 @@

      Set up SparkSubmitOperator for each query

      for query in queries: taskid = f'sparksubmit{query}' - scriptpath = '/home/haneefa/airflow/dags/customoperators/samplesqlspark.py'
      + script      path = '/home/haneefa/airflow/dags/customoperators/samplesqlspark.py'
      query      num = queries.index(query) + 1

      spark_task = SparkSubmitOperator(
@@ -154636,12 +155287,12 @@ 
      Set up SparkSubmitOperator for each query
      
 
      I was referring to fluentd openlineage proxy which lets users copy the event and send it to multiple backend. Fluentd has a list of out-of-the box output plugins containing BigQuery, S3, Redshift and others (https://www.fluentd.org/dataoutputs)
      
                     
                     
      
-                            
      Screenshot 2023-11-30 at 12.45.45.png
      
+                            
      Screenshot 2023-11-30 at 12.45.45.png
      
                             
                                 
     
-        
-            
+            
         
@@ -154727,7 +155378,7 @@ 
      Set up SparkSubmitOperator for each query
      
                     
      Hi all. I’m Stefan, and I’m after some directional advice:
      
 
 
      Context:
-• I drive an open source project called Hamilton. TL;DR: it’s an opinionated way to write python code to express dataflows, e.g. great for feature engineering, data processing, doing LLM workflows, etc. in python. 
      
+• I drive an open source project called Hamilton. TL;DR: it’s an opinionated way to write python code to express dataflows, e.g. great for feature engineering, data processing, doing LLM workflows, etc. in python.
      
 • One of the features is that you get “lineage as code”, e.g. you can get “column/dataframe level” lineage for pandas (or pyspark) code. So given a dataflow definition, and execution of a Hamilton dataflow, we can emit this code provenance for any artifacts generated — and this works where ever python runs.
 Ask:
 • As I understand open lineage was built more for artifact to artifact lineage. E.g. this table -> table -> ML model, etc. Question, for people who use/consume lineage, would this extra level of granularity (e.g. the code that ran to create an artifact) that we can provide with Hamilton, be interesting to emit as part of an open lineage event? (e.g. see inside your python airflow task, or spark job). I’m trying to determine how to prioritize an open lineage integration, and whether someone would pick up Hamilton because of it.
@@ -155406,7 +156057,7 @@ 
      Set up SparkSubmitOperator for each query
      
 
 <hr />
 
-<p>SPDX-License-Identifier: Apache-2.0 <br />
+<p>SPDX-License-Identifier: Apache-2.0<br />
 Copyright 2018-2023 contributors to the OpenLineage project</p>
                                 
      
                                 
@@ -155474,7 +156125,7 @@ 
      Set up SparkSubmitOperator for each query
      
                     
      
         
      
             
-                
+                
             
             
                 
      
@@ -156534,7 +157185,7 @@ 
      Set up SparkSubmitOperator for each query
      
                     
      
         
      
             
-                
+                
             
             
                 
      
@@ -156598,7 +157249,7 @@ 
      Set up SparkSubmitOperator for each query
      
                     
      
         
      
             
-                
+                
             
             
                 
      
@@ -156665,7 +157316,7 @@ 
      Set up SparkSubmitOperator for each query
      
                     
      *Thread Reply:* This text file contains a total of 10-11 events, including the start and completion events of one of my notebook runs. The process is simply reading from a Hive location and performing a full load to another Hive location.
      
                     
                     
      
-                            
      OpenLineage_Events_Facet_Test.txt
      
+                            
      OpenLineage_Events_Facet_Test.txt
      
                             
                                 
     
@@ -156739,7 +157390,7 @@ 
      Set up SparkSubmitOperator for each query
      
                     
      
         
      
             
-                
+                
             
             
                 
      
@@ -156872,7 +157523,7 @@ 
      Set up SparkSubmitOperator for each query
      
                     
      
         
      
             
-                
+                
             
             
                 
      
@@ -159391,12 +160042,12 @@ 
      ./bin/spark-submit --class "SparkTest" --master local[**] --jars ```
      
 Thanks 🙏
      
                     
                     
      
-                            
      Screenshot 2023-12-18 at 11.00.49.png
      
+                            
      Screenshot 2023-12-18 at 11.00.49.png
      
                             
                                 
     
-        
-            
+            
         
@@ -159624,7 +160275,7 @@ 
      ./bin/spark-submit --class "SparkTest" --master local[**] --jars ```
      
 
 <hr />
 
-<p>SPDX-License-Identifier: Apache-2.0 <br />
+<p>SPDX-License-Identifier: Apache-2.0<br />
 Copyright 2018-2023 contributors to the OpenLineage project</p>
                                 
      
                                 
@@ -160537,12 +161188,12 @@ 
      ./bin/spark-submit --class "SparkTest" --master local[**] --jars ```
      
                     
      *Thread Reply:* in Admin > Plugins can you see whether you have OpenLineageProviderPlugin and if so, are there listeners?
      
                     
                     
      
-                            
      image.png
      
+                            
      image.png
      
                             
                                 
     
-        
-            
+            
         
@@ -160641,7 +161292,7 @@ 
      ./bin/spark-submit --class "SparkTest" --master local[**] --jars ```
      
                     
      *Thread Reply:* Dont
      
                     
                     
      
-                            
      airflow.cfg
      
+                            
      airflow.cfg
      
                             
                                 
     
@@ -160702,7 +161353,7 @@ 
      ./bin/spark-submit --class "SparkTest" --master local[**] --jars ```
      
                     
      *Thread Reply:* 
      
                     
                     
      
-                            
      .env
      
+                            
      .env
      
                             
                                 
     
@@ -161714,7 +162365,7 @@ 
      ./bin/spark-submit --class "SparkTest" --master local[**] --jars ```
      
 
 <hr />
 
-<p>SPDX-License-Identifier: Apache-2.0 <br />
+<p>SPDX-License-Identifier: Apache-2.0<br />
 Copyright 2018-2023 contributors to the OpenLineage project</p>
                                 
      
                                 
@@ -162936,12 +163587,12 @@ 
      ./bin/spark-submit --class "SparkTest" --master local[**] --jars ```
      
 I've created a pdf with some code samples and OL inputs and output attributes.
      
                     
                     
      
-                            
      Test Merge Operations for OL - Databricks.pdf
      
+                            
      Test Merge Operations for OL - Databricks.pdf
      
                             
                                 
     
-        
-            
+            
         
@@ -162949,7 +163600,7 @@ 
      ./bin/spark-submit --class "SparkTest" --master local[**] --jars ```
      
                             
                         
      
                     
      
-                            
      metadata.json
      
+                            
      metadata.json
      
                             
                                 
     
@@ -163330,7 +163981,7 @@ 
      ./bin/spark-submit --class "SparkTest" --master local[**] --jars ```
      
 
 <hr />
 
-<p>SPDX-License-Identifier: Apache-2.0 <br />
+<p>SPDX-License-Identifier: Apache-2.0<br />
 Copyright 2018-2023 contributors to the OpenLineage project</p>
                                 
      
                                 
@@ -163433,7 +164084,7 @@ 
      ./bin/spark-submit --class "SparkTest" --master local[**] --jars ```
      
                     
      
         
      
             
-                
+                
             
             
                 
      
@@ -163572,12 +164223,12 @@ 
      ./bin/spark-submit --class "SparkTest" --master local[**] --jars ```
      
                                 
                                 
      #2186 [PROPOSAL] Formalizing transformation types
      
                                 
      
-                                    <p><strong>Purpose:</strong> <br />
+                                    <p><strong>Purpose:</strong><br />
 Automatic support of lineage use cases highly depends on the type of transformations between columns. Oftentimes the use cases to be supported pose a significant risk (compliance, impact analysis) and false negatives are highly undesirable. This means that the taxonomy of transformation types supported by OpenLineage has to be tightly specified. Otherwise OpenLineage consumers cannot build reliable automation on top of the transformation types, and therefore not reliably support these high-risk use cases. This is a proposal for modifying the column-level lineage facet with a tightly specified taxonomy of transformation types.</p>
 
 <p><strong>Proposed implementation</strong></p>
 
-<ol><li>Change transformationType definition: <br />
+<ol><li>Change transformationType definition:<br />
 From:</li>
 </ol>
 
@@ -164795,7 +165446,7 @@ 
      ./bin/spark-submit --class "SparkTest" --master local[**] --jars ```
      
                     
      
         
      
             
-                
+                
             
             
                 
      
@@ -164859,7 +165510,7 @@ 
      ./bin/spark-submit --class "SparkTest" --master local[**] --jars ```
      
                     
      
         
      
             
-                
+                
             
             
                 
      
@@ -164913,7 +165564,7 @@ 
      ./bin/spark-submit --class "SparkTest" --master local[**] --jars ```
      
                     
      
         
      
             
-                
+                
             
             
                 
      
@@ -165024,12 +165675,12 @@ 
      ./bin/spark-submit --class "SparkTest" --master local[**] --jars ```
      
 
      Do we have the functionality to search on the lineage we are getting?
      
                     
                     
      
-                            
      image.png
      
+                            
      image.png
      
                             
                                 
     
-        
-            
+            
         
@@ -165050,7 +165701,7 @@ 
      ./bin/spark-submit --class "SparkTest" --master local[**] --jars ```
      
                     
      
         
      
             
-                
+                
             
             
                 
      
@@ -165134,7 +165785,7 @@ 
      ./bin/spark-submit --class "SparkTest" --master local[**] --jars ```
      
                     
      
         
      
             
-                
+                
             
             
                 
      
@@ -165478,6 +166129,85 @@ 
      ./bin/spark-submit --class "SparkTest" --master local[**] --jars ```
      
         
             
                 
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      David Goss
+                     (david.goss@matillion.com)
+                
      
+                
      2024-02-07 09:49:43
      
+                
      
+                    
      *Thread Reply:* Sorry for the late reply @Julien Le Dem. So we have a custom dataset facet internally right now which is doing a couple of different jobs:
+• datasetType captures TABLE vs VIEW etc. We’re getting into more types with APIs, files etc so that will expand
+• nameParts captures the component parts of the dataset name where it’s a fully-qualified table name for example - so like DATABASE.SCHEMA.TABLE would be in there as ["TABLE", "SCHEMA", "DATABASE"] - this might seem weird but it’s useful for us to not have to parse this back out later especially where the canonical name has quotes, escapes etc
+{
+  "$schema": "<https://json-schema.org/draft/2020-12/schema>",
+  "$id": "****************/MatillionDatasetFacet.json",
+  "title": "Matillion Dataset Facet Schema",
+  "description": "Matillion Dataset Facet Schema",
+  "type": "object",
+  "properties": {
+    "type": {
+      "type": "string",
+      "description": "The type of the dataset",
+      "enum": [
+        "TABLE",
+        "VIEW",
+        "UNKNOWN"
+      ]
+    },
+    "nameParts": {
+      "type": "array",
+      "description": "The component parts of the dataset name (e.g. table, schema, database), from lowest to highest level",
+      "items": {
+        "type": "string"
+      }
+    }
+  }
+}
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      David Goss
+                     (david.goss@matillion.com)
+                
      
+                
      2024-02-07 09:50:46
      
+                
      
+                    
      *Thread Reply:* (This is just how we’re handling some stuff internally really - the idea is we can use our custom facet where needed and will pivot to standard facets if/when they start to exist for those use cases.)
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
                     
      
         
      
             
@@ -165548,7 +166278,7 @@ 
      ./bin/spark-submit --class "SparkTest" --master local[**] --jars ```
      
 
 <hr />
 
-<p>SPDX-License-Identifier: Apache-2.0 <br />
+<p>SPDX-License-Identifier: Apache-2.0<br />
 Copyright 2018-2023 contributors to the OpenLineage project</p>
                                 
      
                                 
@@ -166053,12 +166783,12 @@ 
      ./bin/spark-submit --class "SparkTest" --master local[**] --jars ```
      
                     
      *Thread Reply:* 
      
                     
                     
      
-                            
      image.png
      
+                            
      image.png
      
                             
                                 
     
-        
-            
+            
         
@@ -166263,7 +166993,7 @@ 
      Gradle 8.5
      
                         
      
                     
                     
      
-                            
      .tool-versions
      
+                            
      .tool-versions
      
                             
                                 
     
@@ -166869,12 +167599,12 @@ 
      Gradle 8.5
      
                     
      any suggestions on naming for Graph API sources from outlook? I pull a lot of data from email attachments with Airflow. generally I am passing a resource (email address), the mailbox, and subfolder. from there I list messages and find attachments
      
                     
                     
      
-                            
      image.png
      
+                            
      image.png
      
                             
                                 
     
-        
-            
+            
         
@@ -167620,11 +168350,11 @@ 
      Gradle 8.5
      
                                 
                                 
      #372 spark: Cover streaming LogicalPlans
      
                                 
      
-                                    <p>Cover implementations of LogicalPlan in <br />
-org.apache.spark.sql.execution.streaming.continuous <br />
-WriteToContinuousDataSource <br />
-org.apache.spark.sql.execution.streaming.sources <br />
-MemoryPlan <br />
+                                    <p>Cover implementations of LogicalPlan in<br />
+org.apache.spark.sql.execution.streaming.continuous<br />
+WriteToContinuousDataSource<br />
+org.apache.spark.sql.execution.streaming.sources<br />
+MemoryPlan<br />
 WriteToMicroBatchDataSource</p>
                                 
      
                                 
@@ -167718,12 +168448,12 @@ 
      Gradle 8.5
      
                     
      Hello team I see the following issue when i install apache-airflow-providers-openlineage==1.4.0
      
                     
                     
      
-                            
      Screenshot 2024-01-22 at 1.43.27 PM.png
      
+                            
      Screenshot 2024-01-22 at 1.43.27 PM.png
      
                             
                                 
     
-        
-            
+            
         
@@ -167944,12 +168674,12 @@ 
      Gradle 8.5
      
 is there any solution?
      
                     
                     
      
-                            
      Screenshot 2024-01-25 170958.png
      
+                            
      Screenshot 2024-01-25 170958.png
      
                             
                                 
     
-        
-            
+            
         
@@ -168059,13 +168789,13 @@ 
      Gradle 8.5
      
 <p><strong>Getting started</strong>
 <strong>Quickstart</strong></p>
 
-<p>The fastest way to get started testing Spark and OpenLineage is to use the docker-compose files included <br />
+<p>The fastest way to get started testing Spark and OpenLineage is to use the docker-compose files included<br />
 in the project. From the Spark integration directory ($OPENLINEAGE_ROOT/integration/spark), execute</p>
 
 <pre><code>docker-compose up
 </code></pre>
 
-<p>This will start Marquez as an Openlineage client and Jupyter Spark notebook on localhost:8888. On startup, the notebook container logs will show a list of URLs <br />
+<p>This will start Marquez as an Openlineage client and Jupyter Spark notebook on localhost:8888. On startup, the notebook container logs will show a list of URLs<br />
 including an access token, such as</p>
 
 <pre><code>notebook_1  |     To access the notebook, open this file in a browser:
@@ -168075,8 +168805,8 @@ 
      Gradle 8.5
      
 notebook_1  |      or &lt;http://127.0.0.1:8888/?token=XXXXXX&gt;
 </code></pre>
 
-<p>Copy the URL with the localhost IP and paste it into your browser window to begin creating a new Jupyter <br />
-Spark notebook (see the <docs|https://jupyter-docker-stacks.readthedocs.io/en/latest/> for info on <br />
+<p>Copy the URL with the localhost IP and paste it into your browser window to begin creating a new Jupyter<br />
+Spark notebook (see the <docs|https://jupyter-docker-stacks.readthedocs.io/en/latest/> for info on<br />
 using the Jupyter docker image).</p>
 
 <p><strong>OpenLineageSparkListener as a plain Spark Listener</strong></p>
@@ -168100,9 +168830,9 @@ 
      Gradle 8.5
      
 <pre><code>gradle shadowJar
 </code></pre>
 
-<p>then reference it in the Jupyter notebook with the following (note that the jar should be built <br />
-<em>before</em> running the <code>docker-compose up</code> step or docker will just mount a dummy folder; once the <br />
-<code>build/libs</code> directory exists, you can repeatedly build the jar without restarting the jupyter <br />
+<p>then reference it in the Jupyter notebook with the following (note that the jar should be built<br />
+<em>before</em> running the <code>docker-compose up</code> step or docker will just mount a dummy folder; once the<br />
+<code>build/libs</code> directory exists, you can repeatedly build the jar without restarting the jupyter<br />
 container):</p>
 
 <pre><code>from pyspark.sql import SparkSession
@@ -168121,7 +168851,7 @@ 
      Gradle 8.5
      
 <p><strong>Arguments</strong>
 <strong>Spark Listener</strong></p>
 
-<p>The SparkListener reads its configuration from SparkConf parameters. These can be specified on the <br />
+<p>The SparkListener reads its configuration from SparkConf parameters. These can be specified on the<br />
 command line or in the <code>conf/spark-defaults.conf</code> file.</p>
 
 <p>The following parameters can be specified in the Spark configuration:</p>
@@ -168147,7 +168877,7 @@ 
      Gradle 8.5
      
 
 <p><strong>Kinesis</strong></p>
 
-<p>If <code>spark.openlineage.transport.type</code> is set to <code>kinesis</code>, then the below parameters would be read and used when building KinesisProducer. <br />
+<p>If <code>spark.openlineage.transport.type</code> is set to <code>kinesis</code>, then the below parameters would be read and used when building KinesisProducer.<br />
 Also, KinesisTransport depends on you to provide artifact <code>com.amazonaws:amazon_kinesis_producer:0.14.0</code> or compatible on your classpath.</p>
 
 <p><strong>Kafka</strong></p>
@@ -168163,8 +168893,8 @@ 
      Gradle 8.5
      
 
 <p><strong>Preparation</strong></p>
 
-<p>The integration depends on two libraries that are build locally <code>openlineage-java</code> and <code>openlineage-sql-java</code>, <br />
-so before any testing or building of a package you need to publish the appropriate artifacts to local maven repository. <br />
+<p>The integration depends on two libraries that are build locally <code>openlineage-java</code> and <code>openlineage-sql-java</code>,<br />
+so before any testing or building of a package you need to publish the appropriate artifacts to local maven repository.<br />
 To build the packages you need to execute.</p>
 
 <p>To install <code>openlineage-java</code> in local maven repo run:</p>
@@ -168197,12 +168927,12 @@ 
      Gradle 8.5
      
 
 <p><strong>Extending</strong></p>
 
-<p>The Spark library is intended to support extension via custom implementations of a handful <br />
-of interfaces. Nearly every extension interface extends or mimics Scala's <code>PartialFunction</code>. The <br />
-<code>isDefinedAt(Object x)</code> method determines whether a given input is a valid input to the function. <br />
-A default implementation of <code>isDefinedAt(Object x)</code> is provided, which checks the generic type <br />
-arguments of the concrete class, if concrete type arguments are given, and determines if the input <br />
-argument matches the generic type. For example, the following class is automatically defined for an <br />
+<p>The Spark library is intended to support extension via custom implementations of a handful<br />
+of interfaces. Nearly every extension interface extends or mimics Scala's <code>PartialFunction</code>. The<br />
+<code>isDefinedAt(Object x)</code> method determines whether a given input is a valid input to the function.<br />
+A default implementation of <code>isDefinedAt(Object x)</code> is provided, which checks the generic type<br />
+arguments of the concrete class, if concrete type arguments are given, and determines if the input<br />
+argument matches the generic type. For example, the following class is automatically defined for an<br />
 input argument of type <code>MyDataset</code>.</p>
 
 <pre><code>class MyDatasetDetector extends QueryPlanVisitor&lt;MyDataset, OutputDataset&gt; {
@@ -168215,16 +168945,16 @@ 
      Gradle 8.5
      
 
 <h3><shared/src/main/java/io/openlineage/spark/api/OpenLineageEventHandlerFactory.java|<code>OpenLineageEventHandlerFactory</code>&gt;</h3>
 
-<p>This interface defines the main entrypoint to the extension codebase. Custom implementations <br />
-are registered by following Java's <a href="https://docs.oracle.com/javase/8/docs/api/java/util/ServiceLoader.html"><code>ServiceLoader</code> conventions</a>. <br />
-A file called <code>io.openlineage.spark.api.OpenLineageEventHandlerFactory</code> must exist in the <br />
-application or jar's <code>META-INF/service</code> directory. Each line of that file must be the fully <br />
-qualified class name of a concrete implementation of <code>OpenLineageEventHandlerFactory</code>. More than one <br />
-implementation can be present in a single file. This might be useful to separate extensions that <br />
-are targeted toward different environments - e.g., one factory may contain Azure-specific extensions, <br />
+<p>This interface defines the main entrypoint to the extension codebase. Custom implementations<br />
+are registered by following Java's <a href="https://docs.oracle.com/javase/8/docs/api/java/util/ServiceLoader.html"><code>ServiceLoader</code> conventions</a>.<br />
+A file called <code>io.openlineage.spark.api.OpenLineageEventHandlerFactory</code> must exist in the<br />
+application or jar's <code>META-INF/service</code> directory. Each line of that file must be the fully<br />
+qualified class name of a concrete implementation of <code>OpenLineageEventHandlerFactory</code>. More than one<br />
+implementation can be present in a single file. This might be useful to separate extensions that<br />
+are targeted toward different environments - e.g., one factory may contain Azure-specific extensions,<br />
 while another factory may contain GCP extensions.</p>
 
-<p>The <code>OpenLineageEventHandlerFactory</code> interface makes heavy use of default methods. Implementations <br />
+<p>The <code>OpenLineageEventHandlerFactory</code> interface makes heavy use of default methods. Implementations<br />
 may override any or all of the following methods</p>
 
 <p>```
@@ -168254,12 +168984,12 @@ 
      Gradle 8.5
      
                         
      
                     
                     
      
-                            
      image.png
      
+                            
      image.png
      
                             
                                 
     
-        
-            
+            
         
@@ -168382,6 +169112,62 @@ 
      Gradle 8.5
      
                     
                     
                     
+                        
      
+                        👍 jayant joshi
+                        
      
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      jayant joshi
+                     (itsjayantjoshi@gmail.com)
+                
      
+                
      2024-01-30 02:23:06
      
+                
      
+                    
      *Thread Reply:* Thanks @Maciej Obuchowski
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Willy Lulciuc
+                     (willy@datakin.com)
+                
      
+                
      2024-01-30 13:47:41
      
+                
      
+                    
      *Thread Reply:* @jayant joshi have you’ve tried running Marquez via gitpod? see https://github.com/MarquezProject/marquez?tab=readme-ov-file#try-it
      
+                    
+                    
+                    
 		
      
             
      
         
      
@@ -168446,7 +169232,7 @@ 
      Gradle 8.5
      
                     
                     
                         
      
-                        ❤️ Maciej Obuchowski, Abdallah, Eric Veleker, Rodrigo Maia
+                        ❤️ Maciej Obuchowski, Abdallah, Eric Veleker, Rodrigo Maia, tati
                         
      
                     
 		
      
@@ -168499,11 +169285,11 @@ 
      Gradle 8.5
      
                     
                     
                         
      
-                        ❤️ Ross Turk, Harel Shein
+                        ❤️ Ross Turk, Harel Shein, tati, Rodrigo Maia, Maciej Obuchowski, Jarek Potiuk, Mattia Bertorello
                         
      
                     
                         
      
-                        🙌 Francis McGregor-Macdonald, Harel Shein
+                        🙌 Francis McGregor-Macdonald, Harel Shein, tati, Maciej Obuchowski, Jarek Potiuk
                         
      
                     
 		
      
@@ -168537,9 +169323,10147 @@ 
      Gradle 8.5
      
                         
      
                     
                         
      
-                        ➕ Harel Shein
+                        ➕ Harel Shein, Maciej Obuchowski
+                        
      
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Balachandru S
+                     (balachandru2468@gmail.com)
+                
      
+                
      2024-01-30 01:16:01
      
+                
      
+                    
      Hi Team,
+I am trying to run a pyspark code through spark-submit command. Find the command which trying below. While running below command I am facing an issue "java.net.UnknownHostException: marquez-api".  Any solution to solve this issue ?
+Note that through jupyter lab I can run same code and create a lineage in the Marquez UI.
      
+
+
      "spark-submit  --conf "spark.extraListeners=io.openlineage.spark.agent.OpenLineageSparkListener"  --packages "io.openlineage:openlineagespark:1.7.0"  --conf "spark.openlineage.transport.type=http"  --conf "spark.openlineage.transport.url= http://marquez-api:5000"  --conf "spark.openlineage.namespace=sparkintegration"  pyspark_etl.py".
      
+                    
+                    
      
+                            
      image.png
      
+                            
+                                
+    
+        
+            
+        
+    
+                            
+                        
      
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Jakub Dardziński
+                     (jakub.dardzinski@getindata.com)
+                
      
+                
      2024-01-30 02:06:27
      
+                
      
+                    
      *Thread Reply:* You've probably been running jupyter lab with docker compose alonng with Marquez containers?
      
+
+
      Try changing to host.docker.internal from marquez-api
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Balachandru S
+                     (balachandru2468@gmail.com)
+                
      
+                
      2024-01-30 02:43:43
      
+                
      
+                    
      *Thread Reply:* Are you mean to say try passing below one ?
+"spark.openlineage.transport.url= http://host.docker.internal:5000"
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Jakub Dardziński
+                     (jakub.dardzinski@getindata.com)
+                
      
+                
      2024-01-30 02:47:25
      
+                
      
+                    
      *Thread Reply:* yes
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Balachandru S
+                     (balachandru2468@gmail.com)
+                
      
+                
      2024-01-30 02:47:40
      
+                
      
+                    
      *Thread Reply:* The above one also throwing a same error.
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Jakub Dardziński
+                     (jakub.dardzinski@getindata.com)
+                
      
+                
      2024-01-30 02:48:00
      
+                
      
+                    
      *Thread Reply:* how do you instantiate marquez?
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Balachandru S
+                     (balachandru2468@gmail.com)
+                
      
+                
      2024-01-30 02:49:56
      
+                
      
+                    
      *Thread Reply:* Using below two sources:
+git clone https://github.com/OpenLineage/OpenLineage
      
+
+
      docker run --network sparkdefault -p 3000:3000 -e MARQUEZHOST=marquez-api -e MARQUEZ_PORT=5000 --link marquez-api:marquez-api marquezproject/marquez-web:0.44.0
      
+                    
      
+                            
+                            
+                            
+                                
+                                
      OpenLineage/OpenLineage
      
+                                
      
+                                    An Open Standard for lineage metadata collection
+                                
      
+                                
+                                    
      
+                                        
      Website
      
+                                        <http://openlineage.io>
+                                    
      
+                                
+                                    
      
+                                        
      Stars
      
+                                        1496
+                                    
      
+                                
+                                
+    
+        
+    
+                                
+                                
+                            
+                        
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Jakub Dardziński
+                     (jakub.dardzinski@getindata.com)
+                
      
+                
      2024-01-30 02:50:56
      
+                
      
+                    
      *Thread Reply:* so you need to pass additional -p 5000:5000 flag
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Balachandru S
+                     (balachandru2468@gmail.com)
+                
      
+                
      2024-01-30 03:02:02
      
+                
      
+                    
      *Thread Reply:* Are you mean like this  --conf "spark.openlineage.transport.port=5000" in the spark submit command ?
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Jakub Dardziński
+                     (jakub.dardzinski@getindata.com)
+                
      
+                
      2024-01-30 03:04:03
      
+                
      
+                    
      *Thread Reply:* no, in docker run command
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Jakub Dardziński
+                     (jakub.dardzinski@getindata.com)
+                
      
+                
      2024-01-30 03:04:59
      
+                
      
+                    
      *Thread Reply:* you need to expose port 5000 for marquez-api container
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Jakub Dardziński
+                     (jakub.dardzinski@getindata.com)
+                
      
+                
      2024-01-30 03:26:49
      
+                
      
+                    
      *Thread Reply:* and if you expose localhost:5000 should be available too
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Balachandru S
+                     (balachandru2468@gmail.com)
+                
      
+                
      2024-01-30 04:03:53
      
+                
      
+                    
      *Thread Reply:* Find the attached localhost 5000 & 5001 port results. Note that while running same code in the jupyter notebook, I could see lineage on the Marquez UI. For running a code through spark-submit only I am facing an issue.
      
+                    
+                    
      
+                            
      image.png
      
+                            
+                                
+    
+        
+            
+        
+    
+                            
+                        
      
+                    
      
+                            
      image.png
      
+                            
+                                
+    
+        
+            
+        
+    
+                            
+                        
      
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Jakub Dardziński
+                     (jakub.dardzinski@getindata.com)
+                
      
+                
      2024-01-30 04:04:48
      
+                
      
+                    
      *Thread Reply:* yeah but are you running jupyter from console or in docker?
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Balachandru S
+                     (balachandru2468@gmail.com)
+                
      
+                
      2024-01-30 04:20:34
      
+                
      
+                    
      *Thread Reply:* I am using the jupyter one which comes with Open Lineage docker container
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Balachandru S
+                     (balachandru2468@gmail.com)
+                
      
+                
      2024-01-31 04:25:13
      
+                
      
+                    
      *Thread Reply:* Hi @Jakub Dardziński, good afternoon. Any solution on this one ?
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Paweł Leszczyński
+                     (pawel.leszczynski@getindata.com)
+                
      
+                
      2024-01-31 04:43:13
      
+                
      
+                    
      *Thread Reply:* this is some workshop copy-paste tutorial https://github.com/OpenLineage/workshops/blob/main/spark/column-lineage-oct-2022.ipynb i did in the past.
      
+
+
      you can try the following code snippet:
+import json,requests
+marquez_url = "<http://host.docker.internal:5000>" ## this may depend on your local setup
+if (requests.get("{}/api/v1/namespaces".format(marquez_url)).status_code == 200):
+    print("Marquez is OK.")
+else:
+    print("Cannot connect to Marquez")
+ to see if docker network is setup properly
      
+                    
      
+                            
+                            
+                            
+                                
+                                
      <https://github.com/OpenLineage/workshops/blob/main/spark/column-lineage-oct-2022.ipynb | column-lineage-oct-2022.ipynb>
      
+                                
      
+                                    <p><code>
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "8867600e",
+   "metadata": {},
+   "source": [
+    "# Column Lineage Demo - October 2022\n",
+    "\n",
+    "In this document we provide a demo of current implementation of Column Level Lineage within Marquez and Openlinege \n",
+    "\n",
+    "## Envirnonment setup \n",
+    "\n",
+    " ** I've tested this demo on `Mac 12.6` with `Docker Desktop 4.12.0`. \n",
+    " ** I've setup Spark Jupyter environment like described here [here](&lt;https://openlineage.io/docs/integrations/spark/quickstart_local&gt;)\n",
+    " ** Run Marquez using the latest source code which contains the column level lineage feature.\n",
+    "</code>\n",
+    " git clone <a href="https://github.com/MarquezProject/marquez.git">https://github.com/MarquezProject/marquez.git</a>\n",
+    " cd marquez\n",
+    " ./docker/up.sh --build\n",
+    " ``<code>\n",
+    "\n",
+    "&gt; Once an October 2022 version of Marquez is released, testing this with Marquez docker image may be an easier option.  \n",
+    " \n",
+    "Let's check first if Marquez instance is properly running under a defined address. Returned status code should be 200."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 42,
+   "id": "b43f2f98",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Marquez is OK.\n"
+     ]
+    }
+   ],
+   "source": [
+    "import json,requests\n",
+    "marquez_url = \"&lt;http://host.docker.internal:5000\&gt;" ## this may depend on your local setup\n",
+    "if (requests.get(\"{}/api/v1/namespaces\".format(marquez_url)).status_code == 200):\n",
+    "    print(\"Marquez is OK.\")\n",
+    "else:\n",
+    "    print(\"Cannot connect to Marquez\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "036c5b2f",
+   "metadata": {},
+   "source": [
+    "Let's create the warehouse directory first."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "51e8b4f9",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%mkdir /home/jovyan/notebooks/spark-warehouse"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "7fe60ddb",
+   "metadata": {},
+   "source": [
+    "If Marquez connection is OK, we can start a Spark context with OpenLineage pointed to Marquez"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 43,
+   "id": "914e5905",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from pyspark.sql import SparkSession\n",
+    "\n",
+    "spark = (SparkSession.builder.master('local')\n",
+    "         .appName('sample_spark')\n",
+    "         .config('spark.extraListeners', 'io.openlineage.spark.agent.OpenLineageSparkListener')\n",
+    "         .config('spark.jars.packages', 'io.openlineage:openlineage_spark:0.15.1')\n",
+    "         .config('spark.openlineage.url', '{}/api/v1/namespaces/column-lineage/'.format(marquez_url))\n",
+    "         .getOrCreate())"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "9bc086ab",
+   "metadata": {},
+   "source": [
+    "Let's clear Docker's existing warehouse, in case the previous data exist."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 44,
+   "id": "a011bf15",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%rm -rf /home/jovyan/notebooks/spark-warehouse/**"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "dfc940c6",
+   "metadata": {},
+   "source": [
+    "## Run example Spark Job"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e3e7f340",
+   "metadata": {},
+   "source": [
+    "Let's create now four datasets:</code>dataset<em>a<code>,</code>dataset</em>b<code>,</code>dataset<em>c<code>,</code>dataset</em>d<code>:\n",
+    " **</code>dataset<em>a<code>has to columns</code>col</em>1<code>and</code>col<em>2<code>filled with some data,\n",
+    " **</code>dataset</em>b<code>has one column</code>col<em>3<code>and is created from</code>dataset</em>a<code>,\n",
+    " **</code>dataset<em>c<code>with</code>col</em>4<code>and</code>dataset<em>d<code>with</code>col</em>5<code>are created from</code>dataset<em>b<code>\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 45,
+   "id": "f401c1e1",
+   "metadata": {
+    "scrolled": true
+   },
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "22/10/11 15:46:09 WARN HadoopFSUtils: The directory file:/home/jovyan/notebooks/spark-warehouse/dataset_c was not found. Was it deleted very recently?\n",
+      "22/10/11 15:46:10 WARN HadoopFSUtils: The directory file:/home/jovyan/notebooks/spark-warehouse/dataset_d was not found. Was it deleted very recently?\n"
+     ]
+    }
+   ],
+   "source": [
+    "spark.createDataFrame([\n",
+    "    {'col_1': 1, 'col_2': 2},\n",
+    "    {'col_1': 3, 'col_2': 4}\n",
+    "]).write.mode(\"overwrite\").saveAsTable('dataset_a')\n",
+    "spark.sql(\"SELECT col_1 + col_2 AS col_3 FROM dataset_a\").write.mode(\"overwrite\").saveAsTable('dataset_b')\n",
+    "spark.sql(\"SELECT col_3 AS col_4 FROM dataset_b\").write.mode(\"overwrite\").saveAsTable('dataset_c')\n",
+    "spark.sql(\"SELECT col_3 AS col_5 FROM dataset_b\").write.mode(\"overwrite\").saveAsTable('dataset_d')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4143ccb8",
+   "metadata": {},
+   "source": [
+    "This should result in following column lineage graph:\n",
+    " **</code>col</em>3<code>is created out of</code>col<em>1<code>and</code>col</em>2<code>,\n",
+    " **</code>col<em>4<code>and</code>col</em>5<code>depend on</code>col<em>3<code>."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "94092c13",
+   "metadata": {},
+   "source": [
+    "## Marquez API "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "40563cf7",
+   "metadata": {},
+   "source": [
+    "### Get dataset resource with column lineage included\n",
+    "\n",
+    "First we may list some example datasets:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 46,
+   "id": "f9abcf87",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "{\n",
+      "  \"namespace\": \"file\",\n",
+      "  \"name\": \"/home/jovyan/notebooks/spark-warehouse/dataset_a\"\n",
+      "}\n"
+     ]
+    }
+   ],
+   "source": [
+    "datasets = requests.get(\"{}/api/v1/namespaces/file/datasets\".format(marquez_url)).json()\n",
+    "print(json.dumps(datasets[\"datasets\"][0][\"id\"], indent=2))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "f4b64488",
+   "metadata": {},
+   "source": [
+    "Let's try now to fetch a specific dataset:\n",
+    " ** ****namespace****:</code>file<code>,\n",
+    " ** ****name****:</code>/home/jovyan/notebooks/spark-warehouse/dataset</em>c<code>\n",
+    " \n",
+    "We need to encode dataset name to be able to pass it through URL."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "id": "1e2c674a",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import urllib\n",
+    "encoded_name = urllib.parse.quote_plus(\"/home/jovyan/notebooks/spark-warehouse/dataset_c\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "61dbcf02",
+   "metadata": {},
+   "source": [
+    "</code>dataset<em>c<code>was created from a single column</code>col</em>3<code>in</code>dataset_b<code>, so its column lineage section should only contain a single field. "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 47,
+   "id": "486a7e2a",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "[\n",
+      "  {\n",
+      "    \"name\": \"col_4\",\n",
+      "    \"inputFields\": [\n",
+      "      {\n",
+      "        \"namespace\": \"file\",\n",
+      "        \"dataset\": \"/home/jovyan/notebooks/spark-warehouse/dataset_b\",\n",
+      "        \"field\": \"col_3\"\n",
+      "      }\n",
+      "    ],\n",
+      "    \"transformationDescription\": null,\n",
+      "    \"transformationType\": null\n",
+      "  }\n",
+      "]\n"
+     ]
+    }
+   ],
+   "source": [
+    "dataset = requests.get(\"{}/api/v1/namespaces/file/datasets/{}\".format(marquez_url, encoded_name)).json()\n",
+    "print(json.dumps(dataset[\"columnLineage\"], indent=2))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3f24b59d",
+   "metadata": {},
+   "source": [
+    "Fields</code>transformationDescription<code>and</code>transformationType` are available in the OpenLineage standard specification but not implemented in Spark integration (which is the only one).\n",
+    "\n",
+    "Colu…</p>
+                                
      
+                                
+                                
+    
+        
+    
+                                
+                                
+                                    
      
+                                        
+                                        <a href="https://github.com/OpenLineage/workshops">OpenLineage/workshops</a>
+                                    
      
+                                
+                            
+                        
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Paweł Leszczyński
+                     (pawel.leszczynski@getindata.com)
+                
      
+                
      2024-01-31 04:44:05
      
+                
      
+                    
      *Thread Reply:* the other parts of workshop refer to old version of connector, but please verify first if everything with your docker network is fine before running real Spark job
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Balachandru S
+                     (balachandru2468@gmail.com)
+                
      
+                
      2024-01-31 04:53:29
      
+                
      
+                    
      *Thread Reply:* Sure, thanks.
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Balachandru S
+                     (balachandru2468@gmail.com)
+                
      
+                
      2024-01-31 04:54:55
      
+                
      
+                    
      *Thread Reply:* From your code, I could see marquez-api is running successfully at "http://marquez-api:5000". Find attached screenshot.
      
+                    
+                    
      
+                            
      image.png
      
+                            
+                                
+    
+        
+            
+        
+    
+                            
+                        
      
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Balachandru S
+                     (balachandru2468@gmail.com)
+                
      
+                
      2024-01-31 04:57:08
      
+                
      
+                    
      *Thread Reply:* And I could create a lineage for spark in the Jupyter notebook successfully. But when I am submitting a job through spark-submit I am facing an issue "ERROR EventEmitter: Could not emit lineage w/ exception
+io.openlineage.client.OpenLineageClientException: java.net.UnknownHostException: marquez-api".
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Paweł Leszczyński
+                     (pawel.leszczynski@getindata.com)
+                
      
+                
      2024-01-31 04:57:44
      
+                
      
+                    
      *Thread Reply:* great. do you run spark-submit from within the same docker container?
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Paweł Leszczyński
+                     (pawel.leszczynski@getindata.com)
+                
      
+                
      2024-01-31 04:58:08
      
+                
      
+                    
      *Thread Reply:* you can open terminal tab within jupyter to run it
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Balachandru S
+                     (balachandru2468@gmail.com)
+                
      
+                
      2024-01-31 05:02:05
      
+                
      
+                    
      *Thread Reply:* Sure, thanks. Let me try running it.
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Balachandru S
+                     (balachandru2468@gmail.com)
+                
      
+                
      2024-01-31 07:52:22
      
+                
      
+                    
      *Thread Reply:* Thanks @Paweł Leszczyński, it ran successfully in the jupyter terminal.
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Mayur Singal
+                     (mayur.s@deuexsolutions.com)
+                
      
+                
      2024-01-30 05:50:58
      
+                
      
+                    
      Hi Team, I'm trying out OpenLineage with databricks and I'm not seeing expected results, need help to figure out what's the issue
      
+
+
      I'm following this quickstart guide: https://openlineage.io/docs/integrations/spark/quickstart_databricks/
      
+
+
      more details in the thread 🧵
      
+                    
      
+                            
      openlineage.io
      
+                            
+                            
+                                
+                                
      Quickstart with Databricks | OpenLineage
      
+                                
      
+                                    OpenLineage's Spark Integration can be installed on Databricks leveraging init scripts. Please note, Databricks on Google Cloud does not currently support the DBFS CLI, so the proposed solution will not work on Google Cloud until that feature is enabled.
+                                
      
+                                
+                                
+    
+        
+    
+                                
+                                    
      Original URL: https://openlineage.io/docs/integrations/spark/quickstart_databricks/
      
+                                
+                                
+                            
+                        
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Mayur Singal
+                     (mayur.s@deuexsolutions.com)
+                
      
+                
      2024-01-30 05:52:09
      
+                
      
+                    
      *Thread Reply:* I executed this piece of code as per the quickstart guide but the result generated on marquez is not as expected:
+spark.createDataFrame([
+    {'a': 1, 'b': 2},
+    {'a': 3, 'b': 4}
+]).write.mode("overwrite").saveAsTable("default.temp")
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Mayur Singal
+                     (mayur.s@deuexsolutions.com)
+                
      
+                
      2024-01-30 05:53:58
      
+                
      
+                    
      *Thread Reply:* the quickstart guide shows this example and it produces the result with a output node in the results, But when I run this in databricks I see no output node generated.
      
+                    
+                    
      
+                            
      image.png
      
+                            
+                                
+    
+        
+            
+        
+    
+                            
+                        
      
+                    
      
+                            
      image.png
      
+                            
+                                
+    
+        
+            
+        
+    
+                            
+                        
      
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Mayur Singal
+                     (mayur.s@deuexsolutions.com)
+                
      
+                
      2024-01-30 05:56:02
      
+                
      
+                    
      *Thread Reply:* same thing happens when I run this piece of code
      
+
+
      ```sourcetable = "mayurtable"
+destinationtable = "onkartable"
      
+
+
      Read the data from the source table into a DataFrame
      
+
+
      sourcedf = spark.sql(f"SELECT id FROM {sourcetable}")
      
+
+
      Show the source DataFrame
      
+
+
      source_df.show()
      
+
+
      Write the data from the source DataFrame to the destination table
      
+
+
      sourcedf.write.mode("overwrite").saveAsTable(destinationtable)``
+where I'm transferring data from one table to another and would expect a lineage betweenmayurtableandonkartable`  few events gets captured while running this piece of code but none of them has any output node
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Mayur Singal
+                     (mayur.s@deuexsolutions.com)
+                
      
+                
      2024-01-30 05:57:51
      
+                
      
+                    
      *Thread Reply:* as a result onkar_table  as a dataset was never recorded hence lineage between mayur_table  and onkar_table  was not recorded as well
      
+                    
+                    
      
+                            
      image.png
      
+                            
+                                
+    
+        
+            
+        
+    
+                            
+                        
      
+                    
      
+                            
      image.png
      
+                            
+                                
+    
+        
+            
+        
+    
+                            
                         
      
                     
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Mayur Singal
+                     (mayur.s@deuexsolutions.com)
+                
      
+                
      2024-01-30 05:58:27
      
+                
      
+                    
      *Thread Reply:* I'm using OpenLineage jar version 1.7.0 also tried this out with version 1.8.0 but had no luck
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Mayur Singal
+                     (mayur.s@deuexsolutions.com)
+                
      
+                
      2024-01-30 05:59:17
      
+                
      
+                    
      *Thread Reply:* any help would be appreciated, not sure if this could be a configuration issue
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Rodrigo Maia
+                     (rodrigo.maia@manta.io)
+                
      
+                
      2024-01-30 06:06:40
      
+                
      
+                    
      *Thread Reply:* are you using UC catalog?
      
+                    
+                    
+                    
+                        
      
+                        🙏 Mayur Singal
+                        
      
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Rodrigo Maia
+                     (rodrigo.maia@manta.io)
+                
      
+                
      2024-01-30 06:07:45
      
+                
      
+                    
      *Thread Reply:* ive reported an issue while ago about "saveAsTable(destination_table)" when using Unity
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Mayur Singal
+                     (mayur.s@deuexsolutions.com)
+                
      
+                
      2024-01-30 06:12:52
      
+                
      
+                    
      *Thread Reply:* Umm, yes UC is enabled on my instance but I'm performing this operation on default hive_metastore catalog
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Mayur Singal
+                     (mayur.s@deuexsolutions.com)
+                
      
+                
      2024-01-30 06:23:59
      
+                
      
+                    
      *Thread Reply:* just to add some more context, running this query via notebook also doesn't generate any lineage
      
+
+
      %sql
+insert into onkar_table select id from mayur_table;
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Maciej Obuchowski
+                     (maciej.obuchowski@getindata.com)
+                
      
+                
      2024-01-30 07:11:52
      
+                
      
+                    
      *Thread Reply:* any relevant logs @Mayur Singal? Might be regular issue with databricks where they replace OSS Spark class with their own implementation silently
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Mayur Singal
+                     (mayur.s@deuexsolutions.com)
+                
      
+                
      2024-01-30 07:30:42
      
+                
      
+                    
      *Thread Reply:* I had a similar thought, I checked the logs there is no relevant information over there don't see any errors as well.
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Mayur Singal
+                     (mayur.s@deuexsolutions.com)
+                
      
+                
      2024-02-01 05:10:34
      
+                
      
+                    
      *Thread Reply:* Hi Team, please do let me know if I need to open any ticket for this or could this be a configuration issue on my side!
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Maciej Obuchowski
+                     (maciej.obuchowski@getindata.com)
+                
      
+                
      2024-02-01 05:28:05
      
+                
      
+                    
      *Thread Reply:* Can you try to set log level to debug and running again?
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Maciej Obuchowski
+                     (maciej.obuchowski@getindata.com)
+                
      
+                
      2024-02-01 05:29:15
      
+                
      
+                    
      *Thread Reply:* Would be interesting what happens in code here https://github.com/OpenLineage/OpenLineage/blob/31f8ce588526e9c7c4bc7d849699cb7ce2[…]ent/lifecycle/plan/column/CreateReplaceInputDatasetBuilder.java
      
+                    
      
+                            
+                            
+                            
+                                
+                                
      <https://github.com/OpenLineage/OpenLineage/blob/31f8ce588526e9c7c4bc7d849699cb7ce2969c8f/integration/spark/spark34/src/main/java/io/openlineage/spark34/agent/lifecycle/plan/column/CreateReplaceInputDatasetBuilder.java | CreateReplaceInputDatasetBuilder.java>
      
+                                
      
+                                    <pre><code>public class CreateReplaceInputDatasetBuilder
+</code></pre>
+                                
      
+                                
+                                
+    
+        
+    
+                                
+                                
+                                    
      
+                                        
+                                        <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a>
+                                    
      
+                                
+                            
+                        
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Rodrigo Maia
+                     (rodrigo.maia@manta.io)
+                
      
+                
      2024-02-05 08:24:09
      
+                
      
+                    
      *Thread Reply:* I can try that and let you know
      
+                    
+                    
+                    
+                        
      
+                        👍 Maciej Obuchowski, Mayur Singal
+                        
      
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Balachandru S
+                     (balachandru2468@gmail.com)
+                
      
+                
      2024-01-30 08:14:51
      
+                
      
+                    
      Hi Team,
+I want to view column level lineage for pyspark code. So I have sample pyspark ETL code and I tried to run on the Open Lineage. In the Marquez I could see high level lineage diagram like attached screenshot. Is any dependency I need to add to get a column level lineage ?.
+Thanks.
      
+                    
+                    
      
+                            
      image.png
      
+                            
+                                
+    
+        
+            
+        
+    
+                            
+                        
      
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Maciej Obuchowski
+                     (maciej.obuchowski@getindata.com)
+                
      
+                
      2024-01-30 08:55:29
      
+                
      
+                    
      *Thread Reply:* > Is any dependency I need to add to get a column level lineage ?.
+No, you need to wait for Marquez release that contains this PR 🙂
      
+                    
      
+                            
+                            
+                            
+                                
+                                
      #2725 Web: Updates for Table and Column Lineage
      
+                                
      
+                                    <p><strong>Problem</strong></p>
+
+<p>We currently have a feature gap for column lineage where we don't have a display for it. In order to correctly display this we needed to have something to handle nested graphs so that columns can be nested under tables.</p>
+
+<p>This PR both creates a new page for column lineage and an updated view for lineage with a common set of shared principles.</p>
+
+<p><strong>Solution</strong>
+<strong>Feature Set</strong></p>
+
+<p>• Redesigned lineage page
+• New column lineage page
+• Refresh button (lineage + column lineage)
+• Compact mode for lineage display
+• Zooming and framing capabilities and zoom controls
+• Open view for graphs with a side panel instead of a draggable bar that limited graph navigation and visualization.
+• Shared action menu for graph context and actions
+• Tuning down the graph depth and preserving more in the url.</p>
+
+<p><strong>Technical</strong></p>
+
+<p>• New client side rendering engine with out of the box support for arbitrarily nested graphs (subgraphs) and hot swappable rendering strategies. There is also an easy to use and type safe framework to construct node types
+• New custom React based d3 manipulation framework for zooming, panning, transitions, framing and focal points that uses React paradigms instead of direct DOM access like a conventional d3 approach
+• A Minimap that allows for context awareness in a larger graph along with auto positioning extents</p>
+
+<p>Marquez.Lineage.mov</p>
+
+<p><a href="https://private-user-images.githubusercontent.com/7514204/293756591-0c5ce570-6422-4469-9a36-790573fe57ca.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MDY2MjMyMzEsIm5iZiI6MTcwNjYyMjkzMSwicGF0aCI6Ii83NTE0MjA0LzI5Mzc1NjU5MS0wYzVjZTU3MC02NDIyLTQ0NjktOWEzNi03OTA1NzNmZTU3Y2EucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDEzMCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDAxMzBUMTM1NTMxWiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9YjUwMGRjZjNjMGUwNjYzYTc2MjMwMTIwMmMzZGFjYzc1NjAwY2I1NWVjZThlY2UxODNlMDZmMmQ4MzA2ZmY5NiZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.NJ7ldOslCT9IjML4Hyo3SLn3Hw8FggtgBeazfCpLRRw">image</a><br />
+<a href="https://private-user-images.githubusercontent.com/7514204/293756603-a5b6d6e8-fde6-4ab0-b123-205e0e821fc5.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MDY2MjMyMzEsIm5iZiI6MTcwNjYyMjkzMSwicGF0aCI6Ii83NTE0MjA0LzI5Mzc1NjYwMy1hNWI2ZDZlOC1mZGU2LTRhYjAtYjEyMy0yMDVlMGU4MjFmYzUucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDEzMCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDAxMzBUMTM1NTMxWiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9OTU3NmU1MmM4Njk5MGM4MWZhNzQzOTY4YTRhNDllOWI2OWNhYzUzZjkzNzBjNGJmNjhmMDFhMmFjNmVmMzkwMCZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.Bm1Nw3lpGJp6E0pf_NK8WEKvjnU3uI7OCWPX-yPVCWY">image</a><br />
+<a href="https://private-user-images.githubusercontent.com/7514204/293756610-9bfacc0e-1279-49dd-ba8d-67c64663c9a5.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MDY2MjMyMzEsIm5iZiI6MTcwNjYyMjkzMSwicGF0aCI6Ii83NTE0MjA0LzI5Mzc1NjYxMC05YmZhY2MwZS0xMjc5LTQ5ZGQtYmE4ZC02N2M2NDY2M2M5YTUucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDEzMCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDAxMzBUMTM1NTMxWiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9Yjg5NDZkNWU4MzhjMDdiMGMyNGEzNzQzMjFiOTk0YzljZWJiMzBlM2UxMGE1OWI0MjA0MDEyYWZmZmY1YzEzZCZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.ac2K--QZObd0wLYiFjwIiW25WHd-ZQdZeAss0Fgmsl8">image</a></p>
+
+<p>One-line summary:<br />
+New views for both table and column lineage with shared new engine. Marks column lineage as <code>beta</code> for preview.</p>
+
+<p><strong>Checklist</strong></p>
+
+<p>☐ You've <a href="https://github.com/MarquezProject/marquez/blob/main/CONTRIBUTING.md#sign-your-work">signed-off</a> your work
+☐ Your changes are accompanied by tests (<em>if relevant</em>)
+☐ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained
+☐ You've updated any relevant documentation (<em>if relevant</em>)
+☐ You've included a one-line summary of your change for the <a href="https://github.com/MarquezProject/marquez/blob/main/CHANGELOG.md#unreleased"><code>CHANGELOG.md</code></a> (<em>Depending on the change, this may not be necessary</em>).
+☐ You've versioned your <code>.sql</code> database schema migration according to <a href="https://flywaydb.org/documentation/concepts/migrations#naming">Flyway's naming convention</a> (<em>if relevant</em>)
+☐ You've included a <a href="https://github.com/MarquezProject/marquez/blob/main/CONTRIBUTING.md#copyright--license">header</a> in any source code files (<em>if relevant</em>)</p>
+                                
      
+                                
+                                    
      
+                                        
      Labels
      
+                                        docs, web
+                                    
      
+                                
+                                    
      
+                                        
      Comments
      
+                                        9
+                                    
      
+                                
+                                
+    
+        
+    
+                                
+                                
+                                    
      
+                                        
+                                        <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a>
+                                    
      
+                                
+                            
+                        
      
+                    
+                    
+                    
+                        
      
+                        👍 Balachandru S
+                        
      
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      jayant joshi
+                     (itsjayantjoshi@gmail.com)
+                
      
+                
      2024-01-30 08:59:00
      
+                
      
+                    
      *Thread Reply:* I am also very much interested to see column level lineage using pyspark script in visual graph that located in the PR.
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      RaghavanAP
+                     (raghavan.panneerselvam@wavicledata.com)
+                
      
+                
      2024-01-30 09:03:06
      
+                
      
+                    
      *Thread Reply:* Is there any tentative date for the next release which includes Column level lineage using spark integration?
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Willy Lulciuc
+                     (willy@datakin.com)
+                
      
+                
      2024-01-30 13:42:37
      
+                
      
+                    
      *Thread Reply:* The Marquez team is working to having an RC by end of next month for users to try! https://marquezproject.slack.com/archives/C01E8MQGJP7/p1706636124360629?thread_ts=1706621358.017329&cid=C01E8MQGJP7
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Balachandru S
+                     (balachandru2468@gmail.com)
+                
      
+                
      2024-01-30 23:44:51
      
+                
      
+                    
      *Thread Reply:* Can we get visual representation of column level lineage using any other data consumers of Amundsen or Egeria or Apache atlas ?
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Michael Robinson
+                     (michael.robinson@astronomer.io)
+                
      
+                
      2024-01-30 12:00:06
      
+                
      
+                    
      @channel
+Our first London meetup, at Confluent's offices in London, is tomorrow! https://openlineage.slack.com/archives/C01CK9T7HKR/p1706193796047049
      
+                    
      
+                            
+                            
+                                
      
+                                    
+                                        
+                                    }
+                                    
+                                    Michael Robinson
+                                    (https://openlineage.slack.com/team/U02LXF3HUN7)
+                                
      
+                            
+                            
+                                
+                                
      
+                                
      
+                                    <p>@channel
+Our first London meetup is happening next Wednesday, Jan. 31, at Confluent's offices in Covent Garden. Click through to the Meetup page to sign up and view an up-to-date agenda, featuring talks by @Abdallah, Kirill Kulikov at Confluent, and @Paweł Leszczyński! <a href="https://openlineage.slack.com/archives/C01CK9T7HKR/p1704736501486239">https://openlineage.slack.com/archives/C01CK9T7HKR/p1704736501486239</a></p>
+                                
      
+                                
+                                
+    
+        
+    
+                                
+                                    
      Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1706193796047049
      
+                                
+                                
+                                    
      
+                                        
+                                        Slack Conversation
+                                    
      
+                                
+                            
+                        
      
+                    
+                    
+                    
+                        
      
+                        ❤️ Jarek Potiuk, Maciej Obuchowski, Mattia Bertorello, Willy Lulciuc, Abdallah, Harel Shein
+                        
      
+                    
+                        
      
+                        🙌 Jarek Potiuk, Maciej Obuchowski, Mattia Bertorello, Abdallah
+                        
      
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Athitya Kumar
+                     (athityakumar@gmail.com)
+                
      
+                
      2024-01-30 13:40:41
      
+                
      
+                    
      Hey team. Is column/attribute level lineage supported for input/topic Kafka topic ports in the OpenLineage Flink listener?
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Dheeraj
+                     (dheeraj.athrey@gmail.com)
+                
      
+                
      2024-01-31 01:28:10
      
+                
      
+                    
      Hello all
+I am using airflow and dbt. I want to know if there is any tool using open lineage that actually provides the following feature.
      
+
+
      If I pick a job J1, I want to be able to see the real time job start/run/finish status of all the jobs J1 is dependent on in the lineage.
      
+
+
      Any other tool/framework that would help me achieve this is also helpful.
      
+
+
      Thank you
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Jakub Dardziński
+                     (jakub.dardzinski@getindata.com)
+                
      
+                
      2024-01-31 02:13:41
      
+                
      
+                    
      *Thread Reply:* what is a job in this case? dbt run/Airflow DagRun?
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Dheeraj
+                     (dheeraj.athrey@gmail.com)
+                
      
+                
      2024-01-31 02:37:41
      
+                
      
+                    
      *Thread Reply:* By job, I meant an airflow dag.
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Jakub Dardziński
+                     (jakub.dardzinski@getindata.com)
+                
      
+                
      2024-01-31 02:38:42
      
+                
      
+                    
      *Thread Reply:* @tati might say more as project maintainer but you could take a look into https://github.com/astronomer/astronomer-cosmos
+it transforms dbt project into Airflow DAG and plays really nice with OL too
      
+                    
      
+                            
+                            
+                            
+                                
+                                
      astronomer/astronomer-cosmos
      
+                                
      
+                                    Run your dbt Core projects as Apache Airflow DAGs and Task Groups with a few lines of code
+                                
      
+                                
+                                    
      
+                                        
      Website
      
+                                        <https://astronomer.github.io/astronomer-cosmos/>
+                                    
      
+                                
+                                    
      
+                                        
      Stars
      
+                                        371
+                                    
      
+                                
+                                
+    
+        
+    
+                                
+                                
+                            
+                        
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      RaghavanAP
+                     (raghavan.panneerselvam@wavicledata.com)
+                
      
+                
      2024-01-31 02:14:31
      
+                
      
+                    
      Hi Team,
+Am running the pyspark script through spark-submit command and am facing the error as "Could not emit lineage". Can you please help on this error?
+spark-submit command: 
+spark-submit --conf "spark.extraListeners=io.openlineage.spark.agent.OpenLineageSparkListener" --packages "io.openlineage:openlineagespark:1.7.0" --conf "spark.openlineage.transport.type=http" --conf "spark.openlineage.transport.url=http://host.docker.internal:5000" --conf "spark.openlineage.namespace=sparkintegration" --conf "spark.pyspark.python=python" pyspark_etl.py
      
+
+
      Error Screenshot:
      
+                    
+                    
      
+                            
      ol -error .png
      
+                            
+                                
+    
+        
+            
+        
+    
+                            
+                        
      
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Mattia Bertorello
+                     (mattia.bertorello@booking.com)
+                
      
+                
      2024-01-31 08:31:22
      
+                
      
+                    
      *Thread Reply:* Hi,
      
+
+
      The host of Marquez is probably wrong  "spark.openlineage.transport.url=<http://host.docker.internal:5000>"  Check on your machine where Marquez is exposed.
+Thanks
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Balachandru S
+                     (balachandru2468@gmail.com)
+                
      
+                
      2024-01-31 08:17:46
      
+                
      
+                    
      Hi Team,
+I need to config AWS CLI on the Jupyter lab (which comes with open lineage docker container) to read and write an AWS S3 files through pyspark code. For this I am running this command "sudo ./aws/install". While running this command it's prompt me to password. I not sure what is the password here. Can you please help me on this ?. Here I just need sudo password for default jupyter lab (which comes with open lineage docker container).
+Thanks.
      
+                    
+                    
      
+                            
      image.png
      
+                            
+                                
+    
+        
+            
+        
+    
+                            
+                        
      
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Mattia Bertorello
+                     (mattia.bertorello@booking.com)
+                
      
+                
      2024-01-31 08:26:46
      
+                
      
+                    
      *Thread Reply:* Hi,
+Did you try to pass this environment variable  -e GRANT_SUDO=yes ?
+https://jupyter-docker-stacks.readthedocs.io/en/latest/using/common.html#permission-specific-configurations
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Balachandru S
+                     (balachandru2468@gmail.com)
+                
      
+                
      2024-01-31 08:33:26
      
+                
      
+                    
      *Thread Reply:* While composing up an open lineage docker-compose.yml. It showed the path to access jupyter lab, through the path I am accessing it. I didn't run any command externally. Find the attached screenshot.
      
+                    
+                    
      
+                            
      image.png
      
+                            
+                                
+    
+        
+            
+        
+    
+                            
+                        
      
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Mattia Bertorello
+                     (mattia.bertorello@booking.com)
+                
      
+                
      2024-01-31 08:36:41
      
+                
      
+                    
      *Thread Reply:* Mine was a suggestion to add this variable GRANT_SUDO=yes  in the docker compose if you need sudo access. And in general it's not a OpenLineage problem you should look into the documentation of the jupyter docker image
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Balachandru S
+                     (balachandru2468@gmail.com)
+                
      
+                
      2024-01-31 08:38:07
      
+                
      
+                    
      *Thread Reply:* Sure, will do that. Thanks,
      
+                    
+                    
+                    
+                        
      
+                        👍 Mattia Bertorello
+                        
      
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Balachandru S
+                     (balachandru2468@gmail.com)
+                
      
+                
      2024-02-02 04:18:20
      
+                
      
+                    
      *Thread Reply:* Hi @Mattia Bertorello, good afternoon
+I just tried to inspect the notebook container, there I could "GRANT_SUDO=yes". And after passing this also it's asking the password. Find the attached screenshot. Thanks.
      
+                    
+                    
      
+                            
      image.png
      
+                            
+                                
+    
+        
+            
+        
+    
+                            
+                        
      
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Michael Robinson
+                     (michael.robinson@astronomer.io)
+                
      
+                
      2024-01-31 10:46:34
      
+                
      
+                    
      @channel
+This is happening this evening! https://openlineage.slack.com/archives/C01CK9T7HKR/p1706193796047049
      
+                    
      
+                            
+                            
+                                
      
+                                    
+                                        
+                                    }
+                                    
+                                    Michael Robinson
+                                    (https://openlineage.slack.com/team/U02LXF3HUN7)
+                                
      
+                            
+                            
+                                
+                                
      
+                                
      
+                                    <p>@channel
+Our first London meetup is happening next Wednesday, Jan. 31, at Confluent's offices in Covent Garden. Click through to the Meetup page to sign up and view an up-to-date agenda, featuring talks by @Abdallah, Kirill Kulikov at Confluent, and @Paweł Leszczyński! <a href="https://openlineage.slack.com/archives/C01CK9T7HKR/p1704736501486239">https://openlineage.slack.com/archives/C01CK9T7HKR/p1704736501486239</a></p>
+                                
      
+                                
+                                
+    
+        
+    
+                                
+                                    
      Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1706193796047049
      
+                                
+                                
+                                    
      
+                                        
+                                        Slack Conversation
+                                    
      
+                                
+                            
+                        
      
+                    
+                    
+                    
+                        
      
+                        🎉 Harel Shein, Jakub Dardziński, Maciej Obuchowski, Ernie Ostic, harsh loomba
+                        
      
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Abhinav Ajith
+                     (abhinavajith0968@gmail.com)
+                
      
+                
      2024-02-01 01:37:31
      
+                
      
+                    
      Hi Team
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Abhinav Ajith
+                     (abhinavajith0968@gmail.com)
+                
      
+                
      2024-02-01 01:38:27
      
+                
      
+                    
      I am trying to run open lineage for sql query. Any Idea on how to use the openlineage-sql package
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Jakub Dardziński
+                     (jakub.dardzinski@getindata.com)
+                
      
+                
      2024-02-01 04:44:04
      
+                
      
+                    
      *Thread Reply:* Hey, could please tell more about your use case?
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Suhas Shenoy
+                     (ksuhasshenoy@gmail.com)
+                
      
+                
      2024-02-01 04:01:35
      
+                
      
+                    
      Hi everyone.  I am trying to integrate openlineage with apache airflow version > 2.7. I followed this guide: https://airflow.apache.org/docs/apache-airflow-providers-openlineage/stable/guides/user.html. This guide mentions that no changes are required in the dag file. I am using BashOperator in the dag file but openlineage does not seem to emit the events. What am I missing?
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Jakub Dardziński
+                     (jakub.dardzinski@getindata.com)
+                
      
+                
      2024-02-01 04:02:52
      
+                
      
+                    
      *Thread Reply:* BashOperator can only pass lineage if you set inlets or outlets manually
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Suhas Shenoy
+                     (ksuhasshenoy@gmail.com)
+                
      
+                
      2024-02-01 04:25:37
      
+                
      
+                    
      *Thread Reply:* Thanks for the reply Jakub. I created a dag with the code of example in this url: https://openlineage.io/docs/integrations/airflow/manual. But still no luck generating events.
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Suhas Shenoy
+                     (ksuhasshenoy@gmail.com)
+                
      
+                
      2024-02-01 04:35:22
      
+                
      
+                    
      *Thread Reply:* I am using airflow version 2.7.1. In my airflow.cfg file, there was no [openlineage] section and I had to add it. Are these two related in any way? @Jakub Dardziński
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Jakub Dardziński
+                     (jakub.dardzinski@getindata.com)
+                
      
+                
      2024-02-01 04:43:14
      
+                
      
+                    
      *Thread Reply:* I ran the bash example DAG you linked and I can confirm I’m getting the lineage
      
+
+
      Could you please confirm you’re sending events to Marquez at all? Can you see Airflow jobs created in Marquez but without inputs/outputs?
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Suhas Shenoy
+                     (ksuhasshenoy@gmail.com)
+                
      
+                
      2024-02-01 04:45:21
      
+                
      
+                    
      *Thread Reply:* I am not sending the events to Marquez.  I have an endpoint defined in the localhost where I am trying to receive the event
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Jakub Dardziński
+                     (jakub.dardzinski@getindata.com)
+                
      
+                
      2024-02-01 04:45:38
      
+                
      
+                    
      *Thread Reply:* I see, so are you receiving any events?
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Suhas Shenoy
+                     (ksuhasshenoy@gmail.com)
+                
      
+                
      2024-02-01 04:46:09
      
+                
      
+                    
      *Thread Reply:* No. I am not receiving anything in that endpoint.
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Jakub Dardziński
+                     (jakub.dardzinski@getindata.com)
+                
      
+                
      2024-02-01 04:46:54
      
+                
      
+                    
      *Thread Reply:* Would you want to share your [openlineage] section?
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Suhas Shenoy
+                     (ksuhasshenoy@gmail.com)
+                
      
+                
      2024-02-01 04:47:50
      
+                
      
+                    
      *Thread Reply:* Sure.
      
+
+
      [openlineage]
+transport = '{"type": "http", "url": "http://localhost:8082/event"}'
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Jakub Dardziński
+                     (jakub.dardzinski@getindata.com)
+                
      
+                
      2024-02-01 04:50:02
      
+                
      
+                    
      *Thread Reply:* and what endpoint is available in your local server? if it’s /event then you should set following:
      
+
+
      [openlineage]
+transport = ‘{“type”: “http”, “url”: “http://localhost:8082”, “endpoint”: “event”}’
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Suhas Shenoy
+                     (ksuhasshenoy@gmail.com)
+                
      
+                
      2024-02-01 05:15:12
      
+                
      
+                    
      *Thread Reply:* Yes the endpoint is /event and I made this change. Still no luck. Not only the http transport but console transport is also not working.
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Jakub Dardziński
+                     (jakub.dardzinski@getindata.com)
+                
      
+                
      2024-02-01 05:24:23
      
+                
      
+                    
      *Thread Reply:* Could you please check what’s in the Airflow UI under Admin &gt; Plugins?
      
+
+
      listeners should be there under OpenLineageProviderPlugin
      
+                    
+                    
      
+                            
      image.png
      
+                            
+                                
+    
+        
+            
+        
+    
+                            
+                        
      
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Maciej Obuchowski
+                     (maciej.obuchowski@getindata.com)
+                
      
+                
      2024-02-01 05:26:11
      
+                
      
+                    
      *Thread Reply:* @Suhas Shenoy if you're using Astro Runtime image locally then you additionally need to set env variable OPENLINEAGE_DISABLED=false due to bug in some versions of the runtime
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Suhas Shenoy
+                     (ksuhasshenoy@gmail.com)
+                
      
+                
      2024-02-01 05:31:24
      
+                
      
+                    
      *Thread Reply:* There is no listeners attribute in OpenLineageProviderPlugin. And I am not using Astro Runtime image. I installed airflow following this documentation: https://airflow.apache.org/docs/apache-airflow/2.7.1/installation/installing-from-pypi.html
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Suhas Shenoy
+                     (ksuhasshenoy@gmail.com)
+                
      
+                
      2024-02-01 05:46:54
      
+                
      
+                    
      *Thread Reply:* How do I make that listener attribute set to OpenLineageListener?
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Maciej Obuchowski
+                     (maciej.obuchowski@getindata.com)
+                
      
+                
      2024-02-01 05:51:48
      
+                
      
+                    
      *Thread Reply:* do you see OpenLineageProviderPlugin then?
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Maciej Obuchowski
+                     (maciej.obuchowski@getindata.com)
+                
      
+                
      2024-02-01 05:52:42
      
+                
      
+                    
      *Thread Reply:* can you try setting
+OPENLINEAGE_DISABLED=false and use OPENLINEAGE_URL=<http://localhost:8082> instead of setting it in config?
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Jakub Dardziński
+                     (jakub.dardzinski@getindata.com)
+                
      
+                
      2024-02-01 05:53:42
      
+                
      
+                    
      *Thread Reply:* I'd rather bet airflow.cfg is not read or it's in the wrong path
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Suhas Shenoy
+                     (ksuhasshenoy@gmail.com)
+                
      
+                
      2024-02-01 05:57:29
      
+                
      
+                    
      *Thread Reply:* This is the snapshot of my Plugins. I will also try with the configs which you mentioned.
      
+                    
+                    
      
+                            
      Screenshot 2024-02-01 at 4.24.41 PM.png
      
+                            
+                                
+    
+        
+            
+        
+    
+                            
+                        
      
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Maciej Obuchowski
+                     (maciej.obuchowski@getindata.com)
+                
      
+                
      2024-02-01 06:14:17
      
+                
      
+                    
      *Thread Reply:* ah, on Airflow 2.7 it does not show the listeners row yet
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Jakub Dardziński
+                     (jakub.dardzinski@getindata.com)
+                
      
+                
      2024-02-01 06:15:39
      
+                
      
+                    
      *Thread Reply:* if you do have access to Airflow’s console you could check what’s the output of below:
+```from airflow.configuration import conf
      
+
+
      print(conf.get('openlineage', 'transport'))```
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Suhas Shenoy
+                     (ksuhasshenoy@gmail.com)
+                
      
+                
      2024-02-01 06:26:52
      
+                
      
+                    
      *Thread Reply:* This was the output I got
+'{"type": "http", "url": "<http://localhost:8082>", "endpoint": "event"}'
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Suhas Shenoy
+                     (ksuhasshenoy@gmail.com)
+                
      
+                
      2024-02-01 09:12:34
      
+                
      
+                    
      *Thread Reply:* I am running Apache airflow in a virtual environment. Will it matter in any way?
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Maciej Obuchowski
+                     (maciej.obuchowski@getindata.com)
+                
      
+                
      2024-02-01 09:14:13
      
+                
      
+                    
      *Thread Reply:* have you tried OPENLINEAGE_DISABLED=false and OPENLINEAGE_URL=<http://localhost:8082> ?
+does virtual environment mean docker?
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Jakub Dardziński
+                     (jakub.dardzinski@getindata.com)
+                
      
+                
      2024-02-01 09:15:45
      
+                
      
+                    
      *Thread Reply:* eventually try checking what’s the output from:
+```from airflow.providers.openlineage.plugins.openlineage import isdisabled
      
+
+
      print(isdisabled())```
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Suhas Shenoy
+                     (ksuhasshenoy@gmail.com)
+                
      
+                
      2024-02-01 09:16:42
      
+                
      
+                    
      *Thread Reply:* Yes I tried both the things.
      
+
+
      No not docker. I have created virtual environment using
+python -m venv venv_name
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Suhas Shenoy
+                     (ksuhasshenoy@gmail.com)
+                
      
+                
      2024-02-01 09:18:09
      
+                
      
+                    
      *Thread Reply:* The output of the code block is false.
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Jakub Dardziński
+                     (jakub.dardzinski@getindata.com)
+                
      
+                
      2024-02-01 09:35:54
      
+                
      
+                    
      *Thread Reply:* that means you have listener enabled
      
+
+
      what I can think of is to set airflow logging level to debug and see what logs say
      
+                    
+                    
+                    
+                        
      
+                        ☝️ Maciej Obuchowski
+                        
      
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Suhas Shenoy
+                     (ksuhasshenoy@gmail.com)
+                
      
+                
      2024-02-01 23:21:36
      
+                
      
+                    
      *Thread Reply:* This is the log from scheduler terminal. I found 3 logs related to openlineage
      
+
+
      ```DEBUG - Loading EntryPoint(name='providerinfo', value='airflow.providers.openlineage.getproviderinfo:getproviderinfo', group='apacheairflow_provider') from package apache-airflow-providers-openlineage
      
+
+
      DEBUG - Importing entry_point plugin openlineage
      
+
+
      DEBUG - Creating module airflow.macros.OpenLineageProviderPlugin```
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Suhas Shenoy
+                     (ksuhasshenoy@gmail.com)
+                
      
+                
      2024-02-01 23:32:23
      
+                
      
+                    
      *Thread Reply:* And this is the log related to openlineage I found in the task1: https://pastebin.com/qbeYFTuJ
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Suhas Shenoy
+                     (ksuhasshenoy@gmail.com)
+                
      
+                
      2024-02-02 06:13:36
      
+                
      
+                    
      *Thread Reply:* Hi Team. Any reason why the extractor is failing to extract the metadata?
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Jakub Dardziński
+                     (jakub.dardzinski@getindata.com)
+                
      
+                
      2024-02-02 06:16:49
      
+                
      
+                    
      *Thread Reply:* Are you running the same DAG as in https://openlineage.io/docs/integrations/airflow/manual/?
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Suhas Shenoy
+                     (ksuhasshenoy@gmail.com)
+                
      
+                
      2024-02-02 06:18:18
      
+                
      
+                    
      *Thread Reply:* Yes.
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      priya narayana
+                     (n.priya88@gmail.com)
+                
      
+                
      2024-02-01 06:59:24
      
+                
      
+                    
      Hi Team!  When i run a job in Dataproc Bigquery job with openlineage 1.4.1 version I am seeing events which has anonymous tables and i cannot see proper lineage event in console with inputs and outputs . can you please tell me which version of openlineage version can help me with right table name in input and output complete event
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      priya narayana
+                     (n.priya88@gmail.com)
+                
      
+                
      2024-02-02 10:28:23
      
+                
      
+                    
      *Thread Reply:* Can someone guide me here
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      priya narayana
+                     (n.priya88@gmail.com)
+                
      
+                
      2024-02-02 10:28:38
      
+                
      
+                    
      *Thread Reply:* @Maciej Obuchowski @Paweł Leszczyński @Michael Robinson
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      ldacey
+                     (lance.dacey2@sutherlandglobal.com)
+                
      
+                
      2024-02-01 13:02:45
      
+                
      
+                    
      any thoughts on how sources from a REST API should be organized, or if they even should be set up as a dataset in OpenLineage? I am querying various endpoints and saving the results to jsonl files on GCS. those get read and ultimately loaded as parquet files in various delta tables (certain unnested columns become a completely separate table)
      
+
+
      https://something.something.com/api/v2/ticket_audits
      
+
+
      https://developer.zendesk.com/api-reference/ticketing/tickets/ticket_audits/
      
+                    
      
+                            
      developer.zendesk.com
      
+                            
+                            
+                                
+                                
      Ticket Audits
      
+                                
      
+                                    Developer documentation for products at Zendesk
+                                
      
+                                
+                                
+    
+        
+    
+                                
+                                    
      Original URL: https://developer.zendesk.com/api-reference/ticketing/tickets/ticket_audits/
      
+                                
+                                
+                            
+                        
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      ldacey
+                     (lance.dacey2@sutherlandglobal.com)
+                
      
+                
      2024-02-01 13:06:14
      
+                
      
+                    
      for the input dataset:
+namespace would be "https://something.something.com"
+resource would be "/api/v2/ticketaudits"
+not sure about the name, just "ticketaudits"?
      
+
+
      maybe the uri would include the entire URL with query parameters, or those parameters could be a separate custom facet.
      
+
+
      output dataset would be the GCS bucket and file?
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Michael Robinson
+                     (michael.robinson@astronomer.io)
+                
      
+                
      2024-02-01 15:42:01
      
+                
      
+                    
      @channel
+The latest issue of OpenLineage News is available now, featuring a rundown of the recent releases, updates to the Airflow Provider, events, proposals, and more.
+To get the newsletter directly in your inbox each month, sign up here.
      
+                    
      
+                            
      openlineage.us14.list-manage.com
      
+                            
+                            
+                                
+                                
      OpenLineage Project
      
+                                
      
+                                    OpenLineage Project Email Forms
+                                
      
+                                
+                                
+    
+        
+    
+                                
+                                    
      Original URL: http://bit.ly/OL_news
      
+                                
+                                
+                            
+                        
      
+                    
+                    
+                    
+                        
      
+                        🚀 Jakub Dardziński, Harel Shein, tati
+                        
      
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      RaghavanAP
+                     (raghavan.panneerselvam@wavicledata.com)
+                
      
+                
      2024-02-02 04:09:43
      
+                
      
+                    
      Hi Team,
+My objective is to configure AWS with open lineage in windows os , to read the file from s3. Can you please help me with required steps or share any documents or links for reference?
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Balachandru S
+                     (balachandru2468@gmail.com)
+                
      
+                
      2024-02-02 06:05:22
      
+                
      
+                    
      Hi Team,
+I need to execute sudo command in the jupyter notebook terminal (which comes with open lineage). Find the configuration details in the attached screenshot and I tried changing the password through the command like attached screenshot. After changing the password, I could use the created password for jupyter notebook login. But not for the jupyter terminal login. I could see usename is jovyan for both. Cany anyone help me to get a password for the jupyter terminal ?.
+Thanks.
      
+                    
+                    
      
+                            
      image.png
      
+                            
+                                
+    
+        
+            
+        
+    
+                            
+                        
      
+                    
      
+                            
      image.png
      
+                            
+                                
+    
+        
+            
+        
+    
+                            
+                        
      
+                    
      
+                            
      image.png
      
+                            
+                                
+    
+        
+            
+        
+    
+                            
+                        
      
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Michael Robinson
+                     (michael.robinson@astronomer.io)
+                
      
+                
      2024-02-02 15:13:10
      
+                
      
+                    
      @channel
+This month’s TSC meeting is next Thursday the 8th at 10am PT. On the tentative agenda:
+• announcements
+• recent releases
+• Coming soon: simplified job hierarchy in Spark @Maciej Obuchowski 
+• Flink integration updates @Maciej Obuchowski 
+• open discussion
+• more (TBA)
+More info and the meeting link can be found on the website. All are welcome! Do you have a discussion topic, use case or integration you’d like to demo? DM me to be added to the agenda.
      
+                    
      
+                            
      openlineage.io
      
+                            
+                            
+                                
+                                
      TSC Meetings | OpenLineage
      
+                                
      
+                                    The OpenLineage Technical Steering Committee meets monthly, and is open to all.
+                                
      
+                                
+                                
+    
+        
+    
+                                
+                                    
      Original URL: https://openlineage.io/meetings/
      
+                                
+                                
+                            
+                        
      
+                    
+                    
+                    
+                        
      
+                        👍 Maciej Obuchowski, Mattia Bertorello
+                        
      
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Peter Huang
+                     (huangzhenqiu0825@gmail.com)
+                
      
+                
      2024-02-08 12:48:48
      
+                
      
+                    
      *Thread Reply:* @Maciej Obuchowski I am not able to attend the meeting this time. Please help to update on our agreement on the flink community about the flink lineage listener APIs.
      
+                    
+                    
+                    
+                        
      
+                        👍 Maciej Obuchowski
+                        
      
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      priya narayana
+                     (n.priya88@gmail.com)
+                
      
+                
      2024-02-05 04:08:31
      
+                
      
+                    
      Hi Team , When i run jobs in Dataproc - Target host is not specified error is coming , when i set --properties 'dataproc:dataproc.lineage.enabled=true'. Can you tell what settings am i missing
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Jakub Dardziński
+                     (jakub.dardzinski@getindata.com)
+                
      
+                
      2024-02-05 04:42:48
      
+                
      
+                    
      *Thread Reply:* hey, we don't develop Dataproc integration directly, this seems like Google service issue and should be resolved by their support
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Rodrigo Maia
+                     (rodrigo.maia@manta.io)
+                
      
+                
      2024-02-05 10:56:52
      
+                
      
+                    
      HI! Is anybody here working with AWS Glue + OpenLineage (Spark)?
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Michael Robinson
+                     (michael.robinson@astronomer.io)
+                
      
+                
      2024-02-05 13:29:05
      
+                
      
+                    
      *Thread Reply:* There's been some progress on support for Glue recently: https://github.com/OpenLineage/OpenLineage/pull/2283
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Maciej Obuchowski
+                     (maciej.obuchowski@getindata.com)
+                
      
+                
      2024-02-06 07:32:07
      
+                
      
+                    
      *Thread Reply:* Glue Spark overall is working, but I think Glue Data Catalog is not
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Rodrigo Maia
+                     (rodrigo.maia@manta.io)
+                
      
+                
      2024-02-07 05:37:06
      
+                
      
+                    
      *Thread Reply:* I'll try to configure an instance with OL latest version to validate the results in terms of the jobs and the catalog. Thank you
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Suraj Gupta
+                     (suraj.gupta@atlan.com)
+                
      
+                
      2024-02-06 02:20:47
      
+                
      
+                    
      Do we have any list/documentation of all the connectors supported by Spark + OpenLineage?
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Ruchira Prasad
+                     (ruchiraprasad@gmail.com)
+                
      
+                
      2024-02-07 12:35:55
      
+                
      
+                    
      Hi Team,
+I tried to set up marquez on my Windows PC by using Docker Desktop by referring this document. https://openlineage.io/getting-started/
+Once set up, it's not start the "marquez-api" by giving an error as "org.postgresql.util.PSQLException: FATAL: password authentication failed for user "marquez""
+Do you have any idea how to fix this?
      
+                    
+                    
      
+                            
      image.png
      
+                            
+                                
+    
+        
+            
+        
+    
+                            
+                        
      
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Jakub Dardziński
+                     (jakub.dardzinski@getindata.com)
+                
      
+                
      2024-02-07 12:43:13
      
+                
      
+                    
      *Thread Reply:* could you please check marquez-db logs?
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Ruchira Prasad
+                     (ruchiraprasad@gmail.com)
+                
      
+                
      2024-02-07 12:48:09
      
+                
      
+                    
      *Thread Reply:* It has this
+FATAL:  password authentication failed for user "marquez"
+DETAIL:  Role "marquez" does not exist.
      
+                    
+                    
      
+                            
      image.png
      
+                            
+                                
+    
+        
+            
+        
+    
+                            
+                        
      
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Jakub Dardziński
+                     (jakub.dardzinski@getindata.com)
+                
      
+                
      2024-02-07 12:49:56
      
+                
      
+                    
      *Thread Reply:* no error related to init script?
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Ruchira Prasad
+                     (ruchiraprasad@gmail.com)
+                
      
+                
      2024-02-07 13:24:37
      
+                
      
+                    
      *Thread Reply:* Probably you might ask this.
      
+                    
+                    
      
+                            
      image.png
      
+                            
+                                
+    
+        
+            
+        
+    
+                            
+                        
      
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Jakub Dardziński
+                     (jakub.dardzinski@getindata.com)
+                
      
+                
      2024-02-07 13:26:53
      
+                
      
+                    
      *Thread Reply:* as you're on windows you may have wrong crlf setting in git. you can either fix the script manually or change git settings
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Ruchira Prasad
+                     (ruchiraprasad@gmail.com)
+                
      
+                
      2024-02-08 00:51:24
      
+                
      
+                    
      *Thread Reply:* I set git config --global core.autocrlf false. Then the line endings remain as LF.
+With this, the above error was gone. But it has an authentication error as below.
      
+                    
+                    
      
+                            
      image.png
      
+                            
+                                
+    
+        
+            
+        
+    
+                            
+                        
      
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Ruchira Prasad
+                     (ruchiraprasad@gmail.com)
+                
      
+                
      2024-02-08 10:59:25
      
+                
      
+                    
      *Thread Reply:* Isn't this some code issue in this repo @Jakub Dardziński
+https://github.com/MarquezProject/marquez
      
+                    
      
+                            
+                            
+                            
+                                
+                                
      MarquezProject/marquez
      
+                                
      
+                                    Collect, aggregate, and visualize a data ecosystem's metadata
+                                
      
+                                
+                                    
      
+                                        
      Website
      
+                                        <https://marquezproject.ai>
+                                    
      
+                                
+                                    
      
+                                        
      Stars
      
+                                        1551
+                                    
      
+                                
+                                
+    
+        
+    
+                                
+                                
+                            
+                        
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Jakub Dardziński
+                     (jakub.dardzinski@getindata.com)
+                
      
+                
      2024-02-08 11:18:40
      
+                
      
+                    
      *Thread Reply:* did you try docker compose down and up again?
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Ruchira Prasad
+                     (ruchiraprasad@gmail.com)
+                
      
+                
      2024-02-08 11:56:03
      
+                
      
+                    
      *Thread Reply:* Yes. didn't work.
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Michael Robinson
+                     (michael.robinson@astronomer.io)
+                
      
+                
      2024-02-07 13:47:08
      
+                
      
+                    
      @channel
+This month's TSC meeting, open to all, is tomorrow! Note the recent additions to the agenda 👀. https://openlineage.slack.com/archives/C01CK9T7HKR/p1706904790913259
      
+                    
      
+                            
+                            
+                                
      
+                                    
+                                        
+                                    }
+                                    
+                                    Michael Robinson
+                                    (https://openlineage.slack.com/team/U02LXF3HUN7)
+                                
      
+                            
+                            
+                                
+                                
      
+                                
      
+                                    <p><strong>@channel</strong>
+This month’s TSC meeting is next Thursday the 8th at 10am PT. On the tentative agenda:
+• announcements
+• recent releases
+• Coming soon: simplified job hierarchy in Spark @Maciej Obuchowski 
+• Flink integration updates @Maciej Obuchowski 
+• open discussion
+• more (TBA)
+More info and the meeting link can be found on the <a href="https://openlineage.io/meetings/">website</a>. All are welcome! Do you have a discussion topic, use case or integration you’d like to demo? DM me to be added to the agenda.</p>
+                                
      
+                                
+                                
+    
+        
+    
+                                
+                                    
      Original URL: https://openlineage.slack.com/archives/C01CK9T7HKR/p1706904790913259
      
+                                
+                                
+                                    
      
+                                        
+                                        Slack Conversation
+                                    
      
+                                
+                            
+                        
      
+                    
+                    
+                    
+                        
      
+                        👍 Ernie Ostic, Jakub Dardziński, Maciej Obuchowski, Mattia Bertorello
+                        
      
+                    
+                        
      
+                        ❤️ Harel Shein, alexandre bergere
+                        
      
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Sheeri Cabral (Collibra)
+                     (sheeri.cabral@collibra.com)
+                
      
+                
      2024-02-08 09:25:41
      
+                
      
+                    
      HIRING! Hello OpenLineage community members! Collibra is hiring for 2 Software Engineers (one regular, one senior) in Czech Republic, for our Lineage ETL team. These jobs are open because one engineer is leaving to hike the Pacific Coast Trail, and another is backfill (an engineer left a different lineage team and an engineer from ETL team is moving to that team).
      
+
+
      I am the Product Manager for this team, so you’d be working with me daily. Ask me anything (in a thread here or a DM). Creating an OpenLineage consumer is our next project starting later this month :D
      
+
+
      https://www.linkedin.com/feed/update/urn:li:activity:7161361969311084544/
      
+                    
      
+                            
      linkedin.com
      
+                            
+                            
+                                
+                                
      Sheeri Cabral :watermelon: on LinkedIn: Senior Software Engineer (Python), Lineage
      
+                                
      
+                                    Hello! If you&amp;#39;re a software engineer in Czech Republic and are looking for a job, we have 2 job openings for the product line I am a product manager for. 
+
+If…
+                                
      
+                                
+                                
+    
+        
+            
+        
+    
+                                
+                                    
      Original URL: https://www.linkedin.com/feed/update/urn:li:activity:7161361969311084544/
      
+                                
+                                
+                            
+                        
      
+                    
+                    
+                    
+                        
      
+                        🔥 Maciej Obuchowski, Kacper Muda, Michael Robinson, Harel Shein, Mattia Bertorello, Paweł Leszczyński
+                        
      
+                    
+                        
      
+                        :flag_cz: Maciej Obuchowski, Harel Shein, Paweł Leszczyński
+                        
      
+                    
+                        
      
+                        🥾 Mattia Bertorello
+                        
      
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Michael Robinson
+                     (michael.robinson@astronomer.io)
+                
      
+                
      2024-02-08 09:47:28
      
+                
      
+                    
      *Thread Reply:* Exciting news all around! Thanks for the update.
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Harel Shein
+                     (harel.shein@gmail.com)
+                
      
+                
      2024-02-08 09:53:30
      
+                
      
+                    
      *Thread Reply:* Amazing news @Sheeri Cabral (Collibra) 🎉 
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Harel Shein
+                     (harel.shein@gmail.com)
+                
      
+                
      2024-02-08 09:53:44
      
+                
      
+                    
      *Thread Reply:* Cross posting to #jobs 
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Sheeri Cabral (Collibra)
+                     (sheeri.cabral@collibra.com)
+                
      
+                
      2024-02-08 10:00:31
      
+                
      
+                    
      *Thread Reply:* Oh! I forgot about that channel!
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      ldacey
+                     (lance.dacey2@sutherlandglobal.com)
+                
      
+                
      2024-02-08 18:49:02
      
+                
      
+                    
      any general recommendations for naming buckets in GCS? i have a separate bucket per client (unrelated to each other).
      
+
+
      
+
+
      then i normally have folders like:
      
+
+
      sourcename/raw (csv/json data)
+sourcename/bronze (append only delta table)
+sourcename/silver (no duplicates)
+sourcename/gold (additional filters/aggregations)
      
+
+
      I kind of like having an OL dataset per raw file so I can see the schema and some metadata about what happened to that file in Marquez, but it makes my Namespace dataset view kind of messy. perhaps I should separate the raw data into a separate bucket (meaning a separate namespace)? I know that Dataplex zones are based on "raw" and "curated" so maybe that makes sense?
      
+
+
      it seems like having a separate bucket per source might be a bit much (, , ) since we have many hundreds of unique sources.
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Polly
+                    
+                
      
+                
      2024-02-09 11:30:45
      
+                
      
+                    
      @Harel Shein has a polly for you! Votes in this polly are anonymous 🔒.
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Sheeri Cabral (Collibra)
+                     (sheeri.cabral@collibra.com)
+                
      
+                
      2024-02-15 08:30:17
      
+                
      
+                    
      *Thread Reply:* FYI the EST times aren’t always correct, e.g. 1300 am. (I skipped over some 12 am EST times because I didn’t notice it really meant noon, 12 pm)
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Harel Shein
+                     (harel.shein@gmail.com)
+                
      
+                
      2024-02-16 13:37:31
      
+                
      
+                    
      *Thread Reply:* oh, oops. sorry about that 🤦‍♂️
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Harel Shein
+                     (harel.shein@gmail.com)
+                
      
+                
      2024-02-09 11:33:30
      
+                
      
+                    
      Hi all @here,
+We are considering changing the times for our monthly TSC meetings to allow more members of the community to participate.
+Please vote according to your preferred meeting days and times.
+Thank you!
      
+                    
+                    
+                    
+                        
      
+                        🔥 Paweł Leszczyński, Mateusz Kozioł, Michael Robinson, Sheeri Cabral (Collibra)
+                        
      
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Maciej Obuchowski
+                     (maciej.obuchowski@getindata.com)
+                
      
+                
      2024-02-16 13:30:35
      
+                
      
+                    
      *Thread Reply:* So, are we switching?
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Harel Shein
+                     (harel.shein@gmail.com)
+                
      
+                
      2024-02-16 13:40:03
      
+                
      
+                    
      *Thread Reply:* looks like Wednesday 9:30am PST / 12:30pm EST / 6:30pm CET / 11:00pm IST is the winner!
      
+                    
+                    
+                    
+                        
      
+                        👍 Maciej Obuchowski
+                        
      
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Wajdi Fathallah
+                     (wajdi@siffletdata.com)
+                
      
+                
      2024-02-13 08:14:30
      
+                
      
+                    
      Hi team,
+I really like the way you define URI in https://github.com/OpenLineage/OpenLineage/blob/main/spec/Naming.md and would like to know:
      
+
+
      1. any recommendations to extend this list and support other data sources? 
      2. do you plan to support assets other that Tables, Jobs ? for instance BI dashboard, AI models etc..? 
        3. 
+
      
+                    
      
+                            
+                            
+                            
+                                
+                                
      <https://github.com/OpenLineage/OpenLineage/blob/main/spec/Naming.md | Naming.md>
      
+                                
      
+                                    <p><strong>Naming</strong></p>
+
+<p>We define the unique name strategy per resource to ensure it is followed uniformly independently of who is producing<br />
+metadata, so we can connect lineage from various sources.</p>
+
+<p>Both Jobs and Datasets are in their own namespaces. Job namespaces are related to their schedulers. The namespace for a<br />
+dataset is the unique name for its datasource.</p>
+
+<p><strong>Datasets</strong></p>
+
+<p>The namespace and name of a datasource can be combined to form a URI (scheme:[//authority]path)</p>
+
+<p>• Namespace = scheme:[//authority] (the datasource)
+• Name = path (the datasets)</p>
+
+<p><strong>Data warehouses/data bases</strong></p>
+
+<p>Datasets are called tables. Tables are organized into databases and schemas.</p>
+
+<p><strong>Postgres:</strong></p>
+
+<p>Datasource hierarchy:</p>
+
+<p>• Host
+• Port</p>
+
+<p>Naming hierarchy:</p>
+
+<p>• Database
+• Schema
+• Table</p>
+
+<p>Identifier:</p>
+
+<p>• Namespace: postgres://{host}:{port} of the service instance.
+    • Scheme = postgres
+    • Authority = {host}:{port}
+• Unique name: {database}.{schema}.{table}
+    • URI = postgres://{host}:{port}/{database}.{schema}.{table}</p>
+
+<p><strong>MySQL:</strong></p>
+
+<p>Datasource hierarchy:</p>
+
+<p>• Host
+• Port</p>
+
+<p>Naming hierarchy:</p>
+
+<p>• Database
+• Table</p>
+
+<p>Identifier:</p>
+
+<p>• Namespace: mysql://{host}:{port} of the service instance.
+    • Scheme = mysql
+    • Authority = {host}:{port}
+• Unique name: {database}.{table}
+    • URI = mysql://{host}:{port}/{database}.{table}</p>
+
+<p><strong>Trino:</strong></p>
+
+<p>Datasource hierarchy:</p>
+
+<p>• Host
+• Port</p>
+
+<p>Naming hierarchy:</p>
+
+<p>• Catalog
+• Schema
+• Table</p>
+
+<p>Identifier:</p>
+
+<p>• Namespace: trino://{host}:{port} of the service instance.
+    • Scheme = trino
+    • Authority = {host}:{port}
+• Unique name: {catalog}.{schema}.{table}
+    • URI = trino://{host}:{port}/{catalog}.{schema}.{table}</p>
+
+<p><strong>Redshift:</strong></p>
+
+<p>Datasource hierarchy:</p>
+
+<p>• Host: examplecluster.<XXXXXXXXXXXX>.<a href="http://us-west-2.redshift.amazonaws.com">us-west-2.redshift.amazonaws.com</a>
+• Port: 5439</p>
+
+<p>OR</p>
+
+<p>• Cluster identifier
+• Region name
+• Port (defaults to 5439)</p>
+
+<p>Naming hierarchy:</p>
+
+<p>• Database
+• Schema
+• Table</p>
+
+<p>One can interact with Redshift using SQL or Data API. The combination of cluster identifier and region name is the only<br />
+common unique ID available to both.</p>
+
+<p>Identifier:</p>
+
+<p>• Namespace: redshift://{cluster<em>identifier}.{region</em>name}:{port} of the cluster instance.
+    • Scheme = redshift
+    • Authority = {cluster<em>identifier}.{region</em>name}:{port}
+• Unique name: {database}.{schema}.{table}
+    • URI = redshift://{cluster<em>identifier}.{region</em>name}:{port}/{database}.{schema}.{table}</p>
+
+<p><strong>Athena:</strong></p>
+
+<p>Datasource hierarchy:</p>
+
+<p>• Host: athena.{region_name}.<a href="http://amazonaws.com">amazonaws.com</a></p>
+
+<p>Naming hierarchy:</p>
+
+<p>• Catalog
+• Database
+• Table</p>
+
+<p>Identifier:</p>
+
+<p>• Namespace: <awsathena://athena>.{region<em>name}.<a href="http://amazonaws.com">amazonaws.com</a> of the service instance.
+    • Scheme = awsathena
+    • Authority = athena.{region</em>name}.<a href="http://amazonaws.com">amazonaws.com</a>
+• Unique name: {catalog}.{database}.{table}
+    • URI = <awsathena://athena>.{region_name}.<a href="http://amazonaws.com/{catalog}.{database}.{table}|amazonaws.com/{catalog}.{database}.{table}">http://amazonaws.com/{catalog}.{database}.{table}|amazonaws.com/{catalog}.{database}.{table}</a></p>
+
+<p><strong>Snowflake</strong></p>
+
+<p>See:</p>
+
+<p>• <a href="https://docs.snowflake.com/en/user-guide/admin-account-identifier|Account Identifiers | Snowflake Documentation">https://docs.snowflake.com/en/user-guide/admin-account-identifier|Account Identifiers | Snowflake Documentation</a>
+• <a href="https://docs.snowflake.com/en/sql-reference/identifiers.html|Object Identifiers | Snowflake Documentation">https://docs.snowflake.com/en/sql-reference/identifiers.html|Object Identifiers | Snowflake Documentation</a></p>
+
+<p>Datasource hierarchy:</p>
+
+<p>• account identifier (composite of organization name and account name)</p>
+
+<p>Naming hierarchy:</p>
+
+<p>• Database: {database name} =&gt; unique across the account
+• Schema: {schema name} =&gt; unique within the database
+• Table: {table name} =&gt; unique within the schema</p>
+
+<p>Identifier:</p>
+
+<p>• Namespace: snowflake://{organization name}-{account name}
+    • Scheme = snowflake
+    • Authority = {organization name}-{account name}
+• Name: {database}.{schema}.{table}
+    • URI = snowflake://{organization name}-{account name}/{database}.{schema}.{table}</p>
+
+<p>Snowflake resolves and stores names for databases, schemas, tables and columns differently depending on how they are<br />
+<a href="https://docs.snowflake.com/en/sql-reference/identifiers-syntax">expressed in statements</a> (e.g. unquoted vs quoted). The<br />
+representation of names in OpenLineage events should be based on the canonical name that Snowflake stores. Specifically:</p>
+
+<p>• For dataset names, each period-delimited part (database/schema/table) should be in the simplest form it would take in<br />
+    a statement i.e. quoted only if necessary. For example, a table <code>My Table</code> in schema <code>MY_SCHEMA</code> and in database<br />
+    <code>MY_DATABASE</code> would be represented as <code>MY_DATABASE.MY_SCHEMA."My Table"</code>. If in doubt, check<br />
+    <a href="https://docs.snowflake.com/en/sql-reference/account-usage/access_history">Snowflake's <code>ACCESS_HISTORY</code> view</a> to see<br />
+    how <code>objectName</code> is formed for a given table.
+• For column names, the canonical name should always be used verbatim.</p>
+
+<p><strong>BigQuery</strong></p>
+
+<p>See:<br />
+<a href="https://cloud.google.com/resource-manager/docs/creating-managing-projects|Creating and managing projects | Resource Manager Documentation">https://cloud.google.com/resource-manager/docs/creating-managing-projects|Creating and managing projects | Resource Manager Documentation</a><br />
+<a href="https://cloud.google.com/bigquery/docs/datasets-intro|Introduction to datasets | BigQuery">https://cloud.google.com/bigquery/docs/datasets-intro|Introduction to datasets | BigQuery</a><br />
+<a href="https://cloud.google.com/bigquery/docs/tables-intro|Introduction to tables | BigQuery">https://cloud.google.com/bigquery/docs/tables-intro|Introduction to tables | BigQuery</a></p>
+
+<p>Datasource hierarchy:</p>
+
+<p>• bigquery</p>
+
+<p>Naming hierarchy:</p>
+
+<p>• Project Name: {project name} =&gt; is not unique
+• Project number: {project number} =&gt; numeric: is unique across Google cloud
+• Project ID: {project id} =&gt; readable: is unique across Google cloud
+• dataset: {dataset name} =&gt; is unique within a project
+• table: {table name} =&gt; is unique within a dataset</p>
+
+<p>Identifier:</p>
+
+<p>• Namespace: bigquery
+    • Scheme = bigquery
+    • Authority =
+• Unique name: {project id}.{dataset name}.{table name}
+    • URI = bigquery:{project id}.{dataset name}.{table name}</p>
+
+<p><strong>Azure Synapse:</strong></p>
+
+<p>Datasource hierarchy:</p>
+
+<p>• Host: <XXXXXXXXXXXX>.<a href="http://sql.azuresynapse.net">sql.azuresynapse.net</a>
+• Port: 1433
+• Database: SQLPool1</p>
+
+<p>Naming hierarchy:</p>
+
+<p>• Schema
+• Table</p>
+
+<p>Identifier:</p>
+
+<p>• Namespace: sqlserver://{host}:{port};database={database};
+    • Scheme = sqlserver
+    • Authority = {host}:{port}
+• Unique name: {schema}.{table}
+    • URI = sqlserver://{host}:{port};database={database}/{schema}.{table}</p>
+
+<p><strong>Azure Cosmos DB:</strong></p>
+
+<p>Datasource hierarchy: <azurecosmos://%s.documents.azure.com/dbs/%s></p>
+
+<p>• Host: <XXXXXXXXXXXX>.<a href="http://documents.azure.com">documents.azure.com</a>
+• Database</p>
+
+<p>Naming hierarchy:</p>
+
+<p>• Schema
+• Table</p>
+
+<p>Identifier:</p>
+
+<p>• Namespace: azurecosmos://{host}/dbs/{database}
+    • Scheme = azurecosmos
+    • Authority = {host}
+• Unique name: /colls/{table}
+    • URI = azurecosmos://{host}.<a href="http://documents.azure.com/dbs/{database}/colls/{table}|documents.azure.com/dbs/{database}/colls/{table}">http://documents.azure.com/dbs/{database}/colls/{table}|documents.azure.com/dbs/{database}/colls/{table}</a></p>
+
+<p><strong>Azure Data Explorer:</strong></p>
+
+<p>Datasource hierarchy:</p>
+
+<p>• Host: <clustername>.<clusterlocation>
+• Database
+• Table</p>
+
+<p>Naming hierarchy:</p>
+
+<p>• Database
+• Table</p>
+
+<p>Identifier:</p>
+
+<p>• Namespace: azurekusto://{host}.<a href="http://kusto.windows.net/{database}|kusto.windows.net/{database}">http://kusto.windows.net/{database}|kusto.windows.net/{database}</a>
+    • Scheme = azurekusto
+• Unique name: {database}/{table}
+    • URI = azurekusto://{host}.<a href="http://kusto.windows.net/{database}/{table}|kusto.windows.net/{database}/{table}">http://kusto.windows.net/{database}/{table}|kusto.windows.net/{database}/{table}</a></p>
+
+<p><strong>Distributed file systems/blob stores</strong>
+<strong>GCS</strong></p>
+
+<p>Datasource hierarchy: none, naming is global</p>
+
+<p>Naming hierarchy:</p>
+
+<p>• bucket name =&gt; globally unique
+• Path</p>
+
+<p>Identifier :</p>
+
+<p>• Namespace: gs://{bucket name}
+    • Scheme = gs
+    • Authority = {bucket name}
+• Unique name: {path}
+    • URI = gs://{bucket name}{path}</p>
+
+<p><strong>S3</strong></p>
+
+<p>Naming hierarchy:</p>
+
+<p>• bucket name =&gt; globally unique
+• Path</p>
+
+<p>Identifier :</p>
+
+<p>• Namespace: s3://{bucket name}
+    • Scheme = s3
+    • Authority = {bucket name}
+• Unique name: {path}
+    • URI = s3://{bucket name}{path}</p>
+
+<p><strong>HDFS</strong></p>
+
+<p>Naming hierarchy:</p>
+
+<p>• Namenode: host + port
+• Path</p>
+
+<p>Identifier :</p>
+
+<p>• Namespace: hdfs://{namenode host}:{namenode port}
+    • Scheme = hdfs
+    • Authority = {namenode host}:{namenode port}
+• Unique name: {path}
+    • URI = hdfs://{namenode host}:{namenode port}{path}</p>
+
+<p><strong>DBFS (Databricks File System)</strong></p>
+
+<p>Naming hierarchy:</p>
+
+<p>• workspace name: globally unique
+• Path</p>
+
+<p>Identifier :</p>
+
+<p>• Namespace: hdfs://{workspace name}
+    • Scheme = hdfs
+    • Authority = workspace name
+• Unique name: {path}
+    • URI = hdfs://{workspace nam…</p>
+                                
      
+                                
+                                
+    
+        
+    
+                                
+                                
+                                    
      
+                                        
+                                        <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a>
+                                    
      
+                                
+                            
+                        
      
+                    
+                    
+                    
+                        
      
+                        👍 Athitya Kumar, alexandre bergere
+                        
      
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Julien Le Dem
+                     (julien@apache.org)
+                
      
+                
      2024-02-13 19:51:22
      
+                
      
+                    
      *Thread Reply:* Hello Wajdi, nice to see you!
      
+
+
      1. If you wan to add a missing datasource to that document, you should open an issue to propose it. The main requirement is they need to be unique and canonical. Once we agree on the format you can open a PR and add it to the spec.
      2. So far the idea, is to model AI model and dashboards as datasets. an AI model is an asset with metadata that is produced by a job. It will have specific facets. Let us know what you think. You are welcome to push that discussion further if you want to add those to the spec. 
        3. 
+
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Wajdi Fathallah
+                     (wajdi@siffletdata.com)
+                
      
+                
      2024-02-14 09:29:00
      
+                
      
+                    
      *Thread Reply:* Thanks for your answer @Julien Le Dem
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Balachandru S
+                     (balachandru2468@gmail.com)
+                
      
+                
      2024-02-14 01:06:05
      
+                
      
+                    
      Hi Team,
+We are getting the column-level lineage for pyspark code with the help of OpenLineage.
+We are interested to know the limitation here, is there any limitation in creating column-level lineage for pyspark code like lines of code, number of input data sources, number of output data sources or pyspark operations (filter, join & extra) in the OpenLineage ?.
+We have gone through the OpenLineage documentation, from the documentation we could only get supported spark versions and data source types alone. Thanks.
      
+                    
+                    
      
+                            
      image.png
      
+                            
+                                
+    
+        
+            
+        
+    
+                            
+                        
      
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Maciej Obuchowski
+                     (maciej.obuchowski@getindata.com)
+                
      
+                
      2024-02-14 05:48:43
      
+                
      
+                    
      *Thread Reply:* Spark 2.4 does not support Column-level lineage.
      
+
+
      > number of output data sources
+In Spark model, each output creates additional action that goes through general Spark compilation and execution model - we follow that, so you'll end up with multiple events.
      
+
+
      One thing we're not supporting for CLL (...maybe yet?) is using mapping to Scala object using DeserializeToObject logical plan, as we're breaking CLL chain there.
      
+
+
      > We are interested to know the limitation here
+If anything, it depends on the type of data sources. Some LogicalPlans (like streaming ones) are not supported, as well as external ones. We're working on API for external connectors to implement, so we can have stable support for them.
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Balachandru S
+                     (balachandru2468@gmail.com)
+                
      
+                
      2024-02-14 08:31:42
      
+                
      
+                    
      *Thread Reply:* Thanks @Maciej Obuchowski.
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Sheeri Cabral (Collibra)
+                     (sheeri.cabral@collibra.com)
+                
      
+                
      2024-02-15 08:31:10
      
+                
      
+                    
      *Thread Reply:* follow-up - what versions of spark - if any - create column-level lineage?
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Maciej Obuchowski
+                     (maciej.obuchowski@getindata.com)
+                
      
+                
      2024-02-15 09:15:02
      
+                
      
+                    
      *Thread Reply:* 3.0+ 🙂 
+For now, it can depend on the connector though - Paweł is working on a better way to do it. https://github.com/OpenLineage/OpenLineage/pull/2272
      
+                    
      
+                            
+                            
+                            
+                                
+                                
      #2272 [SPARK] lineage metadata extraction built-in within Spark extensions
      
+                                
      
+                                    <p><strong>Problem</strong></p>
+
+<p>Support for lineage extraction built-in within Spark extensions.</p>
+
+<p>This PR adds <code>spark-interfaces-scala</code> package which allows lineage extraction to be implemented within Spark extensions (like iceberg, delta, gcs, etc). Openlineage integration, when traversing query plan, verifies if nodes implement defined interfaces. If so, interface methods are used to extract lineage.</p>
+
+<p>Please refer to <code>README.md</code> within the PR for more details.</p>
+
+<p><strong>Solution</strong></p>
+
+<p>PR contains extra artifact with interfaces as well as Spark integration internals</p>
+
+<blockquote>
+  <p><strong>Note:</strong> All schema changes require discussion. Please <a href="https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue">link the issue</a> for context.</p>
+</blockquote>
+
+<p>☐ Your change modifies the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json">core</a> OpenLineage model
+☐ Your change modifies one or more OpenLineage <a href="https://github.com/OpenLineage/OpenLineage/tree/main/spec/facets">facets</a></p>
+
+<p>If you're contributing a new integration, please specify the scope of the integration and how/where it has been tested (e.g., Apache Spark integration supports <code>S3</code> and <code>GCS</code> filesystem operations, tested with AWS EMR).</p>
+
+<p><strong>One-line summary:</strong>
+<strong>Checklist</strong></p>
+
+<p>☑︎ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work
+☑︎ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a>
+☑︎ Your changes are accompanied by tests (<em>if relevant</em>)
+☐ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained
+☐ You've updated any relevant documentation (<em>if relevant</em>)
+☐ Your comment includes a one-liner for the changelog about the specific purpose of the change (<em>if necessary</em>)
+☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>)
+☐ You've added a <a href="https://github.com/OpenLineage/OpenLineage/tree/main/.github/header_templates.md">header</a> to source files (<em>if relevant</em>)</p>
+
+<hr />
+
+<p>SPDX-License-Identifier: Apache-2.0<br />
+Copyright 2018-2023 contributors to the OpenLineage project</p>
+                                
      
+                                
+                                    
      
+                                        
      Labels
      
+                                        documentation, integration/spark, ci, integration/flink, tests, streaming
+                                    
      
+                                
+                                    
      
+                                        
      Comments
      
+                                        3
+                                    
      
+                                
+                                
+    
+        
+    
+                                
+                                
+                                    
      
+                                        
+                                        <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a>
+                                    
      
+                                
+                            
+                        
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Michael Robinson
+                     (michael.robinson@astronomer.io)
+                
      
+                
      2024-02-14 09:26:47
      
+                
      
+                    
      @channel
+The Marquez Project, the reference implementation for OpenLineage, has released an RC featuring column lineage support among other significant web UI improvements. For those unfamiliar with our sister project, Marquez is a highly scalable backend, API and UI for metadata management that implements the OpenLineage spec. In addition to being a powerful, platform-agnostic solution for metadata management, it's always been a great way to get started with OpenLineage and explore what the capabilities are. Now it's an even better reflection of those capabilities and, in particular, the strengths of the Spark integration and Airflow Provider. Whether you're already a seasoned Marquez user or you're new to the project, please consider trying out the RC. It would be very helpful to know what your experience is like and about any issues you run into. Thanks!
      
+
+
      Resources
+Release: https://github.com/MarquezProject/marquez/releases/tag/0.45.0-rc.1
+Changelog: https://github.com/MarquezProject/marquez/blob/0.45.0-rc.1/CHANGELOG.md
+Commit history: https://github.com/MarquezProject/marquez/compare/0.43.1...0.45.0-rc.1
+Maven: https://oss.sonatype.org/#nexus-search;quick~marquez
+PyPI: https://pypi.org/project/marquez-python/
+Docker: https://hub.docker.com/u/marquezproject
      
+                    
+                    
+                    
+                        
      
+                        🚀 Mattia Bertorello, Jakub Dardziński, Andy Alseth, Maciej Obuchowski, Willy Lulciuc, Minkyu Park, Sheeri Cabral (Collibra)
+                        
      
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Michael Robinson
+                     (michael.robinson@astronomer.io)
+                
      
+                
      2024-02-14 11:07:48
      
+                
      
+                    
      *Thread Reply:* 
      
+                    
+                    
      
+                            
      Screenshot 2024-02-14 at 11.02.31.png
      
+                            
+                                
+    
+        
+            
+        
+    
+                            
+                        
      
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Michael Robinson
+                     (michael.robinson@astronomer.io)
+                
      
+                
      2024-02-14 11:08:02
      
+                
      
+                    
      *Thread Reply:* 
      
+                    
+                    
      
+                            
      Screenshot 2024-02-14 at 11.06.27.png
      
+                            
+                                
+    
+        
+            
+        
+    
+                            
+                        
      
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Michael Robinson
+                     (michael.robinson@astronomer.io)
+                
      
+                
      2024-02-14 11:08:16
      
+                
      
+                    
      *Thread Reply:* 
      
+                    
+                    
      
+                            
      Screenshot 2024-02-14 at 11.06.55.png
      
+                            
+                                
+    
+        
+            
+        
+    
+                            
+                        
      
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Willy Lulciuc
+                     (willy@datakin.com)
+                
      
+                
      2024-02-14 13:38:31
      
+                
      
+                    
      *Thread Reply:* For those interested in learning more about Marquez, I hold office hours every Tue 9:30AM PST! You can join here 👉 https://astronomer.zoom.us/j/84548968341?pwd=Asb8rpLuhbSalGP9i4BYHd1UXfPQe1.1
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      ldacey
+                     (lance.dacey2@sutherlandglobal.com)
+                
      
+                
      2024-02-15 10:11:11
      
+                
      
+                    
      *Thread Reply:* looks sharp. it seems to have fixed some issues I might have had in 0.44 which did not show all connections
      
+
+
      I did an airflow backfill job which redownloaded all files from a SFTP (191 files) and each of those are a separate OL dataset. in this view I clicked on a single file, but because it is connected to the "extract" airflow task, it shows all of the files that task downloaded as well (dynamic mapped tasks in Airflow)
      
+                    
+                    
      
+                            
      image.png
      
+                            
+                                
+    
+        
+            
+        
+    
+                            
+                        
      
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      ldacey
+                     (lance.dacey2@sutherlandglobal.com)
+                
      
+                
      2024-02-15 10:16:07
      
+                
      
+                    
      *Thread Reply:* but day to day there is normally just one file downloaded so that should be a cleaner view in the future I assume? that input dataset (SFTP file) should only refer to the output dataset on GCS
      
+
+
      [sftp_file] --> [GCS file in landing folder] --> [GCS file in raw folder (renamed), only added if the checksum doesn't exist] -> [bronze delta table on GCS] --> [silver delta tableon GCS] --> [gold delta table on ADLS gen2]
      
+
+
      I am just considering whether I should proceed with file level datasets if it will make the UI too messy and complex. we do not control changes in the schema from clients, so on one hand it is nice to track
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Zacay Daushin
+                     (zacayd@octopai.com)
+                
      
+                
      2024-02-14 09:32:43
      
+                
      
+                    
      is this also parses python code for column level?
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Jakub Dardziński
+                     (jakub.dardzinski@getindata.com)
+                
      
+                
      2024-02-14 09:34:22
      
+                
      
+                    
      *Thread Reply:* Marquez is a reference backend for OpenLineage. What Michael highlighted is its ability to visualize column lineage.
      
+
+
      Parsing Python code is, I think, undoable in sustainable way. What we’re aiming at is to use hook-level lineage which is work in progress. This work happens in Airflow OpenLineage provider.
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Sheeri Cabral (Collibra)
+                     (sheeri.cabral@collibra.com)
+                
      
+                
      2024-02-15 08:32:26
      
+                
      
+                    
      *Thread Reply:* (and just to clarify - does the airflow openlineage provider generate column-level lineage info?)
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Jakub Dardziński
+                     (jakub.dardzinski@getindata.com)
+                
      
+                
      2024-02-15 08:47:24
      
+                
      
+                    
      *Thread Reply:* it does with exceptions
+we even have that info now in the main repo page: https://github.com/OpenLineage/OpenLineage?tab=readme-ov-file#integration-matrix 🙂
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Minkyu Park
+                     (minkyu.park.200@gmail.com)
+                
      
+                
      2024-02-14 18:51:09
      
+                
      
+                    
      Hi all, greetings again from my new account with personal email. It's been a while to see you all 😄
      
+                    
+                    
+                    
+                        
      
+                        👋 Jakub Dardziński, Paweł Leszczyński, Maciej Obuchowski
+                        
      
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Harel Shein
+                     (harel.shein@gmail.com)
+                
      
+                
      2024-02-14 19:48:48
      
+                
      
+                    
      *Thread Reply:* Good to see you here @Minkyu Park ❤️ 
      
+                    
+                    
+                    
+                        
      
+                        ❤️ Minkyu Park
+                        
      
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Michael Robinson
+                     (michael.robinson@astronomer.io)
+                
      
+                
      2024-02-14 19:50:45
      
+                
      
+                    
      *Thread Reply:* welcome back, @Minkyu Park!
      
+                    
+                    
+                    
+                        
      
+                        ❤️ Minkyu Park
+                        
      
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Maciej Obuchowski
+                     (maciej.obuchowski@getindata.com)
+                
      
+                
      2024-02-15 05:01:24
      
+                
      
+                    
      *Thread Reply:* Hello @Minkyu Park 🙂
      
+                    
+                    
+                    
+                        
      
+                        ❤️ Minkyu Park
+                        
      
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Sheeri Cabral (Collibra)
+                     (sheeri.cabral@collibra.com)
+                
      
+                
      2024-02-15 08:33:21
      
+                
      
+                    
      *Thread Reply:* Welcome back Minkyu!
      
+
+
      (Does open source need some presentations on ‘how best to contribute to open source’ including ‘use personal emails/github whenever possible because your job may not last forever’?)
      
+                    
+                    
+                    
+                        
      
+                        🤣 Minkyu Park
+                        
      
+                    
+                        
      
+                        👍 Minkyu Park
+                        
      
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Minkyu Park
+                     (minkyu.park.200@gmail.com)
+                
      
+                
      2024-02-15 18:30:46
      
+                
      
+                    
      Are extractors in OL airflow integration still available from airflow 2.7+ OL providers, or should it be implemented separately in Airflow?
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Jakub Dardziński
+                     (jakub.dardzinski@getindata.com)
+                
      
+                
      2024-02-15 18:32:04
      
+                
      
+                    
      *Thread Reply:* They’re still available, you can register them with [openlineage] extractors or (AIRFLOW__OPENLINEAGE__EXTRACTORS  env var) - see: https://airflow.apache.org/docs/apache-airflow-providers-openlineage/stable/configurations-ref.html#extractors
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Jakub Dardziński
+                     (jakub.dardzinski@getindata.com)
+                
      
+                
      2024-02-15 18:32:45
      
+                
      
+                    
      *Thread Reply:* however, we encourage to use get_openlineage_facets_on_** methods just like we do within all providers now
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Minkyu Park
+                     (minkyu.park.200@gmail.com)
+                
      
+                
      2024-02-15 18:35:46
      
+                
      
+                    
      *Thread Reply:* In order to use get_openlineage_facets_on_** with the DefaultExtractor, the operator itself should implement the method, right?
+https://github.com/apache/airflow/blob/main/airflow/providers/openlineage/extractors/base.py#L115
      
+                    
      
+                            
+                            
+                            
+                                
+                                
      <https://github.com/apache/airflow/blob/main/airflow/providers/openlineage/extractors/base.py | base.py>
      
+                                
      
+                                    <pre><code>            return self._get_openlineage_facets(self.operator.get_openlineage_facets_on_start)  # type: ignore
+</code></pre>
+                                
      
+                                
+                                
+    
+        
+    
+                                
+                                
+                                    
      
+                                        
+                                        <a href="https://github.com/apache/airflow">apache/airflow</a>
+                                    
      
+                                
+                            
+                        
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Jakub Dardziński
+                     (jakub.dardzinski@getindata.com)
+                
      
+                
      2024-02-15 18:36:52
      
+                
      
+                    
      *Thread Reply:* correct, you can find multiple examples, like https://github.com/apache/airflow/blob/main/airflow/providers/amazon/aws/operators/athena.py#L208
      
+                    
      
+                            
+                            
+                            
+                                
+                                
      <https://github.com/apache/airflow/blob/main/airflow/providers/amazon/aws/operators/athena.py | athena.py>
      
+                                
      
+                                    <pre><code>    def get_openlineage_facets_on_start(self) -&gt; OperatorLineage:
+</code></pre>
+                                
      
+                                
+                                
+    
+        
+    
+                                
+                                
+                                    
      
+                                        
+                                        <a href="https://github.com/apache/airflow">apache/airflow</a>
+                                    
      
+                                
+                            
+                        
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Minkyu Park
+                     (minkyu.park.200@gmail.com)
+                
      
+                
      2024-02-15 18:37:12
      
+                
      
+                    
      *Thread Reply:* 👍
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Jackson Goerner
+                     (jgoerner@squareup.com)
+                
      
+                
      2024-02-15 19:24:14
      
+                
      
+                    
      👋 Hello! I'm trying to integrate our airflow service with open lineage, and while the standard transports work, I'm having trouble using a custom transport (I'm sure its a silly mistake). Details are in 🧵
      
+                    
+                    
+                    
+                        
      
+                        ✅ Jackson Goerner
+                        
      
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Jackson Goerner
+                     (jgoerner@squareup.com)
+                
      
+                
      2024-02-15 19:24:30
      
+                
      
+                    
      *Thread Reply:* The reason I need a custom transport is we have an api which requires some authentication with aws incognito, which when executing on the box seems to work fine (I can instantiate the transport with config, and emit events and that works all good.)
      
+
+
      From what I understand the type transport configuration should be the fully qualified class name, which I've done. Using a standard file transport worked fine, and I saw the files being created as tasks we're executed. However, replacing my openlineage.yml file with:
      
+
+
      transport:
+  type: lineage_utils.api_transport.Transport
+  env: dev
+  endpoint: lineage/data
+Has stopped working. The file containing the class is /usr/share/airflow/plugins/lineage_utils/api_transport.py, and the first thing that file does is just touch a log file, which it isn't even doing, so I think the issue is something to do with the fully qualified class name.
      
+
+
      The airflow worker service has the following standard configuration:
      
+
+
      [Service]
+Environment="PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/share/airflow/venv/bin:/usr/local/bin"
+Environment="PYTHONPATH=${PYTHONPATH}:/usr/share/airflow/plugins"
+EnvironmentFile=/etc/sysconfig/airflow
+User=airflow
+Group=airflow
+Type=simple
+ExecStart=/usr/share/airflow/venv/bin/airflow celery worker -D -l /var/log/airflow/airflow-worker.log  --stdout /var/log/airflow/airflow-worker-stdout.log --stderr /var/log/airflow/airflow-worker-stderr.log -q default
+Restart=on-failure
+RestartSec=10s
+Which should add the plugins directory to the PYTHONPATH (which is working for the dags themselves). Changing the qualified path to plugins.lineage_utils.api_transport.Transport also didn't work.
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Maciej Obuchowski
+                     (maciej.obuchowski@getindata.com)
+                
      
+                
      2024-02-16 03:46:01
      
+                
      
+                    
      *Thread Reply:* Any logs/exceptions you can share?
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Jakub Dardziński
+                     (jakub.dardzinski@getindata.com)
+                
      
+                
      2024-02-16 03:52:39
      
+                
      
+                    
      *Thread Reply:* https://github.com/OpenLineage/OpenLineage/blob/main/client/python/openlineage/client/utils.py#L23
+it should log warning if there are issues with importing the class
      
+                    
      
+                            
+                            
+                            
+                                
+                                
      <https://github.com/OpenLineage/OpenLineage/blob/main/client/python/openlineage/client/utils.py | utils.py>
      
+                                
      
+                                    <pre><code>        log.warning(e)
+</code></pre>
+                                
      
+                                
+                                
+    
+        
+    
+                                
+                                
+                                    
      
+                                        
+                                        <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a>
+                                    
      
+                                
+                            
+                        
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Jackson Goerner
+                     (jgoerner@squareup.com)
+                
      
+                
      2024-02-19 19:11:28
      
+                
      
+                    
      *Thread Reply:* Thanks for pointing towards the logs - the issue was with my (faulty) interpretation of the docs here. I just copied the base transport/config definitions and implemented the methods, rather than subclassing the base classes, which caused issues. Everything is all working now! Thanks a bunch.
      
+                    
      
+                            
      openlineage.io
      
+                            
+                            
+                                
+                                
      Python | OpenLineage
      
+                                
      
+                                    Overview
+                                
      
+                                
+                                
+    
+        
+    
+                                
+                                    
      Original URL: https://openlineage.io/docs/client/python/#custom-transport-type
      
+                                
+                                
+                            
+                        
      
+                    
+                    
+                    
+                        
      
+                        🚀 Jakub Dardziński
+                        
      
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Simran Suri
+                     (mailsimransuri@gmail.com)
+                
      
+                
      2024-02-19 01:26:05
      
+                
      
+                    
      Hi, I am currently incorporating my Spark app name explicitly within the Spark code and further trying to build lineage. During this process, I've noticed certain transformations occurring in the job name in the event. Specifically, hyphens are changing into underscores, and uppercase letters are being converted into underscores followed by lowercase letters.
+I would appreciate more detailed information on these naming conventions. Are there specific scenarios or considerations we should be mindful of when adding the Spark app name to ensure same job name in events?
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Maciej Obuchowski
+                     (maciej.obuchowski@getindata.com)
+                
      
+                
      2024-02-19 08:43:45
      
+                
      
+                    
      *Thread Reply:* Best I can do is point you to where it happened: https://github.com/MarquezProject/marquez/pull/1191
+and code that does it now: https://github.com/OpenLineage/OpenLineage/blob/7f1dd20d32a2048c6e8608679e0ee5e928[…]openlineage/spark/agent/lifecycle/SparkSQLExecutionContext.java
      
+                    
      
+                            
+                            
+                            
+                                
+                                
      #1191 Update spark job name to reflect spark application name and execution node
      
+                                
      
+                                    
+                                
      
+                                
+                                
+    
+        
+    
+                                
+                                
+                                    
      
+                                        
+                                        <a href="https://github.com/MarquezProject/marquez">MarquezProject/marquez</a>
+                                    
      
+                                
+                            
+                        
      
+                    
      
+                            
+                            
+                            
+                                
+                                
      <https://github.com/OpenLineage/OpenLineage/blob/7f1dd20d32a2048c6e8608679e0ee5e9283f2922/integration/spark/app/src/main/java/io/openlineage/spark/agent/lifecycle/SparkSQLExecutionContext.java | SparkSQLExecutionContext.java>
      
+                                
      
+                                    
+                                
      
+                                
+                                
+    
+        
+    
+                                
+                                
+                                    
      
+                                        
+                                        <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a>
+                                    
      
+                                
+                            
+                        
      
+                    
+                    
+                    
+                        
      
+                        👍 Simran Suri
+                        
      
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Maciej Obuchowski
+                     (maciej.obuchowski@getindata.com)
+                
      
+                
      2024-02-19 08:37:48
      
+                
      
+                    
      Hey all - with @Damien Hawes as we get closer to releasing Spark integration with support for Scala 2.13 we think of requiring specifying Scala version in the artifact name. This is generally standard procedure for libraries containing part in Scala - for example Iceberg integration for Spark or Spark itself.
      
+
+
      The reason we ask is because Scala 2.12 compatible version currently does not have the suffix: for OpenLineage 1.8 the maven artifact coordinates are
+&lt;dependency&gt;
+    &lt;groupId&gt;io.openlineage&lt;/groupId&gt;
+    &lt;artifactId&gt;openlineage-spark&lt;/artifactId&gt;
+    &lt;version&gt;1.8.0&lt;/version&gt;
+&lt;/dependency&gt;
+and for 1.9 would be
+&lt;dependency&gt;
+    &lt;groupId&gt;io.openlineage&lt;/groupId&gt;
+    &lt;artifactId&gt;openlineage-spark_2.12&lt;/artifactId&gt;
+    &lt;version&gt;1.9.0&lt;/version&gt;
+&lt;/dependency&gt;
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Damien Hawes
+                     (damien.hawes@booking.com)
+                
      
+                
      2024-02-19 08:40:37
      
+                
      
+                    
      *Thread Reply:* It should be noted that the 2.13 variant would have these coordinates:
+&lt;dependency&gt;
+    &lt;groupId&gt;io.openlineage&lt;/groupId&gt;
+    &lt;artifactId&gt;openlineage-spark_2.13&lt;/artifactId&gt;
+    &lt;version&gt;1.9.0&lt;/version&gt;
+&lt;/dependency&gt;
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Balachandru S
+                     (balachandru2468@gmail.com)
+                
      
+                
      2024-02-21 03:53:45
      
+                
      
+                    
      Hi Team,
+I ran pyspark code in the Jupyter notebook (which comes with OpenLineage) and created lineage diagrams in the Marquez successfully. But I have a doubt instead of using Jupyter notebook (comes with OpenLineage), can use use my locally installed spark to create a lineage?. Thanks.
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Kacper Muda
+                     (kacper.muda@getindata.com)
+                
      
+                
      2024-02-21 04:14:11
      
+                
      
+                    
      *Thread Reply:* Hey, yes it's possible to use OpenLineage Spark integration outside of Jupyter notebook. Check out the docs on that integration.
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Balachandru S
+                     (balachandru2468@gmail.com)
+                
      
+                
      2024-02-21 04:26:51
      
+                
      
+                    
      *Thread Reply:* Thanks @Kacper Muda
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Rodrigo Maia
+                     (rodrigo.maia@manta.io)
+                
      
+                
      2024-02-21 04:43:19
      
+                
      
+                    
      *Thread Reply:* @Balachandru S its easy as moving the jar OL spark listener to the JAR spark folder and configuring spark as usual.
      
+
+
      or something like:
      
+
+
      SparkSession.builder
+        .appName("example")
+        .config("spark.jars.packages", "io.openlineage:openlineage_spark:1.8.0")
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Balachandru S
+                     (balachandru2468@gmail.com)
+                
      
+                
      2024-02-21 09:12:19
      
+                
      
+                    
      *Thread Reply:* Thanks @Rodrigo Maia
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Abdallah
+                     (abdallah@terrab.me)
+                
      
+                
      2024-02-21 06:17:11
      
+                
      
+                    
      Hi team,
+I hope you are all doing well.
+After the recent contributions, some of which addressed significant issues, may I kindly request a new release, please?
+Thank you very much for your time.
      
+                    
+                    
+                    
+                        
      
+                        ➕ Abdallah, Sophie LY, Yannick Libert, Tristan GUEZENNEC -CROIX-, Michael Robinson, Jakub Dardziński, Maciej Obuchowski, Damien Hawes
+                        
      
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Rodrigo Maia
+                     (rodrigo.maia@manta.io)
+                
      
+                
      2024-02-21 06:53:36
      
+                
      
+                    
      *Thread Reply:* Tha would be awesome
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Abdallah
+                     (abdallah@terrab.me)
+                
      
+                
      2024-02-21 11:37:46
      
+                
      
+                    
      *Thread Reply:* @Maciej Obuchowski
+Do you consider any release soon ?
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Maciej Obuchowski
+                     (maciej.obuchowski@getindata.com)
+                
      
+                
      2024-02-21 12:20:15
      
+                
      
+                    
      *Thread Reply:* @Michael Robinson is working on it 🙂
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Abdallah
+                     (abdallah@terrab.me)
+                
      
+                
      2024-02-21 12:20:45
      
+                
      
+                    
      *Thread Reply:* Owh ! Thank you !
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Rodrigo Maia
+                     (rodrigo.maia@manta.io)
+                
      
+                
      2024-02-21 06:51:43
      
+                
      
+                    
      Bug on Unity Catalog Support on OpenLineage for Databricks:
      
+
+
      Im struggling to generate any output when running pyspark, reading and writing to Unity Catalog on Databricks, while input facets are created properly with correct symlinks:
      
+
+
      example code:
+df = spark.read.table("rodrigo.default.brazil_universities")
+df.write.mode("overwrite").saveAsTable("rodrigo.default.brazil_universities_temp1")
+lineage json payload for all events (start and complete - it never emits a running event):
+..."outputs":[]}
+What i noticed from the error logs:
      
+
+
      If im writing to the default schema:
+24/02/21 11:50:49 INFO PlanUtils: apply method failed with
+org.apache.spark.SparkException: There is no Credential Scope. Current env: Driver
+    at com.databricks.unity.UCSDriver$Manager.$anonfun$currentScopeId$3(UCSDriver.scala:131)
+    at scala.Option.getOrElse(Option.scala:189)
+    at com.databricks.unity.UCSDriver$Manager.currentScopeId(UCSDriver.scala:131)
+    at com.databricks.unity.UCSDriver$Manager.currentScope(UCSDriver.scala:134)
+    at com.databricks.unity.UnityCredentialScope$.currentScope(UnityCredentialScope.scala:100)
+    at com.databricks.unity.UnityCredentialScope$.getSAMRegistry(UnityCredentialScope.scala:120)
+    at com.databricks.unity.SAMRegistry$.registerSAM(SAMRegistry.scala:322)
+    at com.databricks.unity.SAMRegistry$.registerDefaultSAM(SAMRegistry.scala:338)
+    at org.apache.spark.sql.catalyst.catalog.SessionCatalogImpl.defaultTablePath(SessionCatalog.scala:1200)
+    at com.databricks.sql.managedcatalog.ManagedCatalogSessionCatalog.defaultTablePath(ManagedCatalogSessionCatalog.scala:991)
+    at io.openlineage.spark3.agent.lifecycle.plan.catalog.AbstractDatabricksHandler.getDatasetIdentifier(AbstractDatabricksHandler.java:92)
+    at io.openlineage.spark3.agent.lifecycle.plan.catalog.CatalogUtils3.lambda$getDatasetIdentifier$2(CatalogUtils3.java:61)
+........
+If im writing to other schema (not the default):
+24/02/21 11:47:16 INFO PlanUtils: apply method failed with
+org.apache.spark.SparkException: There is no Credential Scope. Current env: Driver
+    at com.databricks.unity.UCSDriver$Manager.$anonfun$currentScopeId$3(UCSDriver.scala:131)
+    at scala.Option.getOrElse(Option.scala:189)
+    at com.databricks.unity.UCSDriver$Manager.currentScopeId(UCSDriver.scala:131)
+    at com.databricks.unity.UCSDriver$Manager.currentScope(UCSDriver.scala:134)
+    at com.databricks.unity.UnityCredentialScope$.currentScope(UnityCredentialScope.scala:100)
+    at com.databricks.unity.UnityCredentialScope$.getSAMRegistry(UnityCredentialScope.scala:120)
+    at com.databricks.unity.SAMRegistry$.registerSAM(SAMRegistry.scala:322)
+    at com.databricks.unity.SAMRegistry$.registerDefaultSAM(SAMRegistry.scala:338)
+    at org.apache.spark.sql.catalyst.catalog.SessionCatalogImpl.defaultTablePath(SessionCatalog.scala:1200)
+    at com.databricks.sql.managedcatalog.ManagedCatalogSessionCatalog.defaultTablePath(ManagedCatalogSessionCatalog.scala:991)
+    at io.openlineage.spark3.agent.lifecycle.plan.catalog.AbstractDatabricksHandler.getDatasetIdentifier(AbstractDatabricksHandler.java:92)
+    at io.openlineage.spark3.agent.lifecycle.plan.catalog.CatalogUtils3.lambda$getDatasetIdentifier$2(CatalogUtils3.java:61)
+..........
+Has anyone successfully used Unity Catalog and produced  the output facet?
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Maciej Obuchowski
+                     (maciej.obuchowski@getindata.com)
+                
      
+                
      2024-02-21 07:07:03
      
+                
      
+                    
      *Thread Reply:* I would call this feature request rather than bug 🙂
      
+                    
+                    
+                    
+                        
      
+                        💡 Abdallah
+                        
      
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Rodrigo Maia
+                     (rodrigo.maia@manta.io)
+                
      
+                
      2024-02-21 09:03:10
      
+                
      
+                    
      *Thread Reply:* i really wish i could help on that
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Matthew Paras
+                     (matthewparas2020@u.northwestern.edu)
+                
      
+                
      2024-02-21 13:18:10
      
+                
      
+                    
      *Thread Reply:* I have a fix for this!
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Matthew Paras
+                     (matthewparas2020@u.northwestern.edu)
+                
      
+                
      2024-02-21 13:18:45
      
+                
      
+                    
      *Thread Reply:* Let me create an issue on github, I'm happy to make the change in a PR as well
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Maciej Obuchowski
+                     (maciej.obuchowski@getindata.com)
+                
      
+                
      2024-02-21 13:20:28
      
+                
      
+                    
      *Thread Reply:* Great to hear that!
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Matthew Paras
+                     (matthewparas2020@u.northwestern.edu)
+                
      
+                
      2024-02-21 13:21:30
      
+                
      
+                    
      *Thread Reply:* Should I do issue + PR or just straight to PR?
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Maciej Obuchowski
+                     (maciej.obuchowski@getindata.com)
+                
      
+                
      2024-02-21 13:43:34
      
+                
      
+                    
      *Thread Reply:* Straight to PR is good for me, assuming you don't plan to break other things as part of the solution 🙂
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Matthew Paras
+                     (matthewparas2020@u.northwestern.edu)
+                
      
+                
      2024-02-21 13:51:37
      
+                
      
+                    
      *Thread Reply:* It is a fairly small change and we've been running this patched version on databricks for a few weeks now, so cautiously optimistic that it doesn't break anything else 😄
      
+                    
+                    
+                    
+                        
      
+                        🙌 Jakub Dardziński, Maciej Obuchowski, Rodrigo Maia
+                        
      
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Matthew Paras
+                     (matthewparas2020@u.northwestern.edu)
+                
      
+                
      2024-02-21 21:58:40
      
+                
      
+                    
      *Thread Reply:* https://github.com/OpenLineage/OpenLineage/pull/2453
      
+                    
      
+                            
+                            
+                            
+                                
+                                
      #2453 Address outputs missing with unity catalog
      
+                                
      
+                                    <p><strong>Problem</strong></p>
+
+<p>👋 Thanks for opening a pull request! Please include a brief summary of the problem your change is trying to solve, or bug fix. If your change fixes a bug or you'd like to provide context on why you're making the change, please <a href="https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue">link the issue</a> as follows:</p>
+
+<p>Closes: #ISSUE-NUMBER</p>
+
+<p><strong>Solution</strong></p>
+
+<p>Please describe your change as it relates to the problem, or bug fix, as well as any dependencies. If your change requires a schema change, please describe the schema modification(s) and whether it's a <em>backwards-incompatible</em> or <em>backwards-compatible</em> change, then select one of the following:</p>
+
+<blockquote>
+  <p><strong>Note:</strong> All schema changes require discussion. Please <a href="https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue">link the issue</a> for context.</p>
+</blockquote>
+
+<p>☐ Your change modifies the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json">core</a> OpenLineage model
+☐ Your change modifies one or more OpenLineage <a href="https://github.com/OpenLineage/OpenLineage/tree/main/spec/facets">facets</a></p>
+
+<p>If you're contributing a new integration, please specify the scope of the integration and how/where it has been tested (e.g., Apache Spark integration supports <code>S3</code> and <code>GCS</code> filesystem operations, tested with AWS EMR).</p>
+
+<p><strong>One-line summary:</strong>
+<strong>Checklist</strong></p>
+
+<p>☐ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work
+☐ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a>
+☐ Your changes are accompanied by tests (<em>if relevant</em>)
+☐ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained
+☐ You've updated any relevant documentation (<em>if relevant</em>)
+☐ Your comment includes a one-liner for the changelog about the specific purpose of the change (<em>if necessary</em>)
+☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (<em>if relevant</em>)
+☐ You've added a <a href="https://github.com/OpenLineage/OpenLineage/tree/main/.github/header_templates.md">header</a> to source files (<em>if relevant</em>)</p>
+
+<hr />
+
+<p>SPDX-License-Identifier: Apache-2.0<br />
+Copyright 2018-2023 contributors to the OpenLineage project</p>
+                                
      
+                                
+                                    
      
+                                        
      Labels
      
+                                        integration/spark
+                                    
      
+                                
+                                    
      
+                                        
      Comments
      
+                                        1
+                                    
      
+                                
+                                
+    
+        
+    
+                                
+                                
+                                    
      
+                                        
+                                        <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a>
+                                    
      
+                                
+                            
+                        
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Rodrigo Maia
+                     (rodrigo.maia@manta.io)
+                
      
+                
      2024-02-21 09:03:49
      
+                
      
+                    
      💡Idea for next group meeting 
+A workshop would come in handy to help newbies start contributing to the project 🙂
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Maciej Obuchowski
+                     (maciej.obuchowski@getindata.com)
+                
      
+                
      2024-02-21 09:13:41
      
+                
      
+                    
      *Thread Reply:* I think this is what you're looking for https://www.youtube.com/watch?v=SXebBTVcY4Q
      
+                    
      
+                            
      YouTube
      
+                            
+                                
      
+                                    
+                                        
+                                    }
+                                    
+                                    OpenLineage Project
+                                    (https://www.youtube.com/@openlineageproject6897)
+                                
      
+                            
+                            
+                                
+                                
      How to Become an OpenLineage-Spark Contributor | November 29, 2023
      
+                                
      
+                                    
+                                
      
+                                
+                                
+    
+        
+            
+        
+    
+                                
+                                    
      Original URL: https://www.youtube.com/watch?v=SXebBTVcY4Q
      
+                                
+                                
+                            
+                        
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Jakub Dardziński
+                     (jakub.dardzinski@getindata.com)
+                
      
+                
      2024-02-21 09:15:47
      
+                
      
+                    
      *Thread Reply:* What kind of guidance would you like to see?
      
+
+
      There are several places that introduce some help:
+https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md
+https://openlineage.io/docs/development/developing/
+each integration has its own readme that writes down basic dev e.g.
+https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/README.md
      
+
+
      and above link from Maciej 🙂
      
+                    
      
+                            
+                            
+                            
+                                
+                                
      <https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md | CONTRIBUTING.md>
      
+                                
      
+                                    
+                                
      
+                                
+                                
+    
+        
+    
+                                
+                                
+                                    
      
+                                        
+                                        <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a>
+                                    
      
+                                
+                            
+                        
      
+                    
      
+                            
+                            
+                            
+                                
+                                
      <https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/README.md | README.md>
      
+                                
      
+                                    
+                                
      
+                                
+                                
+    
+        
+    
+                                
+                                
+                                    
      
+                                        
+                                        <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a>
+                                    
      
+                                
+                            
+                        
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Max Zheng
+                     (mzheng@plaid.com)
+                
      
+                
      2024-02-21 13:52:07
      
+                
      
+                    
      Hi, I'm running into an odd "deadlock" with small datasets (eg. < 300 rows input/output) with Spark/OpenLineage with Kafka sink. The problem doesn't seem deterministic but is frequent enough where we've had to disable the listener on all of our jobs, and when it happens from driver logs and Spark UI it seems like the job hangs forever (there's a running query but no jobs are generated in the Spark UI, and no driver logs are generated at all). It doesn't seem like disk/memory/CPU utilization is high when this happens (< 20% memory utilization from our Prometheus metrics)
      
+
+
      Are there any debug logs I can enable to get a better sense of whats happening? Any advice on how to debug this? Thanks
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Maciej Obuchowski
+                     (maciej.obuchowski@getindata.com)
+                
      
+                
      2024-02-21 14:28:55
      
+                
      
+                    
      *Thread Reply:* What do you exactly mean by deadlock? Does it happen only with Kafka sink? How are you running the job, is it Databricks, EMR, some other provider, or some custom setup? Is it PySpark or Scala job? What Spark API are you using - RDDs, Data frame, Spark SQL?
      
+
+
      Generally any logging we do is to driver logs, so it does not help that they are missing 😞
+One thing that would be helpful is serialized logical plan of the job.
      
+
+
      Quickly looking at the code, I don't see obvious candidate where something went wrong
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Max Zheng
+                     (mzheng@plaid.com)
+                
      
+                
      2024-02-21 14:36:59
      
+                
      
+                    
      *Thread Reply:* We are running on EMR 6.10.1, I've only been able to reproduce with Kafka sink but doing some more thorough testing with console sink now
      
+
+
      • pyspark job
+• DataFrame API 
+By deadlock I mean there's a running query in the Spark UI, but there's no active job/stages - the job just seems to be idle
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Max Zheng
+                     (mzheng@plaid.com)
+                
      
+                
      2024-02-21 14:48:18
      
+                
      
+                    
      *Thread Reply:* == Parsed Logical Plan ==
+SaveIntoDataSourceCommand org.apache.hudi.Spark32PlusDefaultSource@6e5f1c4f, Map(hoodie.copyonwrite.record.size.estimate -&gt; 57, hoodie.insert.shuffle.parallelism -&gt; 1500, path -&gt; {path}, hoodie.datasource.write.precombine.field -&gt; _autogenerated_primary_key, hoodie.bootstrap.index.enable -&gt; false, hoodie.metadata.enable -&gt; true, hoodie.metrics.graphite.metric.prefix -&gt; lake_production, hoodie.index.type -&gt; SIMPLE, hoodie.datasource.write.operation -&gt; upsert, hoodie.metrics.reporter.type -&gt; GRAPHITE, hoodie.datasource.write.recordkey.field -&gt; _autogenerated_primary_key, hoodie.table.name -&gt; {table_name}, hoodie.datasource.write.table.type -&gt; COPY_ON_WRITE, hoodie.datasource.write.hive_style_partitioning -&gt; true, hoodie.metrics.graphite.host -&gt; {host}, hoodie.datasource.write.table.name -&gt; {table_name}, hoodie.populate.meta.fields -&gt; false, hoodie.metrics.graphite.port -&gt; 9109, hoodie.metrics.on -&gt; true, hoodie.datasource.write.keygenerator.class -&gt; org.apache.hudi.keygen.NonpartitionedKeyGenerator, hoodie.upsert.shuffle.parallelism -&gt; 1500, hoodie.datasource.write.partitionpath.field -&gt; ), Overwrite
+   +- Project [c1#3, c2#4, a#5, c3#6, c4#7, c5#8, c6#9, uuid(Some(-217463645859800419)) AS _autogenerated_primary_key#17]
+      +- Relation [c1#3,c2#4,a#5,c3#6,c4#7,c5#8,c6#9] parquet
+Here's the parsed logical plan for the stuck running query (its been 8 hours and typically finishes in < 1 minute, assuming this will never finish). The job is just rewriting a parquet input from S3 as an Apache Hudi table in S3.
      
+                    
+                    
+                    
+                        
      
+                        👍 Maciej Obuchowski
+                        
      
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Maciej Obuchowski
+                     (maciej.obuchowski@getindata.com)
+                
      
+                
      2024-02-21 15:08:34
      
+                
      
+                    
      *Thread Reply:* Ah, by Kafka sink you mean OpenLineage writing events to Kafka, not Spark writing to Kafka.
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Max Zheng
+                     (mzheng@plaid.com)
+                
      
+                
      2024-02-21 15:09:31
      
+                
      
+                    
      *Thread Reply:* Correct, sorry for the ambiguity 😅
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Maciej Obuchowski
+                     (maciej.obuchowski@getindata.com)
+                
      
+                
      2024-02-21 15:20:53
      
+                
      
+                    
      *Thread Reply:* Would be great if you could check that with console transport. If it does not fix the issue, my guess is that something weird happens with Hudi, as we don't support it explicitly
      
+                    
+                    
+                    
+                        
      
+                        👍 Max Zheng
+                        
      
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Max Zheng
+                     (mzheng@plaid.com)
+                
      
+                
      2024-02-21 15:54:10
      
+                
      
+                    
      *Thread Reply:* Sounds good, will do
      
+
+
      I'm confused on how a listener could affect the running job ... wonder if its something on onJobStart https://github.com/OpenLineage/OpenLineage/blob/1.8.0/integration/spark/app/src/ma[…]n/java/io/openlineage/spark/agent/OpenLineageSparkListener.java
      
+                    
      
+                            
+                            
+                            
+                                
+                                
      <https://github.com/OpenLineage/OpenLineage/blob/1.8.0/integration/spark/app/src/main/java/io/openlineage/spark/agent/OpenLineageSparkListener.java | OpenLineageSparkListener.java>
      
+                                
      
+                                    <pre><code>  public void onJobStart(SparkListenerJobStart jobStart) {
+</code></pre>
+                                
      
+                                
+                                
+    
+        
+    
+                                
+                                
+                                    
      
+                                        
+                                        <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a>
+                                    
      
+                                
+                            
+                        
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Maciej Obuchowski
+                     (maciej.obuchowski@getindata.com)
+                
      
+                
      2024-02-21 15:56:32
      
+                
      
+                    
      *Thread Reply:* I'm also not sure, but deadlock sure sounds like something that theoretically could happen - SaveIntoDataSourceVisitor for example uses createRelation if it can't recognize particular relation type
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Maciej Obuchowski
+                     (maciej.obuchowski@getindata.com)
+                
      
+                
      2024-02-21 15:56:43
      
+                
      
+                    
      *Thread Reply:* Which it does not with Hudi
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Max Zheng
+                     (mzheng@plaid.com)
+                
      
+                
      2024-02-21 16:00:48
      
+                
      
+                    
      *Thread Reply:* Got it hmm
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Max Zheng
+                     (mzheng@plaid.com)
+                
      
+                
      2024-02-21 17:19:26
      
+                
      
+                    
      *Thread Reply:* Same issue with
+spark.openlineage.transport.type console
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Max Zheng
+                     (mzheng@plaid.com)
+                
      
+                
      2024-02-21 17:54:28
      
+                
      
+                    
      *Thread Reply:* Looking at the thread dump of the driver it seems like spark-listener-group-shared is running
+org.apache.hudi.DataSourceOptionsHelper$.&lt;init&gt;(DataSourceOptions.scala:731)
+org.apache.hudi.DataSourceOptionsHelper$.&lt;clinit&gt;(DataSourceOptions.scala)
+org.apache.hudi.DataSourceReadOptions$.&lt;init&gt;(DataSourceOptions.scala:143)
+org.apache.hudi.DataSourceReadOptions$.&lt;clinit&gt;(DataSourceOptions.scala)
+org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:75)
+org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:68)
+io.openlineage.spark.agent.lifecycle.plan.SaveIntoDataSourceCommandVisitor.apply(SaveIntoDataSourceCommandVisitor.java:139)
+io.openlineage.spark.agent.lifecycle.plan.SaveIntoDataSourceCommandVisitor.apply(SaveIntoDataSourceCommandVisitor.java:46)
+io.openlineage.spark.api.AbstractQueryPlanDatasetBuilder$1.apply(AbstractQueryPlanDatasetBuilder.java:94)
+io.openlineage.spark.api.AbstractQueryPlanDatasetBuilder$1.apply(AbstractQueryPlanDatasetBuilder.java:85)
+io.openlineage.spark.agent.util.PlanUtils.safeApply(PlanUtils.java:279)
+io.openlineage.spark.api.AbstractQueryPlanDatasetBuilder.lambda$apply$0(AbstractQueryPlanDatasetBuilder.java:75)
+io.openlineage.spark.api.AbstractQueryPlanDatasetBuilder$$Lambda$4382/1107261117.apply(Unknown Source)
+java.util.Optional.map(Optional.java:215)
+io.openlineage.spark.api.AbstractQueryPlanDatasetBuilder.apply(AbstractQueryPlanDatasetBuilder.java:67)
+io.openlineage.spark.api.AbstractQueryPlanDatasetBuilder.apply(AbstractQueryPlanDatasetBuilder.java:39)
+io.openlineage.spark.agent.util.PlanUtils.safeApply(PlanUtils.java:279)
+io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.lambda$null$23(OpenLineageRunEventBuilder.java:453)
+io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder$$Lambda$4380/2129506005.apply(Unknown Source)
+...
+If an event listener is stuck like here, does it prevent the next job from starting?
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Max Zheng
+                     (mzheng@plaid.com)
+                
      
+                
      2024-02-21 19:08:51
      
+                
      
+                    
      *Thread Reply:* It seems like another thread and the Spark listener thread are both trying to access a singleton from Hudi in the stuck state. This seems like the problem to me though I am not familiar with Scala
      
+
+
      org.apache.hudi.DataSourceWriteOptions$.&lt;init&gt;(DataSourceOptions.scala:400)
+org.apache.hudi.DataSourceWriteOptions$.&lt;clinit&gt;(DataSourceOptions.scala)
+org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:141)
+org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:47)
+org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:75) =&gt; holding Monitor(org.apache.spark.sql.execution.command.ExecutedCommandExec@213524349})
+org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:73)
+org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:84)
+...
+org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:860)
+org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:390)
+org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:363)
+org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:239)
+Seems to be stuck on https://github.com/apache/hudi/blob/release-0.12.2/hudi-spark-datasource/hudi-spar[…]k-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala
      
+
+
      (I've asked in the Hudi Slack about this)
      
+                    
      
+                            
+                            
+                            
+                                
+                                
      <https://github.com/apache/hudi/blob/release-0.12.2/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala | DataSourceOptions.scala>
      
+                                
      
+                                    <pre><code>    DataSourceWriteOptions.TABLE_TYPE,
+</code></pre>
+                                
      
+                                
+                                
+    
+        
+    
+                                
+                                
+                                    
      
+                                        
+                                        <a href="https://github.com/apache/hudi">apache/hudi</a>
+                                    
      
+                                
+                            
+                        
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Maciej Obuchowski
+                     (maciej.obuchowski@getindata.com)
+                
      
+                
      2024-02-22 06:00:53
      
+                
      
+                    
      *Thread Reply:* So yeah - the approach from our side would be to implement HudiRelationVisitor that refrains from calling createRelation because it can utilize some unique data present in Hudi's DefaultSource .
      
+
+
      Looking at the Hudi code, it seems we can get the destination path from options: https://github.com/apache/hudi/blob/3a97b01c0263c4790ffa958b865c682f40b4ada4/hudi-[…]spark-common/src/main/scala/org/apache/hudi/DefaultSource.scala
+However the various path operation might require calling remote filesystem. This would require tests if we can safely call for example TablePathUtils.getTablePath
      
+                    
      
+                            
+                            
+                            
+                                
+                                
      <https://github.com/apache/hudi/blob/3a97b01c0263c4790ffa958b865c682f40b4ada4/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DefaultSource.scala | DefaultSource.scala>
      
+                                
      
+                                    
+                                
      
+                                
+                                
+    
+        
+    
+                                
+                                
+                                    
      
+                                        
+                                        <a href="https://github.com/apache/hudi">apache/hudi</a>
+                                    
      
+                                
+                            
+                        
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Maciej Obuchowski
+                     (maciej.obuchowski@getindata.com)
+                
      
+                
      2024-02-22 06:02:58
      
+                
      
+                    
      *Thread Reply:* Also not exactly sure why it deadlocks on reading config options? I can't exactly see where it breaks, since the code line numbers don't align with main anymore 🙂
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Max Zheng
+                     (mzheng@plaid.com)
+                
      
+                
      2024-02-22 12:31:37
      
+                
      
+                    
      *Thread Reply:* I suspect its because the two singletons being loaded reference each other? ie. DataSourceWriteOptions and DataSourceOptionsHelper
      
+
+
      I did manage to workaround this by doing this dumb fake read at Spark session initialization
+try:
+        spark.read.format("hudi").load("dummy")
+    except Exception:
+        pass
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Maciej Obuchowski
+                     (maciej.obuchowski@getindata.com)
+                
      
+                
      2024-02-22 12:33:01
      
+                
      
+                    
      *Thread Reply:* maybe, but why does that happen reliably?
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Max Zheng
+                     (mzheng@plaid.com)
+                
      
+                
      2024-02-22 12:34:06
      
+                
      
+                    
      *Thread Reply:* I have no idea, but over 3 runs it was always the same lines in the thread dump
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Ruchira Prasad
+                     (ruchiraprasad@gmail.com)
+                
      
+                
      2024-02-22 06:25:40
      
+                
      
+                    
      Is it possible to integrate the SQL Server Integration Service (SSIS) data pipeline with OpenLineage Marquez? If so give me some reference or guideline on how to implement it.
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Damien Hawes
+                     (damien.hawes@booking.com)
+                
      
+                
      2024-02-22 06:28:23
      
+                
      
+                    
      *Thread Reply:* Oh. It's been a while (> 10 years) since I last used SSIS (SQL Server 2008R2 and SQL Server 2012), however, if you're able to obtain the SQL queries it executes (assuming regular SQL transforms), you'd be able to run it through the openlineage-sql parser.
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Jakub Dardziński
+                     (jakub.dardzinski@getindata.com)
+                
      
+                
      2024-02-22 06:31:09
      
+                
      
+                    
      *Thread Reply:* It is possible, however in Airflow for instance there has not been added such support.
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Damien Hawes
+                     (damien.hawes@booking.com)
+                
      
+                
      2024-02-22 06:43:05
      
+                
      
+                    
      *Thread Reply:* Additionally, you could parse the DTSX file (DTSX is an XML-like file), to extract the data flow out of it, and adapt it to the OpenLineage format.
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Damien Hawes
+                     (damien.hawes@booking.com)
+                
      
+                
      2024-02-22 06:43:23
      
+                
      
+                    
      *Thread Reply:* Though, importantly, this would probably be a Job Event and not a Run Event.
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Athitya Kumar
+                     (athityakumar@gmail.com)
+                
      
+                
      2024-02-22 08:10:57
      
+                
      
+                    
      Hey team. We have a use-case where we would like to leverage the transports of openlineage (console, http, kafka etc) & enhance/customise the spark metrics/events published by the OL spark listener.
      
+
+
      For example, OL spark listener currently publishes lineage-related info like inputs, outputs, column-level lineage etc - we want to add more job/task-level metrics etc that's available from the raw Spark Bus events. What'd be the best way to go about it, to add our custom logic & publish custom fields in the OL event payload to the OL transports?
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Kacper Muda
+                     (kacper.muda@getindata.com)
+                
      
+                
      2024-02-22 08:13:08
      
+                
      
+                    
      *Thread Reply:* Hey, have You checked the docs about Custom Facets? It can help You attach any custom logic You might have, including:
+> more job/task-level metrics etc
+to an OpenLineage Event.
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Athitya Kumar
+                     (athityakumar@gmail.com)
+                
      
+                
      2024-02-22 08:17:02
      
+                
      
+                    
      *Thread Reply:* @Kacper Muda - Yup, I checked the above doc. But I thought it explains more on the "how" part from a schema / POJO perspective.
      
+
+
      I was more interested to see where would be the best place to make these CustomFacet and logic changes. Should it be via extending the OL Spark Listener class & overriding/super, or some other way to inject our custom business logic which would be used by OL when it prepares the events?
      
+                    
+                    
+                    
+                        
      
+                        👍 Kacper Muda
+                        
      
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Maciej Obuchowski
+                     (maciej.obuchowski@getindata.com)
+                
      
+                
      2024-02-22 08:32:58
      
+                
      
+                    
      *Thread Reply:* @Athitya Kumar https://github.com/OpenLineage/OpenLineage/blob/31f8ce588526e9c7c4bc7d849699cb7ce2[…]/src/main/java/io/openlineage/spark/api/CustomFacetBuilder.java
      
+                    
      
+                            
+                            
+                            
+                                
+                                
      <https://github.com/OpenLineage/OpenLineage/blob/31f8ce588526e9c7c4bc7d849699cb7ce2969c8f/integration/spark/shared/src/main/java/io/openlineage/spark/api/CustomFacetBuilder.java | CustomFacetBuilder.java>
      
+                                
      
+                                    <pre><code>    implements AbstractPartial&lt;Object&gt;, BiConsumer&lt;Object, BiConsumer&lt;String, ? super F&gt;&gt; {
+</code></pre>
+                                
      
+                                
+                                
+    
+        
+    
+                                
+                                
+                                    
      
+                                        
+                                        <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a>
+                                    
      
+                                
+                            
+                        
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Athitya Kumar
+                     (athityakumar@gmail.com)
+                
      
+                
      2024-02-22 08:51:37
      
+                
      
+                    
      *Thread Reply:* @Maciej Obuchowski - Is the recommendation to use this CustomFacetBuilder by extending/overriding OL spark listener's methods like onTaskEnd / onJobEnd etc? I remember seeing something regarding ServiceLoader and being able to "inject" our own logic earlier - but didn't get to explore that completely
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Maciej Obuchowski
+                     (maciej.obuchowski@getindata.com)
+                
      
+                
      2024-02-22 08:59:28
      
+                
      
+                    
      *Thread Reply:* @Athitya Kumar the idea is to implement OpenLineageEventHandlerFactory in your own code, implementing methods where you add your own CustomFacetBuilders , then provide it in separate JAR for ServiceLoader with META-INF file.
      
+
+
      You can see example in tests:
+Implementation that adds TestRunFacet
+META-INF file
      
+                    
      
+                            
+                            
+                            
+                                
+                                
      <https://github.com/OpenLineage/OpenLineage/blob/31f8ce588526e9c7c4bc7d849699cb7ce2969c8f/integration/spark/app/src/test/java/io/openlineage/spark/agent/util/TestOpenLineageEventHandlerFactory.java | TestOpenLineageEventHandlerFactory.java>
      
+                                
      
+                                    
+                                
      
+                                
+                                
+    
+        
+    
+                                
+                                
+                                    
      
+                                        
+                                        <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a>
+                                    
      
+                                
+                            
+                        
      
+                    
      
+                            
+                            
+                            
+                                
+                                
      <https://github.com/OpenLineage/OpenLineage/blob/9d31aa6f0177a60ba6b85812abd46159e5cdae74/integration/spark/app/src/test/resources/META-INF/services/io.openlineage.spark.api.OpenLineageEventHandlerFactory | io.openlineage.spark.api.OpenLineageEventHandlerFactory>
      
+                                
      
+                                    
+                                
      
+                                
+                                
+    
+        
+    
+                                
+                                
+                                    
      
+                                        
+                                        <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a>
+                                    
      
+                                
+                            
+                        
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Maciej Obuchowski
+                     (maciej.obuchowski@getindata.com)
+                
      
+                
      2024-02-22 09:01:02
      
+                
      
+                    
      *Thread Reply:* your CustomFacetBuilder can specify on which Spark events it listens to
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Monisha SL
+                     (monisha.sl@philips.com)
+                
      
+                
      2024-02-22 08:30:17
      
+                
      
+                    
      Is it possible to integrate OpenLineage with Glue Jobs written in PySpark. If yes, please help me with reference documents
      
+                    
+                    
+                    
+                        
      
+                        ✅ Rodrigo Maia, Monisha SL
+                        
      
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Rodrigo Maia
+                     (rodrigo.maia@manta.io)
+                
      
+                
      2024-02-22 08:48:43
      
+                
      
+                    
      *Thread Reply:* it is, but there is no support for a glue catalog.
+Host the OL jar in S3
+On your pyspark code put the configs for spark and the reference for Extra Jars glue big configuration pointing to your S3/jar file
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Monisha SL
+                     (monisha.sl@philips.com)
+                
      
+                
      2024-02-26 09:03:35
      
+                
      
+                    
      *Thread Reply:* @Rodrigo Maia I am unable to start the OL server on EC2 instance. Am trying to connect to OL hosted on EC2 from Glue to provide the OL URL. But unable to bring up the OL Server.
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Rodrigo Maia
+                     (rodrigo.maia@manta.io)
+                
      
+                
      2024-02-26 10:23:28
      
+                
      
+                    
      *Thread Reply:* here is an axample:
      
+                    
+                    
      
+                            
      image.png
      
+                            
+                                
+    
+        
+            
+        
+    
+                            
+                        
      
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Monisha SL
+                     (monisha.sl@philips.com)
+                
      
+                
      2024-02-26 11:17:42
      
+                
      
+                    
      *Thread Reply:* @Rodrigo Maia Thanks this helped the job to run successfully. But am quite confused on where to find the lineage generated? Does this needs to be integrated with Marquez UI? Apologies for so many questions, am new to this piece of work.
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Rodrigo Maia
+                     (rodrigo.maia@manta.io)
+                
      
+                
      2024-02-26 11:22:45
      
+                
      
+                    
      *Thread Reply:* is this case, the spark config is set to console. you can check on the logs in cloudwatch. I havent tried with Marquez
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Monisha SL
+                     (monisha.sl@philips.com)
+                
      
+                
      2024-02-26 11:24:41
      
+                
      
+                    
      *Thread Reply:* I can find it on the logs, looking for UI to showcase the lineage. Any help will be appreciated. Thank you.
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      soumilshah1995
+                     (shahsoumil519@gmail.com)
+                
      
+                
      2024-02-22 18:52:31
      
+                
      
+                    
      Hello there
+I am new here I am trying to run docker example given on GH im facing following issue
+im running on Mac M2
      
+
+
      ```git clone git@github.com:MarquezProject/marquez.git && cd marquez
      
+
+
      ./docker/up.sh```
      
+
+
      (venv) soumilshah@Soumils-MBP marquez % ./docker/up.sh              
+...creating volumes: marquez_data, marquez_db-conf, marquez_db-init, marquez_db-backup
+                                             Successfully copied 7.17kB to volumes-provisioner:/data/wait-for-it.sh
+Added files to volume marquez_data: wait-for-it.sh
+                                             Successfully copied 2.05kB to volumes-provisioner:/db-conf/postgresql.conf
+Added files to volume marquez_db-conf: postgresql.conf
+                                             Successfully copied 2.05kB to volumes-provisioner:/db-init/init-db.sh
+Added files to volume marquez_db-init: init-db.sh
+DONE!
+[+] Running 39/3
+ ✔ web 12 layers [⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿]      0B/0B      Pulled                                          31.9s 
+ ✔ db 14 layers [⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿]      0B/0B      Pulled                                         15.9s 
+ ✔ api 10 layers [⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿]      0B/0B      Pulled                                             8.8s 
+[+] Building 0.0s (0/0)                                                          docker:desktop-linux
+[+] Running 6/2
+ ✔ Network marquez_default                                                                                                                            Created0.0s                                      0.7s 
+ ✔ Container marquez-db                                                                                                                               Created0.7s 
+ ✔ Container marquez-api                                                                                                                              Created0.0s 
+ ! api The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested 0.0s 
+ ✔ Container marquez-web                                                                                                                              Created0.0s 
+ ! web The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested 0.0s 
+Attaching to marquez-api, marquez-db, marquez-web
+Error response from daemon: Ports are not available: exposing port TCP 0.0.0.0:5000 -> 0.0.0.0:0: listen tcp 0.0.0.0:5000: bind: address already in use
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Maciej Obuchowski
+                     (maciej.obuchowski@getindata.com)
+                
      
+                
      2024-02-23 08:28:14
      
+                
      
+                    
      *Thread Reply:* AFAIK on newer Macs Airplay Receiver  runs on that port - please change the port to something else, unused or disable Airplay Receiver
      
+                    
+                    
+                    
+                        
      
+                        ✅ soumilshah1995
+                        
      
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      soumilshah1995
+                     (shahsoumil519@gmail.com)
+                
      
+                
      2024-02-23 18:37:07
      
+                
      
+                    
      *Thread Reply:* Thanks
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Max Zheng
+                     (mzheng@plaid.com)
+                
      
+                
      2024-02-23 02:05:58
      
+                
      
+                    
      Hi, I'm taking a look at lineage data from Spark and there's a weird event type called {spark_application_name}.map_partitions_parallel_collection  which has 1 input (an S3 path) and 1 output which is strangely the same S3 path as the input - anyone have idea what this event is?
      
+
+
      spark.logicalPlan=None, spark_properties=None
+are both None for this event which seems kind of weird
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Max Zheng
+                     (mzheng@plaid.com)
+                
      
+                
      2024-02-23 02:09:39
      
+                
      
+                    
      *Thread Reply:* Could be some oddity with using Apache Hudi but the Spark execution plan just shows a SaveIntoDataSourceCommand (which has a correct looking OpenLineage event)
      
+                    
+                    
+                    
+                        
      
+                        👍 Maciej Obuchowski
+                        
      
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Max Zheng
+                     (mzheng@plaid.com)
+                
      
+                
      2024-02-23 02:09:53
      
+                
      
+                    
      *Thread Reply:* Just found this extra event that was generated very odd
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Maciej Obuchowski
+                     (maciej.obuchowski@getindata.com)
+                
      
+                
      2024-02-23 06:45:26
      
+                
      
+                    
      *Thread Reply:* this is RDD event I think
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Maciej Obuchowski
+                     (maciej.obuchowski@getindata.com)
+                
      
+                
      2024-02-23 06:46:16
      
+                
      
+                    
      *Thread Reply:* https://github.com/OpenLineage/OpenLineage/blob/31f8ce588526e9c7c4bc7d849699cb7ce2[…]src/main/java/io/openlineage/spark/agent/util/RddPathUtils.java
      
+                    
      
+                            
+                            
+                            
+                                
+                                
      <https://github.com/OpenLineage/OpenLineage/blob/31f8ce588526e9c7c4bc7d849699cb7ce2969c8f/integration/spark/shared/src/main/java/io/openlineage/spark/agent/util/RddPathUtils.java | RddPathUtils.java>
      
+                                
      
+                                    <pre><code>  static class MapPartitionsRDDExtractor implements RddPathExtractor&lt;MapPartitionsRDD&gt; {
+</code></pre>
+                                
      
+                                
+                                
+    
+        
+    
+                                
+                                
+                                    
      
+                                        
+                                        <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a>
+                                    
      
+                                
+                            
+                        
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Maciej Obuchowski
+                     (maciej.obuchowski@getindata.com)
+                
      
+                
      2024-02-23 06:53:52
      
+                
      
+                    
      *Thread Reply:* RDD events work kinda differently, they go through RddExecutionContext and not SparkSQLExecutionContext - LogicalPlan is a concept of Spark SQL https://jaceklaskowski.gitbooks.io/mastering-spark-sql/content/spark-sql-QueryExecution.html
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Maciej Obuchowski
+                     (maciej.obuchowski@getindata.com)
+                
      
+                
      2024-02-23 06:54:27
      
+                
      
+                    
      *Thread Reply:* on the other hand, it would be probably good to attach Spark Properties facet
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Max Zheng
+                     (mzheng@plaid.com)
+                
      
+                
      2024-02-23 12:51:49
      
+                
      
+                    
      *Thread Reply:* Ah I see
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Max Zheng
+                     (mzheng@plaid.com)
+                
      
+                
      2024-02-23 12:53:08
      
+                
      
+                    
      *Thread Reply:* Its a little weird the output seems to be created erroneously, it definitely isn't writing anything to the input path
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Maciej Obuchowski
+                     (maciej.obuchowski@getindata.com)
+                
      
+                
      2024-02-23 15:35:53
      
+                
      
+                    
      *Thread Reply:* Would be nice to look at it more as part of more proper Hudi support
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Max Zheng
+                     (mzheng@plaid.com)
+                
      
+                
      2024-02-23 16:40:02
      
+                
      
+                    
      *Thread Reply:* Yep, and thanks for the help in explaining things/code pointers 🙂
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Balachandru S
+                     (balachandru2468@gmail.com)
+                
      
+                
      2024-02-23 04:43:49
      
+                
      
+                    
      Hi Team,
+I want to generate table-level and column-level lineage for pyspark script. With the help of OpenLineage I can see lineage in the Marquez. Now I am interested to know the limitations here, can I create a lineage for pyspark script with 50+ number of sources and targets?. Are there any limitations in the source and target file count in the spark job?
+Like the above one, is there any limitation for spark job depth also?. Thanks.
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Maciej Obuchowski
+                     (maciej.obuchowski@getindata.com)
+                
      
+                
      2024-02-23 06:44:50
      
+                
      
+                    
      *Thread Reply:* I don't think there's hard limitation, we've seen it working for one job where serialized size of logical plan was >100MB
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      jayant joshi
+                     (itsjayantjoshi@gmail.com)
+                
      
+                
      2024-02-23 08:09:09
      
+                
      
+                    
      Hi Team, If any transformation changes happening in the Spark job, can we track this under data lineage version history?
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Maciej Obuchowski
+                     (maciej.obuchowski@getindata.com)
+                
      
+                
      2024-02-23 08:26:11
      
+                
      
+                    
      *Thread Reply:* Do you mean column-level lineage? If yes, then we generally have that for Spark.
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Balachandru S
+                     (balachandru2468@gmail.com)
+                
      
+                
      2024-02-23 08:42:38
      
+                
      
+                    
      *Thread Reply:* @Maciej Obuchowski if I am running the same spark job multiple times with each time some transformation changes, can we trace transformation changes between runs?. Thanks.
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Maciej Obuchowski
+                     (maciej.obuchowski@getindata.com)
+                
      
+                
      2024-02-26 05:36:30
      
+                
      
+                    
      *Thread Reply:* Do you want to track exactly how the transformation happens? I would look at spark.logicalPlan facet, however it's raw information, and not sure how hard it is to get anything meaningful.
+Do you want to just track end result - how the resulting columns are formed? I would look at column-level lineage facet.
      
+
+
      If that answer does not satisfies you - we process transformation information as part of generating column-level lineage, you can take look at that code.
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Balachandru S
+                     (balachandru2468@gmail.com)
+                
      
+                
      2024-02-26 05:56:06
      
+                
      
+                    
      *Thread Reply:* Thanks @Maciej Obuchowski
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Hubert Boguski
+                     (hubert.boguski@spglobal.com)
+                
      
+                
      2024-02-23 15:52:57
      
+                
      
+                    
      Hi everyone! Want to say this is a really awesome and much needed effort for the data community. Just wanted to double check my understanding here-
      
+
+
      TLDR: is there a way to achieve this with python operators in a DAG with multiple python operators: To propagate the job hierarchy, tasks must send their own run ids so that the downstream process can report the ParentRunFacet with the proper run id.
      
+
+
      Is there a way to have the multiple Python Operators (multiple python tasks) in a DAG trace job hierachy in Marquez. Right now I am invoking GET request and receiving a response in one task and passing the first tasks output to another task, but the jobs in Marquez shows up as different jobs (job hierarchy not shown). I understand there could be some extractor classes that could be implemented, but just wondering if there is anything simpler out there to show hierarchy with Python Operators 😅
      
+
+
      I read somewhere that we can use the parent Id to link these jobs, but I had no luck (picture of docs)
      
+
+
      I also saw a youtube video https://www.youtube.com/watch?v=yrSDngUhj8U
+But as I understand this is still a work in progress
      
+
+
      I know there are some supported extractors already, but they dont support my use case. Thanks in advance!!!
      
+                    
      
+                            
      YouTube
      
+                            
+                                
      
+                                    
+                                        
+                                    }
+                                    
+                                    OpenLineage Project
+                                    (https://www.youtube.com/@openlineageproject6897)
+                                
      
+                            
+                            
+                                
+                                
      Extracting Lineage from PythonOperator | November 29, 2023
      
+                                
      
+                                    
+                                
      
+                                
+                                
+    
+        
+            
+        
+    
+                                
+                                    
      Original URL: https://www.youtube.com/watch?v=yrSDngUhj8U
      
+                                
+                                
+                            
+                        
      
+                    
+                    
      
+                            
      image.png
      
+                            
+                                
+    
+        
+            
+        
+    
+                            
+                        
      
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Jakub Dardziński
+                     (jakub.dardzinski@getindata.com)
+                
      
+                
      2024-02-26 05:34:03
      
+                
      
+                    
      *Thread Reply:* Hey, I’m not exactly sure what you’d like to achieve.
      
+
+
      First, let’s explain what’s the hierarchy. When Airflow TaskInstance is running it is considered run for which DagRun is its parent run. Similarly if task spawns other jobs, they should set TaskInstance as their parent run. More about the idea here: https://openlineage.io/docs/spec/job-hierarchy
      
+
+
      If you’d like to emit additional events within PythonOperator’s callback and set  this task as parent run for those additional events you should take another approach:
+• here’s how task uuid is generated: https://github.com/apache/airflow/blob/main/airflow/providers/openlineage/plugins/listener.py#L79
+• based on the above you could use build_task_instance_run_id method with additional information taken from context  (that is dag_id, task_id, execution_date, try_number)
+Code snippet you provided allows to point to DAG as parent run.
      
+                    
      
+                            
      openlineage.io
      
+                            
+                            
+                                
+                                
      Job Hierarchy | OpenLineage
      
+                                
      
+                                    In a complex environment, where there are thousands of processing jobs daily, there can be a lot of chaos.
+                                
      
+                                
+                                
+    
+        
+    
+                                
+                                    
      Original URL: https://openlineage.io/docs/spec/job-hierarchy
      
+                                
+                                
+                            
+                        
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Jakub Dardziński
+                     (jakub.dardzinski@getindata.com)
+                
      
+                
      2024-02-26 05:35:02
      
+                
      
+                    
      *Thread Reply:* Extracting lineage from PythonOperator is still in progress, proposal should be public soon.
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Jakub Dardziński
+                     (jakub.dardzinski@getindata.com)
+                
      
+                
      2024-02-26 05:36:11
      
+                
      
+                    
      *Thread Reply:* Maybe I’m mistaken what you mean by job hierarchy. What would you expect to be shown as a graph?
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Suhas Shenoy
+                     (ksuhasshenoy@gmail.com)
+                
      
+                
      2024-02-25 23:51:38
      
+                
      
+                    
      Hi team, Just wanted to know if we can we configure openlineage to send events to two endpoints?
      
+                    
+                    
+                    
+                        
      
+                        👍 Derya Meral
+                        
      
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Kacper Muda
+                     (kacper.muda@getindata.com)
+                
      
+                
      2024-02-26 02:57:35
      
+                
      
+                    
      *Thread Reply:* Hi, i don't think it's possible with a built-in HTTP Transport, but You can always implement a custom Transport, that will suit Your needs 🙂
      
+                    
+                    
+                    
+                        
      
+                        👍 Suhas Shenoy, Derya Meral
+                        
      
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Paweł Leszczyński
+                     (pawel.leszczynski@getindata.com)
+                
      
+                
      2024-02-26 03:27:59
      
+                
      
+                    
      *Thread Reply:* Hi @Suhas Shenoy, this was one of the motivations behind fluentD proxy -> https://github.com/OpenLineage/OpenLineage/tree/main/proxy/fluentd
      
+                    
+                    
+                    
+                        
      
+                        ❤️ Kacper Muda, Derya Meral
+                        
      
+                    
+                        
      
+                        👍 Suhas Shenoy, Jakub Dardziński, Mattia Bertorello, Derya Meral
+                        
      
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Derya Meral
+                     (drderyameral@gmail.com)
+                
      
+                
      2024-02-26 11:24:36
      
+                
      
+                    
      *Thread Reply:* I was wondering the same thing and joined the slack to ask this. In our case, we're wondering whether it would be possible to use both Marquez and Datahub with our Airflow. Thanks for the pointer, @Kacper Muda!
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Max Zheng
+                     (mzheng@plaid.com)
+                
      
+                
      2024-02-26 12:37:43
      
+                
      
+                    
      Does anyone know how to disable the logger for:
+4/02/23 09:08:01 INFO AsyncEventQueue: Process of event SparkListenerJobStart
+It seems like spark.redaction.regex isn't respected by this logger which causes secrets to get logged when Spark OpenLineage is enabled
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Max Zheng
+                     (mzheng@plaid.com)
+                
      
+                
      2024-02-26 12:51:28
      
+                
      
+                    
      I also ran into an error Py4JNetworkError("Answer from Java side is empty") very frequently on some jobs. It seems like this is caused by our Spark sessions being explicitly stopped via a spark.stop() and then OpenLineageSparkListener trying to access the stopped Spark session, which causes the application to fail (I suspect in https://github.com/OpenLineage/OpenLineage/blob/5298e8a30a8168dd8096f334e5d484812a[…]park/agent/lifecycle/plan/SaveIntoDataSourceCommandVisitor.java)
      
+
+
      Should session already closed exceptions be handled gracefully there? Part of this is likely due to Hudi not being supported well (it seems like its very slow for some jobs and doesn't generate any events before crashing) but this seems like a race condition that can generally cause crashes
      
+                    
      
+                            
+                            
+                            
+                                
+                                
      <https://github.com/OpenLineage/OpenLineage/blob/5298e8a30a8168dd8096f334e5d484812a48aa84/integration/spark/shared/src/main/java/io/openlineage/spark/agent/lifecycle/plan/SaveIntoDataSourceCommandVisitor.java | SaveIntoDataSourceCommandVisitor.java>
      
+                                
      
+                                    <pre><code>        relation = p.createRelation(sqlContext, command.options());
+</code></pre>
+                                
      
+                                
+                                
+    
+        
+    
+                                
+                                
+                                    
      
+                                        
+                                        <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage/OpenLineage</a>
+                                    
      
+                                
+                            
+                        
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Paweł Leszczyński
+                     (pawel.leszczynski@getindata.com)
+                
      
+                
      2024-02-27 02:26:01
      
+                
      
+                    
      *Thread Reply:* Hi Max,
      
+
+
      Do you have any logs on OpenLineageSparkListener crashing? All the exceptions from OpenLineageSparkListener trying to build an event should be caught and we do have a test for this. Please provide more details on that. How do you know it's failing? The desired behaviour is to log and exception and proceed.
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Max Zheng
+                     (mzheng@plaid.com)
+                
      
+                
      2024-02-26 12:52:47
      
+                
      
+                    
      Lastly, would disabling facets improve performance? eg. disabling spark.logicalPlan
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Paweł Leszczyński
+                     (pawel.leszczynski@getindata.com)
+                
      
+                
      2024-02-27 02:26:44
      
+                
      
+                    
      *Thread Reply:* Disabling spark.LogicalPlan may improve performance of populating OL event. It's disabled by default in recent version (the one released yesterday). You can also use circuit breaker feature if you are worried about Ol integration affecting Spark jobs
      
+                    
+                    
+                    
+                        
      
+                        🤩 Yannick Libert
+                        
      
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Yannick Libert
+                     (yannick.libert.partner@decathlon.com)
+                
      
+                
      2024-02-27 05:20:13
      
+                
      
+                    
      *Thread Reply:* This feature is going to be so useful for us! Love it!
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Michael Robinson
+                     (michael.robinson@astronomer.io)
+                
      
+                
      2024-02-26 14:23:37
      
+                
      
+                    
      @channel
+We released OpenLineage 1.9.1, featuring:
+• Airflow: add support for JobTypeJobFacet properties #2412 @mattiabertorello
+• dbt: add support for JobTypeJobFacet properties #2411 @mattiabertorello
+• Flink: support Flink Kafka dynamic source and sink #2417 @HuangZhenQiu
+• Flink: support multi-topic Kafka Sink #2372 @pawel-big-lebowski
+• Flink: support lineage for JDBC connector #2436 @HuangZhenQiu
+• Flink: add common config gradle plugin #2461 @HuangZhenQiu
+• Java: extend circuit breaker loaded with ServiceLoader #2435 @pawel-big-lebowski
+• Spark: integration now emits intermediate, application level events wrapping entire job execution #2371 @mobuchowski
+• Spark: support built-in lineage within DataSourceV2Relation #2394 @pawel-big-lebowski
+• Spark: add support for JobTypeJobFacet properties #2410 @mattiabertorello
+• Spark: stop sending spark.LogicalPlan facet by default #2433 @pawel-big-lebowski
+• Spark/Flink/Java: circuit breaker #2407 @pawel-big-lebowski
+• Spark: add the capability to publish Scala 2.12 and 2.13 variants of openlineage-spark #2446 @d-m-h
+A large number of changes and bug fixes were also included.
+Thanks to all our contributors with a special shout-out to @Damien Hawes, who contributed >10 PRs to this release!
+Release: https://github.com/OpenLineage/OpenLineage/releases/tag/1.9.1
+Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md
+Commit history: https://github.com/OpenLineage/OpenLineage/compare/1.8.0...1.9.1
+Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage
+PyPI: https://pypi.org/project/openlineage-python/
      
+                    
+                    
+                    
+                        
      
+                        🚀 Jakub Dardziński, Jackson Goerner, Abdallah, Yannick Libert, Mattia Bertorello, Tristan GUEZENNEC -CROIX-, Fabio Manganiello
+                        
      
+                    
+                        
      
+                        🎉 Abdallah, Mattia Bertorello
+                        
      
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Jakub Dardziński
+                     (jakub.dardzinski@getindata.com)
+                
      
+                
      2024-02-26 14:33:27
      
+                
      
+                    
      *Thread Reply:* Oudstanding work @Damien Hawes 👏
      
+                    
+                    
+                    
+                        
      
+                        ➕ Michael Robinson, Mattia Bertorello, Fabio Manganiello
+                        
      
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Abdallah
+                     (abdallah@terrab.me)
+                
      
+                
      2024-02-27 00:39:29
      
+                
      
+                    
      *Thread Reply:* Thank you 👏👏
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Derya Meral
+                     (drderyameral@gmail.com)
+                
      
+                
      2024-02-26 15:04:33
      
+                
      
+                    
      Hi all, I'm working on a local Airflow-OpenLineage-Marquez integration using Airflow 2.7.3 and python 3.10. Everything seems to be installed correctly with the appropriate settings. I'm seeing events, jobs, tasks trickle into the UI. I'm using the PostgresOperator. When it's time for the SQL code to be parsed, I'm seeing the following in my Airflow logs:
+[2024-02-26, 19:43:17 UTC] {sql.py:457} INFO - Running statement: SELECT CURRENT_SCHEMA;, parameters: None
+[2024-02-26, 19:43:17 UTC] {base.py:152} WARNING - OpenLineage provider method failed to extract data from provider. 
+[2024-02-26, 19:43:17 UTC] {manager.py:198} WARNING - Extractor returns non-valid metadata: None
+Can anyone give me pointers on why exactly this might be happening? I've tried also with the SQLExecuteQueryOperator, same result. I previously got a Marquez setup to work with the external OpenLineage package for Airflow with Airflow 2.6.1. But I'm struggling with this newer integrated OpenLineage version
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Jakub Dardziński
+                     (jakub.dardzinski@getindata.com)
+                
      
+                
      2024-02-26 15:10:21
      
+                
      
+                    
      *Thread Reply:* Does this happen for some particular SQL but works for other?
+Also, my understanding is that it worked with openlineage-airflow on Airflow 2.6.1 (the same code)?
+What version of OL provider are you using?
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Derya Meral
+                     (drderyameral@gmail.com)
+                
      
+                
      2024-02-26 15:20:22
      
+                
      
+                    
      *Thread Reply:* I've been using one toy DAG and have only tried with the two operators mentioned. Currently, my team's code doesn't use provider operators so it would not really work well with OL.
      
+
+
      Yes, it worked with Airflow 2.6.1. Same code.
      
+
+
      Right now, I'm using apache-airflow-providers-openlineage==1.5.0 and the other OL dependencies are at 1.9.1.
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Jakub Dardziński
+                     (jakub.dardzinski@getindata.com)
+                
      
+                
      2024-02-26 15:21:00
      
+                
      
+                    
      *Thread Reply:* Would you want to share the SQL statement?
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Derya Meral
+                     (drderyameral@gmail.com)
+                
      
+                
      2024-02-26 15:31:42
      
+                
      
+                    
      *Thread Reply:* It has some PII in it, but it's basically in the form of:
+```DROP TABLE IF EXISTS usersmeral.keyrelations;
      
+
+
      CREATE TABLE usersmeral.keyrelations AS
      
+
+
      WITH
+staff AS ( SELECT ...)
+,enabled AS (SELECT ...)
+SELECT ... 
+FROM public.borrowers 
+LEFT JOIN ...;```
+We're splitting the query with sqlparse.split() and feed it to a PostgresOperator.
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Derya Meral
+                     (drderyameral@gmail.com)
+                
      
+                
      2024-02-27 09:26:41
      
+                
      
+                    
      *Thread Reply:* I thought I should share our configs in case I'm missing something:
+```[openlineage]
+disabled = False
+disabledforoperators = 
      
+
+
      namespace =
      
+
+
      extractors =
      
+
+
      config_path = /opt/airflow/openlineage.yml
+transport = 
      
+
+
      disablesourcecode = ```
      
+                    
+                    
+                    
+		
      
+            
      
+        
      
+    
      
+                
+            
+        
+            
+                
+                    
      
+        
      
+            
+                
+            
+            
+                
      
+            
+                
      Derya Meral
+                     (drderyameral@gmail.com)
+                
      
+                
      2024-02-27 09:27:20
      
+                
      
+                    
      *Thread Reply:* The YAML file:
+transport:
+  type: http
+  url: <http://marquez:5000>
      
+                    
+                    
+                    
 		
      
             
      
         
      

From 1c6fa3c3a828543943fb9f062edb7af5a045a4b1 Mon Sep 17 00:00:00 2001
From: merobi-hub 
Date: Tue, 27 Feb 2024 11:18:40 -0500
Subject: [PATCH 2/2] Add more instructions to readme.

Signed-off-by: merobi-hub 
---
 README.md | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/README.md b/README.md
index 75ca69b..c52b13c 100644
--- a/README.md
+++ b/README.md
@@ -1,5 +1,9 @@
 # OpenLineage Slack Archive
 
+To get an export (entire history): workspace>Tools & Settings>Workspace settings
+
+To get the HTML with slack-export-viewer (from root): `python3 app.py -z /abs/path/to/export/dir --html-only`
+
 Browse the archive at https://openlineage.github.io/slack-archives/.
 
 ## Acknowledgements