[sbt-gen] sbt plugin for openapi to endpoints codegen #3225

hochgi · 2024-11-30T20:00:26Z

/fixes #3180
/claim #3180

based on a plugin I collaborated with @runtologist on, in another closed repo
automatic mode integrated in regular ~compile continuous compilation mode, manual trigger via direct task call
static openapi files read from src/main/oapi/**
dynamic download/create/… of openapi files via regular use of sourceGenerators scoped to the custom ZIOpenApi configuration. Files should reside under ZIOpenApi / sourceManaged (which is by default evaluated as target/scala-{cross version}/src_managed/oapi/ in same dir as package convention (see example in tests).
package prefix determined by relative path under /src/main/oapi/ => allows for multiple different spec files in different packages in a conventional and expected manner
sbt config to control codegen
using sbt caching mechanism to avoid re-generating sources from unchanged spec files on every compile

known gaps:

for some (currently unknown) reason, using .dependsOn(…) does not work, and I was forced to use zio-http-gen as an external dependency
cross compilation fails, currently commented out, so we only support sbt 1.x (sbt 2 is not yet released anyway)
~~only "static" openapi spec files are supported as of this first commit. Ideally, we should allow for dynamic generation of spec files like any other generated sources.~~ (done!)
collisions of generated sources from different spec files is not checked (but users should really use different packages for different services they're generating endpoints for).
only supporting Compile (main) configuration - generating endpoints for e.g. Test (by placing openapi files under src/test/oapi/) will not work out of the box (but is it really needed anyway?)
no README/docs yet - should probably add to the site, and document after we converge on actual functionality (I assume review might change enough to render any docs I'll write now obsolete)

codecov-commenter · 2024-12-01T12:11:13Z

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 61.12%. Comparing base (f164c90) to head (bcd4333).
Report is 32 commits behind head on main.

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3225      +/-   ##
==========================================
- Coverage   65.09%   61.12%   -3.97%     
==========================================
  Files         161      161              
  Lines       10551    11386     +835     
  Branches     2024     1998      -26     
==========================================
+ Hits         6868     6960      +92     
- Misses       3683     4426     +743

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

zio-http-gen-sbt-plugin/src/main/scala/zio/http/gen/sbt/Format.scala

build.sbt

hochgi · 2024-12-02T07:10:23Z

build.sbt

+    // crossScalaVersions := Seq(Scala212, Scala213),
+    // pluginCrossBuild / sbtVersion := {
+    //   // Not sure why version 1.5.8 specifically is set to minimum version,
+    //   // but when I searched the plugins under sbt org, this seems like the most common:
+    //   // https://github.com/search?q=org%3Asbt%20pluginCrossBuild&type=code
+    //   scalaBinaryVersion.value match {
+    //     case "2.12" => "1.5.8"
+    //     case _      => "2.0.0-M2"
+    //   }
+    // },


this does not work in CI.
Since sbt 2.x is not published yet (only milestone versions currently available),
I commented it out.
When sbt 2.0 comes out, this should be commented in with stable version, and it should just work.

Why does it not work with 2 M2? I saw other plugins having cross-builds
If it does not, please put this code in an issue, not in a comment.

I don't know why.
maybe those versions are published to another repo, which we need to add another resolver for?
might be as simple as that, I didn't checked.

an issue would be best to put such TODOs

M2 is on maven central. See https://mvnrepository.com/artifact/org.scala-sbt/sbt

I'll open an issue then.
should I wait until this is merged?

hochgi · 2024-12-02T07:12:48Z

build.sbt

+      // For now, and until this issue is resolved, we'll just use latest stable published version.
+      //
+      // FIXME: replace the dependency with .dependsOn(zioHttpGen) syntax
+      "dev.zio" %% "zio-http-gen" % "3.0.1",


I tried, and wasted a few hours trying to figure out why it won't work.
In a clean multi module project, it works.
Only in zio-http repo it acts weird.

similar plugin modules as part of a larger build work in other repos (playframework, tapir, …)

Any idea what's special here?

idk
But this is no option, if the plugin is in the same repo. I actually started myself to work on this issue and and problems with the sbt setup. So I created a different repo. Maybe we should do that.

Its solveable. same structure works in playframework and tapir projects.
It's also easier to catch integration issues.
If something changes in the zio-http-gen module which breaks the plugin, ideally we would want to catch that right away.

I'll try to spend some more time on this :/

@987Nabil turns out it works. but only under ++2.12.x.
Which is expected. When cross version is set to default 2.13, and we try to resolve only 2.12 modules it fails and resorts to trying to download.

So this should work, and we only need carful CI handling for only building/testing the plugin under ++2.12.x

zio-http-gen-sbt-plugin/src/main/scala/zio/http/gen/sbt/ZioHttpCodegen.scala

hochgi · 2024-12-02T07:38:52Z

zio-http-gen-sbt-plugin/src/sbt-test/zio-http-sbt-codegen/dynamic/build.sbt

+
+      val outFile = (ZIOpenApi / sourceManaged).value / "gov" / "uspto" / "ibd" / "api.json"
+      val http = Gigahorse.http(Gigahorse.config)
+      val request = Gigahorse.url("https://developer.uspto.gov/ibd-api/v3/api-docs")


arbitrary openapi spec. I tried so many, and every single one I checked had usage of features we don't support in zio-http-gen yet.
not the official swagger example: https://petstore3.swagger.io
nothing in this collection (didn't do an exhaustive check though): https://petstore.dev
So I tried looking for the most minimal example I could find, and even the one I found didn't work out of the box.
So I pruned the JSON, to only be left with the acceptable APIs definitions.

In retrospect, it kind of nice to have this as an example.
Most services we deal with have many APIs, and we don't use all of them.
So no need to generate code for stuff we're not using.

I may have came up with this example out of necessity, but now I think it's actually good to have it.

Also, I don't like the idea of tests depending on stuff outside of the build (other than conventional stuff like library dependencies of course)

But this is how I expect users to actually use it. so… ¯\_(ツ)_/¯

zio-http-gen-sbt-plugin/src/sbt-test/zio-http-sbt-codegen/dynamic/test

987Nabil · 2024-12-02T11:40:20Z

Do I see it right, that compile always triggers the generation? I'd like it to be opt out

hochgi · 2024-12-02T12:27:32Z

Do I see it right, that compile always triggers the generation? I'd like it to be opt out

I assume you want this since codegen could take a lot of time, and you wouldn't want to waste it on every incremental compilation?
If that's the case, this is covered by sbt task caching mechanism I added.
If there's no change in openapi files, then there's no need to re-generate.

Is there any other reason I didn't think of?

@runtologist

* based on a plugin I collaborated with @runtologist on, in another closed repo * automatic mode integrated in regular '~compile' continuous compilation mode, manual trigger via direct task call * openapi files read from src/main/oapi/** * package prefix determined by relative path under /src/main/oapi/ => allows for multiple different spec files in different packages in a conventional and expected manner * sbt config to control codegen * using sbt caching mechanism to avoid re-generating sources from unchanged spec files on every compile * added support for dynamic openapi spec files via sourceGenerators key (scoped to the new custom configuration) * tests via `scripted` known gaps: * collisions of generated sources from different spec files is not checked (one will override the other). * only supporting 'Compile' (main) configuration (is any other configuration, e.g, 'Test', really needed anyway?)

hochgi requested review from jdegoes, vigoo and 987Nabil as code owners November 30, 2024 20:00

algora-pbc bot mentioned this pull request Nov 30, 2024

Create a OpenAPI to Endpoint Code Gen sbt plugin #3180

Open

algora-pbc bot added the 🙋 Bounty claim label Nov 30, 2024

hochgi force-pushed the sbt-gen-issue-3180-add-sbt-plugin-to-generate-code-from-openapi branch 2 times, most recently from 2513bfa to a0f8acb Compare December 1, 2024 11:45

hochgi force-pushed the sbt-gen-issue-3180-add-sbt-plugin-to-generate-code-from-openapi branch from d342817 to 8aac9bb Compare December 1, 2024 14:17

hochgi commented Dec 2, 2024

View reviewed changes

zio-http-gen-sbt-plugin/src/main/scala/zio/http/gen/sbt/Format.scala Show resolved Hide resolved

hochgi commented Dec 2, 2024

View reviewed changes

build.sbt Outdated Show resolved Hide resolved

hochgi commented Dec 2, 2024

View reviewed changes

zio-http-gen-sbt-plugin/src/main/scala/zio/http/gen/sbt/ZioHttpCodegen.scala Outdated Show resolved Hide resolved

hochgi commented Dec 2, 2024

View reviewed changes

zio-http-gen-sbt-plugin/src/sbt-test/zio-http-sbt-codegen/dynamic/test Outdated Show resolved Hide resolved

hochgi force-pushed the sbt-gen-issue-3180-add-sbt-plugin-to-generate-code-from-openapi branch 6 times, most recently from e73fbd8 to 5544618 Compare December 2, 2024 19:01

hochgi force-pushed the sbt-gen-issue-3180-add-sbt-plugin-to-generate-code-from-openapi branch from 5e24f44 to 34543c8 Compare December 14, 2024 17:34

hochgi added 5 commits December 15, 2024 09:50

mark a flaky test as such

db44345

still flaky. try harder

0f0a908

flaky test is consistently failing now. Ignored + comment

514c5b0

mark another flaky test as such

4313750

hochgi force-pushed the sbt-gen-issue-3180-add-sbt-plugin-to-generate-code-from-openapi branch from 34543c8 to 4313750 Compare December 15, 2024 07:51

githubWorkflowGenerate

3d109bc

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[sbt-gen] sbt plugin for openapi to endpoints codegen #3225

[sbt-gen] sbt plugin for openapi to endpoints codegen #3225

hochgi commented Nov 30, 2024 •

edited

Loading

codecov-commenter commented Dec 1, 2024

hochgi Dec 2, 2024

987Nabil Dec 2, 2024

hochgi Dec 2, 2024

987Nabil Dec 2, 2024

hochgi Dec 2, 2024

hochgi Dec 2, 2024

987Nabil Dec 2, 2024

hochgi Dec 2, 2024

hochgi Dec 2, 2024

hochgi Dec 2, 2024

hochgi Dec 2, 2024

987Nabil commented Dec 2, 2024

hochgi commented Dec 2, 2024

[sbt-gen] sbt plugin for openapi to endpoints codegen #3225

Are you sure you want to change the base?

[sbt-gen] sbt plugin for openapi to endpoints codegen #3225

Conversation

hochgi commented Nov 30, 2024 • edited Loading

codecov-commenter commented Dec 1, 2024

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

987Nabil commented Dec 2, 2024

hochgi commented Dec 2, 2024

hochgi commented Nov 30, 2024 •

edited

Loading