Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rename otp-shaded-jar artifact and fix deployment to Maven Central #6331

Merged
merged 3 commits into from
Dec 13, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/performance-test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,7 @@ jobs:
- name: Build graph
if: matrix.profile == 'core' || github.ref == 'refs/heads/dev-2.x'
run: |
cp shaded-jar/target/otp-*-SNAPSHOT-shaded.jar otp.jar
cp otp-shaded/target/otp-shaded-*-SNAPSHOT.jar otp.jar
java -Xmx32G -jar otp.jar --build --save test/performance/${{ matrix.location }}/

- name: Run speed test
Expand Down
15 changes: 10 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,10 +31,11 @@ We run a speed test (included in the code) to measure the performance for every

## Repository layout

The main Java server code is in `application/src/main/`. OTP also includes a Javascript client based on the
MapLibre mapping library in `client/src/`. This client is now used for testing, with most major
deployments building custom clients from reusable components. The Maven build produces a unified ("shaded")
JAR file at `shaded-jar/target/otp-VERSION.jar` containing all necessary code and dependencies to run OpenTripPlanner.
The main Java server code is in `application/src/main/`. OTP also includes a Javascript client
based on the MapLibre mapping library in `client/src/`. This client is now used for testing, with
most major deployments building custom clients from reusable components. The Maven build produces a
unified ("shaded") JAR file at `otp-shaded/target/otp-shaded-VERSION.jar` containing all necessary
code and dependencies to run OpenTripPlanner.

Additional information and instructions are available in
the [main documentation](http://docs.opentripplanner.org/en/dev-2.x/), including a
Expand All @@ -59,7 +60,11 @@ the world.

## Getting in touch

The fastest way to get help is to use our [Gitter chat room](https://gitter.im/opentripplanner/OpenTripPlanner) where most of the core developers are. Bug reports may be filed via the Github [issue tracker](https://github.com/openplans/OpenTripPlanner/issues). The OpenTripPlanner [mailing list](http://groups.google.com/group/opentripplanner-users) is used almost exclusively for project announcements. The mailing list and issue tracker are not intended for support questions or discussions. Please use the chat for this purpose. Other details of [project governance](http://docs.opentripplanner.org/en/dev-2.x/Governance/) can be found in the main documentation.
The fastest way to get help is to use our [Gitter chat room](https://gitter.im/opentripplanner/OpenTripPlanner) where most of the core developers
are. Bug reports may be filed via the Github [issue tracker](https://github.com/openplans/OpenTripPlanner/issues). The OpenTripPlanner [mailing list](http://groups.google.com/group/opentripplanner-users)
is used almost exclusively for project announcements. The mailing list and issue tracker are not
intended for support questions or discussions. Please use the chat for this purpose. Other details
of [project governance](http://docs.opentripplanner.org/en/dev-2.x/Governance/) can be found in the main documentation.

## OTP Ecosystem

Expand Down
18 changes: 9 additions & 9 deletions doc/user/Basic-Tutorial.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,9 @@ JAR containing all other libraries needed for OTP to work, and is available from
repository. You will be able to go
to [the OTP directory at Maven Central](https://repo1.maven.org/maven2/org/opentripplanner/otp/),
navigate to
the [directory of releases](https://repo1.maven.org/maven2/org/opentripplanner/otp/2.6.0/),
the [directory of releases](https://repo1.maven.org/maven2/org/opentripplanner/otp-shaded/2.7.0/),
and download
the [file with `shaded.jar` suffix](https://repo1.maven.org/maven2/org/opentripplanner/otp/2.6.0/otp-2.6.0-shaded.jar)
the [file with `otp-shaded` artifactId](https://repo1.maven.org/maven2/org/opentripplanner/otp-shaded/2.7.0/otp-shaded-2.7.0.jar)
.

You may also want to get your own copy of the OTP source code
Expand Down Expand Up @@ -95,7 +95,7 @@ for cropping and filtering OSM data.

## Starting OTP

A typical command to start OTP looks like `java -Xmx2G -jar otp.shaded.jar <options>`. The
A typical command to start OTP looks like `java -Xmx2G -jar otp-shaded-VERSION.jar <options>`. The
`-Xmx` parameter sets the limit on how much memory OTP is allowed to consume. GTFS and OSM data sets
are often very large, and OTP is relatively memory-hungry. You will need at least 1GB of memory when
working with the Portland TriMet data set, and several gigabytes for larger inputs.
Expand Down Expand Up @@ -129,7 +129,7 @@ below and in other tutorials.
The simplest way to use OTP is to build a graph in a single step and start a server immediately,
without saving it to disk. The command to do so is:

$ java -Xmx2G -jar otp-2.6.0-shaded.jar --build --serve /home/username/otp
$ java -Xmx2G -jar otp-shaded-2.7.0.jar --build --serve /home/username/otp

where `/home/username/otp` should be the directory where you put your configuration and input files.

Expand All @@ -154,13 +154,13 @@ build a graph from street and transit data then save it to a file using the `--b
command line parameters together. If for example your current working directory (`.`) contains the
input files and the OTP JAR file, you can use this command:

$ java -Xmx2G -jar otp-2.6.0-shaded.jar --build --save .
$ java -Xmx2G -jar otp-shaded-2.7.0.jar --build --save .

This will produce a file called `graph.obj` in the same directory as the inputs. The server can then
be started later using the `--load` parameter, and will read this file instead of building the graph
from scratch:

$ java -Xmx2G -jar otp-2.6.0-shaded.jar --load .
$ java -Xmx2G -jar otp-shaded-2.7.0.jar --load .

Another reason to perform these two phases separately is that the building process loads the entire
GTFS and OSM data sets into memory, so can require significantly more memory than just running a
Expand All @@ -177,16 +177,16 @@ graph once, and then layer transit data on top of the streets to make the final
Again assuming the input files and OTP JAR file are in the current working directory, you can build
a street graph with OSM and elevation data only (ignoring transit input files) with this command:

$ java -Xmx2G -jar otp-2.6.0-shaded.jar --buildStreet .
$ java -Xmx2G -jar otp-shaded-2.7.0.jar --buildStreet .

Then, to build a graph layering transit data on top of the saved street graph (built using the
previous command):

$ java -Xmx2G -jar otp-2.6.0-shaded.jar --loadStreet --save .
$ java -Xmx2G -jar otp-shaded-2.7.0.jar --loadStreet --save .

Finally, the server can be started using the `--load` parameter:

$ java -Xmx2G -jar otp-2.6.0-shaded.jar --load .
$ java -Xmx2G -jar otp-shaded-2.7.0.jar --load .

## Command Line Switches

Expand Down
2 changes: 1 addition & 1 deletion doc/user/Developers-Guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -232,7 +232,7 @@ Releases are performed off the master branch, and are tagged with git annotated

OpenTripPlanner is currently configured such that builds including releases upload JAR files to
GitHub Packages. This is not the most convenient place for end users to find and download the files.
Therefore we also attach a stand-alone "shaded" JAR to the GitHub tag/release page, and have
Therefore, we also attach a stand-alone "shaded" JAR to the GitHub tag/release page. We have
historically also uploaded Maven artifacts to Maven Central including compiled and source code JARs
as well as the "shaded" JAR containing all dependencies, allowing stand-alone usage. This release
process is handled by the Sonatype Nexus Staging plugin, which is no longer configured in the
Expand Down
25 changes: 11 additions & 14 deletions doc/user/Getting-OTP.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@ OpenTripPlanner is distributed as a single stand-alone runnable JAR file. We cre
release page on GitHub for each release version, and also deploy them to the Maven Central
repository. You can go to
the [release pages on GitHub](https://github.com/opentripplanner/OpenTripPlanner/releases)
or [the OTP directory at Maven Central](https://repo1.maven.org/maven2/org/opentripplanner/otp/),
navigate to the highest version number, and download the file whose name ends with `shaded.jar`.
or [the OTP directory at Maven Central](https://repo1.maven.org/maven2/org/opentripplanner/otp-shaded/),
navigate to the highest version number, and download the jar file `otp-shaded-VERSION.jar`.

Note that version numbers like `v2.1.0-rc1` or `v2.6.0-SNAPSHOT` refer to development builds _
before_ the release version `v2.6.0`. The existence of a build `vX.Y.Z-SNAPSHOT` does not mean
Expand Down Expand Up @@ -64,8 +64,8 @@ OTP. If all goes well you should see a success message like the following:
[INFO] ------------------------------------------------------------------------
```

This build process should produce a JAR file called `otp-x.y.z-shaded.jar` in the
`shaded-jar/target/` directory which contains all the compiled OTP classes and their dependencies
This build process should produce a JAR file called `otp-shaded-x.y.z.jar` in the
`otp-shaded/target/` directory which contains all the compiled OTP classes and their dependencies
(the external libraries they use). The shell script called 'otp' in the root of the cloned repository
will start the main class of that JAR file under a Java virtual machine, so after the Maven build
completes you should be able to run `./otp --help` and see an OTP help message including command line
Expand All @@ -92,14 +92,11 @@ git clean -df
mvn clean package -DskipTests
```

Please note that the build process creates two distinct versions of the OTP JAR file. The one ending
in `-shaded.jar`
is much bigger because it contains copies of all the external libraries that OTP uses. It serves as
a stand-alone runnable distribution of OTP. The one with a version number but without the
word `shaded`
contains only OTP itself, without any external dependencies. This JAR is useful when OTP is included
as a component in some other project, where we want the dependency management system to gather all
the external libraries automatically.
Please note that the build process multiple jar files. The `otp-shaded-VERSION.jar` is much bigger
because it contains copies of all the external libraries that OTP uses. It serves as a stand-alone
runnable distribution of OTP. The other jar files are regular Java jar files. These JAR files are
useful when OTP is included as a component in some other project, where we want the dependency
management system to gather all the external libraries automatically.

## Maven Repository

Expand All @@ -110,8 +107,8 @@ file) to the Maven repository, from which it can be automatically included in ot

This repository is machine-readable (by Maven or other build systems) and also provides human
readable directory listings via HTTP. You can fetch an OTP JAR from this repository by constructing
the proper URL for the release you want. For example, release 2.6.0 will be found
at `https://repo1.maven.org/maven2/org/opentripplanner/otp/2.6.0/otp-2.6.0-shaded.jar`.
the proper URL for the release you want. For example, release 2.7.0 will be found
at `https://repo1.maven.org/maven2/org/opentripplanner/otp-shaded/2.7.0/otp-shaded-2.7.0-shaded.jar`.

To make use of OTP in another Maven project, you must specify it as a dependency in that
project's `pom.xml`:
Expand Down
95 changes: 49 additions & 46 deletions shaded-jar/pom.xml → otp-shaded/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,10 @@
<artifactId>otp-root</artifactId>
<version>2.7.0-SNAPSHOT</version>
</parent>
<artifactId>shaded-jar</artifactId>
<packaging>pom</packaging>
<artifactId>otp-shaded</artifactId>
<name>OpenTripPlanner - Shaded Jar</name>

<properties>
<maven.source>skip</maven.source>
<skipShadeJar>false</skipShadeJar>
</properties>

Expand All @@ -27,39 +25,25 @@
</dependencies>

<build>
<!-- finalName>otp-${project.version}-shaded</finalName -->
<plugins>
<!--
There used to be a dependency-plugin:copy-dependencies plugin entry here, but the
shade-plugin will explode the dependencies even if they aren't manually copied in.
-->
<plugin>
<!-- We want to create a standalone jar that can be run on the command
line. Java does not really allow this - you cannot place jars inside of jars.
You must either provide all the dependency jars to the user (usually lib/
under the directory containing the runnable jar) or explode all the jars
and repackage them into a single jar. The problem is that while class files
are nicely organized into the package namespace and should not collide, the
META-INF directories of the jars will collide. Maven's standard assembly
plugin does not account for this and will just clobber metadata. This then
causes runtime errors, particularly with Spring. Instead, we use the shade
plugin which has transformers that will for example append files of the same
name rather than overwrite them in the combined JAR. NB: Don't use a version
of the shade plugin older than 1.3.2, as it fixed MSHADE-76 (files not merged
properly if some input files are missing a terminating newline) -->
<groupId>org.apache.maven.plugins</groupId>
<!--
We want to create a standalone jar that can be run on the command line. Java does
not really allow this - you cannot place jars inside of jars. You must either
provide all the dependency jars to the user (usually lib/ under the directory
containing the runnable jar) or explode all the jars and repackage them into a
single jar. The problem is that while class files are nicely organized into the
package namespace and should not collide, the META-INF directories of the jars will
collide. Maven's standard assembly plugin does not account for this and will just
clobber metadata. This then causes runtime errors, particularly with Spring.
Instead, we use the shade plugin which has transformers that will for example
append files of the same name rather than overwrite them in the combined JAR.
-->
<artifactId>maven-shade-plugin</artifactId>
<configuration>
<!--
We need to set the final name for two reasons:
- We want the artifact to have another name than the maven module.
It should be "otp" not "shaded-jar"
- For some odd reason the maven-shade-plugin uses the ${project.packaging}
as the fila extension. So, without it, the file name is
"otp-SNAPSHOT-shaded.pom". Changing the packaging of the project is not
what we want either.
-->
<outputFile>${project.build.directory}/otp-${project.version}-shaded.jar</outputFile>
<skip>${skipShadeJar}</skip>
<shadedArtifactAttached>false</shadedArtifactAttached>
<createDependencyReducedPom>false</createDependencyReducedPom>
<filters>
<filter>
<!-- exclude signatures from merged JAR to avoid invalid signature messages -->
Expand All @@ -71,20 +55,6 @@
</excludes>
</filter>
</filters>
<skip>${skipShadeJar}</skip>
<!-- The shaded JAR will not be the main artifact for the project, it will be attached
for deployment in the way source and docs are. -->
<shadedArtifactAttached>true</shadedArtifactAttached>
<!-- shadedClassifierName>shaded</shadedClassifierName -->
<createDependencyReducedPom>false</createDependencyReducedPom>
<!-- MinimizeJar removes unused classes, (classes not imported explicitly by name).
We have eliminated most Jersey auto-scanning, but there is still some need for include
filters to force-include classes that are dynamically loaded by name/auto-scanned. -->
<!-- This roughly halves the size of the OTP JAR, bringing it down to around 20 MB.
<minimizeJar>true</minimizeJar>
<filters> <filter> <artifact>com.sun.jersey:*</artifact> <includes> <include>**</include>
</includes> </filter> <filter> <artifact>org.opentripplanner:*</artifact>
<includes> <include>**</include> </includes> </filter> </filters> -->
<transformers>
<transformer
implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
Expand Down Expand Up @@ -115,6 +85,39 @@
</execution>
</executions>
</plugin>


<!--
DISABLED PLUGINS


Turn off default plugins for package type jar. These plugins will normally run fine
even when there is no source, but print warnings in the Maven console log.

An alternative approach is to set "pom" as package-type and use the
build-helper-maven-plugin to attach the shaded jar as the main artifact. This also
has side effects. The biggest problem is that the published "pom.xml" does not indicate
that the main artifact is a jar file - it has package type pom.
-->

<!--
We would like to skip generating an empty jar file, but the maven-shade-plugin
fails if we set "skipIfEmpty=true". Not sure if this is a bug or if the
maven-shade-plugin needs the generated files/maven meta info from the jar plugin.
<plugin>
<artifactId>maven-jar-plugin</artifactId>
<configuration>
<skipIfEmpty>true</skipIfEmpty>
</configuration>
</plugin>
-->
<plugin>
<groupId>com.hubspot.maven.plugins</groupId>
<artifactId>prettier-maven-plugin</artifactId>
<configuration>
<skip>true</skip>
</configuration>
</plugin>
</plugins>
</build>
</project>
8 changes: 6 additions & 2 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,7 @@
<module>raptor</module>
<module>gtfs-realtime-protobuf</module>
<module>application</module>
<module>shaded-jar</module>
<module>otp-shaded</module>
</modules>


Expand All @@ -110,6 +110,11 @@
<artifactId>maven-shade-plugin</artifactId>
<version>3.6.0</version>
</plugin>
<plugin>
<groupId>com.hubspot.maven.plugins</groupId>
<artifactId>prettier-maven-plugin</artifactId>
<version>${plugin.prettier.version}</version>
</plugin>
</plugins>
</pluginManagement>
<plugins>
Expand Down Expand Up @@ -290,7 +295,6 @@
<plugin>
<groupId>com.hubspot.maven.plugins</groupId>
<artifactId>prettier-maven-plugin</artifactId>
<version>${plugin.prettier.version}</version>
<configuration>
<skip>${plugin.prettier.skip}</skip>
<prettierJavaVersion>2.0.0</prettierJavaVersion>
Expand Down
2 changes: 1 addition & 1 deletion script/otp
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,6 @@
# Standalone OTP can build a graph, run an OTP API server,
# or any combination of these.

JAR_FILE="$(dirname $0)/target/${project.build.finalName}-shaded.jar"
JAR_FILE="$(dirname $0)/../otp-shaded/target/otp-shaded-2.7.0-SNAPSHOT.jar"

java -Xmx8G -jar "${JAR_FILE}" "$@"
Loading
Loading