GTFS transformer seems to suppress pickup_type and drop_off_type

[sebastianknopf]

Summary:

I am using the gtfs-transformer-cli for running several transformations. After transforming my GTFS feed, the columns pickup_type, drop_off_type and shape_dist_traveled are missing in the transformed feed. Even if I am running to transformations which should affect stop times at all.

Steps to reproduce:

See the following files attached:

• gtfs-static.zip > The original GTFS-Feed

gtfs-static-transformations.txt > The file containing the transformations

gtfs-static-transformed.zip > The finally transformed GTFS feed

• gtfs-static-transformations-full.txt > A file containing several more transformations which are currently done in my production environment

Expected behavior:

The columns are not removed.

Observed behavior:
The columns mentioned above are removed entirely, maybe some other columns may be affected too.

Platform:
Test Environment: Windows 10 using Java 17.0.6
Prod Environment: Ubuntu 24.04 using Java in a Docker container (exoplatform/jdk)

[leonardehrenfried]

Hello, I'm the (relatively) new maintainer of this library.

I have not looked into it but that sounds like a bug.

Are you interested in contributing a fix?

[sebastianknopf]

Hi Leonard,

thank you for ypur reply. I will take a deeper look into the code and debug it as soon I will have a little time left up for it.

If I find a possible bug, I will report it here and then we can discuss a possible solution.

[sebastianknopf]

It seems like have found the cause for removing the columns. If you take a look into org.onebusaway.gtfs.serialization.GtfsWriter, you will find call to the method excludeOptionalAndMissingFields(...) in line 80. This call is responsible for removing all columns containing only default or empty values based on the GTFS model.

I have done follwing tests:

• Provide a GTFS feed with pickup_type=1 set in stop_times.txt > Works fine, pickup_type is not removed anymore
• Commenting the mentioned line 80 > Works also fine

In my opinion, this is not a real bug in onebusaway-gtfs-modules, but in my own converter for providing GTFS data :-) Normally, the output should contain some lines with pickup_type other than default / 0.

Anyway, what do you think about an additional flag as option in command line and the API for specifying whether the empty or default columns should be removed or not?

[leonardehrenfried]

If it's covered with unit tests, I would be ok with it.

[sebastianknopf]

Thank you for your reply. I will try an approach as soon as I have a little more time for it.