Specify the behaviour of TripUpdate.schedule_relationship = ADDED, and un-deprecate REPLACEMENT #504

miklcct · 2024-09-25T15:50:23Z

The use of TripUpdate.schedule_relationship = ADDED was unspecified and different producers / consumers used it in different ways. For example, it is sometimes used to specify additional departures on an existing route, but it is also used to specify departures which can't be matched to any existing trips.

This PR attempts to specify the behaviour of ADDED, and un-deprecate REPLACEMENT, based on the implementation of OpenTripPlanner which specifies the whole journey to be added or replaced. Additional fields, such as headsigns, and pickup / drop-off types, are introduced as required to support the full specification of completely new trips.

`ADDED`

In this proposal, TripUpdate.schedule_relationship = ADDED should be used to add trips which do not duplicate an existing trip. Such trips are considered to be unrelated to any existing trips in the GTFS Static and can serve an arbitrary pattern, including completely new patterns not found in the GTFS Static.

A typical use case is for relief trips for extra demand, typically after big events.

As with the current OpenTripPlanner implementation, trip_id in the TripDescriptor for added trips must be completely new (not found in GTFS static) and unique, and a start_date should also be specified as well (I am not using the word "must" here because it is permitted not to specify start_date to match scheduled trips, in this case the trip is assumed to run today).

The whole journey of the added trip must be specified, in stop order, as StopTimeUpdates inside the TripUpdate without any omission. Fields are added to TripProperties and StopTimeProperties for esssential information such as names, headsigns, pickup / drop off types.

`REPLACEMENT`

I propose to un-deprecate TripUpdate.schedule_relationship = REPLACEMENT as well. It works in the same way as ADDED, apart from that the TripDescriptor must match one instance of a scheduled trip (like other values of TripUpdate.schedule_relationship), and that instance is replaced with the complete replacement trip specified in form of StopTimeUpdates like an added trip. The original stop times in the GTFS static are not considered by the replacement trip in any form to avoid confusion. The replacement trip can serve an arbitrary pattern with an arbitrary schedule, the only expectation is that the passenger should associate the replacement trip to actually be a replacement of the original trip.

A typical use case is for short-term timetable change, or short-term (near real-time) diversion, where the fact that the trip_id remains the same can be used by journey planners to notify the user that the booked service has been changed. (In particular, I have successfully used this feature to handle real-time train diversions in GB in OpenTripPlanner and route users to alight at diverted stops, which is something neither Google Maps nor Citymapper can do now)

This is the behaviour implemented in OpenTripPlanner, which is equivalent to deleting the matched trip, and processing the replacement TripUpdate as an ADDED trip mentioned above.

Relationship to `TripModification`

TripModification provides a way to modify trips en-masse by specifying a list of trip IDs where the same detour can be applied. However, it is not suited to change the schedule on a per-trip basis, replacing the trip with a completely different schedule after any diversions with different running times (common due to pathing constraints on railways).

It should be forbidden to modify the same trip via a REPLACEMENT trip update and also via a TripModification.

…elds in TripProperties and StopTimeProperties to support fields needed for such trips

…hip=REPLACEMENT

gcamp · 2024-09-26T02:54:34Z

Nice to see some movement in that direction!

I think you used a markdown editor that changed formatting on a lot of tables which makes it hard to see the actual diff from your proposal. Would it be possible to fix that?

You put a lot in the PR description that's not actually in the proposed changes. Is that just to start the discussion? Some of it is quite consequential, like the whole journey of the added trip must be specified.

I'm a bit puzzled on how a consumer is supposed to ingest ADDED changes like this with arbitrary trips with no more information than an headsign. Which route is that on? Is those added trips supported only on existing routes in the GTFS? If the answer is no, we're getting quite close to the service change proposal : https://bit.ly/gtfs-service-changes-v3_1

leonardehrenfried · 2024-09-26T07:35:18Z

Thanks for opening this PR!

OTP has had an implementation of ADDED for a long time but its behaviour is severely underspecified. I'd love to formalise it.

Yes, OTP allows you to create completely new free form trips that have no relation to an existing pattern or trip. It tries match the given route id to an existing one but if none is in the message a dummy one is created. For once, OTP is really following the "just give us what you have, and we will try to work it out" strategy.

The only requirement we have is that the stop ids must match the static GTFS. The question is what should happen when they don't. Should the entire update be dropped or individual stops? Does that even need to be specified?

I agree with what @gcamp said about the markdown tables and the issue description.

Lastly, you might find it easier to get this through review if you split it into two PRs: one for ADDED and one for REPLACEMENT. That's just a guess though.

miklcct · 2024-09-26T08:06:27Z

I think that the requirement for the whole trip to be specified is written in the code. Let me know if it is not clear enough.

I'll fix the formatting later today.

miklcct · 2024-09-26T13:28:44Z

Nice to see some movement in that direction!

I think you used a markdown editor that changed formatting on a lot of tables which makes it hard to see the actual diff from your proposal. Would it be possible to fix that?

You put a lot in the PR description that's not actually in the proposed changes. Is that just to start the discussion? Some of it is quite consequential, like the whole journey of the added trip must be specified.

I'm a bit puzzled on how a consumer is supposed to ingest ADDED changes like this with arbitrary trips with no more information than an headsign. Which route is that on? Is those added trips supported only on existing routes in the GTFS? If the answer is no, we're getting quite close to the service change proposal : https://bit.ly/gtfs-service-changes-v3_1

"The whole journey of the added trip must be specified" is a fact in the core of this PR, noted in the updated definition of StopTimeUpdate:

Updates to StopTimes for the trip (both future, i.e., predictions, and in some cases, past ones, i.e., those that already happened). The updates must be sorted by stop_sequence, and apply for all the following stops of the trip up to the next specified stop_time_update.
If trip.schedule_relationship is SCHEDULED, at least one stop_time_update must be provided for the trip.
If trip.schedule_relationship is ADDED or REPLACEMENT, stop_time_updates must be provided for all stops in the added or replacement trip, and the stop times in the static GTFS are not used.
If the trip is canceled or deleted, no stop_time_updates need to be provided. If stop_time_updates are provided for a canceled or deleted trip then the trip.schedule_relationship takes precedence over any stop_time_updates and their associated schedule_relationship. If the trip is duplicated, stop_time_updates may be provided to indicate real-time information for the new trip.

The route and direction of the ADDED trip is specified in TripDescriptor.route_id. Sorry I didn't make it clear and I'll going to refine the PR. It should not be possible to replace a trip to work on a different route in a REPLACEMENT trip as it may confuse consumers.

…ed or replacement trips

miklcct · 2024-09-26T14:07:13Z

I do not want to specify the behaviour of missing stops at this moment because it may depend on the client's capability for dynamically adding stops via Stop messages. Theoretically the stop_id must refer to a stop in GTFS static, or a stop added via Stop messages.

leonardehrenfried · 2024-09-30T10:32:15Z

So this is pretty much a codifcation of what OTP has been supporting for several years. This would of course be very convenient for us but I would like to hear more voices from the community, in particular producers.

I know that HSL (Helsinki) is using this as both a producer and consumer (OTP) for many years.

MBTA has also indicated that they use ADDED as a producer.

@optionsome @jfabi @sam-hickey-ibigroup

gtfs-realtime/spec/en/reference.md

@leonardehrenfried

Accept suggestion by @leonardehrenfried for definition of TripUpdate.ScheduleRelationship = ADDED Co-authored-by: Leonard Ehrenfried <[email protected]>

formatting fix Co-authored-by: Leonard Ehrenfried <[email protected]>

skinkie · 2024-09-30T11:05:24Z

So this is pretty much a codifcation of what OTP has been supporting for several years. This would of course be very convenient for us but I would like to hear more voices from the community, in particular producers.

Producing it for over 12 years too.

optionsome · 2024-09-30T11:31:46Z

I know that HSL (Helsinki) is using this as both a producer and consumer (OTP) for many years.

HSL doesn't produce or consume ADDED or REPLACEMENT updates currently, if that was what you were referring to.

gtfs-realtime/proto/gtfs-realtime.proto

leonardehrenfried · 2024-09-30T16:15:29Z

What happens when you ADD a trip and then CANCEL it again? Should it be become invisible in the system (DELETED?) or show up as a CANCELLED?

miklcct · 2024-09-30T17:32:25Z

That's a good question. I still need to think about how things will work.

My producer implementation cancels an added trip using TripUpdate.schedule_relationship = ADDED with all StopTimeUpdate having a SKIPPED relationship.

The questions are that:

What if a later version of the full dataset real time feed doesn't contain the ADDED feed? (My intention is that it no longer exists and should be considered as DELETED. A GTFS-RT full dataset should only be applied into the original static data.)
How do I cancel an ADDED trip? I think that a TripUpdate with a trip id not found in static GTFS, schedule_relationship = CANCELED and the original planned stops marked with StopTimeUpdate.schedule_relationship = 'SKIPPED' makes it clear that the trip was added then cancelled. (Use DELETED instead of CANCELED to hide it from the board)

skinkie · 2024-09-30T17:35:05Z

As we are considering FULL_DATASET shouldn't that just replace?

leonardehrenfried · 2024-10-01T06:44:57Z

Actually, @skinkie is right. If you use FULL_DATASET then the moment you fetch the new version of the RT feed the old ADDED trip will completely vanish and it neither exists as DELETED nor CANCELLED. It's like it never existed.

However, once there is movement towards specifying INCREMENTAL we will have to revisit this.

leonardehrenfried · 2024-10-01T06:45:52Z

Does anyone know how Google and Apple handle ADDED?

@bdferris-v2

@eliasmbd I don't know who the relevant person from Apple would be. Could you tag them?

skinkie · 2024-10-01T07:21:20Z

@leonardehrenfried at Google the thing was limited to stop sequences previously observed. Hence if the ADDED trip was an instance of a stop sequence that is part of the database, it could be processed. I don't know if it is already capable of processing a partial instance of a stop sequence.

https://support.google.com/transitpartners/answer/10106497?hl=en#zippy=%2Cadd-with-tripupdates

Co-authored-by: Guillaume Campagna <[email protected]>

…m the change

…placement

miklcct · 2024-12-03T16:17:14Z

I am now in the process of making a producer and will soon update opentripplanner/OpenTripPlanner#6028 to be a consumer of this, such that real-time added train trips and train diversions in Great Britain can be shown correctly in journey planners with the correct headsigns (OTP currently leaves the headsign empty for added trips and uses the original headsign for replacement trips).

After I complete this I will update and post the URL of a demonstration back end.

miklcct · 2024-12-03T16:39:19Z

The below is an example complete journey for a replacement trip with real data which my producer is now generating, including the use of the fields that I propose to add (trip_headsign = "Sutton", trip_short_name = "TL2736", and stop_headsign for some intermediate stations, and also pickup_type and drop_off_type as well, with the MFDZ extension still here for compatibility).

entity {
  id: "MODIFY_G50960_20240603_20241213_ON_20241203"
  trip_update {
    trip {
      trip_id: "G50960_20240603_20241213"
      start_time: "12:07:00"
      start_date: "20241203"
      schedule_relationship: REPLACEMENT
      route_id: "TL"
    }
    stop_time_update {
      stop_sequence: 0
      arrival {
        delay: 0
        time: 1733227620
        scheduled_time: 1733227620
      }
      departure {
        delay: 0
        time: 1733227620
        scheduled_time: 1733227620
      }
      stop_id: "9100STALBCY1"
      stop_time_properties {
        stop_headsign: "Sutton (via Wimbledon)"
        pickup_type: REGULAR
        drop_off_type: REGULAR
        [transit_realtime.stop_time_properties] {
          pickup_type: REGULAR
          dropoff_type: REGULAR
        }
      }
    }
    stop_time_update {
      stop_sequence: 1
      arrival {
        delay: -60
        time: 1733227860
        scheduled_time: 1733227920
      }
      departure {
        delay: 0
        time: 1733227920
        scheduled_time: 1733227920
      }
      stop_id: "9100RADLETT1"
      stop_time_properties {
        stop_headsign: "Sutton (via Wimbledon)"
        pickup_type: REGULAR
        drop_off_type: REGULAR
        [transit_realtime.stop_time_properties] {
          pickup_type: REGULAR
          dropoff_type: REGULAR
        }
      }
    }
    stop_time_update {
      stop_sequence: 2
      arrival {
        delay: -60
        time: 1733228100
        scheduled_time: 1733228160
      }
      departure {
        delay: 0
        time: 1733228160
        scheduled_time: 1733228160
      }
      stop_id: "9100ELTR1"
      stop_time_properties {
        stop_headsign: "Sutton (via Wimbledon)"
        pickup_type: REGULAR
        drop_off_type: REGULAR
        [transit_realtime.stop_time_properties] {
          pickup_type: REGULAR
          dropoff_type: REGULAR
        }
      }
    }
    stop_time_update {
      stop_sequence: 3
      arrival {
        delay: -60
        time: 1733228340
        scheduled_time: 1733228400
      }
      departure {
        delay: 0
        time: 1733228400
        scheduled_time: 1733228400
      }
      stop_id: "9100MLHB1"
      stop_time_properties {
        stop_headsign: "Sutton (via Wimbledon)"
        pickup_type: REGULAR
        drop_off_type: REGULAR
        [transit_realtime.stop_time_properties] {
          pickup_type: REGULAR
          dropoff_type: REGULAR
        }
      }
    }
    stop_time_update {
      stop_sequence: 4
      arrival {
        delay: -120
        time: 1733228520
        scheduled_time: 1733228640
      }
      departure {
        delay: 0
        time: 1733228640
        scheduled_time: 1733228640
      }
      stop_id: "9100HDON1"
      stop_time_properties {
        stop_headsign: "Sutton (via Wimbledon)"
        pickup_type: REGULAR
        drop_off_type: REGULAR
        [transit_realtime.stop_time_properties] {
          pickup_type: REGULAR
          dropoff_type: REGULAR
        }
      }
    }
    stop_time_update {
      stop_sequence: 5
      arrival {
        delay: -60
        time: 1733228700
        scheduled_time: 1733228760
      }
      departure {
        delay: 0
        time: 1733228760
        scheduled_time: 1733228760
      }
      stop_id: "9100BRENTX1"
      stop_time_properties {
        pickup_type: REGULAR
        drop_off_type: REGULAR
        [transit_realtime.stop_time_properties] {
          pickup_type: REGULAR
          dropoff_type: REGULAR
        }
      }
    }
    stop_time_update {
      stop_sequence: 6
      arrival {
        delay: 0
        time: 1733228880
        scheduled_time: 1733228880
      }
      departure {
        delay: -60
        time: 1733228880
        scheduled_time: 1733228940
      }
      stop_id: "9100CRKLWD1"
      stop_time_properties {
        stop_headsign: "Sutton (via Wimbledon)"
        pickup_type: REGULAR
        drop_off_type: REGULAR
        [transit_realtime.stop_time_properties] {
          pickup_type: REGULAR
          dropoff_type: REGULAR
        }
      }
    }
    stop_time_update {
      stop_sequence: 7
      arrival {
        delay: 0
        time: 1733229060
        scheduled_time: 1733229060
      }
      departure {
        delay: 0
        time: 1733229120
        scheduled_time: 1733229120
      }
      stop_id: "9100WHMPSTM1"
      stop_time_properties {
        stop_headsign: "Sutton (via Wimbledon)"
        pickup_type: REGULAR
        drop_off_type: REGULAR
        [transit_realtime.stop_time_properties] {
          pickup_type: REGULAR
          dropoff_type: REGULAR
        }
      }
    }
    stop_time_update {
      stop_sequence: 8
      arrival {
        delay: -60
        time: 1733229360
        scheduled_time: 1733229420
      }
      departure {
        delay: 0
        time: 1733229420
        scheduled_time: 1733229420
      }
      stop_id: "9100KNTSHTN1"
      stop_time_properties {
        stop_headsign: "Sutton (via Wimbledon)"
        pickup_type: REGULAR
        drop_off_type: REGULAR
        [transit_realtime.stop_time_properties] {
          pickup_type: REGULAR
          dropoff_type: REGULAR
        }
      }
    }
    stop_time_update {
      stop_sequence: 9
      arrival {
        delay: -120
        time: 1733229600
        scheduled_time: 1733229720
      }
      departure {
        delay: -60
        time: 1733229660
        scheduled_time: 1733229720
      }
      stop_id: "9100STPXBOXA"
      stop_time_properties {
        pickup_type: REGULAR
        drop_off_type: REGULAR
        [transit_realtime.stop_time_properties] {
          pickup_type: REGULAR
          dropoff_type: REGULAR
        }
      }
    }
    stop_time_update {
      stop_sequence: 10
      arrival {
        delay: -60
        time: 1733229900
        scheduled_time: 1733229960
      }
      departure {
        delay: 0
        time: 1733230020
        scheduled_time: 1733230020
      }
      stop_id: "9100FRNDNLT3"
      stop_time_properties {
        stop_headsign: "Sutton (via Wimbledon)"
        pickup_type: REGULAR
        drop_off_type: REGULAR
        [transit_realtime.stop_time_properties] {
          pickup_type: REGULAR
          dropoff_type: REGULAR
        }
      }
    }
    stop_time_update {
      stop_sequence: 11
      arrival {
        delay: 0
        time: 1733230080
        scheduled_time: 1733230080
      }
      departure {
        delay: 60
        time: 1733230200
        scheduled_time: 1733230140
      }
      stop_id: "9100CTMSLNK2"
      stop_time_properties {
        stop_headsign: "Sutton (via Wimbledon)"
        pickup_type: REGULAR
        drop_off_type: REGULAR
        [transit_realtime.stop_time_properties] {
          pickup_type: REGULAR
          dropoff_type: REGULAR
        }
      }
    }
    stop_time_update {
      stop_sequence: 12
      arrival {
        delay: -60
        time: 1733230200
        scheduled_time: 1733230260
      }
      departure {
        delay: 0
        time: 1733230260
        scheduled_time: 1733230260
      }
      stop_id: "9100BLFR1"
      stop_time_properties {
        stop_headsign: "Sutton (via Wimbledon)"
        pickup_type: REGULAR
        drop_off_type: REGULAR
        [transit_realtime.stop_time_properties] {
          pickup_type: REGULAR
          dropoff_type: REGULAR
        }
      }
    }
    stop_time_update {
      stop_sequence: 13
      arrival {
        delay: -60
        time: 1733230440
        scheduled_time: 1733230500
      }
      departure {
        delay: 0
        time: 1733230500
        scheduled_time: 1733230500
      }
      stop_id: "9100ELPHNAC4"
      stop_time_properties {
        stop_headsign: "Sutton (via Wimbledon)"
        pickup_type: REGULAR
        drop_off_type: REGULAR
        [transit_realtime.stop_time_properties] {
          pickup_type: REGULAR
          dropoff_type: REGULAR
        }
      }
    }
    stop_time_update {
      stop_sequence: 14
      arrival {
        delay: -180
        time: 1733230620
        scheduled_time: 1733230800
      }
      departure {
        delay: 0
        time: 1733230800
        scheduled_time: 1733230800
      }
      stop_id: "9100LBGHJN2"
      stop_time_properties {
        stop_headsign: "Sutton (via Wimbledon)"
        pickup_type: REGULAR
        drop_off_type: REGULAR
        [transit_realtime.stop_time_properties] {
          pickup_type: REGULAR
          dropoff_type: REGULAR
        }
      }
    }
    stop_time_update {
      stop_sequence: 15
      arrival {
        delay: -120
        time: 1733230920
        scheduled_time: 1733231040
      }
      departure {
        delay: 0
        time: 1733231100
        scheduled_time: 1733231100
      }
      stop_id: "9100HERNEH4"
      stop_time_properties {
        stop_headsign: "Sutton (via Wimbledon)"
        pickup_type: REGULAR
        drop_off_type: REGULAR
        [transit_realtime.stop_time_properties] {
          pickup_type: REGULAR
          dropoff_type: REGULAR
        }
      }
    }
    stop_time_update {
      stop_sequence: 16
      arrival {
        delay: -60
        time: 1733231280
        scheduled_time: 1733231340
      }
      departure {
        delay: 0
        time: 1733231400
        scheduled_time: 1733231400
      }
      stop_id: "9100TULSEH2"
      stop_time_properties {
        stop_headsign: "Sutton (via Wimbledon)"
        pickup_type: REGULAR
        drop_off_type: REGULAR
        [transit_realtime.stop_time_properties] {
          pickup_type: REGULAR
          dropoff_type: REGULAR
        }
      }
    }
    stop_time_update {
      stop_sequence: 17
      arrival {
        delay: -60
        time: 1733231580
        scheduled_time: 1733231640
      }
      departure {
        delay: 0
        time: 1733231640
        scheduled_time: 1733231640
      }
      stop_id: "9100STRETHM1"
      stop_time_properties {
        stop_headsign: "Sutton (via Wimbledon)"
        pickup_type: REGULAR
        drop_off_type: REGULAR
        [transit_realtime.stop_time_properties] {
          pickup_type: REGULAR
          dropoff_type: REGULAR
        }
      }
    }
    stop_time_update {
      stop_sequence: 18
      arrival {
        delay: 0
        time: 1733231880
        scheduled_time: 1733231880
      }
      departure {
        delay: -60
        time: 1733231880
        scheduled_time: 1733231940
      }
      stop_id: "9100TOOTING2"
      stop_time_properties {
        stop_headsign: "Sutton (via Wimbledon)"
        pickup_type: REGULAR
        drop_off_type: REGULAR
        [transit_realtime.stop_time_properties] {
          pickup_type: REGULAR
          dropoff_type: REGULAR
        }
      }
    }
    stop_time_update {
      stop_sequence: 19
      arrival {
        delay: -60
        time: 1733232060
        scheduled_time: 1733232120
      }
      departure {
        delay: 0
        time: 1733232120
        scheduled_time: 1733232120
      }
      stop_id: "9100HYDNSRD2"
      stop_time_properties {
        stop_headsign: "Sutton (via Wimbledon)"
        pickup_type: REGULAR
        drop_off_type: REGULAR
        [transit_realtime.stop_time_properties] {
          pickup_type: REGULAR
          dropoff_type: REGULAR
        }
      }
    }
    stop_time_update {
      stop_sequence: 20
      arrival {
        delay: 0
        time: 1733232240
        scheduled_time: 1733232240
      }
      departure {
        delay: 0
        time: 1733232300
        scheduled_time: 1733232300
      }
      stop_id: "9100WIMBLDN9"
      stop_time_properties {
        pickup_type: REGULAR
        drop_off_type: REGULAR
        [transit_realtime.stop_time_properties] {
          pickup_type: REGULAR
          dropoff_type: REGULAR
        }
      }
    }
    stop_time_update {
      stop_sequence: 21
      arrival {
        delay: -60
        time: 1733232420
        scheduled_time: 1733232480
      }
      departure {
        delay: 0
        time: 1733232480
        scheduled_time: 1733232480
      }
      stop_id: "9100WIMLCHS2"
      stop_time_properties {
        pickup_type: REGULAR
        drop_off_type: REGULAR
        [transit_realtime.stop_time_properties] {
          pickup_type: REGULAR
          dropoff_type: REGULAR
        }
      }
    }
    stop_time_update {
      stop_sequence: 22
      arrival {
        delay: -60
        time: 1733232540
        scheduled_time: 1733232600
      }
      departure {
        delay: 0
        time: 1733232600
        scheduled_time: 1733232600
      }
      stop_id: "9100SMERTON2"
      stop_time_properties {
        pickup_type: REGULAR
        drop_off_type: REGULAR
        [transit_realtime.stop_time_properties] {
          pickup_type: REGULAR
          dropoff_type: REGULAR
        }
      }
    }
    stop_time_update {
      stop_sequence: 23
      arrival {
        delay: 0
        time: 1733232780
        scheduled_time: 1733232780
      }
      departure {
        delay: 0
        time: 1733232780
        scheduled_time: 1733232780
      }
      stop_id: "9100MORDENS2"
      stop_time_properties {
        pickup_type: REGULAR
        drop_off_type: REGULAR
        [transit_realtime.stop_time_properties] {
          pickup_type: REGULAR
          dropoff_type: REGULAR
        }
      }
    }
    stop_time_update {
      stop_sequence: 24
      arrival {
        delay: 0
        time: 1733232900
        scheduled_time: 1733232900
      }
      departure {
        delay: 0
        time: 1733232900
        scheduled_time: 1733232900
      }
      stop_id: "9100SHLIER2"
      stop_time_properties {
        pickup_type: REGULAR
        drop_off_type: REGULAR
        [transit_realtime.stop_time_properties] {
          pickup_type: REGULAR
          dropoff_type: REGULAR
        }
      }
    }
    stop_time_update {
      stop_sequence: 25
      arrival {
        delay: 0
        time: 1733233020
        scheduled_time: 1733233020
      }
      departure {
        delay: 0
        time: 1733233020
        scheduled_time: 1733233020
      }
      stop_id: "9100SUTTONC2"
      stop_time_properties {
        pickup_type: REGULAR
        drop_off_type: REGULAR
        [transit_realtime.stop_time_properties] {
          pickup_type: REGULAR
          dropoff_type: REGULAR
        }
      }
    }
    stop_time_update {
      stop_sequence: 26
      arrival {
        delay: -60
        time: 1733233140
        scheduled_time: 1733233200
      }
      departure {
        delay: 0
        time: 1733233200
        scheduled_time: 1733233200
      }
      stop_id: "9100WSUTTON2"
      stop_time_properties {
        pickup_type: REGULAR
        drop_off_type: REGULAR
        [transit_realtime.stop_time_properties] {
          pickup_type: REGULAR
          dropoff_type: REGULAR
        }
      }
    }
    stop_time_update {
      stop_sequence: 27
      arrival {
        delay: 0
        time: 1733233440
        scheduled_time: 1733233440
      }
      departure {
        delay: 0
        time: 1733233440
        scheduled_time: 1733233440
      }
      stop_id: "9100SUTTON1"
      stop_time_properties {
        pickup_type: REGULAR
        drop_off_type: REGULAR
        [transit_realtime.stop_time_properties] {
          pickup_type: REGULAR
          dropoff_type: REGULAR
        }
      }
    }
    trip_properties {
      trip_headsign: "Sutton"
      trip_short_name: "TL2736"
    }
  }
}

miklcct · 2024-12-04T18:24:04Z

opentripplanner/OpenTripPlanner#6028 is now ready for demonstration, and our test server at https://test.open-trip-planner.jnction.co.uk/ should be ready by tomorrow morning for consumption of our updated RT feed.

If there are any train diversions / truncations in the GB National Rail network, they should start showing the correct headsigns (according to National Rail Enquiries) tomorrow.

leonardehrenfried · 2024-12-04T20:23:11Z

This is now stripped down to the essentials that @miklcct is actually using. I can also confirm that OTP had an implementation of these fields for many years and I'm glad it's now formally proposed to be added to the standard.

I'm happy to recommend a vote on this.

skinkie · 2024-12-04T21:00:14Z

@Arilith @sven4all

miklcct · 2024-12-04T22:04:42Z

Can anyone please tag NSW Government (Sydney Trains)? They are using replacement trips for diverted trains as well.

https://opendataforum.transport.nsw.gov.au/t/real-time-trip-update-missing-schedule-relationship-field/1537

In particular, do they still apply the pre-2014 insanity that the portion of the trip not specified in the update is still taken from the static schedule?

Quoting an old discussion back then:

My Proposal;

For every routepattern alternation, skipped or added we propose the entire
routepattern and alterations must be published by producer.

TripDescriptor.Schedule_Relationship = REPLACEMENT

a-b-c becomes a-b-d

A time=1 SCHEDULED
B time=2 SCHEDULED
C SKIPPED
D time=4 SCHEDULED

a-b-c becomes a-b (the shorten)
A time=1 SCHEDULED
B time=2 SCHEDULED
C SKIPPED

a-b-c becomes z-a-b-c (changed origin)
Z time=1 SCHEDULED
A time=2 SCHEDULED
B time=3 SCHEDULED
C time=4 SCHEDULED

This would make the life of a consumer more easy and effectively drops this
kind of insanity:

"The replacement applies only to the portion of the trip supplied. For
instance, consider a route that goes through stops A,B,C,D,E,F, and a
REPLACEMENT trip provides data for stops A,B,C. Then, the times for stops
D,E,F are still taken from the static schedule."

...while it is perfectly backwards compatible :*)

Stefan

@skinkie do you think I should add the above into the best practice, to recommend listing the skipped stops for replacement trips as well, such that

The passengers are told that the service no longer calls at the original stop
The feed remains compatible with the pre-2014 spec.

Without these skipped stops, this proposal is incompatible with the pre-2014 specification.

skinkie · 2024-12-04T23:04:31Z

@miklcct The 'skipped' discussion is really something where everyone has a different view on, from both technical as traveler perspective. I think a consuming journey planner is interested in the final situation (read: the actual replacement), hence not having to compute the final situation itself is a win. Skipped implies that the consumer has to remove something first. The flip side of the coin is skipped from the traveler perspective, where skipped in essence becomes a service alert on a stop given a specific trip. There are routing engines which do a two pass effort on scheduled and then realtime and with skipped would be able to tag why a route is not taken.

miklcct · 2024-12-05T00:03:55Z

@miklcct The 'skipped' discussion is really something where everyone has a different view on, from both technical as traveler perspective. I think a consuming journey planner is interested in the final situation (read: the actual replacement), hence not having to compute the final situation itself is a win. Skipped implies that the consumer has to remove something first. The flip side of the coin is skipped from the traveler perspective, where skipped in essence becomes a service alert on a stop given a specific trip. There are routing engines which do a two pass effort on scheduled and then realtime and with skipped would be able to tag why a route is not taken.

For a scheduled service, a skipped stop does mean that it needs to be removed from the schedule. However, for added or replacement trips, we specifically instruct the updater not to look at the static schedule at all (for replacement trips, the only thing required is that the original is scheduled to run, and the original stop list is completely disregarded), so the notion of "removing the stop from the schedule" is null and void.

A two pass routing engine will not be process added trips anyway because they are, by definition, not in the schedule.

An added trip with skipped stop can also represent a situation that, for example, an event special is added to the feed one day before a major event and runs A-B-C-D-X, but it decides to skip D on the day because it is overcrowded.

skinkie · 2024-12-06T17:19:21Z

we specifically instruct the updater not to look at the static schedule at all

That has been a change. I think I do want to know what it replaces.

miklcct · 2024-12-06T17:22:51Z

we specifically instruct the updater not to look at the static schedule at all

That has been a change. I think I do want to know what it replaces.

I have never changed this PR w.r.t. the behaviour of replacement journeys. From the very beginning this PR is written in a way that, one running trip is matched, and the original pattern is thrown away and replaced with the complete sequence of stops in the update, just like an added trip.

According to the behaviour in this PR, the original stops for a replaced trip will simply disappear after the replacement is made, unless they are specified in the update.

This is a change from the pre-2014 behaviour, and your recommendation to producers can make feeds compatible with both the pre-2014 behaviour and the behaviour of this PR.

OpenTripPlanner has never supported the pre-2014 behaviour in regard to the replacement trips.

tzujenchanmbd · 2024-12-11T05:17:08Z

Recently we’ve heard from the community about the need for best practices regarding when to use static vs. realtime data. The changes included in this PR (such as adding new trips in real-time) seem to underscore the importance of such guidance. While the goal of ADDED is to address "unscheduled trips", could it be that producers over-rely on ADDED and use it in situations where static data updates should be applied?

(Recent Issue #512, while not directly related to this proposal, illustrates a case of "misuse" of realtime data. The Caltrans report pointed out that some producers rely on service alerts to present "scheduled" holiday services.)

The currently proposed wording for ADDED is: "An extra trip unrelated to any existing trips, for example, to respond to sudden passenger load."

Curious if the community is interested in further discussing here the timing of using ADDED versus updating trips in static data and adding details on best practices? (Issue #113 seems to have had relevant discussions on this.)

Arilith · 2024-12-11T09:45:46Z

Personally I'm greatly in favour of developing the specification for both ADDED and REPLACEMENT trips further. Without the current implementation in OTP for those relations, we (in The Netherlands) would not be able to correctly utilise GTFS-RT for realtime information for our rail network.

Maybe a few examples of our current use-cases will help shed some light on the situation:

Currently, for just the next three days there's 124 trips that could make use of the ADDED schedulerelationship, and 1235! REPLACEMENT trips. These are mostly trips where one journey has been split into two separate parts due to a disruption along the route.

E.g. Journey 1 is Station A - B - C - D - E - F normally.
There's a big disruption at station C.
Journey 1 is modified using SCHEDULED (or REPLACEMENT in the case of platform changes) into Journey 1a with the original trip number and halting at stations A - B and using SKIPPED for C, D, E, F.
Journey 1b is created using ADDED matching onto the original route with a new trip_id, but only halting at stations D - E - F. Using SKIPPED for A - B - C.

This is an extremely common use case in The Netherlands ands happens tens to hundreds of times per day.

Besides, the ADDED relationship is also relatively commonly used for rail replacement services that were unplanned (e.g. busses during extremely big disruptions)

Additionally, we also make great use of the REPLACEMENT trips for indicating platform changes or small diversions. Without using both ADDED and / or REPLACEMENT, platform changes would not work in our current implementation of OpenTripPlanner.

See this file of my GTFS-RT generating code to get a quick overview of some of the hoops we jump through to get GTFS-RT clients to accept 99.95+% of our raw train journey updates.

miklcct · 2024-12-11T14:32:31Z

Recently we’ve heard from the community about the need for best practices regarding when to use static vs. realtime data. The changes included in this PR (such as adding new trips in real-time) seem to underscore the importance of such guidance. While the goal of ADDED is to address "unscheduled trips", could it be that producers over-rely on ADDED and use it in situations where static data updates should be applied?

(Recent Issue #512, while not directly related to this proposal, illustrates a case of "misuse" of realtime data. The Caltrans report pointed out that some producers rely on service alerts to present "scheduled" holiday services.)

The currently proposed wording for ADDED is: "An extra trip unrelated to any existing trips, for example, to respond to sudden passenger load."

Curious if the community is interested in further discussing here the timing of using ADDED versus updating trips in static data and adding details on best practices? (Issue #113 seems to have had relevant discussions on this.)

From my understanding, changes known at least a week in advance should be put in the static, and a day or less should be in the RT only. Can someone please point me out if this is recorded in the best practices?

For example, there is a landslide and an emergency timetable has been applied as a result. This is clearly a RT scenario and the emergency timetable should be ADDED with the original DELETED. However, in the case of non-urgent track defect found on Monday and a closure is decided to take place on the coming Sunday, a new static file should be published to give advance information to passengers.

Sergiodero · 2024-12-11T21:51:00Z

Hi @miklcct!

We noticed the announcement in the GTFS-Realtime Google Group regarding the intention to open a vote next week. While we appreciate the effort and initiative to advance the proposal, from our end at MobilityData we would like to recommend postponing the vote until at least after the holiday season.

Opening the vote now might result in lower participation and fewer reviews, as many community members may be away or have limited availability during this time. We believe this timing adjustment could help ensure a more inclusive and robust decision-making process.

On a related note, could you please clarify which organizations are acting as the first consumer and first producer for this proposal (and the person who represents each of them)? Clarifying this could help confirm the status of the proposal before opening a vote and ensure the change is well-supported by the community. Typically, these roles are fulfilled by different individuals or organizations.

skinkie · 2024-12-11T22:17:36Z

@Arilith I think we can act as both consumer and producer right?

leonardehrenfried · 2024-12-12T07:10:55Z

While it probably works to use REPLACEMENT for platform changes, I think it's worth pointing out that there is also assigned_stop_id to model it without having to replace the entire trip.

skinkie · 2024-12-12T08:12:04Z

While it probably works to use REPLACEMENT for platform changes, I think it's worth pointing out that there is also assigned_stop_id to model it without having to replace the entire trip.

Worth mentioning that for example for the repository owner a platform change historically only works if the replacement platform has an equal parent_station.

miklcct · 2024-12-12T09:13:29Z

While it probably works to use REPLACEMENT for platform changes, I think it's worth pointing out that there is also assigned_stop_id to model it without having to replace the entire trip.

Worth mentioning that for example for the repository owner a platform change historically only works if the replacement platform has an equal parent_station.

This requirement has since been dropped, with the newest requirement that it shouldn't result in a significant change in journey experience for the passenger (i.e. the passenger should treat that as a day-to-day operation detail rather than something not expected normally).

skinkie · 2024-12-12T09:25:31Z

This requirement has since been dropped, with the newest requirement that it shouldn't result in a significant change in journey experience for the passenger (i.e. the passenger should treat that as a day-to-day operation detail rather than something not expected normally).

The standard dropped it or did the consumer implement it?

miklcct · 2024-12-12T09:30:09Z

There is no longer a strict requirement that the assigned_stop_id must be in the same station in the standard.

miklcct · 2024-12-12T16:55:01Z

While it probably works to use REPLACEMENT for platform changes, I think it's worth pointing out that there is also assigned_stop_id to model it without having to replace the entire trip.

This is something OpenTripPlanner doesn't support yet, and I plan to work on it if I have spare time.

miklcct added 3 commits September 25, 2024 16:17

specify ADDED and REPLACEMENT schedule relationships, and add more fi…

dc181de

…elds in TripProperties and StopTimeProperties to support fields needed for such trips

remove the recommendation that ADDED is not specified

b439cf8

disallow TripModification on trips with TripUpdate.schedule_relations…

5a9853f

…hip=REPLACEMENT

eliasmbd added GTFS Realtime Issues and Pull Requests that focus on GTFS Realtime Status: Discussion Issues and Pull Requests that are currently being discussed and reviewed by the community. Support: Needs Review Needs support to review proposal. Support: Needs Feedback labels Sep 25, 2024

revert formatting

21ccde5

miklcct added 3 commits September 26, 2024 14:39

specify that route_id is required for added trips

b6d07cc

Clarify the use of StopTimeUpdate to specify the whole journey in add…

6bc14c8

…ed or replacement trips

Clarify the time and delay for added and replacement trips

87d28c7

leonardehrenfried reviewed Sep 30, 2024

View reviewed changes

gtfs-realtime/spec/en/reference.md Outdated Show resolved Hide resolved

leonardehrenfried reviewed Sep 30, 2024

View reviewed changes

gtfs-realtime/spec/en/reference.md Outdated Show resolved Hide resolved

miklcct and others added 2 commits September 30, 2024 11:47

Update gtfs-realtime/spec/en/reference.md

991caac

Accept suggestion by @leonardehrenfried for definition of TripUpdate.ScheduleRelationship = ADDED Co-authored-by: Leonard Ehrenfried <[email protected]>

Update gtfs-realtime/spec/en/reference.md

fea9f81

formatting fix Co-authored-by: Leonard Ehrenfried <[email protected]>

leonardehrenfried reviewed Sep 30, 2024

View reviewed changes

gtfs-realtime/proto/gtfs-realtime.proto Outdated Show resolved Hide resolved

Update gtfs-realtime/spec/en/reference.md

ade68ae

Co-authored-by: Guillaume Campagna <[email protected]>

miklcct force-pushed the added-replacement branch 2 times, most recently from aba2a4d to c7c47b4 Compare December 3, 2024 12:15

miklcct added 2 commits December 3, 2024 12:17

remove reference to timepoint for scheduled time as it is removed fro…

2f019d8

…m the change

Merge commit 'ade68aecf431db49d3bcbc55dcb43090dfd0b0dc' into added-re…

555a619

…placement

miklcct force-pushed the added-replacement branch from c7c47b4 to 555a619 Compare December 3, 2024 12:18

miklcct requested review from leonardehrenfried and gcamp December 4, 2024 18:33

miklcct mentioned this pull request Dec 4, 2024

OTP gives "INVALID_STOP_SEQUENCE" error for "overcomplete" GTFS-RT data opentripplanner/OpenTripPlanner#4702

Closed

Specify the behaviour of TripUpdate.schedule_relationship = ADDED, and un-deprecate REPLACEMENT #504

Are you sure you want to change the base?

Specify the behaviour of TripUpdate.schedule_relationship = ADDED, and un-deprecate REPLACEMENT #504

Conversation

miklcct commented Sep 25, 2024

ADDED

REPLACEMENT

Relationship to TripModification

gcamp commented Sep 26, 2024

leonardehrenfried commented Sep 26, 2024 • edited Loading

miklcct commented Sep 26, 2024

miklcct commented Sep 26, 2024

miklcct commented Sep 26, 2024

leonardehrenfried commented Sep 30, 2024

skinkie commented Sep 30, 2024

optionsome commented Sep 30, 2024

leonardehrenfried commented Sep 30, 2024

miklcct commented Sep 30, 2024

skinkie commented Sep 30, 2024

leonardehrenfried commented Oct 1, 2024

leonardehrenfried commented Oct 1, 2024 • edited Loading

skinkie commented Oct 1, 2024

miklcct commented Dec 3, 2024 • edited Loading

miklcct commented Dec 3, 2024

miklcct commented Dec 4, 2024 • edited Loading

leonardehrenfried commented Dec 4, 2024

skinkie commented Dec 4, 2024

miklcct commented Dec 4, 2024 • edited Loading

skinkie commented Dec 4, 2024

miklcct commented Dec 5, 2024

skinkie commented Dec 6, 2024

miklcct commented Dec 6, 2024 • edited Loading

tzujenchanmbd commented Dec 11, 2024 • edited Loading

Arilith commented Dec 11, 2024

miklcct commented Dec 11, 2024

Sergiodero commented Dec 11, 2024

skinkie commented Dec 11, 2024

leonardehrenfried commented Dec 12, 2024

skinkie commented Dec 12, 2024

miklcct commented Dec 12, 2024

skinkie commented Dec 12, 2024

miklcct commented Dec 12, 2024 • edited Loading

miklcct commented Dec 12, 2024

`ADDED`

`REPLACEMENT`

Relationship to `TripModification`

leonardehrenfried commented Sep 26, 2024 •

edited

Loading

leonardehrenfried commented Oct 1, 2024 •

edited

Loading

miklcct commented Dec 3, 2024 •

edited

Loading

miklcct commented Dec 4, 2024 •

edited

Loading

miklcct commented Dec 4, 2024 •

edited

Loading

miklcct commented Dec 6, 2024 •

edited

Loading

tzujenchanmbd commented Dec 11, 2024 •

edited

Loading

miklcct commented Dec 12, 2024 •

edited

Loading