BART On-Time Metric

I used to use BART quite a lot back in the day and despite the fact that often trains would be cancelled or whatever, transit enthusiasts and BART itself would frequently publish an on-time percentage that seemed suspiciously high. I was curious what was going on here so that I could reconstruct what was going on and so I opened a CPRA request where I attempted to get to the bottom of this. I think I might have some understanding of what is going on here and I have an example of a day where the discrepancy between my experience and the reported metric is explained.

The Metric

There are many ways one can define a train being on time. As an example of why this is needed, consider a system with short headways, i.e. 5 min or so. Consider there was an issue at 0800 and the train scheduled for that time instead showed up at 0805 and then all trains thereafter were every five minutes. This would show as 0% punctuality in a naïve train number to scheduled time sense, but no rider would interpret it as that. So when you come up with an on-time metric, you sort of need to choose one that accurately represents the rider's experience.

There are a few out there. Here's a few and I'll show how they'll vary with some example data.

1. BART's Passenger On Time (PoT) Metric
2. BART's Train On Time (ToT) Metric
3. NYC MTA's Customer Journey Time Performance Metric

Passenger on Time

This is the one that's commonly reported by BART. Their document, verbatim, says:

Mechanically, I'd say it goes like this, for each fare-gate trip with a tag-in at A and a tag-out at B:

1. Use the tag-out time to determine the most-recent train T from A to B
2. Look up that train's scheduled door-open $t_B^s(T)$
3. Look up that train's actual door-open $t_B^a(T)$
4. Calculate delay, $D_T(A,B)=t_B^a(T)-t_B^s(T)$ in seconds
5. The passenger-trip is considered delayed if $D_T(A,B)>300$

Then the delayed passenger trips, Y, are a subset of all trips, X where

$Y=\{x\in X\mid D_{T(x)}(A_x,B_x)>300\}.$

PoT is then $\frac{|X\setminus Y|}{|X|}$.

You can see the things this doesn't quite reflect:

• Reneges: passengers who decide not to take the train from A after tagging-in because it's too late
• Cancels: trains that just didn't show up
• Extreme lateness: $D_T(A,B)=301$ and $D_T(A,B)\gg 300$ are treated the same.

Now, I'm inferring a few things here, but it mostly matches what they describe. After all, they don't have anything but tag-in and tag-out to verify. The pathological case does manifest on some days. Take for instance, the case of May 10, 2024:

BART @SFBART

We currently have no train service between Richmond and MacArthur due to a wayside equipment problem.

The Yellow line is not impacted.

Millbrae service is provided by a shuttle train between SFO and Millbrae.

Richmond riders should seek other means to get to MacArthur.

May 10, 2024

BART @SFBART

Replying to @SFBART

Orange line service is running MacArthur to Berryessa.

Red line service is not running. For Millbrae riders, take the shuttle train between SFO and Millbrae and transfer to/from the Yellow line (Antioch-SFO).

Yellow, Green, and Blue line service is not impacted.

May 10, 2024

BART @SFBART

Replying to @SFBART

As of 11:10am, we have restored normal train service between Richmond and MacArthur. We will follow up with more details about the cause of the disruption.

May 10, 2024

So there was no train between Richmond and MacArthur for a while. As far as we're concerned, this line didn't exist for 6.5 hours from 0448 to 1110. The PoT metric for that day on BART was 96%, which is completely unremarkable. You can get the data for BART boardings by station for 2024 and you'll see that the usual boardings at the affected stations are missing. Estimating by looking at prior and posterior weeks, you'll note that some 6,600 passengers didn't take the train that day.

So it seems that this is right, absent trips don't affect lateness at all, and indeed this metric now creates the perverse incentive that cancelling trains that would be late will increase their published on-time performance. A Yellow Line train running once every 20 minutes that is 6 minutes late to leave is best cancelled if one wishes to optimize on-time performance, allowing passengers to take the next train which will arrive exactly on time. The rider experience is that they're waiting up to 40 minutes for a train.

Train on Time

BART also defines a different metric for its performance reviews that looks a bit different.

For this one, I'm going to try to reverse-engineer the rule based off that text rather than ask.

For a train, $T\in X$, all scheduled trains for that day:

1. Was T dispatched? If not, it is delayed and $T\in N$
2. Did T run to the end of the line? If not, it is delayed and $T\in E$
3. Did T stop at every station on its route? If not, it is delayed and $T\in S$.
4. Otherwise, look up its scheduled arrival $t_Z^s(T)$ and actual arrival $t_Z^a(T)$ at the last station Z
5. Compute the terminal delay $D_T=t_Z^a(T)-t_Z^s(T)$.
6. If $D_T>300$ seconds, T is delayed

Then the on-time trains, Y, are a subset of $W=X\setminus (N\cup E\cup S)$, where:

$Y=\{T\in W:D_T\le 300\}$

and consequently, the on-time metric ToT is $\frac{|Y|}{|X|}$.

Now, this is a bit harsh. If there's nearly no one on the train by the time it reaches its terminal station but it's a bit late pulling in there, it's going to look awful. In general, it suffers from a lack of passenger-weighting and so on when we're looking for a metric that represents passenger-experienced delay.

Comparison with PoT

One of the quarterly reports for BART shows how these two metrics can diverge so much.

There is quite a big difference in the metrics but that's to be expected

It does seem hard to believe that 80% of passengers can get somewhere on time when half of all trains are late.

Customer Journey Time Performance

Neither of the above metrics really capture what I want, though, which is that when I arrive at a location A, intending to go to a location B, I want to measure the deviance from the scheduled journey time. For instance, if there's a train scheduled at 0812 that arrives at destination at 0827 and I tag in at 0805, I'd want some slop time to make it to the platform, let's say 3 minutes, so that the train that shows up at 0812 and arrives at the destination at 0827 counts as perfectly on-time, and perhaps counts as on-time even if it arrives just short of 0832 (five minutes past the scheduled arrival).

So if I tag in at 0805, and the train is cancelled and the next one that arrives is ten minutes later at 0822 and arrives correctly on time at the destination at 0837, then that's a 10 minute delay for me. PoT would say I just had an on-time arrival, but my subjective experience is that I'm actually quite late.

NY MTA Metrics

It turns out there are systems that actually do something like this. The Office of the State Comptroller in NY audited the MTA and got them to publish metrics they said they would, and they came up with a few things that helped.

In September 2017, Transit introduced the new metrics: Additional Platform Time (APT), the average time that customers wait at a station beyond their scheduled wait time; Additional Train Time (ATT), the average time customers spend on board a train beyond their scheduled travel time; and the sum of these, Additional Journey Time (AJT). AJT is the key component of Customer Journey Time Performance (CJTP), the percentage of customer trips completed within five minutes of the scheduled time.

— New Customer-Focused Subway Metrics, Metropolitan Transportation Authority - New York City Transit

Okay, so the new metrics are described in slightly more detail as:

• Additional Platform Time (APT) - The average added time that customers spend waiting on the platform for a train, compared with their scheduled wait time. APT is measured using a combination of customers’ MetroCard entry data into stations and train departure times from those stations, using information from the real-time train tracking technologies that provide train arrival information.
• Additional Train Time (ATT) - The average additional unanticipated time customers spend onboard the train due to various service issues. Additional Train Time is measured using a combination of customers’ MetroCard entry data into their starting stations and customers’ arrival times at their destination stations, using information from the real-time train tracking technologies that provide train arrival information.
• Additional Journey Time (AJT) - The sum of APT and ATT.
• Customer Journey Time Performance (CJTP) - The percentage of customers whose journeys (waiting and traveltime) are completed within five minutes of their scheduled journey time. This can also be understood as the percentage of trips where AJT is less than five minutes.

The MTA itself has some caveats about this approach that are mostly specific to its own infrastructure but they're not relevant to the idea itself. They do have some assumptions that they think cause this to differ from the customer experience, though I think it seems fine for the most part.

Transit assigns two customers to a start wait time for a train on a platform at 7:00 a.m. and 7:05 a.m., respectively. A train is scheduled to arrive every 10 minutes (6:50 a.m., 7:00 a.m., and 7:10 a.m.). Transit determines that these two customers’ scheduled wait times are 0 and 5 minutes, However, if the 7:00 a.m. train arrived 6 minutes late, then one customer would have waited 6 minutes and the other would have only waited 1 minute rather than the expected 5 minutes. The APT for these two customers would be 6 minutes (7:00 a.m.–7:06 a.m.) and -4 minutes (7:10 a.m.–7:06 a.m.),respectively. However, in the averaging of APT, the -4 minutes will offset customers with +6 minutes.

— New Customer-Focused Subway Metrics, Metropolitan Transportation Authority - New York City Transit

What I'd Love

My ideal system without changing BART's tagging system would have something like the following:

• A station $s$-specific estimated walk time from entrance to platform $t_w(s)$ over all entrances
• An arrival time for a train T, $t_a(T,s)$
• A tag-in time $t_i(x)$ for each ride $x$
• A Dijkstra-optimal route from $s$ to end $e$

Then let's just consider the single-route case for comprehension.

1. Compute the platform-ready time $t_p(x)=t_i(x)+t_w(s)$
2. Select the earliest-scheduled train $T^{*}(x)$ on the route that scheduled to serve $s$ after $t_p(x)$
3. Look up the scheduled arrival time $t_s(T^{*}(x),e)$
4. Using the tag-out time $t_o(x)$, determine the most recent train that arrived prior to $t_o(x)$, $T^{+}(x)$. Its arrival time is $t_a(T^{+}(x),e)$
5. Delay is then $D_x=t_a(T^{+}(x),e)-t_s(T^{*}(x),e)$
6. The trip is delayed if $D_x>300$ seconds

And to avoid the problem of missing trains, estimate the demand from counterfactual days without disruption and consider all those trips delayed. This is all a result of having to report a single number, though. It's probably the case that we're just facing a dimension reduction problem where many different aspects of a trip need to be considered. So perhaps the right approach is to report:

• on-time percentage as calculated above
• estimated number of journeys missed due to disruption
• p90 passenger delay for delayed trips