But if you just want the executive summary results, here is how it shaped up:
Worcester passenger intending to take P506: arrived in Boston about 75 minutes late.
Worcester passenger intending to take P508: arrived in Boston about 60 minutes late.
Worcester to Newton passenger intending to take P512: arrived in Boston about 20 minutes late.
Grafton to Ashland passenger intending to take P506: arrived in Boston about 70 minutes late.
Grafton to Ashland passenger intending to take P508: arrived in Boston about 55 minutes late.
Framingham to Newton passenger intending to take P506: arrived in Boston about 65 minutes late.
Framingham to Natick passenger intending to take P508: arrived in Boston about 50 minutes late.
Framingham to Newton passenger intending to take P510: arrived in Boston about 20 minutes late.
By my estimation, that's about 2,200 passengers at least 55 minutes late. And probably another 600 or more at least 20 minutes late. Wow.
The fascinating thing is that the official statistics will be completely different:
P506 officially arrived about 65 minutes late.
P508 was cancelled, SO IT DOESN'T COUNT when "late trains" are calculated...
P510 officially arrived about 20 minutes late.
P512 officially arrived about 20 minutes late.
The 'official' statistics have no way of factoring in what train people intended to take, or the fact that P506 started in Grafton not Worcester, forcing Worcester passengers onto an even later P512.
The entire 'what happened' story after the read more...
The Worcester layover yard only has space for four trainsets. So P500, P502, P506 and P508 all start in Worcester after having spent the night in the layover yard. P504 is the inbound turn of P501, and P582 is the inbound turn of an outbound 'deadhead' move. For more details on 'turns', read the "Turn Table."
I'm not certain of the order of these events, but I know there were two problems. Knowing that, I'm making an educated guess as to how this all played out. <edited 1/7/16...story below has been confirmed. Some details added.>
The two problems were:
1) Problem with one of the layover sets, which was eventually assigned as P506 (it might not have started the AM as the P506 set). The problem may have been with either the control car (#1820) or the locomotive (#2029). I'm not clear what the exact problem was, but it rendered the set dead.
2) Faulty HEP (Head End Power - see the glossary to learn about that) on the west facing new "HSP46" locomotive (#2008) of the set that was supposed to be used as P508.
I'm not sure if either of these issues delayed P502, but it is possible that the problems started that early.
Problem #1 surfaced first, and they invented a workaround to solve that problem. The intent was to get the 'bad' set operational and use it, albeit as a delayed P506. In the meantime, the equipment for P504 and P582 arrived from Boston, turned, and went inbound with no delays.
At this point, around 6:15 AM, there are only two sets left in the layover yard. They intended to 'fix' problem set #1 and use it as P506. And the other set would be used as P508 as planned.
As you may have noticed, P508 (and P527 in the evening, which uses the same equipment), has been operating with a locomotive at each end. It has been designated as a "priority" train which means it gets special attention and special treatment - and one of the special treatments is two locomotives, as insurance for problems.
But to deal with the dead control car on P506, the extra (east facing) locomotive (#1070) was uncoupled from the P508 set and switched over to P506. P508 still had it's west facing loco (#2008), and now P506 had an east facing locomotive (#1070) at the front of the inbound train. This should have restored both of the remaining sets in Worcester to being operational.
But at some point the HEP on the remaining west facing loco (#2008) on the P508 equipment died (or went into fault), so by moving the extra loco off the P508 set, P508 was now also dead. So essentially the switching of the locomotive just changed which set was dead - there ended up always being one dead set out of the two remaining in the Worcester layover yard after the departure of P500 and P502. And the bad news is that the one dead set that remained in Worcester was the biggest set out of the four layovers - the set usually used for P508. Although on this day it had 7 doubles instead of the usual 8.
This all took place from ~6:30 to ~7:30 AM, and then another problem surfaced: while they were making the switching move, P503 arrived from Boston and occupied the single platform track at Worcester. It 'turns' at Worcester to become inbound P512. So once P506 with the switched east facing locomotive was ready to go, P503 / P512 was 'in the way' at Worcester. So the decision was made to send P506 out immediately and start at Grafton, while the P503 / P512 set stayed at Worcester and departed on time on the schedule of P512. P508 essentially was cancelled at this point (the P506 set needed to operate as P506 instead of operating as P508 because it needed to make all the local stops where people were waiting).
In a cruel twist of fate, the P503 / P512 set that actually originates in the Readville yard early in the morning was a smaller set than usually assigned. Which wouldn't have been a big deal if it was just used as the normal P512. And when it was leaving Boston, it was too early to predict this big meltdown.
So this takes us up to about 7:30 AM - nothing has left Worcester since P582 at 6:05 AM, meaning thousands of passengers waiting for P506 and P508 are starting to get stranded from Worcester to Boston. P506 is on the move from Grafton scooping up the passengers waiting for P506 AND P508. P512 has all the Worcester passengers waiting for P506, P508, and P512.
Our saga continues in Framingham...
P510 is the inbound turn of outbound P507. In a very good move that I think was more luck than plan, the dispatcher had P510 gather all the passengers waiting for delayed P506 and P508 in Framingham. The reason this was a good move is that if P506 had operated ahead of P510, it would have had very long stops with irate passengers not able to board since it was already almost full approaching Framingham. The delays that would have resulted from it having to deal with the overcrowding would have been worse than the delay it further experienced when it had to wait outside of Framingham so that P510 could leave on time.
The P507 / P510 equipment was six doubles, which is considerably larger than usual. I'm not sure if this was coincidence or if they sent a larger set out from Boston anticipating this meltdown. Regardless, all the capacity was needed, and it was lucky that P510 was that big.
At this point (about 8:05 AM) the die was cast and it was just a big gamble to see if all the passengers could fit on the smaller sets trying to gather everyone up. P510 in the lead, scooping up passengers from Framingham to the Newtons who had been waiting for P506, P508, or P510. P506 following close behind, not getting many passengers from Framingham to the Newtons since it wasn't on a schedule and P510 was right in front of it. Which is good since P506 was already jam packed just from the P506 and P508 passengers from Grafton to Framingham. P512 was behind P506, rounding out the procession, and although it had started on schedule, it drifted late due to overcrowding.
By the time this conga line reached Wellesley Farms, the dispatcher decided to try and get P512 onto track 1 to try and get it around P506, since P512 doesn't normally stop at West Newton or Newtonville. But the crew of P510 started to reach beyond capacity and started to communicate with the dispatcher about the need for P506 to pick up passengers that wouldn't be able to get onto P510. This led the dispatcher to warn P512 that it might also be needed to sweep up Newton passengers - even at West Newton or Newtonville.
Actually, it appears that the crew of P510 may have been telling passengers waiting at Wellesley Farms and all three Newtons that P506 was close behind, and they may be better off waiting for that set. The crew on P510 had no idea about the crowding conditions on P506 or P512 - I was getting better information via twitter than the crews. At any rate, it doesn't appear that anyone was denied boarding P510 until Newtonville. At Newtonville, the crew of P510 estimated that there were about 50-60 passengers stranded and not able to get onto P510. Those passengers were able to squeeze onto P506, and P512 did not need to make unscheduled stops at West Newton or Newtonville.
Twitter comments indicate that at least some (and maybe lots) of Newton passengers gave up waiting for the trains, fearful of being denied boarding due to overcrowding. This happened frequently in the reduced schedule service of March 2015, so they have been conditioned to expect the worst in these scenarios.
The conga line arrived in Boston without further drama, although each station stop in Boston was longer than usual due to unloading the overcrowded coaches. P510 arrived about 9:20 AM, P506 about 9:25 AM, and P512 about 9:30 AM.
Can we draw any conclusions from this mess? Here are mine:
1) The system is paper thin fragile. Without a complete extra spare 'protect' set at Worcester, any breakdown of any one train will cause a substantial delay. The extra locomotive on P508 actually saved us - without that, there would have been two broken down trains. But since the extra locomotive was already coupled to P508, uncouping it and switching it to the other set incurred a substantial delay (even if the subsequent breakdown of the other loco on P508 had not happened). So with no extra locomotives or cab cars to keep at distant terminals (or ideally a complete spare 'protect' set), breakdowns can quickly and easily set off a cascading meltdown. Stick an extra locomotive, cab car, or complete set at key distant terminals, and some problems can be averted. But without more layover capacity, and a bunch more capital equipment, extra idle equipment will never happen. There is an extra 'protect' set (with a crew) in Boston, but it doesn't help the early morning meltdowns at the distant terminals.
2) It's unclear what the exact mechanical issues were, but it should be noted that it was a "new" Rotem cab (control) car and 2 new "HSP46" locomotives that were involved in this story as the potentially broken down equipment. Maybe it was all 3 of those pieces of equipment that broke down, but even if it was just 2, it is all 'new' equipment we're talking about here. Anything mechanical can have issues, but it is disappointing that new equipment is breaking down. And I won't blame Keolis without understanding if the mechanical issues are something that could have been prevented with maintenance. Especially with the Rotem cab cars - those have a very bad reputation and Keolis didn't choose them. And it isn't clear (at least to me - maybe somebody knows) that the breakdowns were related to weather, although I can accept the argument that it is a logical inference.
3) Keolis actually did a pretty good job of managing the dynamic situation and trying to get people to Boston. We were all really late, and I can't diminish that, but if we would have had to deal with CSX dispatching the line, things would have been much worse. Having a Keolis dispatcher directly managing the line helps. And I'm not sure if it was just luck, but they managed the capacity / loading issue pretty good - for example, having P510 go ahead of P506 was a good decision. And their attempt to use the extra locomotive off of P508 to fix the P506 set seemed like a good decision at the time. The whole morning sucked, but I'm saying it could have been worse.
4) The decision to send the fixed P506 set out to Grafton and increase the delay to the Worcester passengers seemed cruel and a rude / bad decision at the time. But in hindsight, it only delayed those passengers an extra five minutes, and it helped manage the capacity / loads better.
5) The conductors and crews are as much victims in this as we are. They don't come to work thinking 'who cares, we'll just see what happens today...' - they really don't want to be delayed or cancelled and have to deal with us grumpy (and some rude) passengers. So be sure not to take it out on them. I'm sure they were scrambling as much as they could to work through fixes and come up with workarounds.
6) It is very difficult for crews in the field to gather information about where other trains are and what they have for capacity. As a passenger, I have a much better vantage point because:
a) I'm just riding the train, I don't have anything else to do;
b) I have access to the status apps that show where the trains are. Train crews have no such similar source of info;
c) Twitter posts from other passengers provide great and reliable info about train capacity and loads. Train crews can only see their particular coaches, and have to ask over the radio just to find out what is happening on other coaches on their own train, let alone other trains; and
d) I can listen to my scanner to hear the conversations between all the train crews.
Even the dispatcher has less information - the way the automatic block signal / traffic control dispatch system works, the dispatcher can't actually see exactly where the trains are (I'm not sure if they use PTIS / GPS data as a 'sidebar' source of info, but probably not).
This difficulty in getting timely and accurate information hampers the ability of the dispatcher and more importantly the train master or chief dispatcher to make decisions about what to do. I don't think it was a factor with what we experienced today, but there were times when I could clearly see that I had a better understanding of the overall picture than some train crews. I'm not trying to brag - my point is more that we need to understand that sometimes this lack of information will result in bad decisions. And I'm not sure what the solution is to that, other than to put some kind of supervisor on each train, which isn't feasible.
The other aspect of this communication delay is the lag time between decisions and events in the field and when that can be converted into official @MBTA_CR tweets and/or text messages. The Keolis customer service folks in Boston are at least one more step removed from the information than the dispatchers - there is no way they can be expected to get information at the same time that decisions are being made in the field, especially considering that the customer service folks are managing information for ALL the lines. So while the information from the @MBTA_CR tweet desk was sometimes erroneous as to what really was going to happen, it wasn't necessarily their fault - things were changing quicker than they could reasonably keep up with. The perfect example is the decision to have P510 go ahead of P506 at Framingham - which required all the Framingham passengers to cross from track 2 to track 1. There just wasn't enough time to publish that info via any method since the decision was made and the action executed all within minutes. Guess this justifies my existence trying to get info out to all of you (yes, that was me telling folks to cross over to track 1 when there were no trains yet in Framingham...)!
7) I started to write a detailed rant about 'reform before revenue.' But I'll keep this blog more focused on operations and less on politics. But suffice it to say, I'm not convinced that the MBTA can be fixed without some restructuring of the legacy Big Dig debt, restructuring of the MBTA's annual sources of revenue (taxes, fares, and passes), AND restructuring of the management and operations. But it can't be done without any money. Reform before revenue has been championed for years and it isn't working.
Finally, thanks to all of you for your tweets and messages about what you're seeing! It really helps me piece together the big picture so that I can write silly explanatory blog posts like this.