Quick summary of the early AM delays on Thursday 2/25.

P500 & P502, the first two inbound trains, departed Worcester on time. Both those trains use equipment that originates ("lays over") in Worcester. Once P500 reached somewhere near Wellesley Square, it was blocked by a tree across both tracks. P502 eventually ended up directly behind P500. 

This tree must have fallen down between about ~5:15 AM and 5:45 AM. This is because there were two outbound trains that passed through Wellesley without any problems before P500 found the tree. P501 departs Boston at 4:30 AM and arrives at Worcester at 5:35 AM. P581, a 'non-revenue' / positioning / deadhead move then departs Boston at 4:45 AM and arrives in Worcester at 5:53 AM. So I'm guessing that P581 went through Wellesley sometime around 5:00 to 5:15 AM.

P501 turns at Worcester to become P504, while the non-revenue P581 usually turns at Worcester to become the P582 express. 

At just about the same time that P500 found its path blocked by the downed tree, outbound P501 died somewhere during its outbound trip. Reports indicate the locomotive had a problem with the main reservoir - a component of the compressed air braking system. P581 pulled up behind P501, coupled to it, and shoved it to Worcester.

Perhaps they thought they could fix the problem and keep all the trains operational, but that wasn't possible and they eventually cancelled P504. They also decided to convert P582 to a local to gather up all the stranded and waiting P504 passengers. In either a stroke of luck or good planning, they decided to use the usual P506 equipment (six doubles) which was already at the Worcester layover yard as the delayed and now local P582. P581 / P582 is planned to have 4 doubles and 2 flats, but I'm not sure it has had this consist lately. The P581 equipment was used as P506. 

It took about 40 minutes to clear the tree from the tracks in Wellesley. P500 was on the move around 6:30 AM with P502 directly behind it.

It is interesting to note that P504 and P582 are scheduled to pass through Wellesley around 6:45 AM, so they probably would have been caught at the downed tree / P500 / P502 traffic jam had they operated on time. But P504 and P582 probably only would have been delayed about ~10-15 minutes in this scenario. The P504 passengers were actually delayed about 40 minutes after being forced to take the delayed P582. So P504 and P582 passengers were definitely delayed more than if the downed tree was the only problem. 

The good news was that all the problems were resolved and cleared out by the time P506 and P508 came through and neither of those trains were delayed. 

Alert rider Dmitry Lev noticed a "lonely locomotive trudging west near Yawkey around 10am." I wonder if that locomotive was going to Worcester to retrieve the P501 / P504 equipment to bring it back to Boston for repairs. 

Final answers:
P500 passengers: 40 minutes late to Boston
P502 passengers: 15 minutes late to Boston
passengers intending to take P504: 30 minutes late to Boston
P582 passengers: 40 minutes late to Boston

Just for reference, according to the June 2015 manual passenger counts, each of those trains has the following typical passenger load:
P500: 183
P502: 490
P504: 553
P582: 818
Total passengers delayed: ~2000
These counts are generated by the Keolis customer service agents with the manual 'clickers' counting passengers as they disembark in Boston. Probably the most accurate count that is available, and the most recent data available.

On Wednesday 2/24, there were quite a few @MBTA_CR messages about delays in Worcester due to a disabled Amtrak train. With the help of observant rider @emmalineJen, I think I've pieced together what happened. 

Let's set the stage:

The Amtrak Lake Shore Limited originates in Chicago and travels east to Boston. The train actually gets physically split in Albany, NY, with most of the equipment heading south to New York City and a few coaches, a baggage car, and sometimes a sleeper car (and obviously a locomotive!) heading east to Boston. Anyway, after departing Chicago at 9:30 PM the previous night, the Boston section of the train is due into Worcester every day at 6:44 PM. It then stops at Framingham and Back Bay before arriving at South Station at a scheduled time of 8:01 PM. This train has the unfortunate nickname of "LATE Shore Limited" since it is very frequently delayed. 

On Wednesday night it was not only somewhat late - probably about an hour - but it also became disabled just east of the Worcester station. Unfortunately the Worcester station only has one platform, and the Amtrak train must have broken down in a location that blocked MBTA trains from approaching the station. 

Outbound MBTA train P533 is due to arrive in Worcester at 7:40 PM. It is then scheduled to head back into Boston as inbound P536, scheduled to depart at 7:50 PM. 

Around 7:50 PM, the @MBTA_CR twitter feed started to notify passengers that P536 would be delayed departing Worcester due to the disabled Amtrak train. Eventually, the @MBTA_CR feed notified passengers that P536 would originate at Grafton - and GPS data indicated that the P533 / P536 equipment had never made it to Worcester on the outbound trip, presumably due to the disabled Amtrak train. 

Meanwhile, outbound P535 was due to arrive at Worcester at 8:17 PM and head back into Boston as P538 with a Worcester departure time of 8:30 PM. 

This P535 / P538 set eventually shoved the disabled Amtrak set BACK westbound into the Worcester station, where passengers (including @emmalineJen) were directed to board the MBTA P538 equipment relatively far up the platform (to the east) as the crew finished coupling the two sets together. 

By this time, P536 had made its way back out to Grafton, where GPS data showed it stopped at the Grafton station for a very long time. It only departed Grafton after the departure of outbound P537 on the opposite track. This may explain how Worcester bound passengers on P533 (which became P536) were able to get to Worcester. Since that P533 / P536 equipment was never able to get to Worcester because of the disabled Amtrak train, the P533 Worcester passengers remained on the train until they were able to transfer to outbound P537 at Grafton. 

At about 9:00 PM, the double draft Lake Shore Limited - MBTA P538 combination train departed Worcester, presumably carrying passengers from Chicago, Cleveland, Upstate New York, and MBTA passengers intending to take both P536 and P538 towards Boston. The P536 Worcester passengers were delayed about 1 hour and 10 minutes while the P538 passengers were delayed about 30 minutes. 

The extremely long delay for Worcester originating P536 passengers (and Worcester bound passengers from P533) could presumably have been prevented if Worcester had two platforms and two tracks for passenger trains to service. But even that isn't guaranteed - it is possible that the Amtrak train was in a position that blocked all access west of CP 44 or CP 43 (see the map). Implementing two platforms at Worcester is somewhat difficult due to the freight yard immediately across from the station, but there is space to re-establish a center platform between the current station track and the next closest mainline track. I have heard that it is a long-term goal for the MBTA to restore that center / 2nd platform, but we all know that the MBTA does not have tons of spare cash lying around to knock out capital projects here and there. Maybe someday...

I really have no idea exactly what happened to cause the meltdown on Thursday. But I can make some educated guesses, and I can explain the details of how the system works and how it affects the Commuter Rail operation. 

Before I get into all that, I'll share some helpful links if you just want the quick story of what is planned for Friday morning:

MBTA official statement
Charles River TMA blog with great info about logistics
My copy of the current, normal MBTA Framingham-Worcester schedule, just for reference
My "Turn Table" for the Framingham-Worcester line, which describes what normally happens to outbound equipment

"Tower 1" is the interlocking just outside of South Station that has the switches that allow trains to come into the station platforms and then depart out towards the three primary 'paths' away from South Station. This is where our problem was on Thursday. 

What's an "interlocking?" In common usage, especially in this context, interlocking refers to the collection of switches between sets of signals. Picture an intersection on a busy road - two lanes in each direction meeting each other. But with railroads, the only place to change lanes or directions is through the switches, which would all be in the middle of the intersection (our "interlocking"). The signals at the entrances to the intersection allow traffic through when it is clear and the switches are lined up properly. And the signals are interlocked such that they won't allow trains through unless everything is properly aligned. 

Why is this interlocking called "Tower 1?" Back in the old days, it was actually a tower with people inside operating big levers to move the switch 'points.' One of these control towers has been turned into a museum in Norwalk, CT, and I've always wanted to visit. You can see the old switch levers in a picture on their home page here. 

Here is a fantastic history of Boston's Tower 1, along with some pictures of the original structure. 

Eventually the people and the levers were replaced with automation. The mechanical linkages were swapped out for electrically operated switches controlled by a computerized system. But the name "Tower 1" for this interlocking has persisted, even though the actual tower is long gone. 

Here is a track chart I found from a document related to the South Coast Rail Project:

Note that this track chart shows the planned (hoped?) expansion of South Station. We don't have 18 tracks at South Station yet!

So let's zoom in on the "Tower 1" area:
​

Tracks 7 & 5 on the left are the Framingham-Worcester line. Tracks 3, 1, & 2 just below that are the Providence line (and Needham line). The Fort Point Channel bridge carries the Dorchester Branch towards the Middleboro line (and other Old Colony lines), the Fairmount line, and, most importantly, the Southhampton Street yard where trains are serviced. 

So you can see the interlocking - it's all those crossing tracks (which represent switches) that would allow trains to go to different tracks / platforms at South Station. The signals that control access 'into' the Tower 1 interlocking are on both sides of the interlocking. On the South Station side, there are signals at the end of each platform. You can see them as you look down the platforms - small white lights at ground level. The signals on the other side of the interlocking are approximately where the red numbers are for the Framingham-Worcester and Providence lines, and approximately where the Fort Point Channel bridge is for the Dorchester Branch. Those are the limits of the interlocking - and a train can't enter any part of the interlocking without permission in the form of a 'favorable' signal indication or specific verbal permission (more on that below). 

Joe Pesatauro, spokesman for the MBTA, tweeted out a couple of great pictures of Tower 1 from Thursday evening:

​ 

This is the 'new' Tower 1 with all the automation. No more levers, no more people operating anything. It's all remotely operated by the dispatcher up in the dispatch theatre. There is actually one dispatcher (the "Amtrak Terminal" dispatcher) who pretty much ONLY operates this interlocking - compare that with the Framingham-Worcester line, where one dispatcher operates the whole line and all the interlockings from South Station to Worcester. The Tower 1 and South Station area is very complex and demands a lot of focused management.

​Take a close look at the right hand picture in the tweet. It is arranged similar to the track chart - South Station on the right, with all the 'destinations' out to the left. The Dorchester branch is shown going off to the bottom left as opposed to directly down on the track chart figure, but you get the idea.

Notice the little red light? It says "PLC Failure."

What's a PLC? A PLC is a Programmable Logic Controller. It is NOT something specific to the railroad industry - PLC's are used in all sorts of industries and devices. Essentially it is a type of computer that controls industrial systems by using multiple inputs and outputs.

In this case, the PLC is the replacement for the person in Tower 1. In the old days, the dispatcher would have the person in Tower 1 arrange the switches in a certain pattern to allow a train to move from one place to another. The person would then throw the levers in the correct pattern to achieve the desired result.

Now we have the dispatcher remotely controlling the operation though the "HMI" - Human Machine Interface - essentially a graphical representation of a system. The dispatcher clicks on a screen or toggles buttons on the HMI. The HMI sends the signal down to the PLC and the PLC interprets those commands and operates the switches. The PLC then also operates the signals to 'grant permission' for trains to move through the interlocking.

Clearly something failed with the Tower 1 system Thursday. The "PLC Failure" light indicates that it is a pretty big problem - although we already knew that. What could the problem be? Some of the older PLC's that I work with illuminate a "PLC Failure" light when they are switched off / taken offline. So that light doesn't necessarily mean that the PLC has actually completely failed, although that could be the problem. But it isn't just one switch or one small component of the Tower 1 interlocking that is a problem - they have either manually shutdown the PLC or it has failed. That means that none of the switches can be operated electrically and none of the signals on either side of the interlocking are operative. It also means that the dispatcher probably cannot see on the HMI the usual information the dispatcher gets: position of trains, position of switches, and signal indications. The dispatcher is probably in the dark.

If it was just one switch within the interlocking or one small component, they could easily disconnect that offending part from the PLC / system and continue to operate the system with the rest of the operational components.

Also note that there are lights on the board for "PLC-A" and "PLC-B." I'm guessing, and this is a big guess, that this means that Tower 1 has two PLC's as redundant systems. But it could mean that there are two separate PLC's that control the interlocking. If there are really two redundant PLC's and neither are operational, then the technicians have a really big problem to troubleshoot.

In this instance (PLC shutdown / interlocking 'dark') the signals either show a stop indication or are completely dark. Dark signals have the same meaning to a train engineer as stop - that's a fail safe procedure.

The combination of 'stop' signals and dead switches throughout the interlocking are what led to the massive problems Thursday. The system just isn't designed to be operated manually - there is no good contingency plan for losing the control system, especially at the most critical interlocking on the South Side.

When I walked by South Station Thursday morning, there were workers out in the interlocking manually 'throwing' the switches to align the tracks the way that the dispatcher wanted. The dispatcher would call the crew over the radio and give them a path to create. For example: "C&S crew 165 [that was the name of the crew], next move I need is Dorchester track 2 to Terminal track 9." The work crew would then have to scurry around moving the switches to line up the tracks to achieve that. [Based on my observations, the CP COVE interlocking was functioning properly / normally.]

In accordance with railroad rules, for a train to pass a stop signal, a very specific verbal conversation needs to happen between the dispatcher and the engineer while the train is stopped at the signal. Thursday morning, the dispatcher was issuing "Rule 241" permissions (see definition of Rule 241 in my glossary) to trains to move through the interlocking after the switches were lined up.

[Actually if you really want to learn a lot more background on this whole topic, spend some time with the following definitions in the glossary:
"CETC"
"Dispatcher"
"NORAC"
"S&I"]

Obviously all of this manual intervention requires a huge amount of effort and time. Just talking to the trains to give them Rule 241 permission takes quite a bit of time. The trains have to be stopped, which doesn't necessarily happen on a normal day - the interlocking can usually be arranged so that incoming trains don't need to stop and can proceed with a 'favorable' signal directly into the platform tracks. The failure of the automated system greatly reduces the volume of trains that can be moved through the interlocking. Probably down to 10 or 20% of normal moves (just my guess).

If you study the track chart, you can quickly realize another problem. With the Tower 1 interlocking not functional, there is no way to efficiently move trains between the different branches. The COVE interlocking appears to be a good spot for at least moving trains between the Providence and Framingham-Worcester lines, but that is somewhat deceptive. One problem is that there is only enough room between COVE and Tower 1 to fit about 5-6 cars (how I know that is another long story from a few years ago. Remind me to blog about that some day). So to get a longer train from one line to the other would require moving past the Tower 1 signal - and that means a Rule 241 move with all the time consuming issues. The other problem is that moving trains between Providence and Framingham-Worcester doesn't really accomplish a lot - and doesn't get them into the South Station platforms.

Another big problem is the inability to move train sets from both the Providence and Framingham-Worcester lines over to the Dorchester branch so they can get over to the yard for service. As well as mandated service inspections, the trains need to be fueled, and the fueling operations are usually only conducted at the Southhampton Street yard. Cleaning the coaches is also important, but not the critical issue for getting the train sets to the yard.

Luckily there is enough room for trains to be 'stacked' between Back Bay and the Tower 1 signal, on both the Providence line and the Framingham-Worcester line. But, as you can see on the larger track chart, there is no way to switch the trains onto different tracks between Yawkey and the COVE interlocking on the Framingham-Worcester line. On Thursday morning, inbound trains on the Framingham-Worcester line pulled past Back Bay station after unloading their passengers and then just waited between Back Bay and the Tower 1 signals. You may have seen the helicopter TV shots of those trains or the pictures of those trains at the far end of the Back Bay platforms.

Many of you noticed that there was one train set on the Framingham-Worcester line that was 'backwards' - it had its locomotive on the inbound front end, and a control coach on the outbound front end. That was train P507- P510 in the morning commute. My guess is that train set used the short little curved track that connects the Dorchester branch to the Providence line near the COVE interlocking. If you look at the track chart you can find it - it's the one track that doesn't require a train to go through Tower 1 and South Station to get from the Dorchester branch to the Providence line (and then the Framingham-Worcester via the COVE interlocking). My guess is that train, with the control coach arranged at the front of the train as it proceeded up the Dorchester Branch (so it would pull into South Station in the typical 'normal' orientation), was re-routed across that track (which has a nickname that I can't think of right now...) and then as it went around that 'curve' it ended up with the control coach headed toward Worcester. And then that train stayed on the Framingham-Worcester all day - pointing the 'wrong' direction but just going back and forth between Back Bay and Framingham (or Worcester).

All of this also serves to explain why they have the Providence trains going into and out of South Station, while other trains are not. A number of factors for this:
1) They can set a bunch of switches lined up for moving trains to and from the Providence line and not to have frequently change those alignments. Setting it up minimizes the number of switches needed to be 'thrown' for each train move.
2) The Providence line leads to the middle (somewhat) of the Tower 1 interlocking. Easier to route trains to each side of South Station with that arrangement.
3) Limiting the moves to only one particular line allows the dispatcher and crews to handle the workload, esp. the issuing of Rule 241 permissions.
4) The Providence line has the most ridership, so having those trains go into South Station allows them to provide better service to the most number of people.
5) The Providence line is also the Amtrak Northeast Corridor to New York and Washington DC. Having the interlocking lined up to the Providence line also allows them to more easily move Amtrak trains (which is important in the grander scheme of moving people around the country, even if we don't think so).

What about Amtrak - why is this their fault? The MBTA actually owns the tracks from South Station to the Rhode Island border along the Northeast Corridor / Providence line. But since those are shared with Amtrak, the MBTA and Amtrak arrived at a mutually beneficial solution, which was memorialized in a contract. Instead of paying 'rent' or a fee to use the tracks, Amtrak agreed to maintain the tracks and all associated equipment - such as interlockings - in addition to performing dispatching for both the South Station terminal and the Providence line. I'm not actually sure who owns the Tower 1 PLC and the equipment - it might be the MBTA or Amtrak.

Amtrak is now trying to change that agreement, but for now, it is still in effect. There are number of practical and logical reasons why this arrangement makes sense:
1) Amtrak owns the 'catenary' - the overhead wires to supply electric power to the Amtrak electric trains. Having one entity responsible for maintaining the tracks and another for the catenary would result in complicated coordination being required to do anything on the line.
2) The tracks need to be maintained to a higher standard than other lines since the Acela trains travel at 150 mph for certain stretches of the line. MBTA / Keolis don't require this higher standard track, so they shouldn't have to pay for that maintenance.
3) see comment below. I got this item wrong: The dispatch system incorporates power management for the catenary. So the dispatcher manages not only the train traffic, but also certain aspects of electrical load management for the electric Amtrak trains. MBTA / Keolis have no need for this, so having their dispatcher do this would require either compensation or a different Amtrak employee to do this function.

Therefore, Amtrak is responsible for the operation and maintenance of the Tower 1 interlocking and all associated components. The MBTA and Keolis have no responsibility or ability to intervene with this Amtrak system. That isn't necessarily a bad thing - the Northeast Corridor is the cash cow for Amtrak that subsidizes their entire nationwide system. So it gets as good treatment as it can and is a high priority for Amtrak, although just like everything else across the country, there is a backlog of needed capital spending for the Northeast Corridor. I'm not sure how that might affect this particular PLC, but we all know the MBTA isn't sitting on a pile of money to keep things maintained to the level they should be either.

I have no idea how old the Tower 1 PLC is, or what the condition of it is. Most of the switches and track within the Tower 1 interlocking were replaced as a result of the Big Dig construction in the early 2000's, but that was ~15 years ago now.

Hopefully they can get it fixed quickly!

Wow. Nuclear meltdown of a day. 

Here's what we know (about Framingham-Worcester):

P529 is cancelled. That train usually carries about 741 passengers. They probably will go to P531.
P583 is cancelled. That train usually carries about 873 passengers. Do they leave early and get on P527? Or go to P531? Either option is problematic. 

Plan on getting on all other trains at Back Bay. Some trains might be able to start at South Station, but it won't be reliable to expect that. They have to get manually switched out of South Station and thru the non-functional interlocking at "Tower 1." Some trains will probably just originate at Back Bay - never having gone to South Station (perhaps all day). 

P527 usually has about 1227 passengers on a train with 8 double deckers. That's a capacity of about 1440 seats. But who knows if that set is in position to make that run. If it gets any passengers from P583, it will be crushingly overcrowded. 
P531 usually has about 504 passengers. Who knows what size set they use for it - maybe a six double set? That would be a capacity of about 1080 passengers. If it gets passengers from both P529 and P583, it could also easily become crushingly overcrowded. 

Another way to look at it: From 5 PM to 6 PM, usually outbound load is 3,345 passengers. With 2 of the 4 usual trains cancelled, it will be very ugly tonight. 

Leave early and expect overcrowding and big delays. And remember it isn't the fault of the conductors. Give them a break. 

Friday morning will be a problem - even if the "Tower 1" problem is fixed, the train sets will be out of position, and there probably aren't enough hours or crews to get everything reorganized in one night (esp. if Tower 1 is still not working). 

This blog has some great logistical tips:
http://www.charlesrivertma.org/cambridge-transportation-news/2016/2/18/whats-going-on-with-south-station.html

If you're a new reader here, don't miss my analysis of the new draft schedule below. 

On Wednesday, February 3, 2016, the MBTA and massDOT held the Natick version of the continuing public hearings regarding the proposed new commuter rail schedules. There wasn't anything big and different than the Worcester hearing, although there was a lot more vocal opposition and concern about some of the proposed changes. 

Brian Benson of the MetroWest Daily News wrote a good article summarizing the meeting. He has another article from the day before outlining the position of the Natick Board of Selectmen. 

As with the Worcester meeting, it was great to see senior MBTA management at the meeting. They were very willing to engage in detailed exchanges, and they seemed very open to the feedback being provided. 

At the beginning of the meeting, there were only about 9 general public attendees. The actual location was somewhat difficult to find on the Walnut Hill School campus, so that may have delayed people. By the end of the evening, I estimate there were probably about 30 general public attendees. There also appeared to be about 6 MBTA / massDOT staff, 1 reporter, and 4 government representatives. There were also one or two Keoils employees, I think, but again, they did not interact much with the public.  

There were no handouts provided by the MBTA. I gave out printed copies of my version of the draft Framingham-Worcester schedule.

Continued after the read more...

Cyrus Moulton asked me to pose this question:

Would you pay a surcharge to ride the bullet train? 

What if the surcharge only applied to single ride cash tickets, and passengers with passes (or tickets purchased on the mobile app) did not incur a surcharge? 

Please post your thoughts in the comments...

If you're a new reader here, don't miss my analysis of the new draft schedule below. 

On Monday February 1, 2016, MBTA and massDOT held the Worcester version of the continuing public hearings regarding the potential fare increases and the proposed new commuter rail schedules. The MBTA tweeted out a picture of the hearing, and my bald spot made a cameo appearance:

Cyrus Moulton at the Worcester Telegram & Gazette does a much better job than this blog with his concise article about the meeting. I think he accurately captured the mood of the meeting (and not just because he quoted me!).  

If you really want my long winded observations and thoughts, click on the read more: