Before I get into all that, I'll share some helpful links if you just want the quick story of what is planned for Friday morning:
MBTA official statement
Charles River TMA blog with great info about logistics
My copy of the current, normal MBTA Framingham-Worcester schedule, just for reference
My "Turn Table" for the Framingham-Worcester line, which describes what normally happens to outbound equipment
"Tower 1" is the interlocking just outside of South Station that has the switches that allow trains to come into the station platforms and then depart out towards the three primary 'paths' away from South Station. This is where our problem was on Thursday.
What's an "interlocking?" In common usage, especially in this context, interlocking refers to the collection of switches between sets of signals. Picture an intersection on a busy road - two lanes in each direction meeting each other. But with railroads, the only place to change lanes or directions is through the switches, which would all be in the middle of the intersection (our "interlocking"). The signals at the entrances to the intersection allow traffic through when it is clear and the switches are lined up properly. And the signals are interlocked such that they won't allow trains through unless everything is properly aligned.
Why is this interlocking called "Tower 1?" Back in the old days, it was actually a tower with people inside operating big levers to move the switch 'points.' One of these control towers has been turned into a museum in Norwalk, CT, and I've always wanted to visit. You can see the old switch levers in a picture on their home page here.
Here is a fantastic history of Boston's Tower 1, along with some pictures of the original structure.
Eventually the people and the levers were replaced with automation. The mechanical linkages were swapped out for electrically operated switches controlled by a computerized system. But the name "Tower 1" for this interlocking has persisted, even though the actual tower is long gone.
Here is a track chart I found from a document related to the South Coast Rail Project:
So let's zoom in on the "Tower 1" area:
So you can see the interlocking - it's all those crossing tracks (which represent switches) that would allow trains to go to different tracks / platforms at South Station. The signals that control access 'into' the Tower 1 interlocking are on both sides of the interlocking. On the South Station side, there are signals at the end of each platform. You can see them as you look down the platforms - small white lights at ground level. The signals on the other side of the interlocking are approximately where the red numbers are for the Framingham-Worcester and Providence lines, and approximately where the Fort Point Channel bridge is for the Dorchester Branch. Those are the limits of the interlocking - and a train can't enter any part of the interlocking without permission in the form of a 'favorable' signal indication or specific verbal permission (more on that below).
Joe Pesatauro, spokesman for the MBTA, tweeted out a couple of great pictures of Tower 1 from Thursday evening:
A look inside Amtrak's "Tower 1," the source of all the problems today for @MBTA_CR #MBTA customers pic.twitter.com/1SDXt7wZhD
— Joe Pesaturo (@JoePesaturo) February 19, 2016
Take a close look at the right hand picture in the tweet. It is arranged similar to the track chart - South Station on the right, with all the 'destinations' out to the left. The Dorchester branch is shown going off to the bottom left as opposed to directly down on the track chart figure, but you get the idea.
Notice the little red light? It says "PLC Failure."
What's a PLC? A PLC is a Programmable Logic Controller. It is NOT something specific to the railroad industry - PLC's are used in all sorts of industries and devices. Essentially it is a type of computer that controls industrial systems by using multiple inputs and outputs.
In this case, the PLC is the replacement for the person in Tower 1. In the old days, the dispatcher would have the person in Tower 1 arrange the switches in a certain pattern to allow a train to move from one place to another. The person would then throw the levers in the correct pattern to achieve the desired result.
Now we have the dispatcher remotely controlling the operation though the "HMI" - Human Machine Interface - essentially a graphical representation of a system. The dispatcher clicks on a screen or toggles buttons on the HMI. The HMI sends the signal down to the PLC and the PLC interprets those commands and operates the switches. The PLC then also operates the signals to 'grant permission' for trains to move through the interlocking.
Clearly something failed with the Tower 1 system Thursday. The "PLC Failure" light indicates that it is a pretty big problem - although we already knew that. What could the problem be? Some of the older PLC's that I work with illuminate a "PLC Failure" light when they are switched off / taken offline. So that light doesn't necessarily mean that the PLC has actually completely failed, although that could be the problem. But it isn't just one switch or one small component of the Tower 1 interlocking that is a problem - they have either manually shutdown the PLC or it has failed. That means that none of the switches can be operated electrically and none of the signals on either side of the interlocking are operative. It also means that the dispatcher probably cannot see on the HMI the usual information the dispatcher gets: position of trains, position of switches, and signal indications. The dispatcher is probably in the dark.
If it was just one switch within the interlocking or one small component, they could easily disconnect that offending part from the PLC / system and continue to operate the system with the rest of the operational components.
Also note that there are lights on the board for "PLC-A" and "PLC-B." I'm guessing, and this is a big guess, that this means that Tower 1 has two PLC's as redundant systems. But it could mean that there are two separate PLC's that control the interlocking. If there are really two redundant PLC's and neither are operational, then the technicians have a really big problem to troubleshoot.
In this instance (PLC shutdown / interlocking 'dark') the signals either show a stop indication or are completely dark. Dark signals have the same meaning to a train engineer as stop - that's a fail safe procedure.
The combination of 'stop' signals and dead switches throughout the interlocking are what led to the massive problems Thursday. The system just isn't designed to be operated manually - there is no good contingency plan for losing the control system, especially at the most critical interlocking on the South Side.
When I walked by South Station Thursday morning, there were workers out in the interlocking manually 'throwing' the switches to align the tracks the way that the dispatcher wanted. The dispatcher would call the crew over the radio and give them a path to create. For example: "C&S crew 165 [that was the name of the crew], next move I need is Dorchester track 2 to Terminal track 9." The work crew would then have to scurry around moving the switches to line up the tracks to achieve that. [Based on my observations, the CP COVE interlocking was functioning properly / normally.]
In accordance with railroad rules, for a train to pass a stop signal, a very specific verbal conversation needs to happen between the dispatcher and the engineer while the train is stopped at the signal. Thursday morning, the dispatcher was issuing "Rule 241" permissions (see definition of Rule 241 in my glossary) to trains to move through the interlocking after the switches were lined up.
[Actually if you really want to learn a lot more background on this whole topic, spend some time with the following definitions in the glossary:
"CETC"
"Dispatcher"
"NORAC"
"S&I"]
Obviously all of this manual intervention requires a huge amount of effort and time. Just talking to the trains to give them Rule 241 permission takes quite a bit of time. The trains have to be stopped, which doesn't necessarily happen on a normal day - the interlocking can usually be arranged so that incoming trains don't need to stop and can proceed with a 'favorable' signal directly into the platform tracks. The failure of the automated system greatly reduces the volume of trains that can be moved through the interlocking. Probably down to 10 or 20% of normal moves (just my guess).
If you study the track chart, you can quickly realize another problem. With the Tower 1 interlocking not functional, there is no way to efficiently move trains between the different branches. The COVE interlocking appears to be a good spot for at least moving trains between the Providence and Framingham-Worcester lines, but that is somewhat deceptive. One problem is that there is only enough room between COVE and Tower 1 to fit about 5-6 cars (how I know that is another long story from a few years ago. Remind me to blog about that some day). So to get a longer train from one line to the other would require moving past the Tower 1 signal - and that means a Rule 241 move with all the time consuming issues. The other problem is that moving trains between Providence and Framingham-Worcester doesn't really accomplish a lot - and doesn't get them into the South Station platforms.
Another big problem is the inability to move train sets from both the Providence and Framingham-Worcester lines over to the Dorchester branch so they can get over to the yard for service. As well as mandated service inspections, the trains need to be fueled, and the fueling operations are usually only conducted at the Southhampton Street yard. Cleaning the coaches is also important, but not the critical issue for getting the train sets to the yard.
Luckily there is enough room for trains to be 'stacked' between Back Bay and the Tower 1 signal, on both the Providence line and the Framingham-Worcester line. But, as you can see on the larger track chart, there is no way to switch the trains onto different tracks between Yawkey and the COVE interlocking on the Framingham-Worcester line. On Thursday morning, inbound trains on the Framingham-Worcester line pulled past Back Bay station after unloading their passengers and then just waited between Back Bay and the Tower 1 signals. You may have seen the helicopter TV shots of those trains or the pictures of those trains at the far end of the Back Bay platforms.
Many of you noticed that there was one train set on the Framingham-Worcester line that was 'backwards' - it had its locomotive on the inbound front end, and a control coach on the outbound front end. That was train P507- P510 in the morning commute. My guess is that train set used the short little curved track that connects the Dorchester branch to the Providence line near the COVE interlocking. If you look at the track chart you can find it - it's the one track that doesn't require a train to go through Tower 1 and South Station to get from the Dorchester branch to the Providence line (and then the Framingham-Worcester via the COVE interlocking). My guess is that train, with the control coach arranged at the front of the train as it proceeded up the Dorchester Branch (so it would pull into South Station in the typical 'normal' orientation), was re-routed across that track (which has a nickname that I can't think of right now...) and then as it went around that 'curve' it ended up with the control coach headed toward Worcester. And then that train stayed on the Framingham-Worcester all day - pointing the 'wrong' direction but just going back and forth between Back Bay and Framingham (or Worcester).
All of this also serves to explain why they have the Providence trains going into and out of South Station, while other trains are not. A number of factors for this:
1) They can set a bunch of switches lined up for moving trains to and from the Providence line and not to have frequently change those alignments. Setting it up minimizes the number of switches needed to be 'thrown' for each train move.
2) The Providence line leads to the middle (somewhat) of the Tower 1 interlocking. Easier to route trains to each side of South Station with that arrangement.
3) Limiting the moves to only one particular line allows the dispatcher and crews to handle the workload, esp. the issuing of Rule 241 permissions.
4) The Providence line has the most ridership, so having those trains go into South Station allows them to provide better service to the most number of people.
5) The Providence line is also the Amtrak Northeast Corridor to New York and Washington DC. Having the interlocking lined up to the Providence line also allows them to more easily move Amtrak trains (which is important in the grander scheme of moving people around the country, even if we don't think so).
What about Amtrak - why is this their fault? The MBTA actually owns the tracks from South Station to the Rhode Island border along the Northeast Corridor / Providence line. But since those are shared with Amtrak, the MBTA and Amtrak arrived at a mutually beneficial solution, which was memorialized in a contract. Instead of paying 'rent' or a fee to use the tracks, Amtrak agreed to maintain the tracks and all associated equipment - such as interlockings - in addition to performing dispatching for both the South Station terminal and the Providence line. I'm not actually sure who owns the Tower 1 PLC and the equipment - it might be the MBTA or Amtrak.
Amtrak is now trying to change that agreement, but for now, it is still in effect. There are number of practical and logical reasons why this arrangement makes sense:
1) Amtrak owns the 'catenary' - the overhead wires to supply electric power to the Amtrak electric trains. Having one entity responsible for maintaining the tracks and another for the catenary would result in complicated coordination being required to do anything on the line.
2) The tracks need to be maintained to a higher standard than other lines since the Acela trains travel at 150 mph for certain stretches of the line. MBTA / Keolis don't require this higher standard track, so they shouldn't have to pay for that maintenance.
3) see comment below. I got this item wrong: The dispatch system incorporates power management for the catenary. So the dispatcher manages not only the train traffic, but also certain aspects of electrical load management for the electric Amtrak trains. MBTA / Keolis have no need for this, so having their dispatcher do this would require either compensation or a different Amtrak employee to do this function.
Therefore, Amtrak is responsible for the operation and maintenance of the Tower 1 interlocking and all associated components. The MBTA and Keolis have no responsibility or ability to intervene with this Amtrak system. That isn't necessarily a bad thing - the Northeast Corridor is the cash cow for Amtrak that subsidizes their entire nationwide system. So it gets as good treatment as it can and is a high priority for Amtrak, although just like everything else across the country, there is a backlog of needed capital spending for the Northeast Corridor. I'm not sure how that might affect this particular PLC, but we all know the MBTA isn't sitting on a pile of money to keep things maintained to the level they should be either.
I have no idea how old the Tower 1 PLC is, or what the condition of it is. Most of the switches and track within the Tower 1 interlocking were replaced as a result of the Big Dig construction in the early 2000's, but that was ~15 years ago now.
Hopefully they can get it fixed quickly!