All service stopped

Discussion relating to the operations of MTA MetroNorth Railroad including west of Hudson operations and discussion of CtDOT sponsored rail operations such as Shore Line East and the Springfield to New Haven Hartford Line

Moderators: GirlOnTheTrain, nomis, Jeff Smith, FL9AC

RearOfSignal
Posts: 2546
Joined: Tue Aug 09, 2005 2:31 pm

Re: All service stopped

Post by RearOfSignal » Fri Jan 24, 2014 12:09 pm

runningwithscalpels wrote:Third Rail, the OCC isn't very old at all.


It was just a couple if years ago that they upgraded the whole OCC.
Hurry up and wait at the signal!

Steamboat Willie
Posts: 717
Joined: Mon Aug 11, 2008 8:43 pm
Location: Iron Highway

Re: All service stopped

Post by Steamboat Willie » Fri Jan 24, 2014 1:16 pm

As someone said in here, this was yet another situation where the riding public knew more than the people out in the field working trains. We were told all power was lost in the OCC (which was completely revamped in its entirety a year or so ago) and they were unable to display signals or control the alignment of switches. The signals on the mainline, with the exception of the Danbury did not go dark. And after the power was restored everything was backed up and a lot of equipment were out of cycle. Lot of equipment trains were running after 0001hr to their respected terminals to revert back to normal cycle.

Ridgefielder
Posts: 2609
Joined: Sat Aug 02, 2008 7:22 pm
Location: Harlem Division MP 15

Re: All service stopped

Post by Ridgefielder » Fri Jan 24, 2014 1:22 pm

RearOfSignal wrote:I think most employees here would agree that there are aspects of the railroad that are mismanaged, it's definitely true. However, not everything falls under that category; sometimes things just happen well-managed or not. Often times the public and those who post here mistakenly put an incident in the wrong category without knowing the facts. What happened yesterday was probably a mismanagement, not having redundant systems. However nothing will ever be fail proof, and problems are bound to arise, things that no manager public or private could control.

If they indeed did not have a backup power supply for the OCC, that is absolutely a management failure, and heads should roll at 347 Madison.

spidey3
Posts: 97
Joined: Fri Jun 04, 2010 12:11 pm

Re: All service stopped

Post by spidey3 » Fri Jan 24, 2014 1:33 pm

LoHud.com:
The MTA, Metro-North’s parent agency, said in the press release that the computers that run the railroad’s signal system lost reliable power at 7:45 p.m. when one of two main power supply units was taken out of service for replacement. The technicians didn’t realize that a wire was disconnected on the other main power supply unit. It took more than hour before a backup unit could be connected to power up the system.


This is a classical failure mode. Here's a hint: If you are going to be relying upon your redundancy for a project like this, you should check that your failover device is actually connected correctly and operational. Failure to do so is indicative of poor planning. It also indicates that the redundant power supply was never tested...

RearOfSignal
Posts: 2546
Joined: Tue Aug 09, 2005 2:31 pm

Re: All service stopped

Post by RearOfSignal » Fri Jan 24, 2014 1:33 pm

Ridgefielder wrote:
RearOfSignal wrote:I think most employees here would agree that there are aspects of the railroad that are mismanaged, it's definitely true. However, not everything falls under that category; sometimes things just happen well-managed or not. Often times the public and those who post here mistakenly put an incident in the wrong category without knowing the facts. What happened yesterday was probably a mismanagement, not having redundant systems. However nothing will ever be fail proof, and problems are bound to arise, things that no manager public or private could control.

If they indeed did not have a backup power supply for the OCC, that is absolutely a management failure, and heads should roll at 347 Madison.


MNR just announced that last night's outage was a result of human error. In short someone didn't realize that they pulled the wrong plug during routine maintenance. I guess a skilled manager would have assured this couldn't have happened, but it appears to be a big wet, noisy, brain fart. Even more training and testing and firing of people can't prevent all accidents. But no one wants to hear that.
Hurry up and wait at the signal!

spidey3
Posts: 97
Joined: Fri Jun 04, 2010 12:11 pm

Re: All service stopped

Post by spidey3 » Fri Jan 24, 2014 1:43 pm

Ridgefielder wrote:If they indeed did not have a backup power supply for the OCC, that is absolutely a management failure, and heads should roll at 347 Madison.

From the latest info it appears that they do in fact have dual redundant power supplies, but that due to human error (either yesterday or at installation time) the redundancy was ineffective. My read is that they needed to replace the primary (probably were seeing imminent fault warning, voltage drift, etc.) - but didn't verify that the alternate supply was properly connected and operational.

There are two major failings indicated:
1) Lack of regular testing [or insufficient testing plan] for power outage scenarios
2) Insufficient planning / prediction of potential breakdowns for the power supply replacement task

Both of those are indicative of poor management oversight.

RearOfSignal
Posts: 2546
Joined: Tue Aug 09, 2005 2:31 pm

Re: All service stopped

Post by RearOfSignal » Fri Jan 24, 2014 2:06 pm

spidey3 wrote:
Ridgefielder wrote:If they indeed did not have a backup power supply for the OCC, that is absolutely a management failure, and heads should roll at 347 Madison.

From the latest info it appears that they do in fact have dual redundant power supplies, but that due to human error (either yesterday or at installation time) the redundancy was ineffective. My read is that they needed to replace the primary (probably were seeing imminent fault warning, voltage drift, etc.) - but didn't verify that the alternate supply was properly connected and operational.

There are two major failings indicated:
1) Lack of regular testing [or insufficient testing plan] for power outage scenarios
2) Insufficient planning / prediction of potential breakdowns for the power supply replacement task

Both of those are indicative of poor management oversight.


MNR has plenty of plans for extended signal outage, but it was quicker to fix problem downtown then to setup trains for signal outage. MNR implemented such a plan shortly after hurricane Irene destroyed tracks and signaling equipment.
Hurry up and wait at the signal!

User avatar
Tommy Meehan
Posts: 3340
Joined: Tue May 11, 2004 7:00 am
Location: Yonkers

Re: All service stopped

Post by Tommy Meehan » Fri Jan 24, 2014 2:08 pm

Like they used to say on AM radio-

The hits just keep on comin'!

Clean Cab
Posts: 2893
Joined: Tue Oct 25, 2005 4:18 pm
Location: South Dennis Massachusetts

Re: All service stopped

Post by Clean Cab » Fri Jan 24, 2014 2:14 pm

So many know-it-alls who never worked a day on any railroad making rude comments and offering their uniformed opinions is laughable. You people have no idea how complicated it is to run trains when the entire signal and train tracking system goes down. There were literally hundreds of train on all of MN's territories when the system crashed, and it took over an hour just to identify the location of each individual train. Then they had to issues hundreds of train orders just to get the trains moving slowly. Thankfully when a system like this crashes, all the trains can only go at a very slow speed and safety was never at risk. Of all the problems that MN has had recently, this is not a sign of any symptomatic failure. That's like blaming MN for the power failure at Mt. Vernon. Some things are beyond their control. It's not as easy as some people think. This type of problem has happened on several other railroads without all the hoopla and outrage I see here.
I'm stuck on a sandbar on Cape Cod, and I couldn't be happier!!!

DutchRailnut
Posts: 22147
Joined: Thu Mar 11, 2004 8:02 pm
Location: released from Stalag 13
Contact:

Re: All service stopped

Post by DutchRailnut » Fri Jan 24, 2014 2:36 pm

If Conductors are in charge, why are they promoted to be Engineer???

Retired Triebfahrzeugführer. I am not a moderator.

spidey3
Posts: 97
Joined: Fri Jun 04, 2010 12:11 pm

Re: All service stopped

Post by spidey3 » Fri Jan 24, 2014 2:51 pm

RearOfSignal wrote:MNR has plenty of plans for extended signal outage, but it was quicker to fix problem downtown then to setup trains for signal outage. MNR implemented such a plan shortly after hurricane Irene destroyed tracks and signaling equipment.

No argument with that - but it is not what I am talking about.

I get that you all know rail operations from the inside, and with far greater detail than I do.
I concede that this is your territory.

But this was an IT failure, and as an IT pro, with 25 years experience running mission-critical systems, I can state with full confidence that this is my territory.

And based upon that experience I can tell you that the fact that the computers went down at all is a major failure on the part of the IT hardware folks, and a black eye for their managers.

ACeInTheHole
Posts: 2785
Joined: Fri May 07, 2010 10:31 pm

Re: All service stopped

Post by ACeInTheHole » Fri Jan 24, 2014 3:32 pm

runningwithscalpels wrote:
F-line to Dudley via Park wrote:Jesus Christ, who let NHAirhead re-register.

LOL somehow the argument of needing diesels and RDC's would have been brought up if it were him :)

HAHAHAHAHA.

NH2060
Posts: 1512
Joined: Thu Feb 25, 2010 4:44 pm

Re: All service stopped

Post by NH2060 » Fri Jan 24, 2014 6:02 pm

F-line to Dudley via Park wrote:who let NHAirhead re-register.

If this is the same person then there might be multiple accounts that have already been created; just look at the "date joined" info on each one. Smells fishy if you ask me..
Clean Cab wrote:This type of problem has happened on several other railroads without all the hoopla and outrage I see here.

Lucky for them they don't have politishians like Schumer and Malloy crying foul from the peanut gallery ;-)

ACeInTheHole
Posts: 2785
Joined: Fri May 07, 2010 10:31 pm

Re: All service stopped

Post by ACeInTheHole » Fri Jan 24, 2014 7:11 pm

NH2060 wrote:
F-line to Dudley via Park wrote:who let NHAirhead re-register.

If this is the same person then there might be multiple accounts that have already been created; just look at the "date joined" info on each one. Smells fishy if you ask me..
Clean Cab wrote:This type of problem has happened on several other railroads without all the hoopla and outrage I see here.

Lucky for them they don't have politishians like Schumer and Malloy crying foul from the peanut gallery ;-)

His attitude there in the last post did seem rather snippy like Airhead.. Oh geez.

Head-end View
Posts: 2608
Joined: Thu Mar 11, 2004 10:00 pm
Location: The second row on a SEPTA Silverliner V

Re: All service stopped

Post by Head-end View » Fri Jan 24, 2014 9:30 pm

Clean-cab: Is the "uniformed" opinion that you refer to above the opposite of a "plainclothes" opinion? Just curious...... Maybe a little proof-reading would help?

Post Reply

Return to “MTA Metro-North Railroad and CtDOT Passenger Rail”