All service stopped

runningwithscalpels wrote:Third Rail, the OCC isn't very old at all.

It was just a couple if years ago that they upgraded the whole OCC.

As someone said in here, this was yet another situation where the riding public knew more than the people out in the field working trains. We were told all power was lost in the OCC (which was completely revamped in its entirety a year or so ago) and they were unable to display signals or control the alignment of switches. The signals on the mainline, with the exception of the Danbury did not go dark. And after the power was restored everything was backed up and a lot of equipment were out of cycle. Lot of equipment trains were running after 0001hr to their respected terminals to revert back to normal cycle.

RearOfSignal wrote:I think most employees here would agree that there are aspects of the railroad that are mismanaged, it's definitely true. However, not everything falls under that category; sometimes things just happen well-managed or not. Often times the public and those who post here mistakenly put an incident in the wrong category without knowing the facts. What happened yesterday was probably a mismanagement, not having redundant systems. However nothing will ever be fail proof, and problems are bound to arise, things that no manager public or private could control.

If they indeed did not have a backup power supply for the OCC, that is absolutely a management failure, and heads should roll at 347 Madison.

LoHud.com:

The MTA, Metro-North’s parent agency, said in the press release that the computers that run the railroad’s signal system lost reliable power at 7:45 p.m. when one of two main power supply units was taken out of service for replacement. The technicians didn’t realize that a wire was disconnected on the other main power supply unit. It took more than hour before a backup unit could be connected to power up the system.

This is a classical failure mode. Here's a hint: If you are going to be relying upon your redundancy for a project like this, you should check that your failover device is actually connected correctly and operational. Failure to do so is indicative of poor planning. It also indicates that the redundant power supply was never tested...

Ridgefielder wrote:

RearOfSignal wrote:I think most employees here would agree that there are aspects of the railroad that are mismanaged, it's definitely true. However, not everything falls under that category; sometimes things just happen well-managed or not. Often times the public and those who post here mistakenly put an incident in the wrong category without knowing the facts. What happened yesterday was probably a mismanagement, not having redundant systems. However nothing will ever be fail proof, and problems are bound to arise, things that no manager public or private could control.

If they indeed did not have a backup power supply for the OCC, that is absolutely a management failure, and heads should roll at 347 Madison.

MNR just announced that last night's outage was a result of human error. In short someone didn't realize that they pulled the wrong plug during routine maintenance. I guess a skilled manager would have assured this couldn't have happened, but it appears to be a big wet, noisy, brain fart. Even more training and testing and firing of people can't prevent all accidents. But no one wants to hear that.

Ridgefielder wrote:If they indeed did not have a backup power supply for the OCC, that is absolutely a management failure, and heads should roll at 347 Madison.

From the latest info it appears that they do in fact have dual redundant power supplies, but that due to human error (either yesterday or at installation time) the redundancy was ineffective. My read is that they needed to replace the primary (probably were seeing imminent fault warning, voltage drift, etc.) - but didn't verify that the alternate supply was properly connected and operational.

There are two major failings indicated:
1) Lack of regular testing [or insufficient testing plan] for power outage scenarios
2) Insufficient planning / prediction of potential breakdowns for the power supply replacement task

Both of those are indicative of poor management oversight.

spidey3 wrote:

Ridgefielder wrote:If they indeed did not have a backup power supply for the OCC, that is absolutely a management failure, and heads should roll at 347 Madison.

From the latest info it appears that they do in fact have dual redundant power supplies, but that due to human error (either yesterday or at installation time) the redundancy was ineffective. My read is that they needed to replace the primary (probably were seeing imminent fault warning, voltage drift, etc.) - but didn't verify that the alternate supply was properly connected and operational.

There are two major failings indicated:
1) Lack of regular testing [or insufficient testing plan] for power outage scenarios
2) Insufficient planning / prediction of potential breakdowns for the power supply replacement task

Both of those are indicative of poor management oversight.

MNR has plenty of plans for extended signal outage, but it was quicker to fix problem downtown then to setup trains for signal outage. MNR implemented such a plan shortly after hurricane Irene destroyed tracks and signaling equipment.

Like they used to say on AM radio-

The hits just keep on comin'!

So many know-it-alls who never worked a day on any railroad making rude comments and offering their uniformed opinions is laughable. You people have no idea how complicated it is to run trains when the entire signal and train tracking system goes down. There were literally hundreds of train on all of MN's territories when the system crashed, and it took over an hour just to identify the location of each individual train. Then they had to issues hundreds of train orders just to get the trains moving slowly. Thankfully when a system like this crashes, all the trains can only go at a very slow speed and safety was never at risk. Of all the problems that MN has had recently, this is not a sign of any symptomatic failure. That's like blaming MN for the power failure at Mt. Vernon. Some things are beyond their control. It's not as easy as some people think. This type of problem has happened on several other railroads without all the hoopla and outrage I see here.

http://abclocal.go.com/wabc/story?secti ... id=9405341" onclick="window.open(this.href);return false; for explanation.

RearOfSignal wrote:MNR has plenty of plans for extended signal outage, but it was quicker to fix problem downtown then to setup trains for signal outage. MNR implemented such a plan shortly after hurricane Irene destroyed tracks and signaling equipment.

No argument with that - but it is not what I am talking about.

I get that you all know rail operations from the inside, and with far greater detail than I do.
I concede that this is your territory.

But this was an IT failure, and as an IT pro, with 25 years experience running mission-critical systems, I can state with full confidence that this is my territory.

And based upon that experience I can tell you that the fact that the computers went down at all is a major failure on the part of the IT hardware folks, and a black eye for their managers.

runningwithscalpels wrote:

F-line to Dudley via Park wrote:Jesus Christ, who let NHAirhead re-register.

LOL somehow the argument of needing diesels and RDC's would have been brought up if it were him

HAHAHAHAHA.

F-line to Dudley via Park wrote:who let NHAirhead re-register.

If this is the same person then there might be multiple accounts that have already been created; just look at the "date joined" info on each one. Smells fishy if you ask me..

Clean Cab wrote:This type of problem has happened on several other railroads without all the hoopla and outrage I see here.

Lucky for them they don't have politishians like Schumer and Malloy crying foul from the peanut gallery

NH2060 wrote:

F-line to Dudley via Park wrote:who let NHAirhead re-register.

If this is the same person then there might be multiple accounts that have already been created; just look at the "date joined" info on each one. Smells fishy if you ask me..

Clean Cab wrote:This type of problem has happened on several other railroads without all the hoopla and outrage I see here.

Lucky for them they don't have politishians like Schumer and Malloy crying foul from the peanut gallery

His attitude there in the last post did seem rather snippy like Airhead.. Oh geez.

Clean-cab: Is the "uniformed" opinion that you refer to above the opposite of a "plainclothes" opinion? Just curious...... Maybe a little proof-reading would help?