CUS Amtrak/METRA Service Disruption February 28

Discussion related to Amtrak also known as the National Railroad Passenger Corp.

Moderators: GirlOnTheTrain, Amtrak67 of America, Tadman, gprimr1

CUS Amtrak/METRA Service Disruption February 28

Postby Gilbert B Norman » Fri Mar 01, 2019 7:45 pm

I must admit surprise, even from our regular Chicago area reporters, that this story took so long to report. Here is such from the Chicago Tribune:

Fair Use:

....debacle at Union Station on Thursday was caused by Amtrak’s decision to upgrade its servers during peak service hours and a worker falling on a circuit board, which turned off the computers and led to delays that affected more than 60,000 Chicago-area commuters.

The cause of Thursday’s incident was provided by Sen. Dick Durbin, D-Ill., who spoke to Amtrak CEO and President Richard Anderson. Amtrak spokesman Marc Magliari did not dispute Durbin’s characterization.

Durbin said Anderson admitted that Amtrak made a series of errors, the most significant of which was upgrading servers during its peak hours of service, rather than in the middle of the night when fewer trains are running.

“They tried to do a server upgrade, sadly, in the middle of a busy day, exactly the wrong time to do it,” said Durbin said in a phone interview Friday. A workman fell off a ladder and bumped against a panel, which caused the shutdown, said Durbin, who praised Anderson’s honesty.

A former CEO of Northwest Airlines and Delta Air Lines, Anderson told Durbin that Amtrak decentralized its communications system, which was “not a smart thing to do,” Durbin said. Each of the servers across the country is facing maintenance, and fragmentation does not make things easier, Durbin said. Durbin said Anderson plans to look at centralizing Amtrak’s communications system.

Durbin and U.S. Senator Tammy Duckworth, D-Ill., on Friday sent a joint letter calling on Amtrak to immediately implement necessary changes.


"Spinmeister" (whom I shall not refer to as "Dr. G..") Maglieri did what he could to soften the blow from the disruption to both Amtrak and METRA operations arising from the negligence of an Amtrak employee or contractor.

The matter was partially addressed for Thursday evening "rush"; the concern I hold is that some "Pol" around here will start questioning Amtrak's capacity and competence to have the fate of so many non-Amtrak passengers in their hands.
Gilbert B Norman
 
Posts: 13675
Joined: Fri Mar 12, 2004 6:52 am
Location: Clarendon Hills, IL (BNSF Chicago Sub; MP 18.71)

Re: CUS Amtrak/METRA Service Disruption February 28

Postby mmi16 » Fri Mar 01, 2019 8:45 pm

Sounds like pure scapegoating to me. In a Server Update, why would someone be up on a ladder to fall into equipment?
Never too old to have a happy childhood!
User avatar
mmi16
 
Posts: 947
Joined: Fri Apr 15, 2005 5:18 pm
Location: USA

Re: CUS Amtrak/METRA Service Disruption February 28

Postby BandA » Fri Mar 01, 2019 11:24 pm

Person on ladder could be running new Ethernet or fiber-optic cables or moving existing cables to different racks.
User avatar
BandA
 
Posts: 2551
Joined: Thu Oct 11, 2012 11:47 am

Re: CUS Amtrak/METRA Service Disruption February 28

Postby doepack » Sat Mar 02, 2019 12:48 am

A server upgrade at a busy terminal in mid-afternoon with dozens of movements pending as the evening rush hour beckons. What could possibly go wrong?

I can only hope this mistake isn't repeated, and common sense can prevail next time...
--Dorian--
User avatar
doepack
 
Posts: 2318
Joined: Sat May 07, 2005 5:20 am
Location: UP Geneva Subdivision, MP 25.0

Re: CUS Amtrak/METRA Service Disruption February 28

Postby eolesen » Sat Mar 02, 2019 12:15 pm

Human error, I have no doubt, but falling off a ladder? Unless they had a server in the ceiling, sounds unlikely.

I've done server upgrades in data centers and field closets alike... in a planned upgrade, the cabling is pulled days or weeks earlier, and everything racked & powered days prior to the change. Especially when you're in a shop like Chicago where different trades have to be involved.

Maybe it was someone totally unrelated to the upgrade...
eolesen
 
Posts: 342
Joined: Sat Apr 17, 2010 12:01 am

Re: CUS Amtrak/METRA Service Disruption February 28

Postby Greg Moore » Sat Mar 02, 2019 1:43 pm

Keep in mind a "ladder" can be a 3 step stepstool. I've certainly used those to get to the top machine in a rack etc.

Regardless, with the limited details, we honestly don't now enough. From what little I've read, there's a LOT of details missing.
Check out QuiCR, Quick, Crowdsourced Responses for businesses.
Greg Moore
 
Posts: 5253
Joined: Sat Mar 13, 2004 12:15 am
Location: IT Consultant

Re: CUS Amtrak/METRA Service Disruption February 28

Postby hs3730 » Sat Mar 02, 2019 6:42 pm

I've had to use a ladder to pull the cable slack up above the racks after plugging them in, for the network team to later work their magic and disappear them into the overhead guideways. I'd imagine the reason this was done in the daytime is the same reason I usually have to: a lot of different people all working on their piece of the puzzle, that's a lot of OT if done after hours (especially if 3rd party contractors are involved). 99% of the time nothing goes wrong.
They probably had lots of redundancy in place, but no one can plan for someone falling into a piece of equipment. Even with this setback, continuing the practice of doing work like this during business hours still saves money over the long run.
hs3730
 
Posts: 166
Joined: Mon Dec 10, 2007 8:30 pm
Location: Albany NY

Re: CUS Amtrak/METRA Service Disruption February 28

Postby mtuandrew » Sat Mar 02, 2019 8:09 pm

Mr. Olesen: what are you suggesting when you say it was someone else, and why would Amtrak hide it?
User avatar
mtuandrew
 
Posts: 5340
Joined: Tue Feb 20, 2007 2:59 am
Location: the Manassas Gap Independent Line

Re: CUS Amtrak/METRA Service Disruption February 28

Postby Backshophoss » Sat Mar 02, 2019 9:46 pm

IF you had to do this change over during the normal 1st shift hours ,why not run everything off the backup operations center to allow
the work to not affect train movements or run all effected areas off the maintainer's boards in Mi,CUS and NOL? :wink:
Backshophoss
 
Posts: 5832
Joined: Mon Mar 05, 2012 7:58 pm

Re: CUS Amtrak/METRA Service Disruption February 28

Postby east point » Sat Mar 02, 2019 10:38 pm

Anderson says it should be centralized. So a failure shuts all this down plus the NEC as well?

Save money by working during the day? What is the time line you are thinking of? 40 years?
east point
 
Posts: 1188
Joined: Tue Feb 16, 2016 1:50 pm

Re: CUS Amtrak/METRA Service Disruption February 28

Postby eolesen » Sat Mar 02, 2019 11:05 pm

mtuandrew wrote:Mr. Olesen: what are you suggesting when you say it was someone else, and why would Amtrak hide it?


While Chicago and cover-ups are no strangers, I don't think they're hiding anything. What I'm saying is it could have been a worker unrelated to the server upgrade that caused the damage.
eolesen
 
Posts: 342
Joined: Sat Apr 17, 2010 12:01 am

Re: CUS Amtrak/METRA Service Disruption February 28

Postby twropr » Tue Mar 05, 2019 7:44 pm

Most of us know about the disruption that took place last Thursday between approx. 8:30 am and 9 pm - the circuit board supposedly broken by the worker who fell on it. What has NOT been reported were the delays that took place AFTER the Chicago Terminal District was supposedly reopened.
On Amtrak, CITY OF NEW ORLEANS #59 departed at 9:29 pm; there were no more Amtrak departures until Lincoln Service #307 left at 2:13 Friday morning (7 h 13 min LT), followed by LAKE SHORE LIMITED #48 (4 h 58 min LT). So there were no Amtrak departures for 4 h 44 min. Inquiring minds want to know what happened during this lull.
Trains able to come in: #392 at 10:05 pm, #382 at 12:30 am, #306 at 1:05 am and #355 at 3 am
Andy
twropr
 
Posts: 652
Joined: Tue Dec 29, 2009 6:26 pm

Re: CUS Amtrak/METRA Service Disruption February 28

Postby eolesen » Wed Mar 06, 2019 8:31 am

There was a pretty significant commuter backlog.

Sure, it impacted a couple hundred Amtrak riders, but tens of thousands of Metra riders were impacted, and there's already enough friction between tenant and landlord right now.
eolesen
 
Posts: 342
Joined: Sat Apr 17, 2010 12:01 am

Re: CUS Amtrak/METRA Service Disruption February 28

Postby Tadman » Thu Mar 07, 2019 10:05 am

I spoke to a local person in Chicago that has very good reason to know, and they confirmed the ladder story. That person also said the folks doing an upgrade mid-week were idiots for doing so.
User avatar
Tadman
 
Posts: 9043
Joined: Wed Sep 01, 2004 10:21 am

Re: CUS Amtrak/METRA Service Disruption February 28

Postby n2cbo » Thu Mar 07, 2019 10:13 am

hs3730 wrote:I've had to use a ladder to pull the cable slack up above the racks after plugging them in, for the network team to later work their magic and disappear them into the overhead guideways. I'd imagine the reason this was done in the daytime is the same reason I usually have to: a lot of different people all working on their piece of the puzzle, that's a lot of OT if done after hours (especially if 3rd party contractors are involved). 99% of the time nothing goes wrong.
They probably had lots of redundancy in place, but no one can plan for someone falling into a piece of equipment. Even with this setback, continuing the practice of doing work like this during business hours still saves money over the long run.


Excuse me..... This is what I do for a living. You N E V E R ---- E V E R make A N Y changes to a production system during peak times!!! Especially for critical infrastructure (which this apparently was since it caused all the chaos) What were they thinking???
n2cbo
 
Posts: 414
Joined: Tue Jul 24, 2007 10:29 pm

Next

Return to Amtrak

Who is online

Users browsing this forum: Majestic-12 [Bot] and 7 guests