10/24 systems outage

Sounds like a control center failure at approx 6:30pm. As we dumped air just south of Ivy Ridge I tuned into the north road radio. All trains instructed to stop at nearest station.

As of 645pm individual trains are requesting permission to proceed. Sounds like most are still at a standstill.

Any word on what's going on?

scotty269 wrote:Sounds like a control center failure at approx 6:30pm. As we dumped air just south of Ivy Ridge I tuned into the north road radio. All trains instructed to stop at nearest station.

As of 645pm individual trains are requesting permission to proceed. Sounds like most are still at a standstill.

Any word on what's going on?

From SEPTA's twitter about 2 hours ago

RRD: Trains are operating with delays of up to 30 minutes due to signal issues.

Must have been quite the signal issue. I was stopped for about 25 mins on the inner northbound track at North Broad, while an associate was stuck at the same time northbound at Melrose Park. In the end it wasn't a big deal, but it was unusual.. Guess they had to reboot a system.

oh boy

http://www.philly.com/philly/blogs/in-t ... night.html" onclick="window.open(this.href);return false;

SEPTA Regional Rail riders all experienced an unexpectedly long commute Monday night when dispatchers lost the ability to log into the system that controls signaling on the railroad.

The system's software froze from 6:18 p.m. to 6:38 p.m., said Andrew Busch, a SEPTA spokesman, affecting all 13 of the Regional Rail's lines simultaneously

scotty269 wrote:oh boy

http://www.philly.com/philly/blogs/in-t ... night.html" onclick="window.open(this.href);return false;

SEPTA Regional Rail riders all experienced an unexpectedly long commute Monday night when dispatchers lost the ability to log into the system that controls signaling on the railroad.

The system's software froze from 6:18 p.m. to 6:38 p.m., said Andrew Busch, a SEPTA spokesman, affecting all 13 of the Regional Rail's lines simultaneously

Must have been running on Windows 10 8^).......

n2cbo wrote:

scotty269 wrote:oh boy

http://www.philly.com/philly/blogs/in-t ... night.html" onclick="window.open(this.href);return false;

SEPTA Regional Rail riders all experienced an unexpectedly long commute Monday night when dispatchers lost the ability to log into the system that controls signaling on the railroad.

The system's software froze from 6:18 p.m. to 6:38 p.m., said Andrew Busch, a SEPTA spokesman, affecting all 13 of the Regional Rail's lines simultaneously

Must have been running on Windows 10 8^).......

Funny enough, I did once see an overnight reset of an older model NJ Transit ticket machine and found that the operating system was Windows NT.

JeffersonLeeEng wrote:
Must have been running on Windows 10 8^).......

Damn those Windows updates!

Amazing that this made the news outlets, the system crashed, trains were told to wait, the safest action was taken and it is perceived as a bad thing? At least that's how it reads, btw there was no mention of the Amtrak signal problems the following day that delayed some trains up to 60 minutes!!!

Limited-Clear wrote:Amazing that this made the news outlets, the system crashed, trains were told to wait, the safest action was taken and it is perceived as a bad thing? At least that's how it reads, btw there was no mention of the Amtrak signal problems the following day that delayed some trains up to 60 minutes!!!

The news media seems a bit sensitive to SEPTA's train issues since the SLV debacle. Anything that causes continued long delays to commuters will make the news, no matter how safe or routine or correct the response may seem to others.

The general public also has quite high expectations about public transport. If it were one line that were delayed, I think it would have been a non-issue (your Amtrak example). But the fact that one computer glitch took down the whole network for 30 minutes is going to be unacceptable to quite a lot of people.

Limited-Clear wrote:Amazing that this made the news outlets, the system crashed, trains were told to wait, the safest action was taken and it is perceived as a bad thing? At least that's how it reads, btw there was no mention of the Amtrak signal problems the following day that delayed some trains up to 60 minutes!!!

There were more Amtrak signal problems last night, too. Something on the Trenton Line, I think. I've heard late trains being caused by Amtrak signal problems in the past, but it does seem to happen more often lately. Or am I imagining that?

I feel compelled to add that this type of thing is a big concern of mine in the future--not just with railroads but with infrastructure in general. While there's no evidence (from what's been reported) that this incident was hacker-related, the security of networks such as SEPTAs (though I don't actually know about SEPTA in particular), generally lags far behind capability. In other words, in an effort to automate more "essential" tasks of railroads, we place ourselves at a bit of a risk from hackers who may want to hurt US interests, and economically, shutting down a major rail system is very damaging. A well-coordinated attack could shut down SEPTA, Amtrak, (etc) for days/weeks, and I'm not sure if the aforementioned parties are adequately prepared for that.

Just my two cents.

I doubt that's much of an issue as I assume most signal systems date before the invention of the Internet, so would be on a separate "network". Still, any control system should be physically separated from the Internet (aka "air gapped") and not allowed to be connected to it.

Air gapping is ideal, until you get that ONE manager somewhere up there that absolutely positively must be able to access the system from his home at 2am on a Saturday, because...who knows, but he does.

I fight this all the time with my water/wastewater customers. The typical excuse is "Our VPN has a password, and there's a warning when you log into the system".

Still, I have to wonder about a system where taking down the central computer kills everything. Ok, yeah, it failed safe. I dropped out the computers to a water treatment plant the other week for a few hours. There was no shutdowns, no emergencies, no change at all in the water quality, because the low down controls did their job without the head end computer running. This is more common than you'd think, and with some darn large systems.

The head-end computer should at most be a data collection, and centralized command machine that maybe applies some sanity checking. The local stuff should be able to do its job without it.

I'm guessing this is an Arinc system? I've seen their attempts at SCADA, and heard what they cost. I'm in the wrong industry segment...

If the system hadn't come back online when It was rebooted they would have gone to the old way of moving trains, by paperwork, it would have been slow but trains would have moved.