• Operations Computer failure on Sept 18

  • Discussion of the operations of CSX Transportation, from 1980 to the present. Official site can be found here: CSXT.COM.
Discussion of the operations of CSX Transportation, from 1980 to the present. Official site can be found here: CSXT.COM.

Moderator: MBTA F40PH-2C 1050

  by twropr
 
Am told this happened in the Jax operations center. Apparently Selkirk and the Albany Div. were not affected. What I don't understand is how the regionalized dispatchers in places like Baltimore and Florence were not able to control signals/switches if the outage was in Jax.

Andy
  by 10more years
 
To me, what is more relevant is what happened and what measures are in place to prevent it happening again? They told us the whole system was down, but that could have been just a phrase. You wonder if there is a "back-up" plan in place. Of course, the "TECS" system shuts down every week for an hour for "maintenance" on schedule ? A major railroad practically shuts down for five hours. Almost nothing moves. It was almost like a movie, except no media, no PR and no high speed disaster (or Denzel Washington / Bruce Willis action scenes.)
  by roadster
 
Monday morning our dispatcher were reverifying our Disp. Bulletin messages to m ake sure we had all the required messages due to "computer issues" the night before. Mostly verifying the listing of the Rule 89 Foreman controled work zones within the Subs we're were running on.
  by mmi16
 
twropr wrote:Am told this happened in the Jax operations center. Apparently Selkirk and the Albany Div. were not affected. What I don't understand is how the regionalized dispatchers in places like Baltimore and Florence were not able to control signals/switches if the outage was in Jax.

Andy
The CADS computers that serve Florence, Baltimore, Louisville, Nashville and Jacksonville are still the computers that serviced those locations when they were in Jacksonville. It has not been made public what the cause of the failure was.

Approximately 10 years ago, a piece of malware somehow penetrated the CSX mainframe and overloaded the communications circuits between the Mainframe and other standalone computers such as the Train Dispatching CADS computer. The overloaded communications circuits repeatedly crashed the CADS computers....it took several hours of repeated CADS downtime for the Cyber experts to first figure out what was happening and more hours to attempt to develop a strategy and create programming to defeat the malware. For approximately 12 hours NO trains moved as Dispatchers could not 'track' the trains location and were not in a position to provide the required levels of protection to facilitate movement without signals that could not be lined. After the problem was diagnosed the comm links were disabled and Dispatchers were able to reliably line signals and issue the various authorities to trains and on track workers; however, alternative means had to be utilized to capture and/or distribute data that the CADS system and the Mainframe pass back and forth between themselves.

I understand the heads of several people in the computer security department were looking for new employment the following week.