“It’s Déjà Vu All Over Again” as United Kingdom (UK) flights face disruption … again.

A recurring UK (and international) traveler nightmare

On September 21st, disruptions again are affecting flights in/to/from the UK. As in the major September 8th outage, the cause is a “technical issue” in the National Air Traffic Services (NATS) systems that provide UK air traffic control.

A 1 millisecond event caused the prior September 8th NATS outage. Recovery of normal NATS operations took around 6 hours. The effects caused over 2,000 flight cancellations and more than two days for air travel recovery.

The September 8th event investigation (https://www.nats.aero/news/nats-publishes-preliminary-report-on-technical-incident-of-8-september/) currently pins the cause on a failure for a paused operation to resume correctly. The failure to resume correctly created corrupted output which then caused other system issues. Pausing lower priority or other operations briefly is not unusual. Not resuming normally, however, should be unusual. The defect, called by some a “rare sequence” problem due to the about 1 millisecond window, had been in the software for some time.

The perhaps more concerning problem in the September 8th NATS outage was in software “downstream” from the defect … in this case, in the London Area Control (LAC) system. The LAC system is where the cascading air traffic control issues began. That system detected input data issues but only timed out the connection between NATS and LAC. Investigating engineers initially found no indications of underlying data corruption. Repeated attempts to process corrupted data eventually led to some LAC system functions becoming essentially unavailable.

Why is this interesting from a medical device software perspective?

Medical device disruptions can be caused by similar issues.

If there is an ability to pause and resume medical device software functions, then any pause-resume sequences will likely need both normal mode and abnormal mode testing. Test cases would also have diversity scaled to the risk posed by a resumption error.

An example: an infusion pump has an alarm feature for notifying a health care professional (HCP), e.g., reservoir low that can endanger the patient. The alarm software functionality also allows for the HCP to pause the alarm while resolving the critical event. What if:

• the alarm no longer works after pause?
• the alarm, while paused, preempts subsequent alarms?
• primary pump functions are somehow disrupted by alarm function corruption?
• a cybersecurity vulnerability creates the alarm issue(s)?

Two common ways to attack this type of disruption

There are lots of ways to reduce or prevent software defects. Here are two, one an architectural approach and the other a software detail design approach:

• Architectural diagrams can increase visibility and understanding of the design. Sequence, state, or event-driven architectural diagrams can make identification of key faults much easier.
• Input data checking. Implementing input checks and providing exception processing capabilities is not “cutting edge” engineering. Considering those two aspects should be part of detailed design for any software function.

Other considerations

The full September 8th disruption investigation and report should contain more details and lessons learned for potential application to health systems. Some of the remaining unanswered questions are also good thinking points in general about systems reliability:

• How long had the defective code had been in service?
• Was the defective code recently changed?
• Why was the defect not previously found?
• How often does the event initiation scenario (a manual request into the system) occur?
• Do there need to be better ways of system health monitoring?
• Can the data corruption problem be isolated without restarting the system(s)?
• Are they any vendor issues, as the NATS system is provided by a third party?

You will recognize these questions address system development factors like quality controls, change impact analysis, configuration management, human factors engineering, and failure mode/exception processing … all pertinent to health system development.