ANGEL MACIAS, SOFTWARE ENGINEER
01
At Merlin, one of our design principles is to focus on using technology to build skills that machines can perform better than the human pilots they are assisting. This principle is exemplified in the Automated Checklist (AC) system we’re building to bolster compliance with flight checklists. This innovative technology, though still in the experimental phase, promises to significantly enhance flight safety by automating dual concurrence checks in the flight deck.
Today, pilots rely on printed or electronic checklists to ensure all necessary steps are completed during various phases of flight, such as pre-flight, takeoff, cruising, and landing. These checklists are methodically designed to cover critical systems and procedures, ensuring that nothing is overlooked. In a two-pilot flight deck, pilots are trained to follow a challenge-and-response format, where one pilot reads out an item, and the other confirms its completion. This process relies heavily on human vigilance, which can be fallible, especially under stress or high workload.
In an infamous fatal aviation accident, Spanair Flight 5022 crashed shortly after takeoff at least in part due to improper compliance with two different checklist sequences, resulting in the failure to deploy high lift devices (flaps and slats) that were required for a proper takeoff configuration. This and other similar accidents inspired the development of Merlin’s Automated Checklist system, which takes advantage of machine systems’ ability to never skip a checklist step.
Some items on a checklist are so critical that they require dual concurrence, meaning that in a two-pilot flight deck, both pilots must agree that the item has been completed. When operated by just one pilot, the Automated Checklist system is able to provide real-time verification of checklist items by cross-referencing the pilot’s actions with known procedural requirements. As pilots complete each step, the system independently verifies that the action has been performed correctly. This dual-layer verification process ensures that any missed steps or incorrect procedures are immediately flagged, prompting pilots to correct them before proceeding.
The automation of flight checklists involves multiple major technological elements including:
- A Natural Language Processing (NLP) system to facilitate the verbal challenges and prompts between our system and pilots,
- An intuitive User Interface (UI) system for enabling visual feedback about a checklist’s usage and progress,
- Hardware integration for aircraft signal monitoring, which we’re testing with systems like Merlin’s Certification System Bench,
- A computer vision system for monitoring certain required inputs and minimizing the invasiveness of the Automated Checklist system in development.
The above is an example of a “before takeoff” checklist that requires a mix of physical switch and button movement and visual checks. Digital versions of this exist, but they’re typically not much more than a to-do list of things one would see on a phone app. The Automated Checklist system, by comparison, actively determines the state of the aircraft and interacts with the pilot. To do this, it uses digital signals from other parts of the aircraft and from the computer vision system dedicated to monitoring checklist specific items in the flight deck. Further, it provides an interactive visual user interface along with an NLP system for verbal feedback and concurrence.
The above GIF shows an illustrative snippet of the interface used for Automated Checklists. While inaudible here, the workflow includes a voice audio back-and-forth between the pilot and the machine NLP system similar to the following:
Machine: “Parking brake”
Pilot: “Check”
Machine: “Seat belts”
Pilot: “Secure”
Machine “Flight Controls”
Pilot: “Free and clear”
Machine: “Flaps”
Pilot: “Flaps 20”
As the checklist progresses, the AC system challenges the pilot with a checklist item like “Parking brake” and the pilot responds with a verbal affirmative “Check” to mark that item as done and move forward. The AC system, simultaneously, is listening and looking for an affirmative response to move forward. For certain items, like “flight and engine instruments check” the AC system employs computer vision to monitor the analog signals displayed on the gauges of the airplane. The blue check on that step indicates that the AC system also did its own check to provide dual concurrence with the pilot.
Our Automated Checklist system represents a significant step in improving our flight tools with modern technology. Take it from one of our test pilots, Justin Gillen who told us:
“An automated challenge and response checklist function would be a great way to leverage digital cross checking features to alleviate human error and increase safety during risk-prone checklist phases of flights, especially emergency scenarios where human performance can be heavily saturated.”
If you’re looking to make an impact on the exciting and complex challenges that Merlin is taking on – we’re hiring!