The Boeing 737 MAX: When Humans and Technology Don't Mix

good evening everyone thank you all for coming it's a pleasure to be your keynote speaker tonight um thank you to all the conference staff who put all this together of the NDC conferences are always a great time for the speakers uh so I'm Kyle and I'm here to talk about the Boeing 737 Max a plan that you've probably heard about in the media it was a Hot Topic back in 2019 just before kovitz stole the spotlight uh I nobody actually told me I was going to be the party speaker tonight um I mean I should have seen it coming because in in Oslo last year when I was there uh Henriette told me that I had been declared an edutainment speaker um which is a bit of a blow to my academic self-esteem but I'll take what I can get uh so with that in mind I'm gonna try to switch it up a little bit and keep it a bit lighter than I usually do I mean obviously it's a very serious topic and so uh the party speech was an interesting choice but um I'm happy to make it work so I've been trying to think of some other titles that might be more fun um you know when humans and Technology don't mix it it gets the point across but maybe how to irreparably damage your company in three easy steps how to get a 62 million dollar CEO retirement package it takes 102 years to build a reputation and two crashes to destroy it so when I do this I also like to have some audience interaction because it's fun right it keeps things interesting gets everyone engaged and normally we do this with like a who here has done something is something and get a show of hands but that's boring so we're not going to do that instead in the spirit of where we are when I ask this question if it applies to you School nice and loud with energy similarly uh when I ask somewhat rhetorically was this a good idea or does this sound like a good idea you can either respond with absolutely nice and loud or hell no it is not a good idea so let's do some trial runs here who here is enjoying the conference was it a good idea to come all right who had to fly to get here all right when I was flying on a Boeing plane across the Atlantic to get here I was watching and taking notes on on a documentary about Boeing planes crashing um the passengers around me and the flight attendants were getting a little uncomfortable was that a good idea a mixed reaction on that but yeah so a bit about me I'm Kyle I went to MIT to do my PhD there in aerospace engineering focusing on humans and Aerospace human systems integration basically how we Design Technology uh complex technological systems for humans to be able to operate them efficiently and safely without harming the systems or harming the humans and after I did that I started my own company in victim Labs we focus on prototyping of various systems Hardware software Cloud iot that kind of stuff and we're based in Ottawa which is the beautiful national capital of Canada and I also work with a company called top towel which is a Global Network of Freelancers who are pretty skilled at what they do and they basically bring you clients which is very nice for me what I'm going to talk about today I want to note are things to consider right we're talking about pretty high level Concepts uh there's way too much technical detail for me to be giving prescriptive instructions on what you should or should not do so we're talking about things to think about when you're designing your systems not directions on how to do it and we come to this topic called human factors engineering human systems integration it's all kind of tied together and when I say this word you know people say okay I know what that is that's ux and UI right I do that and that's true it does include those components but it's much more than just that you have your biomechanics how bodies move and operate sensory input how you actually collect information from your different senses often we usually use visual we always use Visual and sometimes audio but when we start talking about more complex systems particularly aircraft we start getting into haptic feedback as well which is touch the sense of touch and we'll get into that a bit more later cognitive processing how you actually process all this information you're getting and what you do with it ergonomics which is how you design things like cockpits such that Pilots can sit in them and actually reach all the different controls under things like high g-forces or other weird conditions and then mental models which is something we're going to come back to again later which is this idea that in your mind you understand what the system is thinking and you're thinking in the same way So This Together encompasses human factors engineering and I also want to touch on life critical systems I think a lot of people come to my talk thinking hey aircraft are cool I want to learn about them and them crashing and well let's get a show of hands who here is here because aircraft are cool okay thank you uh but it's it's it does include aircraft yeah but it's so much more than that and the reason that I think this is kind of relevant to this particular conference where there's been a number of talks on embedded and life critical Mission critical systems is that so many different topics can be considered life critical so many different pieces of software in an airplane for example obviously you have your avionics your engines your Control Systems um fuselage anything that can that if it breaks you're going to crash those are all life critical but it's also everything else right you think about the coffee pot in the galley if that software fails and that boils over and overflows and the liquid drops down through the through the floorboards and shorts out some circuits that is life critical system now right or it burns one of the flight attendants who might be critical for some operation of the aircraft Emergency Services anything to do with them their communication systems or vehicles or tools obviously live critical telecommunications is a is one that people don't often think about this way because I mean hey ktello Communications fail my cell phone doesn't work I can't text my grandma my cat pictures but it's actually much more important because now I can't call Emergency Services right A lot of people elderly people in some countries need to be able to phone their prescriptions into a pharmacy to have them filled and delivered or you can't call your family to come take care of you if you need help so all of a sudden telecommunications as a whole becomes life critical Automotive especially with the Advent of self-driving cars anything that fails there is going to drive you into a wall at 100 kilometers per hour definitely life critical Healthcare not only things like surgical robots IV pumps anything that actually medicates you or does surgery on you but also all the software that is used by healthcare workers to maintain your medical records for example your medical record disappears or the system shows the wrong information and all of a sudden a physician is prescribing you something you're allergic to life critical military also obviously anything they touch is life critical by Design but things can go wrong with the systems they use and there's a lot of research in there so with that who here has ever worked on a life critical system of any type let's say it's about a third of the room right which is pretty decent for a system like this and I also want to say this is not theoretical we're trying to keep this light and fun because it's the party talk but it's also important to keep in mind that this is something serious that actually happened that a lot of people died from it and there are a lot of families that are still mourning that so we just need to keep that in mind as we go through this and I used to say when I first started doing the stock a couple years ago that I'm not assigning blame Boeing please don't sue me yada yada yada but the investigations are done now and we can definitively say that they are at fault um the Boeing was actually criminally charged with uh trying to defraud the FAA and settled that charge for 2.5 billion dollars so let's talk about what actually happened our first flight was Lion Air Flight 610 this is October of 2018 and it crashed into the Java sea off the coast of Indonesia at over 560 kilometers per hour that was the last recorded speed it probably was going much faster than that when it actually hit and that killed all 189 people on board and then about six months later we had Ethiopian Airlines flight 302 uh crashed into terrain which is the ground at 926 kilometers per hour and that killed 157 people on board and this had a huge worldwide impact the 737 Max was grounded worldwide three days after the second crash that included 387 aircraft that had already been delivered that affected 59 Airlines and canceled 8 600 flights per week for 20 months that is a lot of flights that were canceled it resulted in an audit of the FAA regulatory process basically the U.S government said FAA clearly screwed up we need to investigate what happened and that caused quite a quite a kerfuffle in the US government and then one of the more important ones is uncertainty around the world and other regulatory bodies up until this point most governments in their Aviation regulate regulators said okay Boeing aircraft was manufactured in the U.S the U.S has a relatively reliable government so if the FAA says that it's safe we're just going to trust that it's safe and let it fly through our airspace and land at our airports well clearly that wasn't true right because the FAA said this aircraft is fine and so now all of a sudden all these countries around the world are like okay do we actually need to start doing our own evaluations of all these different aircraft and that is a massive undertaking in terms of what actually resulted from that 20 billion dollars in fines for Boeing 60 billion dollars in canceled aircraft orders Boeing is a massive company but 80 billion dollars is still a massive hit to their pocket over seven billion dollars in costs for the Airlines and the uh because the aircraft couldn't fly and 100 million dollars in Lost income for Pilots now that number seems small compared to everything else up there but the pilots Union in the U.S is quite a powerful body that has a lot of sway with the FAA and the Airlines and the uh air manufacturers and there is no better way to piss off a group of Pilots than to deprive them of 100 million dollars so that actually played quite a role in these proceedings looking at the timeline of how we got here so the the 737 is a legendary aircraft right it's the longest flying aircraft that Boeing has produced and it's been around for almost 60 years at this point but in 2006 they said okay this thing's gone through so many revisions it's old it's time to design a new replacement a clean sheet design which means we start with a blank piece of paper nothing gets carried over from the old design but then in 2010 Airbus which is Boeing's primary competitor launches the A320 Neo they announced that this is going to be flying soon now the A320 is important to note is very similar in size and capability and range to the 737 they're direct competitors in the same Market space and Neo here stands for new engine option so basically Airbus which has also been flying the A320 for a long time said we're just gonna take the old engines off we're going to put some bigger more efficient engines on it leave everything else the same and call it a new plane and the airlines love this because it's the plane they know and love except now it's more fuel efficient which is a big impact for them and shortly after that they set up a record of selling these plants 730 at a single Air Show which is way more than any previous number and the month after that American Airlines orders 260 of them and what's really important here is that up until this point American Airlines had exclusively flown Boeing aircraft right American Airlines American aircraft it's a match made in heaven but this new A320 Neo is so appealing that they decided to scrap that and introduce an entirely New Logistics chain from Europe just to be able to fly this aircraft and this scares the hell out of Boeing because it's a sign of bad things to come if one of their biggest customers is now making this change but at the same time American Airlines also orders a bunch more 737s but asks Boeing to put more efficient engines on them and so Boeing says okay we're going to have to make these re-engined 737s for American Airlines anyways so let's just scrap the idea of a new plane and we'll put all our efforts into developing a new version of the 737 which we'll call the max and in 2016 the A320 Neo has its first Revenue flight meaning the first flight operated by an airline is carrying passengers and in 2017 Boeing has their first 737 Max flight so through this process Boeing pretty much accomplished what they were trying to do of keeping up with Airbus and having a new plane to Market why does any of this matter okay more efficient engines are larger this is a picture of the engine on the original 737 right 1.25 meters in diameter it's called a low bypass engine you can see it's quite narrow and this is fine this is what the plane was originally designed for we skip over to the third generation the Next Generation they called it and here we have a much larger high bypass engine that's 1.83 meters in diameter and then we have the 737 Max which is now 2.26 meters in diameter and the question is how do you fit an engine that is almost twice the diameter of the original engine on an aircraft of the same original design where nothing else has changed the answer is you can't you cannot do that in the way that it was intended to it doesn't fit because if you try it looks like this and your engine's scraping along the ground when you are trying to take off and that is not good for the engine so what Boeing does is they shift the engine forward and up like so and so this engine is now basically on the front hanging off the front of the wing but they've raised it up enough that they have the ground clearance they need and we can see in a side-by-side comparison of the different models looking from top to bottom the original on top has a lot of ground clearance Small Engine which is almost directly underneath the wing and then the bottom the 737 Max has a much larger engine that is just hanging off the front of that wing the more important piece here is how far forward that engine is sitting because what happens is that the nacelle the casing around the engine uh actually generates is such a large surface area that it generates lift when the plane is at a high angle of attack meaning that when the plane is pitched up the airflow moving past that engine generates lift and pushes the plane up but because the engine is not under the wing where the center of lift should be it's forward that lift actually pushes the nose of the plane up and causes it to want to pitch up even further so you have this self-reinforcing Loop where the pilot pitches up a little too high this effect happens pitches up more this effect happens more and you get into a very bad situation very quickly and the FAA says no it's not okay you can't do this you can't have a plane that pitches up on its own with no pilot input and is unrecoverable if you don't deal with it and boating Zone test pilots say something similar they say we've been flying this it doesn't feel the same as the old 737 doesn't feel good has these weird handling characteristics we need you to fix it and Boeing tries a couple things they do some Hardware changes some aerodynamic changes to try to to deal with this issue but they can't really without causing major redesigns to the aircraft and so they decide to solve it in software and we come to MCAS the maneuvering characteristics augmentation system MCAS is an algorithm that sits on the flight control computer so in our in our 737 on each side of the cockpit on the external side we have this sensor here and this is called an angle of attack sensor and what it does is it measures the angle of the airflow moving past it relative to the angle of the plane the angle that the plane is attacking the air is literally what that term means and when that plane starts to get into that pitching up situation that angle of attack sensor starts getting a very high reading and it starts sending that data into the flight control computer which is where amcas is sitting and MCAS says oh we're starting to pitch up too high this is what we were designed for we need to fix this situation so it sends a signal to the uh what's called the Jack screw at the back which controls the stabilizers the stabilizers are those little horizontal small wings at the back which affect the trim and so it sends a signal to the screw which basically rotates these things back which at the back of the aircraft generates lift which brings the nose of the aircraft down and so this causes the plane to pitch down like so and it stabilizes and now it's no longer generating that extraneous lift from the engine to sell and we're back in a stable situation that's the theory behind MCAS and if we put this on a nice little systems diagram now I should note I am an infrastructure and back-end developer I don't do front end work I don't do Graphics work so this is not looking great but uh gets the point across so we have the sensors the angle of attack sensors which are on the side which are connected to the flight control computers there are two flight control computers one on each side of the plane and each sensor is attached to its own flight control computer they're not cross-linked and what happens is MCAS is sitting on those flight control computers and it sends a signal to the stabilizers at the back which rotates them we'll take a little break on that to talk about something called type ratings so a type is a category of aircraft they'd all have similar characteristics similar handling characteristics similar performance characteristics and a type rating is a certification that allows a pilot to fly aircraft that are in a type and type ratings are a pain to get they're very expensive and they're very time consuming and Airlines avoid them at all cost because Pilots have to go through a ground school they have to go through a lot of time in a simulator those simulators and many millions of dollars each they have to do an oral exam and they have to do a check ride with an instructor pilot to make sure they've actually learned and memorized all these skills and this whole thing takes a very long time is very expensive not only because of all the equipment but because every hour that a pilot is doing this is an hour that they're not flying for the airline and therefore not generating Revenue so Airlines try to avoid this and so we have a type here the A330 is a is a type and we see there are different variants within that type and the important part is that the A320 is also a type and if we compare two variants within the A320 type we see why they are the same these are two different models of aircraft that look almost identical in the cockpits the controls are the same the layout's the same displays are the same everything's almost identical so pilot doesn't even really need to know which variant they're flying within a type rating the different variants to move from one to the other to to be approved to fly a new one basically a pilot just has to do a brief course on a tablet to go over some very basic differences in performance and weight and such and that's about it no simulator no check ride no exam no Ground School and the reason the A320 Neo was so successful is because they managed to pull this off where they got it into the same type rating as all the previous a320s so all these airlines that were already flying this the older models could buy the new model and all their pilots could just fly it without having to go through all this extra training and simulator time now the 737 is also a type and all the previous variants had fallen within the same type as desired and so the question is could Boeing get the max into the same type category because that's the only way they could be competitive otherwise no one's going to buy it if all their pilots have to go through all this extensive training but the problem is that MCAS is a new control system and a new control SAS system means a pilot needs a different training and different training means it gets a different type rating and a different type rating means it's no longer competitive and this makes Boeing very upset because they know that no one's going to buy their aircraft and um it's just simply not going to sell but they come up with a solution and their solution is you don't tell anyone about MCAS because if you don't know about MCAS you don't need training on MCAS right and if there's no amcast training then you get the same training as the older 737 and if it's the same training it's the same type rating the same typewriting now it's a competitive aircraft and this makes Boeing thrilled and they're very proud of themselves patting themselves in the back what a great solution does this seem like a good idea but that's what they did uh so Boeing decided the 737 Pilots didn't need any extra training didn't even need to know about it wasn't in the flight manuals well not quite they it was originally in the flight manuals and then they were worried about it causing this extra training so they took it out but they left the acronym in the glossary so the acronym is still in the back of the flight manual but nowhere in the manual as it actually mentioned or used so that allowed them to get the common type rating with the previous models so all the pilots could fly the max yeah they added exactly what they were trying to do successful now their justification for doing this uh As Told by Dennis muellenberg who was the CEO of Boeing he said that the system MCAS is fundamentally embedded in the system and the handling qualities of the aircraft it's not a separate system to be trained on basically saying that because a pilot doesn't actually interact with it they don't turn it on turn it off they can't turn it on or off they don't need to know about it because they never actually interact with it a little spoiler alert here Mr Muhlenberg is no longer the CEO of Boeing but it worked their plan worked um training on moving to the new aircraft took a pilot who was already type certified uh one hour on an iPad to fly the new airplane and that is it they're off carrying passengers what's interesting is that some of the airlines they were selling this thing to actually wanted that extra training though they wanted the simulators they wanted simulator time for their pilots but Boeing did not take kindly to that because they were worried that if some Airlines start doing this then some of the aviation authorities are going to say okay I know maybe that should be a requirement and then you have a big problem a text sent from one Boeing Employee to another now friggin lion era might need a Sim to fly the Max and maybe because of their own stupidity I'm scrambling trying to figure out how to unscrew this now idiots is this a good way to talk to your customers so you can see they were quite worried about this and does anyone notice the name of the airline in here yeah that was the first one that crashed so talk about their manual overrides for MCAS this is a cockpit of 737 a couple things to note here on the side of the Yoke you have these little toggle switches called trim switches what they do is they control the electric motor the same one that MCAS uses which rotates the stabilizer to trim the aircraft and then at the bottom you have your trim cutout switches these are just electric switches that completely disconnect the flight control computer from the trim motor so that electric motor does not function anymore when you turn those off then you have this manual trim wheel which is a little wheel which is next to the Pilot's knee and it has a little handle that flips out and this is mechanically linked to the stabilizer right so there's actually a cable that runs back um and so when the stabilizer moves this moves and when this moves stabilizer moves now if we add those to our diagram our trim switch is somewhere up here it sends that signal to the computers which then sends it off to the stabilizer motor and our cutout switch is here it disconnects the entire system and our trim wheel is here it is mechanically linked it cannot be disconnected so details of the flights this is a graph showing the altitude and speed of that flight and this is not a normal flight path as you can see about two minutes in uh there's this first weird situation happens where the pi where the plane drops in altitude suddenly picks up a lot of speed but then they recover and keep climbing and then more like four five minutes in we start this weird oscillation pattern where they're going down and up and down and up and down and up until eventually that overpowers them and they drop down into the ocean here's a little video clip showing what um it looks like in the ocean after a plane crashes there the components to their left are very small and difficult to find a lot of them are underwater bottom of the ocean uh and it makes the investigation extremely difficult and long some often times they can't even find the flight recorder or cockpit voice recorder because they've been damaged so bad or they're so far underwater that they can't be found and that makes the investigation very difficult these were long investigations that have finally come to a close um but it's quite a process so what happened here is that the angle of attack sensor that was connected to the active flight control computer uh failed somehow it started sending erroneous data into the control computer and it could have been a number of things it could have been essentially it was faulty when it was installed it could have been a bird strike that broke it during flight it could have been a number of different things but it failed and what happens then is it sends that signal into MCAS and MCAS is getting this this erroneous data that's saying the plane is pitched up far too high so MCAS activates and that triggers that stabilizer to rotate into the trim down position and it does this for 10 seconds at a time with a five second interval in between so 10 seconds on five seconds off and so it's that's why you're starting to see this oscillation pattern because it's trimming down it stops the pilots are pulling back on the stick as hard as they can so they're trying so it pulls up a bit and then it happens again so Pilots don't understand what's happening so they try to use um their electric trim to counteract it it only works during the five second period where MCAS isn't firing uh and they go back and forth and lose this battle and eventually mcast overpowers them and down they go and so now we can see why that oscillation pattern pattern at the top there is happening so after this happens and they recover the data recorder and actually get some some sensor measurements off of it I believe that updates at eight to eight Hertz so it's a pretty up-to-date um recording of the sensor measurements and Boeing and the FAA get together and they start reviewing this data and they say what could have happened here and they think they know what might have what might have occurred because that angle of attack sensor has very very bizarre data and so they put together what's called an airworthiness directive which is basically a bulletin that gets sent out to all the pilots who are certified to fly this aircraft and it tells you what to do in the event that something like this happens you're supposed to turn off the autopilot and try to pull back on the stick to pull up and then use your mat your electric trim switch if that doesn't work uh turn the those cut off switches to off to disable the electric control altogether and if it's still having a problem then use that manual wheel that we talked about to crank the stabilizer into a trim up position so all the pilots that fly this aircraft have to read this and understand it before they're allowed to keep flying so now we come to our second flight this is a chart again showing the altitude of the aircraft over time and we're starting to see some real similarities here right we're seeing about two minutes in a sudden drop and recovery and drop some recovery in this oscillation pattern again until eventually uh down it goes and this is what it looks like when an aircraft hits the ground at 930 some kilometers per hour uh it's again very difficult to recover anything it's easier than in the water because at least the parts are still there and accessible but it's still a massive undertaking to try to investigate the crash fortunately they found the data recorder quite quickly and were able to access the center data off of it so again what happened well same issue angle of attack sensor busted don't know why but something happened to it MCAS activates same thing but this time those Pilots they've read that directive and they recognize this right so they know that this is what they're supposed to do so they disabled those cutout switches right following the instructions and then they try to use their manual wheel to rotate the stabilizer back into position again following the directive they did what they were supposed to do the problem is that during takeoff is one of the only times that a plane is at Full Throttle in a regular flight and so these Pilots had taken off at the full throttle and then very shortly after takeoff these issues started coming up that it completely distracted them from the normal flight operations and so this plane while they were battling with this for quite a few minutes the plane was still at Full Throttle and eventually got to a speed that was Far higher than it should have been at that point in this flight and because it was moving so quickly and this stabilizer was rotated back so far the flow of air against the stabilizer caused so much force that the pilots did not have the strength to rotate that manual wheel to trim it back because they were battling this massive amount of air pressure against it so they recognized this they cannot turn the wheel and they think that well they can't fix it doing it manually so the only chance they have is to reactivate that Electric System to use the electric assist but of course as soon as they do that MCAS which has still been running in the background the whole time it's just been disconnected from what it's supposed to actuate um gets connected again and the situation gets worse and it drives the nose down into the ground so to recap they modified the existing design which changed the aerodynamics of the aircraft and they couldn't fix it is that a good idea they couldn't solve it without a redesign so they solved the hardware and aerodynamics problem in software is that a good idea and they didn't tell the pilots they wouldn't have to train on it how about that one good idea but the pilots didn't know what's happening right so that's how you get accident number one which is your Lion Air flight they get the airworthiness directive no they they gave you instructions on what to do right but Boeing never actually said what the problem is they still haven't told anyone about MCAS at this point and the pilots are not physically able to follow those instructions so you get accent number two Ethiopian Airlines and that kind of concludes the summary of what happened here and so let's talk about what we actually learned from this and what we need to think about the first question is when do you actually teach users about the inner workings of a complex system not just Aerospace or an aircraft but any kind of very complicated technical system that takes a long time to learn all the details so I want to illustrate this with an example let's say we have a new car and you can tell this presentation was made a couple years ago when that looked like a new car um and this car straight out of the factory has a defect just one defect it's very rare it doesn't happen often unlikely but sometimes it just accelerates on its own and you can't stop it but it's rare don't worry about it now to deal with this we have two options option number one is what is printed in the technical manual very deep down in the details and this is the proper way to fix the problem and it's guaranteed to work so you have to hold the accelerator pedal at its maximum position for four seconds while simultaneously pumping the brake three times at one second intervals and then you have to apply the emergency brake at 33 for eight seconds but after releasing the accelerator but before you release the emergency brake you have to activate the four-way Hazard flashers and apply the horn for 3.6 seconds and if you do this properly if you follow these directions well here's the kicker you have to start doing it within 10 seconds of noticing there's a problem or it's not going to work 10 seconds was the time that the FAA later determined that if the pilots did not respond to the situation within 10 seconds by following the perfect directions there was no chance of recovery 10 second window if you do it within the time window you're guaranteed to resolve your issues it's perfect now this seems ridiculous right but if you've ever seen some of the checklists that Pilots have to go through for certain flight operations they get incredibly complex then you have option number two which is you turn off the engine and you attempt to maintain control of your vehicle while you apply the brakes and to have an 85 chance percent chance of succeeding with this right you might get into a sticky situation and crash but you'll probably be okay now let's say we have two users we have your average Joe he's been driving for 15 years and you know he kind of knows what he's doing and you have Speedy Sally who's an F1 race car driver she spends her entire career every day eight hours a day training on her vehicle given these two options which options do you tell or provide to each of these operators and you generally do something like this Joe has no idea what he's doing there's not a chance that he's ever going to remember a checklist like that and so if the situation happens especially under stress and he's not very well trained to deal with stress he's not going to follow the rules and he's going to screw it up and it's going to crash so even though it has a guaranteed 100 chance of success if you do it right he's not going to do it right so it doesn't really matter so instead we give them the option that's the intuitive option the common sense approach what is not as perfect but it's more likely for him to actually be able to do it because he is not a professional user but Sally on the other hand you know like I said she does this all day every day she can actually memorize a checklist like this and she can practice it and rehearse it on the track and get it down to the point where if it happens within two seconds she'll be able to respond properly and these form two points to end points I want to call Performance versus knowledge curve or complexity versus knowledge curve because often performance and complexity go hand in hand and sticking with our mode of transportation we can plot some points on here if we look at walking for example it's a very low performance method of of movement but a pretty low level of knowledge required to do it then you have cycling you're moving faster but you actually have to learn how to do it and train on it driving a car keep moving up this scale then you have your race car at the top and the idea being that as your performance of your system and complexity of your system increases you need to know more about how it works in order to be able to safely and effectively actually operate it and we see that with our aircraft as well at the bottom end you're starting on your Cessna where most people pick it up as their first plane you start moving into commercial airliners much more complex but much more performance and then you have your fighter jets near the top where it is a career we are spending more time training and you are actually flying in order to be able to operate it safely and this applies not just to Transportation but to all sorts of different things we look at computers right my grandmother has her bill stock laptop CPU easy to use she needs to turn it on or off and back on again and it works fine versus my computer I've overclocked it and to do this you get more performance but you need to know about voltage regulation thermal management water cooling however else you want to do it so you get the performance but you need a lot more knowledge you can see it with your little kid skis for the Bunny Hill versus skis that actually have bindings right you need to know how to use them how to tension them how to fit them properly you see this with your Mac computers and your Linux computers higher performance higher level of knowledge required and this gets me very different reactions depending on the conference I'm at I think this was the right one for this yeah all right was that a good idea but if we put these on our graph we see that it follows the same Trend right this this holds true from most systems and most domains in most Industries so the question here is should Boeing have told the pilots about MCAS I think most of us would now say yeah yeah probably like Pilots are really expected to know and understand most of the systems in their aircraft that they're flying not the Deep technical details of how they actually work at a software level but at least their presence and their impact on flight parameters but Boeing concluded that there would be little risk in the event of an MCAS failure because the pilots and the FAA approved this the pilots would respond to a failure in within three seconds so it's okay if it breaks because the pilots will fix it within three seconds however Boeing the FAA agreed they wouldn't tell the pilots about it uh because Boeing safety analysis expected the pilots to be uh sorry they wouldn't tell them about it even though they were the primary backstop and I want to pause to note that um as you may know Boeing was historically headquartered in Seattle Washington they moved their headquarters to Chicago in the mid to late 90s in a situation that was actually quite closely associated with what happened to Boeing over the following two decades but they actually because they were in the same city they have a long and storied history of covering Boeing uh in the news and they actually want to Pulitzer Prize for for their coverage of the previous generation of the 737 so that's why we're looking at their quotes as a reputable source so they didn't tell the pilots about it they decided not to follow that curve and we got these accidents the next question is how do you ensure that your users actually know what is happening in a complex system and this comes back to some of those things I mentioned about human factors particularly mental models we have this thing called mode confusion particularly in aircraft because it was a field of research that really stemmed around aircraft and spacecraft because in these types of vehicles you have many different modes of the vehicle vehicle can be operating in you have an autopilot mode and you have an auto throttle mode which are not necessarily the same thing you have a takeoff configuration and a landing configuration and in all these different configurations the same buttons and switches and displays can do or show you different things so you could press a button expecting it to do one thing but if it's in a different mode than what you think it is it could do something entirely different and this gets very confusing for Pilots especially when things change without them being aware of it particularly due to Automation in the background and this issue comes up even before you have things like MCAS dancing around back there messing with your control inputs and your flight parameters without you being aware of it and this causes great confusion for your pilots and it is a theme that you see in almost every accident major accident investigation report is that the pilots got confused about what was happening in the aircraft before they crashed very rarely is a crash due to Pure mechanical failure usually it is due to something breaks which causes confusion for the pilots which they could have recovered from if they had really understood what was happening but they didn't do two situation like this and now I'm going to show you a video this is a simulator from an Air France flight that crashed in 2009 unrelated it was on an Airbus um over the uh over the Atlantic Ocean I think it was going from uh Brazil to Paris and this shows you what the cockpit sounds like during one of these emergencies over the next four and a half minutes the stall warning will sound 75 times so that's from uh documentary about the crash but it kind of shows you that even in this situation their the aircraft was stalling and you can hear that stall alarm going off 75 times do you think they might notice after the first time and maybe they understand the situation but instead we're just hammering them with that continuously even though they already know and the problem is that that then interferes with other information that you're trying to tell them such as why you're stalling here's another one this isn't a 747 [Applause] so not only is it loud like those Pilots can barely talk to each other but it's also very stressful right you're inducing a huge amount of adrenaline which is not conducive to trying to perform calm and rational actions that you need to do to recover from this situation and we know that this played a role in the 737 crashes uh there were at least five different alarms going off in those crashes and in in this video you hear too so you know times that by two and a half and you get an idea of what those Pilots were dealing with and they struggle to understand it they couldn't figure out what was going on and those warnings weren't really helping them understand why the plane was actually pitching down and why they couldn't fix it but this isn't a surprise to anyone right this is a known issue right this is what I did a lot of my PhD research in this is what people have been doing researching for 50 years particularly at Nasa for spacecraft and we know that this is a problem based on lessons learned from past Airline accidents the FAA regulations have precise design details on how to deal with these things and they are aimed at ensuring three things that the pilot knows what is going on that it catches their attention and it averts any possible confusion now the second one there is very easy to do as I showed you in that video it's easy to catch someone's attention you just make a loud noise or shake their their control stick but doing that in combination with one and three is much more difficult right how do you get their attention in a way that does not distract them from actually understanding what you're trying to convey to them and these regulations they didn't just come out of nowhere these are Written in Blood right like these are from past accidents and that is why we have these rules in 2014 Boeing convinced the FAA to relax the standards for the 737 Max related to the cockpit alerts that would warn the pilots if something went wrong their justification for this was because the aircraft the airframe is so old and we're only trying to change this one part of the design it would be cost prohibitive for us to try to upgrade this entire thing to meet the standards the FAA said okay and that was it that was the end of it they were completely okay with that and that is one of the reasons that they're at the front at the beginning I said Boeing and the FAA both played equal roles in this in these tragedies this is not a solution like I'm saying I'm not telling you what you should or should not do when you design your systems because obviously this is a very difficult and complex problem that is not solved in any way but there are plenty of good guidelines and standards and and recommendations for how to design Warning Systems in a way that doesn't overload your Pilots so that is just something you need to consider when you're working on a system like this let's talk about manual override particularly for MCAS did MCAS have a manual override was it used appropriately let's talk about levels of automation now this is a concept that has had a very extensive amount of research on it there are different scales ranging from 3 to 12 plus different levels of automation depending on which scientist you ask but I've kind of condensed it down into four primary levels here your first one being a system that provides helpful information so you tell it what you want or it guesses what you want and it provides you information that could help you do it right so your your Google Maps is a good example it's not driving your car for you it's not making you turn left and right on the sidewalk it's just telling you what you should do to meet your objective your second level is automatic control when directed by a human the cruise control in your car it doesn't turn itself on when you turn it on it automates things until you turn it off or it meets some end conditions such as touching the brakes your third level is automatic control unless directed by a human Windows update is a great example it will do its thing and it will restart your computer at the worst possible time unless you manually intervene to stop it from doing so and then your fourth level is automatic control with no human override and this is a much harder to come up with an example for because there are relatively few systems in the world that don't have some big red emergency stop button that'll shut down the automation but where we start to see this is particularly in space flight if you remember the New Horizons probe that went past Pluto and it took all those really cool photos that entire procedure where it did a lot of different movements to aim the camera at different things that was all choreographed in advance and once it started there was no way for a human to to stop it because of latency and even if they did want to stop it there's no way a person can think fast enough to tell it what it should do instead where does MCAS fit on this we could say that it's three level three because it automatically does its thing unless the human turns off that cutoff switch but is that is that cutoff switch really disabling the automation it's not all it's doing is it's unhooking the automation from the thing it's supposed to be controlling but the automation is still running there in the background and by unhooking that or using that breaker that breaks the connection they were also disabling other critical functionality which is their their electric toggle switch that controls the trim so is it really a manual override if you can't use it without disabling a bunch of other critical systems and such that when you reconnect it it's still failing right from the start redundancy not so much to do with humans and Technology but we've spent all this time talking about the plans we need to mention and certainly relevant for what we do in this conference who here can see the redundancy failure in the systems diagram oh man okay a couple hands all right so in case you here can you see this there we go right here this sensor is connected to one computer one computer is active at a time this sensor isn't doing squat over there it's not its data isn't even being used so when that first sensor fails your system's broke right the flight control computers the pilots can manually switch from one to the other in the case of a computer failure but when the sensor breaks the computer doesn't know that the sensor broke and so it can't tell the pilots that the sensor broke and the pilots don't know that they should switch to the other computer so we have a single point of failure here in that angle of attack sensor and we can see that more obviously in this graph these are our two sensors this is data they pulled off the flight data recorder and we can see that the left sensor is the red line the right sensor is the black well we can guess which sensor was connected to the active flight computer at the time so where the where the reading spikes up is where we think that A bird hit it and broke it off but what do you actually do about this well first off do not ever let assist critical system have a single point of failure right we're working on a lot of a lot of critical embedded systems with people in this room I'm sure most of are already aware of this but the less common one is don't let a broken sensor affect good sensors here we had a good sensor that was still getting good data we could have used it but we didn't but what happens when you do that how do you deal with those two sensors disagreeing with each other well ideally you'd add a third sensor and you'd have some sort of consensus mechanism right like a lot of distributed systems use a an approach like this but you can't always do that there could be technical limitations um there could be it could be cost prohibitive it's always a trade-off between cost and and performance so maybe we do something like forcing a manual override we can read the two sensors and we can alert the pilots that hey these things aren't agreeing something is out of whack one of these is broken we don't know which one so we can't automate anything so that you the pilot you need to take over that could work and I would probably would have worked in this situation but the problem here is that imagine you're flying your plane Along on autopilot and all of a sudden the automation kicks off you have no idea why and your plane just starts drifting off to the side you need to take control and you start getting into this mode confusion situation again because now you are not aware of what mode your airplane is is in why the autopilot failed and why you are now required to take control of it so it's hard right this is not again not a solution it's just talking about a lot of these problems you really need to think about because they're all interconnected it's this tangled web of problems where to solve one you make the other worse change management again not so human related unless you're considering the engineers to be the humans in your system in this so MCAS was originally designed to only operate under very specific conditions basically if you're if your aircraft was flying at a fair at cruising speed and somehow pitched up too high and started to get into that reinforcement loop with the lift in the front then the system would kick in and it did this by requiring two different sensors you had the angle of attack sensor that I talked about and a G-Force sensor because if you're flying at a high speed and you pitch up all of a sudden that's quite a G-Force uh change and your sensor will trigger but what happened is the pilots when they were the test pilots when they were doing takeoffs in this aircraft they thought or they said to to Boeing this does this feels weird right um and when MCAS activates we like that it helps with the with the flight control so give us more MCAS right make it happen more often uh it helps us with with controlling the takeoffs so Boeing said okay we're going to have MCAS applied during the takeoff phase as well but the problem is that during takeoff the plane is moving at low speed which means there are no real g-forces to deal with so they had to disable the g-force sensor so that MCAS could work during a takeoff phase so now you have a system like this that is only activated by a single sensor the other issue is that when you're moving at low speed during the takeoff phase when you're moving at low speed a control surface on an aircraft has to move more to have the same effect because there's less air moving over it so they had to increase the power of MCAS as well so when the plants finally entered service MCAS was able to move the tail four times further than was stated in the initial safety analysis document so they did their safety analysis on this system and then they removed one of the triggers and then they quadrupled the power and at no point did they redo the safety analysis how did that get through a change management system retiring a system now I think given the nature of this conference in C plus plus and C A lot of people in here have probably pretty familiar with Legacy Legacy systems right I bet a lot of people in here have been working on Fortran or cobalt-based systems in the not too recent past not too distant past I I think at some point an engineer needs to recognize when something has reached its limits when it is unsafe to push it further when to when trying to add new features is just going to break everything and possibly kill people Boeing had this thought right in 2006 they said we've pushed this plan to its limits we need a new design but because of Market pressures because of poor management because of a culture where people couldn't speak up it made it through as a additional model that was past its design limits it was not built to handle engines of that size so we come to kind of our responsibilities as developers and Engineers what do we do about this unless you're some high level executive in a company like Boeing you're not going to have a whole lot of sway but when someone asks you to develop something and you think should I really quadruple the power of this system or remove this safety check without doing anything about it to evaluate whether it's safe you probably need to raise your voice about that at Boeing they couldn't the engineers actually a number of Engineers tried to do that and resigned from Boeing when they couldn't Boeing at the time their management penalized Engineers from writing safety concerns in in emails in case it could show up in court later they would dock your bonus pay if you wrote up a safety concern in writing that is not a healthy culture to work in and I strongly encourage you if you're working on a system like this and you're facing that kind of issue find a new employer it's a hot Market out there you'll have no problem with it because what I guarantee you don't want is to feel the way a lot of Engineers at Boeing do right now or previous engineers at Boeing with the weight of those deaths on your shoulders for the rest of your career and I'm not saying you need to do all these things for your next WordPress install but if you're working on anything where a life could be involved absolutely there's these are things that need to be considered so with that thank you for your time uh it's been a pleasure talking to you about it hopefully it was fun enough for the party atmosphere um this is my email here email me with anything you want I'm happy to talk about it I'll be here the rest of the evening so come up and talk to me as well if you'd like to chat um there's a QR code for top tell if you're interested you can get a pretty hefty bonus if you have certain skills um thank you for your time since yeah since there's no one really after me other than a C plus plus quiz in half an hour um I'm happy to take any questions that anyone has or would like to chat about so we have a company do we have any people that have been implemented about the could have been implemented oh that's a good question uh there were pardon uh the question was so we discovered this secret unsafe system that had been built into the aircraft have there been any other situations where we have discovered due to an accident or some other cause a different secret system that shouldn't have been there in terms of Secrets yes there are and I know that I've studied them I can't remember specific names but in this aircraft in particular during so when it was grounded obviously Boeing and the FAA we were humiliated by this spent all of their resources thoroughly investigating this plan and what's actually kind of funny about it is that this plane is probably one of the safest aircraft to fly in now because it is one of the only aircraft in the last few decades that has gone through such intense scrutiny and so we actually know that it's probably not going to fail um but during that investigation phase they found quite a few other issues they found metal fragments in the fuel tanks um and some other I think there was another algorithm issue they found as well that they had to fix so yeah there are plenty of problems that get discovered but shouldn't uh FAA extend if they require uh redundancy or performance in theory yeah uh it's doesn't the FAA require redundancy on all components of a plane yeah so it's three systems for critical systems if the safety analysis or the engineering defines it as something that if it fails the plane will crash then it requires three systems so that's generally Hydraulics and certain other flight Control Systems what happened here is that Boeing said this is not a critical system because if it fails the pilots can still fly the plane so it didn't meet that requirement for having to be redundant the same type yeah it's still the same type basically they altered this part I haven't researched as much but I believe what happened is they they altered the MCAS algorithm to not trigger as often and they added redundancy between these two sensors and basically they ended up coming back to their original argument of a pilot doesn't interact with the system so they don't need to train on it hell no yeah yeah with with what ah so uh here let me pull up actually I don't have an Airbus diagram here but um if we look at this so the 737 uh the wings are mounted very low on the fuselage right they're quite low to the ground then it was designed that way for aerodynamic reasons when they were using Small Engines the seven or sorry the A320 is a newer aircraft it's still very old but it was designed with the wings mounted much higher on the fuse sludge so they actually had plenty of space under the wings to mount the newer engines and didn't have to do the whole shift forward and up situation yeah there were against Boeing yep uh the they were charged with trying to defraud the FAA well I'm not exactly sure how the corporate government works governance works I think when a corporation gets charged some of the executives take responsibility for that but um I don't think any individuals were and they were charged but it was settled out of court for 2.5 million and I believe one of the reasons for settling out of court is that if Boeing was had a criminal record they can't provide things for the federal government anymore and Boeing is one of the only providers of certain types of spacecraft and stuff so it was a huge deal yeah yeah sole sourcing it's great any others oh another one I'm not alert I would agree with you but I have no jurisdiction there what's interesting that you mentioned the culture and this is something I don't really have time to talk about in the talk but it's kind of fun to talk about after feel free to go get beer or food or whatever you want by the way um is that so Boeing historically has been the opposite right they have been very engineer driven uh and had a very strong culture of of Engineers bringing concerns forward and having more of a Collegiate relationship instead of a top-down relationship uh in 1997 I believe they merged with mcdougs yeah uh and with Doug McDonald something Douglas yeah and um that company had the opposite culture and even though Boeing I believe acquired Douglas um the Douglas CEO became the Boeing CEO and so that whole management style was replicated throughout Boeing and so you see this trend over the next five ten years of all the really good Engineers bailing because they couldn't they didn't want to be part of this new culture and you get things like the 737 Max yeah just one thing um that illustrates The Importance of Being Honest because well in the short term money of being dishonest at the long run uh it's not just the Boeing but you also have the diesel gate on Volkswagen uh that means that in the long run being dishonest with equality can be very expensive and it can even ruin a company yeah but it's a question of how do you define long run and if you have a CEO and you're already 70 and you're only going to be there for five more years do you really care about the long run or do you care about your retirement package which is based on stock price and it's part of I mean there's a whole bunch of issues around Corporate America that we can go into at a later time but um yeah there's no motivation for long-term uh success really yeah all right I think that's everything thank you again and I'll see you around out there