Being The Human in the Loop

This topic is surprisingly large. And the reason I say surprising is I was surprised by it. So, I'm glad that you've agreed to commit your time to me for the next 6 hours. Thank you. Okay, we're going to do this in 40 minutes, which means yes, there is not enough time for questions within the talk, but if you want to catch me in the breaks and so on, please do. Um so, uh >> [sighs] >> This talk kind came about because I started noticing I guess about a year ago. So, this is the first time just for reference this is the first time I did I'm doing this talk. Um but I've had the idea for the talk for a while cuz I noticed that we were using this phrase. It's okay if you keep your human in the loop. Human in the loop. Human in the loop. And I suddenly thought we're not really interrogating that. We're not really asking questions about that. It's as if the It's as if that's the end of the conversation, not the beginning of the conversation. It's a It's a thought stopper rather than a starter. I would like to investigate this. Um Uh you have various ways of getting in touch with me. Um I'm fairly internet unique name-wise. Um so, feel free to uh connect with me, um stalk me, and whatever. Um but what qualifies me to talk about this? Because I clearly I am talking about AI. But I'm not an AI expert. I I can't tell you the best tricks and tips for your cloud skills. Um you know, I'm familiar enough with it, but there are people who just have way better knowledge than me. So, what what allows me to be up here and talk about this stuff? Well, turns out that that is straight out one of the reasons I can do this. But even better >> [gasps] >> is many years ago I set up a Facebook page. This is kind of the ultimate long game. I set this up in 2013. Being human with moderately successful results. I set this one up. Why did I set this up? So that when you look at my profile, you can see that I live in Bristol and I'm busy being human with moderately successful results. Oh, no, that's not the reason I set it up. It was actually the work at phrase because I work at being human with moderately successful results. I mean, that's a slow long joke and here it is. Thank you. Right, so um ultimately in that sense that qualifies me to talk about that. I have been interested in AI um for a very long time, far longer than this film was around. I won't be talking about this film, but I will be talking a bit about AI. I won't be talking about AI ethics and the morals and all the rest of it because I did not bring my sewer suit. Um we could be doing that for hours. Um I will also not be talking although my name is often associated with bugs and issues. I will not be dealing with um this except to say that there are some issues occasionally. Hopefully nothing quite as drastic as we find in this. Um I'm not going to be talking about putting agents in your code base. >> [snorts] >> I am not going to be talking about MCP. I am not really going to be talking uh about vibe coding. Um and certainly not about Ralph Wiggum loops. Um but there is some stuff that I want to be talking about. Poetry got a mention in my intro. Poetry is a curious and in I won't be By the way, I won't be talking about security either uh except it obliquely. And by the way, poetry is the best way to hack um uh uh one of the best ways to hack an LLM. Uh it turns out that you can put all the rational guardrails. Always remember when you see the word guardrail, work around. Okay, hallucination, bug. I'm translate classic to modern speak. We use a lot of euphemisms. Of course, there's a little bit more to it than that, but sometimes there isn't that much more. Um you'll notice that Emily Dickinson um uh one of one of the great poets, um she was clearly using an LLM. Look at all those M dashes. Uh Yeah. I mean, she even it's even encoded in her name. How shallow can you be? M. Dickinson. Um But I do want to talk at the point about poetry. And the point about many of these things is it's very much about us. It's very much about humans. And this phrasing, this recurrent phrasing. Principles reduce the risk of wrong choices and engender stakeholder and customer customer trust. Businesses are best served by keeping humans in the loop. Maybe this means direct approval, et cetera, et cetera, et cetera. There's a lot of maybes in the second part. The first part is actually fairly interesting business English. In other words, it's trying to avoid saying something. Um that's just a an on-the-fly analysis. We see this phrase popping up. This is where the concept of human in the loop comes into play. We use the term alignment an awful lot. Again, another euphemism. Uh human in the loop refers to a system or process in which a human actively participates in the operation, et cetera, et cetera. This uh this definition here, I want to rest on for a moment. So, I'll give you the full definition. Um So, um from uh Domain Integrated Context Engine, DICE, human in the loop oversight ensures that governance and approvals are in place where risks are high. Now, there's an interesting thing here. This all sounds really good. But notice that there's a subtle pressure. There's an obligation here. It's on you. Um there's a there's a lot of pressure. What does it mean to be this human? Oh. Oh, okay. It's not so I have a great responsibility. I have potentially greater responsibility than I had before. There's a lot of pressure. There's a sense that if anything doesn't work, it's on you. Notice the wording. Look Look at the weight of the wording. This is a what What a lot of this is about. And that notion of pressure and anxiety, it turns out that humans as you kind of highlight, we are a very emotional creatures. We are very emotional creatures and we are susceptible to anxiety and all kinds of stuff. Turns out that we are really bad at guess what? Doing loops. It's a phrase I've used for about the last 20 years. My standard phrasing in running certain training courses and also getting people to automate certain things, humans are bad at loops. We are terrible. We were You know, you don't even have to have ADHD, but honestly that really sets it off. Oh, you're going to ask me to do a thing and then do it again and again and then I'm going to wander off because it's boring. If you want a four loop done, you use a computer. You want something executed reliably every single time, you use a computer. That is what they do really well. We are not good at that. We are really bad at loops. That's intrinsic to our nature. Okay? It's a feature, not a bug. So, putting us in the loop comes with a great deal of silent and unspoken responsibility that I think we have to be aware of. And I don't think people are aware of it, certainly not in the conversations that I'm seeing. There are some people who are aware of it. I was going to put One of the things I wanted to put in here was a slide which had a hamster wheel. That's a loop. That's a lot of people turning into that. We talk about that productivity trap. One of the other ones is I thought, you know, a noose is a loop. But I thought that was a bit morbid. And then I I thought, you know, that's not really how you want to start your morning. Um But then I realized that Cory Doctorow had actually made the same observation um, a couple of years back. And he uses this point. He says humans in the loop present a tantalizing solution to algorithmic misfires, bias, and unexpected areas. So, we'll put human in the loop is the cure response to any objection to putting an imperfect AI. And this is interesting because we are seeing that a little bit. It's not necessarily an objection to the idea that this makes sense. It is the objection to the idea that we are pulling this out as a default rather than entirely trying to understand the situation that led us there and the consequences of making that decision. As I said, we use it as a conversation stopper, not a conversation starter. This has been doing the rounds. I don't know if anybody's seen this. I would I have this really bad habit is that I try and make sure that everything is correctly sourced when I put things into my talks. And I do realize this is a bad habit because I know people who don't do that and I have at times, couple of times, not checked that. But I decided to go and check the veracity of this one. This is from an IBM training manual in 1979. It has been doing the rounds, so it's legit. It's not It's not incorrectly meaned or anything like that. And this is the advice that IBM were giving their people. A computer can never be held accountable, therefore a computer must never make a management decision. Accountability is a double-edged sword. It comes with a great It comes with potential for pressure and anxiety. We need to recognize that. We've also discovered that people are bypassing this one like crazy. Um, so let's go back to us. And the thing that makes us interesting is the fact that we are all individuals and we are also interacting. This is the thing that happens. And you may recognize this particular phrasing. Okay, and we have a particular priority structure that we like to think that we value. So, you know, this particular phrasing I mean, it's Where And where we? 25 years ago last month. Um this was uh the Agile Manifesto was dreamt up. And it's actually the manifesto itself has aged surprisingly well, although not necessarily all the participants have. Um uh we'll leave that as a separate conversation. Uh >> [gasps] >> um there's only so far you can go with a bunch of middle-aged white men. Um uh I speak from a position of authority on that one. Uh so but in terms of wording, and I'm very interested in words and how things are written. This is actually some really good, well-crafted English. There's a lot of manifestos that simply don't um don't do that. But we do have that really interesting notion as a as a device for communicating of telling us what our priorities are. And this is a bit of a problem because what we've noticed is that we seem to have ended up spending a lot of time on processes and tools. And this is I don't I don't even have to talk about AI here. Whether we are talking most conversations in the 2010s and even honestly in the late 2000s and certainly in the 2020s, when you sort of start saying, "Okay, we can let's talk about Agile development." It comes with Agile TM, it comes with safe, it comes with Scrum certification, it comes with somebody giving you a big shopping list of their tooling. And that's not unimportant. In fact, some of these things are important in the wrong way. It's it's it's highly relevant. It's just that that seems to be the lead of the conversation rather than our people. Um so we kind of it would be you know, I would love it if everything you know, if if people did actually pay attention to this stuff. But we've also entered the era where the tools are becoming primarily the thing of interest. Um and Dave Berkus a AI consultant who I met a few years ago and is uh pre-AI and he's moved into this space. He's got some really good insights into how this stuff is going. And this is from his newsletter a couple of weeks ago. >> [laughter] >> Welcome to another week of wondering whether we're building tools or building dependencies. Yeah, just pause on that one for a moment because that's quite interesting. That applies outside the scope of AI, but it also certainly applies within it. Again, I'm inviting you to think about this thing in terms of questions and pauses and have we got everything aligned? Let's use the word that everybody likes using. Um have we got everything aligned with what it means to be human? And so if we're going to talk about roles, let let's talk about this idea. When we talk about assistance, when we talk about AI augmentation, when we talk about the relationship between people doing work and the thing that assists them working, there is a distinction that um is being made. Uh is the idea of the centaur. And a centaur um in Greek mythology um passing into Roman mythology, a creature with head, arms, and torso of a man and the body and legs of a horse. Okay? Um and Cory Doctorow refers to this, in automation theory, a centaur is a person who is assisted by a machine. Okay. The idea of driving a car makes you a centaur, so does using auto complete. You are in charge. Although, along the auto complete path, somewhere something changes. And I've certainly noticed this one a number of years ago and potential failure modes is where we have surrendered to uh we've actually surrendered and outsourced our knowledge and our competency competency and our agency. I have I am blessed with a a staggeringly poor sense of direction. Um GPS, satellite navigation, all of these things are a gift to me. Um uh back back in the day, back in the 2000s, my wife bought me a TomTom for those of you of a particular age. Yes, there was a time before Google Maps. No, Google Maps have not existed since time immemorial. They are not an eternal company. They have a finite uh lifetime. Just as kind of reminder um And I my wife bought me a TomTom with a with basically all the road routes in Western Europe. The reason she bought me this TomTom um was she said, "I'm tired of hearing stories about you getting lost all over Europe, Kevlin." That's how good my sense of direction is. Uh now around that time in the 2000s, I was I was visiting a company in the US. And uh uh the TomTom did not cover um the US and also Google Maps was not at a point that it could be used reliably. Um so I got a GPS satnav with the my rental car. And then there was a problem with the GPS um with the GPS uh and the device started breaking down. The battery could not be recharged. And I suddenly realized having driven the same route for three or four days, I had no idea how to get to work and I also had no idea how to get to the rental company to replace it. I suddenly at that moment I had a realization that actually the thing that was in charge here was not me. I was at best a reverse centaur. The machine was in charge of me. Actually, as an interesting point, what I did following that episode No, my sense of direction did not miraculously improve. What I did following that episode is I always disable the sound on every GPS device that I use. Now that means I make a few mistakes cuz I have attentional issues. But most of the time I am looking at what I'm doing. I'm not I'm not got I'm been not been guided by a voice. I'm using a map and I'm interacting and I'm reflecting in it. I pushed it back slightly to the centaur stage and now have better equa- And this is a really subtle nudge. And this is one I noticed and applied over 15 years ago when doing this. The problem is that we're actually building systems now where not only is there no awareness, but they are deeply, deeply embedded in the reverse Centaur Scientology psychology. Reverse Centaur is a machine head on a human body. A person who is serving as a squishy meat appendage for an uncaring machine. Um perhaps best epitomized by this uh recent uh cartoon uh by Tier Throws, a Dutch cartoonist. Um AI gets to do the music, the writing, the painting, the art, and all the rest of the stuff, and we are merely being um these uh enablers of stuff. I first saw an example of this one uh in uh one of the Belfast airports about 2 or 3 years ago, where I saw that they'd decided to have a robot deliver you your food at one of the restaurants as a nice gimmick, and they had a human sweeping up after it cuz it couldn't get it right. It's just that we have made a fundamental mistake here. Now, I want you to keep your eye on the Amazon delivery guy, cuz maybe what the Amazon delivery guy is delivering you is Agile. Yeah, safe in a box. Um So, let's have a look. We previous previous talk she looked talked about Agile and definitions. So, I want to also understand what we mean by this, given that we are at Agile meets architecture. I'd like to point out it's an adjective, therefore it's a property of a thing. It's an observation that you can make upon a thing. It able to move quickly and easily. That's what my dictionary tells me. And my dictionary is fairly good on this stuff. It reflects existing usage. But there's two ideas up there. That's way too hard for most people and most businesses. That's two things. >> [sighs and gasps] >> Agile, it's a noun. It's a thing you can do, and it's all about quick and fast. It's all about speed. It's not about velocity. Velocity has direction. That's not what I'm saying. What we also get is the practice of it. This is what a lot of people are experiencing. Sprint no longer means time box period. It means a period of exhaustion followed by another period of exhaustion. People are busy. Contained within the word Agile, there's AI. Yeah, today's a today's a lesson. Oh, that's what AI in practice means for a lot of people. So, let's talk about this kind of question that we have, and there is a question that we need to have, which is one about productivity. >> [sighs] >> And when it comes to perception, we're pretty bad at this. Um physicist Richard Feynman, the first principle is that you must not fool yourself, and you are the easiest person to fool. A lot of people will personal experience with um either either they're vibing or they're using a lower level of assistance, and they're telling you, "I'm getting great productivity." Oh, I feel so much more productive. The evidence suggests that this is this the picture is not quite so simple. I mean, in some cases, the we we One of the things that we have learned is a very simple thing. So, some research provides evidence that using artificial intelligence to complete task can improve a person's performance while simultaneously distorting their ability to assess that performance accurately. So, key keywords there, um can improve does not always improve, but the person's awareness and perception of their own skill is wildly distorted. So, when they're getting some improvement, they think they're getting a lot. When they're getting no improvement, they think they're getting improvement. When they're actually doing worse than they otherwise would have done, they still think they're getting improvement. So, there are obviously three actual outcomes on are you getting improvement, but it turns out there's only one, do you feel you're getting improvement? So, your feelings are going to betray you. But, they're not just going to betray you. It turns out that there are a number of other things. This space is quite complex. There are no simple answers here. This is not to do with software development. We I think it's always useful to see what are other people experiencing with AI, and then see are we seeing the same story within software as well, or are we seeing a story within software other people experiencing. It turns out most of the time the story is the same. We're not that special. Um we are human. It turns out this is a human experience. Data reveals 90 96% of C-suite leaders expected AI to boost worker productivity. Uh 77% of employees report AI has increased their workload. So basically there's near unanimity in the belief that we will be better at being machines, because that is the language of productivity. When you use production, that's the language production. Productivity is how you measure a machine, not a human, but nonetheless we have fallen into that trap. But three quarters of employees say it's increased their workload. That's kind of interesting. Well, it's not going to be uniform across this, but a lot of companies are saying it's failing to boost productivity. This is a thing known as Solow's paradox. Um and sometimes individuals are actually getting productivity benefits. And their organizations are not. Which tells you that the problem is somewhere else. Speeding up the bit that doesn't need to be sped up is something that we should be aware of. Um and it turns out that the blocks are elsewhere. Um I was I decided to walk in this morning. It's very nice little walk. Um It's not too much traffic. It's not like living in the center of uh various cities where I um experience lots of traffic a lot of the time. And one of the things I'm always fascinated by I live in Bristol. And I tend to walk into the center. It's about half hour walk. Um I tend to walk into the center and one of the things I like doing is walking past cars at traffic lights and then seeing them race ahead to the next traffic light and then overtaking them. It turns out you can have a Ferrari or a clapped-out 2CV and you will still make the same progress. Except if you're in the Ferrari, you'll think you're making better progress. Okay? And that is the experience a lot of developers are getting. Sometimes an individual vibe coding thinks they're making great performance. Now you put other people, it turns out that having other people may actually not work out the way you expect it does. It's scaling from one to two turns out to be a real game-changer. And then to an organization. So it turns out there's a lot of reasons why we might be experiencing this. Again, there's not one story here. And this [gasps] Here's a Here's a kind of an interesting one. Um an extract from a tweet a few weeks ago. Um about what people are actually doing. This is what I'm going to call AI quit AI quitting. We've heard of quiet quitting, this is AI quitting. Majority of workers have no reason to be super motivated. They want to do their 9:00 to 5:00 and get back to their life. They're not using AI to be 10 times more effective, they're using it to churn out the task with less energy spared. The two people on your team that actually tried are actually now flattened by slop code everyone is producing and they will quit soon. So you've got a bunch of different stories going on. Some people are saying I can kick back. I can now There's a Parkinson's law thing here. I can now do the same work with less effort. But there are also people who feel pressured because the expectations are rising. Both of these things are true and they are happening. So as I said, the picture is nuanced and subtle. Um there was a survey about 18 months ago, can generative AI improve developer productivity? Now this was done from the point of view of using Copilot. Copilot access provided no significant change in efficiency metrics when people looked at the bigger picture. There's a lot of stuff like this. Now, you might say, "Oh, well, that's just Copilot. Let's go and have a look at what Anthropic says." Okay, well, Anthropic didn't say, "Oh, Anthropic." We found AI can speed up some tasks by 80%. Now, if I were an Anthropic salesman, I'd be using this slide. But, it turns out that that's not what's going on. We're finding that different people are having different experiences. But, also people are not evaluating their own experiences objectively, and they're also in different organizational contexts. And it turns out that if you were not the bottleneck, making you faster is actually causing somebody else another problem. But, also there's a duty of care. That throwaway remark about some people producing slop code that other people now have to deal with is real. It's happening, and a lot of open source projects are suffering badly. If anything kills open source, it's going to be AI. I don't think it's going to kill, that's a very strong word, but the it's not a happy relationship that open source has with AI at the moment. That's a different talk. Let's look at something that has looked at a bigger picture and gives us a slightly better reference. So, um the uh the DORA report at the end of last year kind of highlighted really what what is actually going on, which it turns out was not a great surprise for many people. AI's primary role in software development is to amplify. We can replace the word AI with tool. And we can go back through the history of tools. I It's an observation I made a few years ago. How Let me actually just do the just do this quick uh How many people, say in the last 5 years, have been worked on or been associated with a project that has um significant unmanaged technical debt? I don't understand. We have refactoring tools. We have automated refactoring. Refactoring automated refactoring, we've had it for over a quarter of a century in the mainstream tools. How do we still have this problem? Maybe it wasn't a tooling problem. Oh. Oh, mind blown. It's an amplifier. AI magnifies the strengths of high-performing organizations and the dysfunctions of struggling ones. Two years ago, you can find it in a couple of my talks, I made the observation that for some people they are absolutely going to clean the floor with everybody else cuz they understand what they're doing. But this is Sturgeon's law. 90% of anything is crud. There is a tiny minority of people that are doing really well with this stuff. And I said this 2 years ago. And the chances are if you think you're one of them, you're probably not. You might be. I'm not going to This is how probabilities work, but you're probably not. Okay? But the other 90% are either doing what they were already doing but kidding themselves, doing what they're already doing but feeling to more tired, or doing worse than what they were already doing, feeling busier. All of these are possibilities. And the point is I don't think that people are looking closely enough at what this actually means. Observation from Jason Gorman on the same lines. Turns out that the key to being effective with AI coding assistants is being effective without them. Again, this is a non-surprise to a lot of people. It's a It's a skills question. Okay, so let's have a look at what I mentioned the technical debt thing. Clearly the existence of tools was not enough to solve the problem. Um So this is part of GitClear's kind of annual observations. I I actually paired this one back to fit in the talk. I'll just pick what they observed last year on the previous year's data cuz they tend to do a lot of data analysis over 200 million lines of code. Um their main principal focus is on Copilot. Um but that doesn't actually turns out that doesn't make a difference. Developers seem to view AI as a means uh to write more code faster. Okay? Yeah. Through the lens of does more code get written, common sense and research agree, resounding yes. AI assistants do beget more lines. And that would be the problem. That would be great. We've solved it. Done. Move on to the next thing. Has anybody come across this phrasing? Typing is not the bottleneck. Yeah? A couple of you. Sebastian Hermida did a really nice graphic of it back in the 2000s. Now, the thing is that this gets thrown around every now and then. But nobody ever attributes it correctly because it turns out that the person who was responsible for this wording is me back in the 20th century. Literally the 20th I can now say 20th century because you know, or some people might say, "Oh, you know, the 1900s." Um which really makes me feel old. Um typing is not the bottleneck in software development. I started using this phrase sometime around the mid to late 1990s. Primarily in connection with badly written libraries and code generation. And also how teams were scaling. I used this particular phrase to say we're often focusing on the wrong problem. I then wield the phrase out again when there was a kind of an MDA fetish back in the early 2000s. It's just like that you're solving the wrong problem here. I'm not saying this can't help you in some way, but if you think that this was the problem, then you have misunderstood. In fact, there's a very specific piece of wording that I used in 1999 keynote um in Oxford. Typing is not the bottleneck in software development and as development is a thinking profession, removing thinking time is like removing preparation time for a performance artist. You know, why are you wasting all that time practicing every day? The performance is only 90 minutes. That's the only bit that matters. You don't need to do all that other stuff. It looks great on the schedule, but it's self-defeating in practice. The thinking time, the understanding of the domain, the discussions, the what does the customer actually want? Do we even understand what that means? What do we mean by the architecture? What are the implications of this? All this stuff All this stuff has to happen at some point. Your choice is not whether it happens, but when it happens and whether it when it happens um whether it happens effectively. Is it in a rush because everything's gone horribly wrong? Or is it because we actually sat down and we have a shared model of what we're trying to build? Turns out these are the bottlenecks. So, examples of eroding code quality are present in a lot of code bases. Again, I'm not going to say that you can't write good code. There's a lot of people writing and generating absolutely excellent code and they have understood the assignment. But I'm going to say that most people have not or not paying attention to it. And even that And most frustrating is the people who understand that they should be, but their organizational context effectively creates constraints that disallow that. That is that That's the If you don't know you're doing a job, you have a happy life. Okay? The most If you are doing a great job and you're allowed to do a great job, you have a great life. The problem is if you recognize the dissonance, the disparity, that creates a dissonance. If you recognize this could be better, but I am not in a position or do not feel enabled to be able to do that, that is where you get That's where you get the frustration and the anxiety and the dissonance. The frequency of copy and pasted line commits grew um uh faster than our 2024 projection. The percentage of commits with duplicated blocks grew even faster. 2024 was the first year that copy and paste frequency beat moved line frequency. That's really important if you understand refactoring. It basically means we're just going for duplication, to hell with anything else. Movement suggests reallocation of responsibility and reflection and consideration. So, is this news? Have we been concentrating on the right Well, obviously not, otherwise I wouldn't be asking that question. This is Betteridge's law of headlines. So, obviously I'm going to refer to a 20th century book having referred to a 20th century quote. One of the things I like about this book, Fred Brooks, mythical man-month, is that this is a book that was written in the 1970s. That is reflected in the genderism of the title. About a project in the 1960s. This is my which this is my edition from the 1990s, which contains an essay from the 1980s. Which is actually that's one of the important things. Everything I have referred to there, not a single thing is 21st century except for the fact that humans have not changed. How much of what software engineers now do is still devoted to the accidental as opposed to the essential? This is a really good question. He asked this in the mid-80s. In the in an essay there no silver bullet. I would I I would invite you to ask the same question now, over 40 years later. How much of what is the daily work is actually to do with we are building this product for this customer. This is what the customer wants and is directly about that. Not about the other stuff. And frankly, we have a different way of classifying this and organizing this. Let's look at John Seddon. Uh he's a very much a systems thinker. He's not a software guy. I saw him speak at the first Lean Kanban conference a number of years ago. And one of the most enduring observations I took away from that is this idea here, which I talked about for a couple of years then. And then about 5 years ago I started talking again about this stuff. It turns out this is exactly the right timing to be caring about this. Value demand. Right, let's talk about demand. A lot of people are not scrutinizing why are we busy? Or where is our work coming from? What is the nature of my work? So, we have this really interesting one and it kind of goes back to an observation um that uh I guess you know, I didn't have the vocabulary at the time. I'll come back to this one in a moment. A friend of mine, Ray Let's look first of all what John Seddon says, and then we'll kind of loop back to that. Value demand represents the demands customers make for the things they want, things that are of value to them. There's a counterpoint here. Failure demand is demand caused by a failure to do something or to do something right. In other words, the work that you do is to make up for the work that you didn't do, or the work that you didn't do quite right. Let's be very clear, there is no such thing as a 100% efficient or effective system. You will always have failure demand. The question is not do you have it, but how much of it do you have? Okay, as a human being, I generate a certain amount of waste heat. The question is not do I generate waste heat, but how much do I generate? Okay? Um, light bulbs have waste heat. These days, uh they still have waste heat. It turns out they have less waste heat than they used to. Turns out that filament bulbs are very good at producing waste heat, which gives us wonderful things like lava lamps and burnt fingers. But if you actually want energy efficiency, it turns out you have to change something fundamental. You still have inefficiency and waste in that system. It's just a lot less. So, always be careful, cuz sometimes when people sort of see something divided into two and one of them is not good and the other one is good, they want to maximize to 100% the good thing and turn to zero the other That's not how any of this works. But there is this observation, and this is important because we all need to understand what it is that we are scaling. So, my friend Ray he got off to Silicon Valley work for company and he come he'd come back. We went for dinner, and he was just he was telling me over dinner, this is about 20 years ago. He says, like, "Kevin, it's been really good. We've doubled the size of the number of software developers uh in the company. We've gone from a uh an an organization of 30 software developers to 60 software developers." And he's saying this with glee, and I remember just sitting across the table from him going like, "Yeah, Ray, but those 13 new developers, what are they doing? Are they dealing with the problems the first 30 created? Or And the point is, nobody ever asked why are we scaling? This is one of my big beefs with any kind of scaling. You haven't answered the question why. Why is this? The idea of demand is really important. Failure demand and value demand. Of course, it can be more sophisticated, but just partitioning into two already gives you a more conversation you probably didn't have. How much of the work we are doing is actually meaningful and essential? How much of it is fixing bugs, dealing with technical debt, dealing with outages, dealing with all this other stuff? We're so busy with that, we can't actually concentrate on this. Now, when you ask a lot of people, they kind of give you an image a little bit like this. It's like, "Yeah, we we have a bit of failure demand. Yeah, here's the value demand. This is how we spend our time. Yeah, sure we have value demand." Oh, you sweet things. It doesn't look anything like this. I think you know, I'm not going to I don't have hard numbers. I have anecdotal at best. But, certainly one one organization I work with, informally I recognized I said basically if you guys are doing anything more than 1 hour of actual value demand per meeting, 1 hour of value demand per day, you are doing really well. Because when I look at what people are actually doing, it's directly or indirectly dealing with all the problems of everything that you've already got. Okay? That's That's, you know, if you fix that, you would actually be a 10x organization. Okay? That's, you know, that's where your bottleneck is. Your bottleneck comes from two places. One, it's the organizational structure, and two, it's the past. You're building for the future, but you can't even do stuff in the present because of the past is weighing over you. You haven't solved the solved problems. Now, >> [gasps] >> here's scaling. We are so much busier. And that's what a lot of people are measuring. And And that And this is not worthwhile work. This feels feels difficult. Okay, let's let's re- try this. I want to you a meaningful learning progression with time. Now, in an ideal universe, it looks like this. In practice, what often happens is it's going to look like this. Your gains will be more modest, but you are now still busier because you've also scaled your failure demand. But, I've been going through this talk telling you there's always three ways these things go. You can end up with this. We are meeting the same value, but we are busier with all the consequences and the failures, the issues. And when I say failure, I'm talking both bugs, but also other things that are indirect. Having to comprehend or work around stuff that does not work or build or is of poor design and incomprehensible design or is somehow misaligned, that is also failure demand. If you're doing it really badly, you could be this, but you might be thinking you're in the pre- one of the previous categories. So, there is a point here about being a little bit more careful. Now, one of the areas that this kind of raises its head is Have you ever come across Alberto Brandolini? Who knows Alberto? Yeah. Brandolini's law is Alberto coined this one a few years ago, I think 2014, 2013. Brandolini's law is known as the asymmetry principle. The observation that the amount of energy necessary to refute is an order of magnitude greater than needed to produce it. Just last year, I had the epiphany that this was actually what a lot of people were doing with a lot of AI stuff. That's where their work is going. So, let's talk about the nature of work. Siddharth Arora had this lovely piece um early last month. AI fatigue is real and nobody talks about it. This is a paradox. AI reduces the cost of production, but increases the cost of coordination, review, and decision-making, and those call all those costs fall entirely on the human. So, late last year, I saw a little video. I've I've mentioned Claude, I've mentioned Copilot, so let's talk about Google anti-gravity. I saw a little video promoting anti-gravity. Okay. So, Google's kind of like answer to Claude code, and it was just and >> [sighs and gasps] >> in the video, it said, "Congratulations. You have been elevated to a manager of agents." And they were you know, I I had a lot of feelings about this. One is that that's not a thing to be congratulated for. Two is is this what I really got into this whole game for? They should be apologizing saying, "But in spite of this, it's still fun." That for some people, this is an absolutely what they want to hear. But I I was fascinated by the idea that they thought that that was a marketing slogan that would appeal to lots of people. Because it also at that moment made me realize there's another issue. Not only is that not engineering speak, uh not only is that not engineering speak, but it's also is it it it's I I I had a realization that this is very much um the basis of decision fatigue. I suddenly realized we're making lots and lots of decisions. So, I want to kind of close with a couple of observations. Um that that in this sense should allow us to kind of have a little more insight over to over the things that we are doing. Because psychologically going back to this survey let's talk about that being the human. Copilot access was not significant in mitigating the risk of burnout. This was really interesting. Most surveys are not talking about this stuff. They're talking about how many lines of code you're doing, how many tickets you you're burning through. This is really interesting because this is Again, this is this idea of like, "Yeah, we're supposed This is supposed to be helping us." In what way is this helping us if it's not helping us? Have we got proper control? So, this is again is not a question about are we doing the right thing? Is it a good thing or a bad thing? That's too naive. And how what do we mean by effectively working? The idea is that this observation about this being an amplifier, there's a danger here that we are potentially de-skilling ourselves as well. And this is one of the interesting things that I find um from an engineering point of view. The rest of this is more interesting. Research shows AI helps people to do parts of their jobs faster. Other research shows that when people use AI assistants, they become less engaged with their work and reduce the effort they put into doing it. Something we saw reflected more flippantly in in the quote. Um in other words, they offload their thinking to AI. And the paper that this refers to is is from Anthropic. So, you know, hats off. They've done very well this last couple of months in terms of being a moral uh moral agent. Um the research they were referring to was on skill formation. We find that AI use impairs conceptual understanding, code reading, and debugging abilities without delivering significant efficiency gains on average. Our findings suggest AI-enhanced productivity is not a shortcut to competence and AI assistants should be carefully adopted into workflows to preserve skill formation. This is from one of the key companies. They are paying attention. And a few other people are paying attention. I just think we're not yet in the critical mass. This point about learning and understanding your system. Let's have a little bit lightness here. I'm going to I'm going to pick from the um the uh Finnish cartoonist, Samuli Lintula. Dark side of the horse. These days young people are using artificial intelligence to do their homework. And they learn nothing. That just sounds like, you know, old man cloud. When I was their age, I had to learn nothing the hard way. I just love I'm not saying that we were perfect. We absolutely are not. But, this is a multiplier. The multiplication of nothing is quite crazy in this sense. So, I guess I ought to close with a kind of a reflection of architecture. And the architect Yoshio Taniguchi who designed the MoMA I was at MoMA in New York City. Architecture is basically a container of something. That's relevant. The question is what is the something? We might think of it as code. Well, he makes this point and he's talking about buildings, obviously. I hope they will enjoy not so much the teacup but the tea. And this is the whole thing is an architecture is a space. It is a social space. It is a construct. It is not merely a container of code and functionality. It is that. It is a container of our decisions and our conversations. It is a container of the way that we talk about a system, the way that we experience working within it to build it, and how it defines the relations with our customers. And that's quite important from that point of view. So, we need to recognize that to re-engage ourselves, we need to use these tools for one of the things they're really good at. I find many people are looking for one solution when they do stuff. There is nothing more dangerous than an idea when you have only one idea. This is Emilio Rogatchevsky. And this observation from Jules White, first thing I tell people, don't ask for one answer, ask for three or five. The ability to generate options has never been so easy. And yet most people do not do it. The one thing you can do easily is the thing those people are not doing. They are generating the one answer quicker rather than saying, "Hey, what can I learn about this?" As he says, this forces the human being to re-engage and think about what they like and why. Which leads us right back to where we start. Thank you very much. >> [applause]