🎧 Stanford CS230 | Autumn 2025 | Lecture 9: Career Advice in AI

👤 Andrew Ng · EN 19,336w · KR 4,992w

사용법 (30초)
1️⃣ 영문 or 한국어 복사 버튼 터치
2️⃣ 🔗 링크 복사 → Chrome에 붙여넣기 → 열기
3️⃣ + 새 노트북 → 소스 추가 → 붙여넣기
4️⃣ 🎧 Audio Overview → Generate
5️⃣ 차에서 재생 🚗

원문 미리보기 19,336w

What I want to do today is chat with you about career advice in AI. And in previous years, I used to do most of this lecture by myself. But what I thought I'd do today is I'll share just a few thoughts and then hand it over to my good friend Laurence Moroney, who I invited to speak here and kindly agreed to come all the way to San Francisco, he lives in Seattle, to share with us a very broad market landscape for what he's seeing in the job market, as well as tips for career, growing a career in AI. But there was just two slides and then one more thought I want to share with you before I hand it over to Laurence, which is it really feels like the best opportunity, the best time ever to be building with AI and to building a career in AI. A few months ago I noticed in social media, traditional media, there are a few questions about is AI slowing down? People saying, well, it's GPT-5 that good? I think it's actually pretty good. But there are questions about is AI progress slowing down? And I think part of the reason the question was even raised was because if a benchmark for AI is 100% is perfect answers, then if you make rapid progress, at some point, you cannot get above 100% accuracy. But one of the studies that most influenced my thinking was work done by this organization, M-E-T-R, METR that studied as time passes, how complex are the tasks that AI could do as measured by how long it takes a human to do that task? So a few years ago, maybe GPT-2 could do tasks that a human could do in a couple seconds. And then they could do tasks that took a human four seconds, then eight seconds, then, a minute, two minutes, four minutes, and so on. And the study estimates that the length of task AI can do is doubling every seven months. And I think on this metric, I feel optimistic that AI will continue making progress, meaning the complexity of tasks as measured by how long a human takes to do something is doubling rapidly. And same study with a smaller data set seems to show-- same study argued that for AI coding, the doubling time is even shorter, maybe 70 days. So this code that used to take me, I don't know, 10 minutes to write, then 20 minutes to write, 40 minutes to write, and AI could do more and more of that. And so the reasons I think this is a golden age to be building, best time we've ever seen is maybe two themes which are more powerful and faster. So we can all, all of you in this room can now write software that is more powerful than what anyone on the planet could have built a year ago by using AI building blocks. AI building blocks include large language models, radical genetic workflows, voice AI, and of course, deep learning. It turns out that a lot of LLMs have a decent, at least basic understanding of deep learning. So if you have a prompt one of the frontier models to implement a cutting edge neural network for you, try prompting it to implement a transformer network for you. It's actually not bad at helping you use these building blocks to build software quickly. And so we have very powerful building blocks that were very difficult or did not exist a year or two ago. And so you can now build software that does things that no one else on the planet, even the most advanced teams on the planet, could have done. And then also with AI coding, the speed with which you can get software written is much faster than ever before. And I've personally found it as important to stay on the frontier of tools, because the tools for AI coding changes, I don't know, really rapidly. So I feel like since several months ago, my personal number one favorite tool became Cloud Code, moving on from some earlier generations, I think. And then I think since the release of GPT-5, I think OpenAI Codex has actually made tremendous progress. And this morning, Gemini 3 was released, which haven't had time to play with it yet just this morning. It seems like another huge leap forward. So I feel if you ask me every three months what my personal favorite coding tool is, it actually probably changes definitely every six months, but quite possibly every three months. And I find that being half a generation behind in these tools means being, frankly, quite a bit less productive. And I know everyone says AI is moving so fast, everything's changing so fast. But AI coding tools, of all the sectors in AI, many things maybe don't move as fast as the hype says it does, but AI coding tools is one sector where I see the pace of progress is tremendous. And staying at the latest generation of tools, rather than half generation behind makes you more productive. And with our ability to build more powerful software and build it much faster than ever before, I think one piece of advice that I give now, much more strongly now than even a year ago or two years ago, is just go and build stuff. Take classes from Stanford. Take online courses. And additionally your opportunity to build things, and I think Laurence is going to talk about showing them to others, is greater than ever before. But there's one weird implication of this that is maybe not-- is still, I don't know, more and more people are appreciating it, but not widely known, which is the product management bottleneck, which is that when it is increasingly easy to go from a clearly written software spec to a piece of code, then the bottleneck increasingly is deciding what to build or increasingly writing that queer spec for what you actually want to build. When I'm building software, I often think of going through a loop where we'll write some software, write some code sure to use this to get user feedback. I think of this as a PM or product management work. And then based on user feedback, I'll revise my view on what users like, what they don't like. This UI is too difficult. They want this feature. They don't want that feature and change my conception of what to build, and then go around this loop many times to hopefully iterate toward a product that users love. And because of AI coding, the process of building software has become much cheaper and much faster than before. But that ironically shifts the bottleneck to deciding what to build. So some weird trends I'm seeing. In Silicon Valley and in many tech companies, people have often talked about an engineer to product manager, engineer to PM ratio. And you take these ratios with grain of salt, because they're kind of vary all over the place. But you hear companies talk about the Eng to PM ratio of 4 to 1 or 7 to 1 or 8 to 1. This idea that one product manager writing product specs can keep four to eight or some number like that engineer is busy. But because engineering is speeding up, whereas product management is not sped up as far as much by AI as engineering, I'm seeing the Eng to PM ratio trending downward, maybe even two or one to one. So some teams I work with, they proposed headcount was one PM to one engineer, which is a ratio unlike almost all Silicon Valley, certainly traditional Silicon Valley companies. And the other thing I'm seeing is that engineers, they can also shape products that can move really fast where you go one step further, take the engineer, take the PM, and collapse them into a single human. And I find that there are definitely engineers that doing engineering work that don't enjoy talking to users and having that more human, empathetic side of work. But I'm finding increasingly that the subset of engineers that learn to talk to users, get feedback, develop deep empathy for users so that they can make decisions about what to build, those engineers are also the fastest moving people that I'm seeing in Silicon Valley today. And I feel like at the earliest stage of my career, one thing I regretted for years was in one of the roles I had, I went to try to convince a bunch of engineers to do more product work. And I actually made a bunch of really good engineers feel bad for not being good product managers. And that was a mistake I made, regretted that for years. I just shouldn't have done that. And part of me feels like I'm now going back to repeat that exact same mistake. Having said that, I find that the fact that I can write code, but also talk to users to shape what to do, that lets me and the engineers that can do this go much faster. So I think maybe worth taking another look at whether engineers can do a bit more of this work, because then if you're not waiting for someone else to take the product to customers, you can just write code, have a gut for what to do next, and iterate that pace, that velocity of execution is much faster. And then before I hand over to Laurence, just one last thing I want to share, which is in terms of navigating your career, I think one of the most strong predictors for your speed of learning and for your level of success is the people you surround yourself with. I think we're all social creatures. We all learn from people around us. And it turns out there are studies in sociology that show that if your five closest friends are smokers, the odds of you being a smoker is pretty much high. Please don't smoke. It's just an example. I don't know of any study showing that if you're five or 10 closest friends are really hard working, determined people, learning quickly, trying to make the world a better place of AI, that you are more likely to do that too. But it's one of those things that I think is almost certainly true. It's like all of us are inspired by the people around us, and we're able to find a good group of people to work with, that helps drive you forward. In fact, here at Stanford, I feel very fortunate-- the fantastic student body, fantastic group of faculty. And then the other thing that I think we're fortunate to have at Stanford is our connective tissue. So candidly, a lot of the people working and a lot of the cutting edge AI labs, the frontier labs, they were former students of a lot of different Stanford faculty. And so that rich connective tissue candidly means that at Stanford, we often find out about a lot of stuff that's not widely known because of the relationships, the friendships. And when some company does something, one of my friends and the faculty will call up someone to come and say, hey, that's weird. Does this really work? And so that rich connective tissue means that we're all-- just as we try to pull our friends forward, our friends also pull us forward with the knowledge and the connective tissue and this know-how of bleeding edge AI, which unfortunately is not all published on the internet at this moment in time. So I think while you're at Stanford, make those friends, form that rich connective tissue. And there have been a lot of times that just for myself, where, frankly, I was thinking of going in some technical direction. I'd have one or two phone calls with someone really close to research, either Stanford researcher or someone in the frontier lab. They would share something with me that I didn't know before. And that changes the way I choose the technical architecture of a project. So I find that group of friends you surround yourself with, those little pieces of information-- try this. Don't do that-- that's just hype. Ignore the PR. Don't actually try that thing. Those things make a big difference in your ability to steer the direction of your projects. So while you're at Stanford, take advantage of that. This connective tissue that Stanford has, it's actually really unique. There are lots of great universities in the world, but at this moment in time, I don't think there's any-- I don't want to sound like I'm doing PR for Stanford now, but I really think there's no university in the world that is as privileged as Stanford at this moment in time, in terms of the richness of the connective tissue to all of the leading AI groups. But to me, there's also that we're lucky here to have a wonderful community of people to work with and learn from. And for you too. If you apply for jobs, the thing that is much more important for your career success would be if you go to a company, it'll be the people you work with day to day. So here's one story that I've told in previous classes, I repeat, which is there's a Stanford student that I knew this was many years ago, that I knew, and they did really good work at Stanford. I thought they were high flyer. And they applied for a job at a company, and they got a job offer from one of the companies with a hot AI brand. This company refused to tell him which team he would join. They said, oh, come sign up for a job. There's a rotation system, matching system, blah blah blah. Sign on the dotted line first. Then we'll figure out what's a good project for you. Partly because it was a good company. His parents were proud of him for getting a job at this company. This student joined this company hoping to work on exciting AI project. And after he signed on the dotted line, he was assigned to work on the back end Java payment processing system of the company. Nothing against anyone that wants to do Java back end payment processing systems. I think they're great, but this is an AI student that did not get matched to an AI project. And so for about a year, he was really frustrated, and he actually left this company after about a year. The unfortunate thing is, I told this story in CS230 some years back. And then after I was already telling the story in this class, a couple of years later, another student in CS230 went through the same experience with the same company, not Java back end payment processing, but different project. And I think this effect of trying to figure out who you'll be actually working with day to day and making sure you're surrounded by people that inspire you and work on exciting projects, I think that's important. And even completely candid, if a company refuses to tell you what team you'll be assigned to, that does raise a question in my mind of whether or not what will happen. And I think that instead of working for the company with the hottest brand, sometimes if you find a really good team with really hard working, knowledgeable, smart people trying to do good with AI, but the company logo just isn't as hot, I think that often means you actually learn faster and progress your career better because it is after all, we don't learn from the excitement of the company logo when you walk through the door, you learn from the people you deal with day to day. So I just urge you to use that as a huge criteria for your selection process for what you decide to do. But I think number one on my advice is it's become much easier than ever before to build powerful software faster. And what that means is do be responsible. Don't build software that hurts others. And at the same time, there are so many things that each of you can build. And what I find is the number of ideas out in the world is much greater than the number of people with the skill to build them. So I know that finding jobs has gotten tougher for fresh college grads. At the same time, a lot of teams just can't find enough skilled people. And so there are a lot of projects in the world that if you don't build it, I think no one else will build it either. So you don't need to-- so long as you don't harm others, be responsible, there are a lot of things that you don't need to wait for permission. You don't need to wait for someone else to do it first and then you do it. The cost of a failure is much lower than before because you waste a weekend but learn something. That seems fine to me. So I think so let's be responsible, going for trying things out and building lots of things would be the number one most important thing I think would help your careers. And yeah, I think I'm going to say one last thing that is considered not politically correct in some circles, but I'll just say it anyway, which is in some circles, it has become considered not politically correct to encourage others to work hard. I'm going to encourage you to work hard. Now, I think the reason some people don't like that is because there are some people that are in a phase of life where they're not in a position to work hard. So right after my children were born, I was not working hard for a short period of time. And there are people because of an injury or disability, whatever very valid reasons, they're not in a position to work hard at that moment in time. And we should respect them, support them, make sure they're well taken care of even though they're not working hard. Having said that, all of my, say, PhD students have become very successful, I saw every single one of them work incredibly hard. I mean, the 2:00 AM sitting up, hyperparameter tuning, been there, done that. Still doing it some days. And if you are fortunate enough to be in a position in life where you can work really hard, there are so many opportunities to do things right now. If you get excited, as I do, spending evenings and weekends coding and building stuff and getting user feedback, if you lean in and do those things, it will increase your odds of being really successful. So I don't know. Maybe I get into some trouble with some people encouraging me to work hard, but I find that the truth is people that work hard get a lot more done. We should also respect people that don't and people that aren't in a position to do so. But between watching some dumb TV show versus firing your agentic coder on a weekend to try something, I'm going to choose the latter almost every time. Unless I'm watching a show with my kids, sometimes I do that. But you mean-- I hope you do that. All right, so those are the main things I wanted to say. What I want to do is hand the stage over to my good friend Laurence Moroney, who share a lot more about career advice on AI. Let me just quick intro. I've known Laurence for a long time. He's done a lot of online education work, sometimes with me and my teams, taught a lot of people Tensorflow, taught a lot of people PyTorch. He was lead AI advocate at Google for many years, now runs a group at Arm. I've also enjoyed quite a few of his books. This is one of them. He recently also published a new book on PyTorch. This is an excellent book, Introduction to PyTorch. And he's a very sought after speakers in many circles, so I was very grateful when he agreed to come speak to us. Pleasure is all mine. I just want to reinforce something that Andrew was talking about earlier on about choosing the people that you work with being very important. But I also want to show that from the other way around that the company, when they're interviewing you are also choosing you. And the good companies really want to choose the people that they work with also. And I've been doing a lot of mentoring of young people over the last, particularly over the last 18 months, who are hunting for careers for themselves. And I want to tell the story of one young man and this guy, very well educated, great experience, super elite coder. He could do every challenge that was in front of him, and he got laid off from his job in April. He worked in medical software, and medical software business has been changing drastically. Funding has been cut by the Federal government in a number of areas, and he got laid off from his job. And with his experience, with his ability, with his skills, all of these kind of things, he thought that it would be very easy for him to find another job. And the poor young guy had a really terrible April. He got laid off from his job in April. Immediately before that, his girlfriend had broken up with him, and then a couple of weeks later, his dog died. So he was not in a good place. And so I sat down with him after a couple of months and took a look. And he had a spreadsheet of jobs that he was applying to, and he had over 300 jobs that he was tracking in the spreadsheet. And in a number of these jobs, he actually got into the interview process, and he went very deep in the interview process with companies like Meta. Who else? Not Google. It was Meta. There was Microsoft. There was one of the other large tech companies where you do lots and lots of interview loops. And every time towards the end of the loop, he knew he did a great loop. He solved all the coding. He had great conversations with the people, or at least he thought he had. And then every time within a day, the recruiter would call him and say, no, you didn't get the job. And it was like it was heartbreaking. And like I said, 300 plus jobs he had been tracking. So I started working with him to do some mock interviews and to do some fine tuning. Oh, it was Jeff Bezos company, not Amazon. That was one of the other big tech company that he'd interviewed with. And I started working through him and doing some test interviews and all this kind of thing with him. Terrific, terrific candidate couldn't figure out what was going wrong until I decided to try and do a different sort of interview where I gave him a really tough interview. I gave him some tough LeetCode. I gave him some really obscure corner cases in his coding. And I saw how he reacted. And how he reacted was the advice that was given to him in the recruiting pamphlets. And a lot of these recruiting pamphlets will say things like, you're going to have an opportunity to share an opinion, and you've got to stand your ground. You've got to have a backbone. Don't bend. His interpretation of that was to be really, really tough. So I would pick corners. I would pick holes in his code. I'd pick corner cases where things may not work, and I would give him a test of crisis. And this advice that he'd been given to stand his ground ended up making him hostile in these interview environments. And I was looking at this then from the point of view of what Andrew was just talking about, where it's a case of hey, good people, good teams, people that you can work together with. And from the interviewer perspective, if I'm managing this team, this person is that cliched 10x engineer, but I don't want him anywhere near my team because of this attitude. We worked on that. We fine-tuned it. And the strange part is he's a really, really nice guy. It's just this was the advice he was given, and he followed that advice, and he failed so many interviews as a result. So when I gave him the next job that he was interviewing at was at a company where teamwork is very, very highly valued. And the good news is he got the job at that company. He's now working there. He doubled his salary from the job he was laid off from, and he ended up having about-- now he looks back and he had six months of fun employment. But at the time when he was going through all of that, it was a very, very difficult time for him. So the flip side of it, if you're looking at a company and looking at the paper you'd be working with is very, very important. But also realize they are looking at you in the same way. And so if you've gone to a tech interview coaching, and they gave you that advice to stand your ground and have a backbone, it's good to do that. But don't be a jerk while you're doing so. Can you see my slides? OK. So I'm Laurence. I've been working in tech for more decades than ChatGPT thinks there are oars in strawberry. So I've worked in many of the big tech companies. I spent many years at Microsoft, spent many years at Google, also worked in places like Reuters. I've done a lot of work in startups, both in this country and abroad. And so what I really want to talk about today is like to think about what does the career landscape look like today, particularly in AI. Because first of all, what Andrew said about in Stanford, you've got the ability to make use of the networks that you have in Stanford, make use of the prestige that you have, and I say use every weapon you have. Because unfortunately, the landscape right now is not ideal. We've gone through some very difficult times. All you have to do is look at the news, and you can see massive tech layoffs, slowing hiring in tech, and lots of stuff like that. But it's not necessarily a bad thing if you do it the right way. So I want to just have a quick look the job market reality check. Actually out of interest, I don't know. This is a-- are you juniors? You're graduating this year or you're graduating next year or what is the general survey? You're third year of four? Third year of three, I would say. So you're going to be graduating coming summer. How many people are already looking for jobs? OK, quite a few of you. How many people have had success? Nobody. Oh, one. OK. That's good. So you're probably seeing some of these things, the signals out there, junior hiring slowing significantly. When I say junior, I mean graduate level. High-profile layoffs are dominating the headlines. I was at Google a couple of years ago when they had the biggest layoff they'd ever had. We're seeing layoffs at the likes of Amazon, Microsoft, other companies like that. It feels that entry-level positions are scarce, and I'm underlining the word "feels" there, and I want to get into that in a little bit more detail later. And also, competition is fierce. But my question is, should you worry? And I say, no. Because if you can approach things in the right way, if you can approach the job hunting thing in the right way, particularly understanding how rapidly the AI landscape is changing, then I think people with the right mindset will thrive. So what do I mean by that? So as Andrew had mentioned, the AI hiring landscape is changing because the AI industry is changing. The AI industry I-- I actually first got involved in AI back way back in 1992. I worked in it for a little while just before the AI winter. Everything failed drastically, but I got bitten by the AI bug. And then in 2015, when Google were launching TensorFlow, I got pulled right back into it, became part of the whole AI boom, launching TensorFlow, advocating TensorFlow to millions of people, and seeing the changes that happened. But along 2021, 2022, we had a global pandemic. The global pandemic caused a massive industrial slowdown. This massive industrial slowdown meant that companies had to start pivoting towards things that drove revenue and directly drove revenue. And at Google, TensorFlow was an open-source product. It didn't directly drive revenue. We began to scale back. Every company in the world also scaled back on hiring at this time. Then we get to about 2022, 2023. What happens? We begin to come out of the global pandemic. We begin to realize all industries have this massive logjam of non-hiring that they had done or hiring that they hadn't done. And we're also entering a time where AI was exploding on the scene. Thanks to the work of people like Andrew, the world was pivoting and changing to be AI first in just about everything. And every company needed to hire like crazy. Every company then hiring like crazy in 2022, 2023 meant that most companies ended up overhiring. And what that generally meant was people who were not qualified for higher positions usually got higher positions because you had to enter into a bidding war just to be able to get talent. You ended up having talent grabs, and you ended up having stories like the one Andrew told where it's a case here's a person with AI talent, let's grab them, let's throw money at them, let's have them come work for us, and then we'll figure out what we want to do. So as a result, 2022, 2023 all of this massive overhiring happens because of AI and because of the COVID logjam. And then 2024, 2025 is the great wake-up, where a lot of companies realize this over hiring that they had done, they have ended up with a lot of people who are underqualified. I'm sorry. Yeah, underqualified for the job that they were doing. A lot of people ended up getting hired just because they had AI on their resume. And there's a big adjustment going on. And in the light of this big adjustment-- show you-- just one second. In the light of this big adjustment-- oh, you're not saying my slides? OK. And in the light of this big adjustment-- there we go. I think it's because my power. I'm not plugged into power mains. And in the light of this big adjustment, then what has happened is now a lot of companies are much more cautious about AI skills that they're hiring. And if you're coming into that with that mindset and understanding that, realize opportunity is still there, and opportunity is there massively if you approach it strategically. So what I want to talk through today is how you can do exactly that. So I see three pillars of success in the business world and particularly in the AI business world. And nowadays you can't just have AI on your resume and get overhired. Nowadays, not only do you have to be able to tell that you have the mindset of these three pillars of success, but you also have to be able to show. And to be able to show these, that actually has never been a better time. As Andrew demonstrated earlier on, the ability to vibe code things into existence. He doesn't like the word vibe code. I agree with him, but the ability to prompt things into existence, or whatever the word is that we want to use, allows you to be able to show better than ever before. He was talking earlier on about product managers, and he had this time when he got engineers to be product managers, and then those engineers ended up being really bad product managers. I actually interviewed at Google twice and failed twice despite being very successful at Microsoft, authored 20 plus books, taught college courses. I interviewed at Google twice and failed twice because I was interviewing to be a product manager, and then when I interviewed to be an engineer, they hired me and they were like, why didn't you try to join us years ago? So a lot of it is just being a good engineer. You've got the ability to do that and show that nowadays. And with that ratio of engineer to product manager, changing engineering skills are also far more valuable than ever. So the three pillars to success. Number 1, understanding in depth. And I'm going to mean this in two different ways. Number one is academically, to have the understanding and depth academically of machine learning, of particular model architectures, to be able to understand them, to be able to read papers, to be able to understand what's in those papers, and to be able to understand, in particular, how to take that stuff and put it to work. The second part of understanding in depth is really having your finger on the pulse of particular trends and where the signal-to-noise ratio favors signal in those trends. And I'm going to be going into that in a lot more detail a little bit later. Secondly, and also very, very importantly is business focus. So Andrew said something politically incorrect earlier on. I'm going to also say a similar politically incorrect thing. First of all, hard work. Hard work is such a nebulous term that I would say that think about hard work in terms of you are what you measure. There is the whole trend out there. I'm trying to remember, is it 996 or is it 669? 996. 9:00 AM to 9:00 PM, six days a week is a metric of hard work. It's not. There's not a metric of hard work. That's a metric of time spent. So I would encourage everybody, in the same way as Andrew did, to think about hard work. But what hard work is how you measure that hard work. You can work eight hours a day and be incredibly productive. You can work six hours a day and be incredibly productive, but it's the metric of how hard you work and how you measure that. I personally measure that from output, things that I have created in the time that I spent. I joke a lot, but it's true that I've written a lot of books. Andrew held up one. That one that he held up, that he helped me write a little bit, I actually wrote that book in about two months. And people say, well, how do you have time with your jobs and all these kind of things? You must work like 16 hours a day in order to be able to do this. But actually, the key to me being able to write books is baseball. Any baseball fans here? So I love baseball, but if you sit down and try to watch baseball on TV, a match can take like 3 and 1/2 or four hours. So all of my writing I tend to do in baseball season. So I'm like, if I'm going to sit down, I like the Mariners. I'm from Seattle. I like the Dodgers. Nobody booed. OK, good. And so usually one of those is going to be playing at 7 o'clock at night. So instead of sitting in front of the TV, just like watching baseball mindlessly. I'll actually be writing a book while baseball is on in the background. It's a very slow moving game. This is something. That's the hard work in this case. And I would encourage you to try to find areas where you can work hard and produce output. And that's the second pillar here is that business focus, the output that you produce to align that output with the business focus that you want to have and with the work that you want to do. There's an old saying, "Don't dress for the job you have, dress for the one you want." I would say a new angle on that saying would be, don't let your output be for the job you have. Let your output be for the job you want. And if I go back to when I spoke about I failed twice at Google to get in, the third time when I got in, I had actually decided to do to approach this in a different way. And I was interviewing at the time for their cloud team. They were just really launching cloud, and I had just written a book on Java. And so I decided to see what I could do with Java in their cloud. I ended up writing a Java application that ran in their cloud for predicting stock prices using technical analytics and all that kind of stuff. And when it got to the interview, instead of them asking me stupid questions like how many golf balls can fit in a bus, they saw this code. I had put this code. I remember I was producing output for the job I wanted. I'd put this code on my resume, and my entire interview loop was them asking me about my code. So it put the power on me. It gave me the power to communicate about things that I knew, as opposed to going in blind to somebody asking me random questions in the hope that I'll be able to answer them. And it's the same thing I would say in the AI world. The business focus, the ability for you now to prompt code into existence, to prompt products into existence and if you can build those products and line them up with the thing that it is that you want to do, be it a Google or Meta or a startup or any of those kind of things, and have that in-depth understanding not just of your code, but how it aligns to their business, this is a pillar of success in this time and age. And I will also argue that even though it looks like the signals look like there aren't a lot of jobs out there, there are. What there aren't a lot of is a good combination of jobs and people to match them. And then, of course, this bias towards delivery. "Ideas are cheap, execution is everything." I've interviewed many, many people who came in with very, very fluffy ideas and no way to be able to ground them. I've interviewed people who came in with half-baked ideas that they grounded very, very well. Guess which ones got the job? So I would say these three things. Understanding and depth of the academics behind AI of the practicalities behind AI and the things that you need to do. Business focus, focusing on delivery for the business, understanding what the business needs and being able to deliver for that, and again, that bias towards delivery. So a quick pivot. What's it actually like working in AI right now? It's interesting. So as recently as two or three years ago, working in AI was if you could do a thing, you're great. If you can build an image classifier, you're golden. We'll throw six figure salaries and massive stock benefits at you. Unfortunately, that's not the case anymore. It's really a lot of today what you'll see is the P word, production. What can you do for production? What can you do if it's building new models, if it's optimizing models, if it's understanding users, UX is really, really important. Everything is geared towards production. Everything is biased towards production. The history that I told you about, going from the pandemic into the overhiring phase that we'd had, the businesses have pulled back and are optimized towards the bottom line. I have an old saying that the bottom line is that the bottom line is the bottom line, and this is the environment that we're in today. And if you can come in with that mindset when you're talking with companies, that's one of the keys to open the door. One of the things I've seen in the field has been maturing from it used to be really nice that we could do cool things and we could build cool things. Now it's really build useful things. Those useful things can be cool too, by the way, and the results of them can be cool. And the changes that we see that come about as a result of delivering them can be cool. So it's not just coolness for coolness sake, but to focus on delivery, focus on being able to provide value, and then the coolness will follow. I guess what I'm trying to argue. So for realities, number 1, unfortunately nowadays business focus is non-negotiable. Now, let me-- I'm going to be a little bit politically incorrect here again for a moment. I've been working, like I said, for most of the last 35 years in tech. I would say for most of the last 10 years, a lot of large companies, particularly in Silicon Valley have really focused on developing their people above everything. Part of developing their people was bringing their entire self to work. Part of bringing their entire self to work was bringing the things that they care about outside of work. And that led to a lot of activism within companies. Now, please let me underline this. There is nothing wrong with activism. There is nothing wrong with wanting to support causes, not wanting to support causes where of justice. There is absolutely nothing wrong with that. But the overindexing on that, in my experience, has led to a lot of companies getting trapped by having to support activism above business. You've probably seen an example about two years ago of where activists in Google broke into the Google Cloud heads office because they were protesting a country that Google Cloud were doing business with. They broke into his office, they had a sit-in his office, and they used the bathroom all over his desk and stuff like that. This is where activism got out of hand. And as a result, the unfortunate truth is the good signals in that activism are now being lost. Because of those actions, people are being laid off. People are losing jobs. Activism is being stifled, and business focus has become non-negotiable. There's a bit of a pendulum swing going on. And the pendulum that had swung too far towards allowing people to bring their full selves to work is now swinging back in the other direction. We might blame the person in the White House and all that for these kind of things, but it's not solely that. It is that ongoing pendulum there. And I think it's an important part of it, is that you have to realize going into companies now, that business focus is absolutely non-negotiable. Secondly, risk mitigation is part of the job. And I think a very important part of any job, particularly with AI. I think if you can come into AI with a focus and a mindset around understanding the risks of transforming a particular business process to be an AI-oriented one and to help mitigate those risks, I think is really, really powerful. And I would argue in an interview environment, that's the number one skill to have, to have that mindset around you are doing a business transformation from heuristic computing to intelligent computing. Here's the risks. Here's how you mitigate those risks, and here's the mindset behind that. The third part responsibility is evolving. Now responsibility in AI has again changed from a very fluffy definition of let's make sure that the AI works for everybody to a definition of let's make sure that the AI works. Let's make sure that it drives the business. And then let's make sure that it works for everybody. Often that has been inverted over the last few years, and that has led to some famous documented disasters. Let me share one with you. Let's see. I have lots of windows open. OK. Everybody knows image generation, text to image generation. I want to share a-- these were things that happened a couple of years ago with Gemini. So with Gemini, I was doing some testing around this one and I was working heavily on responsible AI. And part of responsible AI is you want to be representative of people. And when you're building something, like if you're a Google, you're indexing information, you really want to make sure that you don't reinforce negative biases. And if you're generating images, it's very easy to reinforce negative biases. So for example, if I said give me an image of a doctor, if the training set primarily has men as doctors, it's more likely to give a man. If I say give me an image of a nurse, if the training set more likely to have women as nurses, it's more likely to give me an image of a woman. But that's reinforcing a negative stereotype. So I wanted to do a test of how Google were trying to overcome that, given that these negative biases are already in the training set. So I said, OK, here's a prompt where I said, "give me a young Asian woman in a cornfield, wearing a summer dress and a straw hat, looking intently at her iPhone," and it gave me these beautiful images. It did a really nice job. And I said, this is a virtual actress I've been working with. I'll share that in a moment. And I say, OK, what if I ask for an Indian one? So I said, OK, whoops, a young Indian woman, same prompt. And it gave me beautiful images of a young Indian woman. Then I was like, OK, what if I want her to be Black? For some reason it only gave me three. I'm not sure why, but it's still adhere to the prompt. So the responsibility was looking really, really good. So then I asked it to give me a Latina. Latina, it gave me four. But yeah, she looks pretty Latina. Maybe the one on the bottom left looks a little bit like Hermione Granger, but on the whole looks pretty good. Then I asked it to give me a Caucasian. What do you think happened? "While I understand your request, I am unable to generate images of people as this could potentially lead to harmful stereotypes and biases." This was a very poorly implemented safety filter, where the safety filter in this case was like looking for the word "Caucasian" or looking for the word "whites" and the results saying it wouldn't do it. I was like, OK, well, let me test the filter a little bit and I said, OK, instead of Caucasian, let me try white. And yet, while I'm unable to fulfill your-- "While I'm able to fulfill your requests, I'm not currently generating images of people." It lied to my face because it had just generate images of people. Anybody know the hack that I used to get it to work? This is a funny one. So I will show you. One moment. I asked it to generate an Irish woman. What do you think it did? It gave me this image of an Irish woman, no problem, in a summer dress, straw hat, looking intently at her phone. What do you notice about this image? She's got red hair in every image. I grew up in Ireland, and Ireland does have the highest proportion of redheads in the world. It's about 8%. But if you're going to draw an image of a person and associate a particular ethnicity with a color of hair, you can begin to see this is massively problematic. There are areas, I believe, in China where the description of a demon is a red-headed person. So what ended up happening here, from the responsible AI perspective, was one very narrow view of the world of what is responsible and what is not responsible. Ended up taking over the model, ended up damaging the reputation of the model and damaging the reputation of the company as a result. In this case, it's borderline offensive to draw all Irish people as having red hair, but that never even entered into the mindset of those that were building the safety filters here. So when I talk about responsibility is evolving, that's the direction that I want to-- sorry, one moment. Let me get my slides back. --that's the direction I want you to think about, that now responsible AI has moved out of very fluffy social issues and into more hard line things that are associated with the business and prevent damaging the reputation of the business. There's a lot of great research out there around responsible AI, and that's the stuff that's been rolled into products. And then, of course, I just showed with Gemini, learning from mistakes is constant questioning at the front. Yes. I also heard that, I didn't verify that to be true, but I incorporated this feature that makes certain races and ethnicities historical objects. Yeah. Yeah. So the question was issues where races and things were mixed in historical context was the same problem. So, for example, if you had a prompt that said, draw me a samurai, the idea was like they didn't want to have-- the engine that changed the prompt to make sure that it was fair would end up saying, give me a mixture of samurai of diverse backgrounds. And then you'd have male and female samurai, samurai of different races and those kind of things. And it was the same prompting that ended up causing the damage that I just demonstrated. So the idea was to intercept your prompts to make sure that the outputs of the model would end up providing something that was more fair when it comes to diverse representation. So it was a very naive solution that ended up being rolled in. That was a few years ago. They've massively improved it since then, but that's when I'm talking about if you're working in the AI space nowadays, that's how responsibility is evolving. You can't just get away with that stuff anymore. That Gemini lesson was a good-- that Gemini example is a good lesson from that. And the mindset of you will make mistakes, so learning from mistakes is a constant ongoing thing. And going back to the people point that Andrew made earlier on, the people around you will make mistakes too. So to have the ability to give them grace when they make mistakes and to work through those mistakes and move on is really, really important and is a reality of AI at work. I've spoken a lot about the business focus advantage, so I'm going to skip over this. So now let's talk about vibe coding. So let's talk about the whole idea of generating code. Now, the meme is out there that it makes engineers less useful by the fact that somebody can just prompt code into existence. There is no smoke without fire, of course, but I would say don't let that meme get you down because that's when you start peeling into these things, that is ultimately not the truth. The more skilled you are as an engineer, the better you become using this type of vibe. Somebody give me another phrase other than vibe coding, using this probe to coding. And I always like to think about this and to try and put you and put people that I speak with into the role of being a trusted advisor for the people that you speak with. So whether you're interviewing with somebody, get yourself into the mindset of being a trusted advisor of the company that you're interviewing for, whether you're consulting or whatever those kind of things are. So when you want to get into the idea of being a trusted advisor, then you really need to understand the implications of generated code. And nobody can understand the implications of generated code better than an engineer. And the metric that I always like to use around that is technical debt. Quick question. Are you familiar with the phrase technical debt? Nobody. OK. Andrew and I were doing a conference in New York on Friday, and I used the phrase, and I saw a lot of blank faces. So I didn't realize that people didn't understand what technical debt is. So let me just take a moment to explain that, because I find it's an excellent framework to help you understand the power of vibe coding. Think about debt the way you normally would. Buying a house. If you buy a house, say, you borrow half a million dollars to buy a house. In a 30-year mortgage, when you're buying that house at half a million dollars, with all the interest that you pay is about double. So you end up paying back the bank about $1 million on half a million owned. So you have 30 years of home ownership at a cost of $1 million in debt. That is probably a good debt to take on, because the value of the house will increase over that time. You're not paying rent over that time, and that million dollars that you're spending on this house over those 30 years is a good debt to take on, because you're getting greater than $1 million worth of value out of it. A bad debt would be an impulse purchase on a high interest credit card. Those pair of shoes, those latest ones I really want to buy them. It's $200. By the time I've paid them off, it's $500. You're not getting $500 worth of benefit out of those shoes. Approaching software development with the same mindset is the right way to go. Every time you build something, you take on debt. It doesn't matter how good it is, there's always going to be bugs. There's always going to be support. There's always going to be new requirements coming in from people. There's always going needs to market it. There's always going needs for feedback. All of these things are debt, every time you do a thing. The only way to avoid debt is to do nothing. So your mindset should then get into when you are creating a thing, whether you're coding it yourself or whether you're vibe coding it or any of these things that you are increasing your amount of technical debt, those things that you need to pay off over time. So the question then becomes, as you vibe code a thing into existence in the same way as buying a thing, is it worth the technical debt that you're taking on? What does technical debt generally look like? Bugs that you need to fix, people that you need to convince to help you maintain the code, documentation that you need to do, features that you need to add, all of these kind of things. You're all very familiar with them. Think about those as that extra work that you need to do beyond your current work. That's the debt that you're taking on. There are soft debt, and there are hard debt. So to me, that would be the number one piece of advice that I give. And it's the one that I give every time I work with companies around vibe coding. And a lot of companies that I speak with, a lot of companies that I consult with-- I do a lot of work with startups, in particular-- they just want to get straight into opening Gemini or GPT or Anthropic and start churning code out. Let's get to a prototype phase very quickly. Let's go to investors. Let's do stuff. It's great. It can be. But debt, debt, debt, debt, debt is always going to be there. How do you manage your debt? A good financier manages their debt and they become rich. A good coder manages their technical debt, and they become rich also. So how do you get the good technical debt? How do you the mortgage instead of the high credit card debt? Well, number one is your objectives. What are they? Are they clear? And have you met them? You knew what you needed to build. You didn't just fire up ChatGPT and start spinning code out. At least I hope you didn't. Think about how you build it. AI was there to help you build it faster. I'm working on my own little startup at the moment in the movie making space. And I've been using code generation almost completely for that. But what I've ended up doing for my clear objectives met box here is that I've started building this application. I've tested it. I've thrown it away. I started again, tested it, thrown it away. Each time my requirements have been improving in my mind. I understand how to do the thing a little bit better, and I can show some of the output of it in a few minutes. But the idea there is that it's always about having those clear objectives and meeting them. And then if you're building out the thing and you're not meeting those objectives, that's still a learning. And there's no harm in throwing it away because code is cheap now in the age of generated code. Finished code, engineered code is not cheap. So get those objectives, make them clear, build it, hit a specific requirement and move on. Is there business value delivered? Is the other part of it. I've seen people vibe coding for hours on things like Replit to build a really, really cool website. And then the answer was, so what? I mean, how is this helping the business? How is this really driving something? It's really cool. Yes, Mr. VP, I know you've never written a line of code in your life, and it's really cool that you've built a website now, but so what? So think about that, and focus on that. And that's how you avoid the bad technical debt. And then, of course, the most understated part of this, and in some ways the most important, particularly if you're working in an organization, is human understanding. The worst technical debt that you can take on is delivering code that nobody understands. Only you understand that, and then you quit and get a better job. And then the company is now dependent on that code. So being able to, as part of the process of building it, to make sure that your code is understandable through documentation, through clear algorithms, through the fact that you've spent some time poring through it to make sure that even simple things like variable names make sense is a really, really important way to avoid bad technical debt. And that bad technical debt, my favorite one is the classic solution looking for a problem. Somebody has an idea. Somebody has a tool. If the only tool you have is a hammer, every problem looks like a nail. And you end up having all of these tools that get vibe coded into existence. I've worked in large organizations where people just vibe coded stuff, checked it into the code base, and then it became really hard to find the good stuff amongst all the bad. Spaghetti code. Of course, poorly structured stuff, particularly when you prompt and prompt and prompt and prompt again, that it can end up getting into all kinds of trouble. My favorite one at the moment that I'm really struggling with is I'm building a macOS application. Anybody ever build in SwiftUI on macOS? OK, a couple. SwiftUI is the default language that Apple use for building for macOS as well as iPhone. But when you look at the training set, the data training sets that are used to train these models, the vast majority of the code is iPhone code, not macOS code. And when I prompt code into existence, it's often given me iOS APIs and those kind of things. Even though I'm in Xcode and I've created a macOS app and it's a macOS template and I'm talking to it in Xcode, it still gives me iOS code, stuff like that. And then if I try to change it using prompting, you end up spiraling into spaghetti code, and you have to end up changing a lot of this stuff manually. And then, of course, the other one that I joked about it earlier, but it's also true, is some of the bad technical debt that you're going to encounter in the workspace is authority over merit. That VP suddenly took out his credit card, subscribed to Replit, and started building stuff in Replit. And guess whose job it is to fix it? So a lot of the advice that I start giving companies and a lot of the words that I would encourage you to start thinking of in being a trusted advisor is to understand this stuff and to manage expectations accordingly. OK, so framework for responsible vibe coding we've just spoke about. So one of the things I want to get into as we're coming soon to a close is the hype cycle. So hype is the most amazing force. I mean, I think it's one of the strongest forces in the universe, and particularly in anything that's hot, such as the fields that I work in that are super hot at the moment and full of hype or AI and crypto-- you should see my Twitter feed-- that the amount of nonsense that's out there is incredible. So one of the things that I would say about the anatomy of hype that you really need to think about is if you are consuming news via social media, that the currency of social media is engagement. Accuracy is not the currency of social media. So I go on to-- even LinkedIn, which is supposed to be the more professional of these, is absolutely overwhelmed with influencers posting things that they've used, Gemini or GPT, to write an engaging post so that they can get engagement and they can get likes. And the engine itself is engineered, excuse the pun, to reward those types of posts. And we end up with that snowball effect of engagement being rewarded. If you are the kind of person who can filter the signal from the noise, and then who can encourage others around the signal and not the noise, that puts you in a huge advantage that makes you very distinctive. It's not as quickly and easily tangible as likes and engagements on social media. But when you're in a one-to-one environment like a job interview, or if you are in a job and you are bringing that signal to the table instead of the noise, that makes you immensely valuable. So coming in with that mindset, coming in with the idea of trying to filter that signal from the noise, trying to understand what is important in current affairs, how you can be a trusted advisor in those things, and how you can really whittle down that noise to help someone is immensely valuable. I want to start with one story. I might be stealing my own thunder. I'll go on to in a moment. So one story. Last year when agents started becoming the key word and everybody saying, in 2025, agent will be the word of the year and the trend of the year, a company in Europe asked me to help them to implement an agent. So let me ask you a question. If a company came up to you and said, please help me implement an agent, what's the correct first question that you ask them? What is an agent for you? OK. That's good. What is an agent for you? I'd actually have a more fundamental question. Yep. What do you want to do? What do you want to do? OK. Even more fundamental. My question was why? Why? And peel that apart. I spoke with the CEO, and he was like, oh. Yeah, everybody's telling me that I'm going to save business costs. And I'm going to be able to do these amazing things. And yeah, my business is going to get better because I have agents. And I'm like, well, who told you that? It was like, oh, yeah, I read this thing on LinkedIn, and I saw this thing on Twitter. And it was like-- and we ended up having that conversation. And it was a difficult conversation because I had to keep peeling apart. And I started asking the questions that you two just mentioned as well, until we really got to the essence of what he wanted to do. And what he really wanted to do, when we take all domain knowledge about AI aside, was that he wanted to make his salespeople more efficient. And I was like, OK, you want to make your salespeople more efficient. Nowhere in that sentence do I hear the word AI, and nowhere in that sentence do I hear the word agent. So now, as a trusted advisor, let me see what I can do to help your salespeople become more efficient. And I'm not going to be an AI Shill or an agent Shill. I just want to say, what do we do to make your salespeople more efficient? If anybody here has ever worked in sales, one of the things you realize what a good salesperson has to do is their homework. Before you have a sales call with somebody, before you have a sales meeting with somebody, you need to check their background. You need to check the company. You need to check the needs of the company. You see it sometimes in the movie that, oh, such and such plays golf. So I'll take them to play golf. It's not really that cliched, but there is a lot of background that needs to be done. So I spoke with him, and I spoke with their leading salespeople and found out that-- and I asked the salespeople, what do you hate most about your job? And they were like, well, I hate the fact that I have to waste all my time going to visit these company websites, going to look up people on LinkedIn. And every website is structured differently. So I can't just have a path through a website that I can follow. I have to take on all this cognitive load. And they were spending about 80% of their time researching and about 20% of their time selling. Oh, and by the way, most salespeople don't get paid very much. They have to make it up by commission, so they're only spending 20% of their time doing the thing that gets them commission directly. So we're like, OK, well, here's something now where we can start thinking about making them more efficient by cutting into that. So we set a goal is like to make salespeople 20% more efficient. And then we could start rolling out the ideas of AI. And then we could start rolling out the ideas of agentic AI. And a quick question what's the difference between AI and agentic AI? OK. So-- yeah. Like a good AI can do some a couple of steps. OK. Yep. Excellent. Yeah. So agentic AI is really about breaking it down into steps, which is good engineering to begin with. But agentic AI, in particular, I find there's a set pattern of steps that if you follow them, you end up with a whole idea of an agent. The first of these steps is to understand intent. We tend to use the words AI, Artificial Intelligence, a lot. But what large language models are really, really good at is also understanding. So if the first step of anything that you want to do is to understand intent. And you can use an LLM to do that to think about this is the task that I need to do. This is how I'm going to do it. Here's the intent. I want to meet Bob Smith and sell widgets to Bob Smith. And this is what I know about Bob Smith. Help me with that intent. The second part then is planning. So you declare to an agent what tools are available to it, browsing the web, searching the web, all of these kind of things. And once you understand your clear intent to be able to go to the step of planning and using those tools for planning, and an LLM is very, very good at then breaking that down into the steps that it needs to do to execute a plan. Search the web with these keywords. Browse this website and find these links, those types of things. Once it's then figured out that plan, then it uses the tools to get to a results. And then once it has the result, the fourth and final step is to reflect on that result. And looking at the results and going back to the intent, did we meet the intent? Yes or no. If we didn't, then go back to that loop. All agent is really broken down into those things. And if you think about breaking any problem down into those four steps, that's when you start building an agent. And that was part of being a trusted advisor, instead of coming in and waving hands and saying, agent this, agent that. Look at this toolkit, save 20%. It's really to break it down into those steps. Se we did. We broke it down into those steps. We built a pilot for the salespeople of this company, and they ended up saving between 10% and 15% of their time, of their wasted time. The doctrine of unintended consequences hit, though, after this. And the unintended consequence was the salespeople were much happier because the average salesperson was making several percentage points more sales in a given week, they were earning more money in a given week, and their job just became a little bit less miserable. And then refinement to that agentic process, to be able to do all of that research for them and to help give them a brief in a few minutes instead of a few hours to help them with the sales process, ended up being like a win-win-win all around. But if you go in being hype led and oh, build an agent for the thing without really peeling apart the business requirements, the why, the what, the how, and all of these kind of things, we ended up like, this company just would have been lost in hype. You've probably seen reports recently. I think McKinsey put one out last week showing that about 85% of AI projects at companies fail. And part of the main reason for that is that they're not well scoped. People are jumping on the hype bandwagon, and they're not really understanding their way through the problem. And I think you know the big brains in this room and the network that you folks have are really key component of being able to succeed is to understand your way through that problem. So that was a hype example around agentic that I was thankfully able to help this company through. Other recent hype examples you've probably seen, the software engineering is dead. My personal favorite, Hollywood is dead or AGI by year end. I was in Saudi Arabia this time last year at a thing called the FYI. And it was a dinner at the FYI, and I sat beside the CEO of a company who I'm not going to name, but this was a CEO of a generative AI company. And at that time he was showing everybody around the table this thing that he'd done, where it was text to video, and he could put in a text prompt and get video out of the prompt and get about six seconds worth of video out of it. A year ago, that was-- I beg your pardon, two years ago. Two years ago, that was hot stuff. Nowadays, obviously, it's quite passé. Anybody can do it. But he made a comment at that table, and it was a lot of media executives at that table was like, by this time next year, from a single prompt, we'll be able to do 90 minutes of video. And so bye-bye, Hollywood. So the whole Hollywood is dead meme, I think, came out of that. First of all, we can't do 90 minutes, even two years later from a prompt. And even if you did, what kind of prompt would be able to tell you a full story of a movie? So this type of hype leads to engagement. This type of hype leads to attention. But my encouragement to you is to peel that apart. Look for the signal. Ask the why question. Ask what question and move on from there. So becoming that trusted advisor. World's drowning in hype. How do you do it? Look at the trends, evaluate them objectively. Look at the genuine opportunities that are out there. There are fashionable distractions. I don't know what the next one is going to be, but there are these distractions that are out there that will get you lots of engagement on social media. Ignore them, and ignore the people that are leaning into them. And then really lean into your skills about explaining technical reality to leadership. One skill that one person coached me in once that I thought was really interesting, because it sounded wrong, but it ended up being right, was whenever you see something like this, try to figure out how to make it as mundane as possible. When you can figure out how to make it as mundane as possible, then you really begin to build the grounding for being able to explain it in detail in ways that people need to understand. If you go and you look at, I think Gemini 3 was released today, but there were leaks earlier this week. And one person leaked that I built a Minecraft clone in a prompt, that kind of stuff. This is the opposite of mundane. This was massively hyping the thing, massively showing. And of course, they didn't. They built a flashy demo. They didn't really build a Minecraft clone. But the idea here is if you can peel that apart to OK, how do I think about what are the mundane things that are happening here? The one that I've been working with a lot recently is video. So text to video prompts, as I've mentioned, instead of the magical, you can do whatever you want all nice and fluffy Hollywood is dead, what is the mundane element of doing text to Video The mundane element of doing text to video is that when you train a model to create video from a text prompt, what it is doing is it's creating a number of successive frames. And each of those successive frames is going to be slightly different from the frame before. And you've trained a model by looking at video to say, well, if in frame 1, the person's hands like this and frame 2 it's like that, then you can predict it moves this way if there's a matching prompt. And suddenly it's become a little bit more mundane, but suddenly they begin to understand it. And then the people who are experts in that specific field, not the technical side of it, are now the ones that will actually be able to come up and do brilliant things with it. So that height navigation strategy-- filter actively, go deep on the fundamentals, get your slides to work. And then, of course, keep your finger on the pulse. The hardest part of that, I think, is the third one is really keeping your finger on the pulse. And that's when you have to wade into those cesspits of people just farming engagement and really try to figure out the signal from the noise there. But I think it's really important for you to be able to do that, to be connected, to understand that. Reading papers is all very good. The signal-to-noise ratio, I think, in reading papers is a lot better. But to understand the landscape that the people that you are advising, they are the ones who are waiting in the cesspools of Twitter and X and LinkedIn. And there's nothing wrong with those platforms in and of themselves, but the stuff that's posted on those platforms. So overall landscape, it is ripe with opportunity, absolutely ripe with opportunity. So I would encourage you, as Andrew did, to continue learning, to continue digging into what you can do and to continue building. But there are risks ahead. Anybody remember the movie Titanic? Remember the famous phrase in that, "iceberg right ahead"? But immediately before that, there's a scene in Titanic-- if we weren't being filmed, I would show it, but I can't for copyright reasons-- where the two guys up in the crow's nest are freezing and talking. And the crow's nest at the top of the ship is where the spotters would be to spot any icebergs in front. And go back and watch the movie again. You'll see the conversation between these two guys is that all they're talking about is how cold they are. And then it cuts away to the crew of the ship who are like, wait, aren't they supposed to have binoculars? And then the crew is like, oh, we left the binoculars behind in port. That framing the whole idea was like, they were so arrogant in being able to move forward that they didn't want to look out for any particular risks. And even though they had people whose job it was to look out for risks, they didn't properly equip or train them. And that, to me, is a really good metaphor for where the AI industry is today. There are risks in front of us. Those risks, the B word, the bubble word you're probably reading in the news is there, are there. To me, though, the opportunity and the things to think about in terms of a bubble are most of you probably don't remember dotcom bubble of the 2000s. But if you think about the dotcom bubble, that was the biggest bubble in history. It bursts, but we're still here. And the people who dotcom rights not only survived, they thrived. Amazon, Google, they did it right. They understood the fundamentals of what it was to build a dotcom. They understood the fundamentals of what it was to build a business on dotcom. And when the bubble of hype burst, they didn't go with it. There was one website, I believe it was pets.com, that they had the mindset of if you build it, they will come. They had Super Bowl commercials around pets.com. They couldn't handle the traffic that they got. And that was the kind of site that when the bubble burst, those were the sites that just evaporated. So that bubble in AI is likely coming. There is always a bubble. So the companies that are doing AI right are the ones, like I said, that won't just avoid the bubble that they will actually thrive post bubble. And the people who are doing AI right, the folks in this room who are thinking about AI and how you bring it to your company, and the advice that you're giving to your company and leaning into that in the right way will also be the ones who not only avoid getting laid off in the bubble crashes, but will be the ones who will thrive through and after the bubble. So anatomy of any bubble, and what I'm seeing in the AI one in particular, is this kind of pyramid. At the top is the hype that I've been talking about. At the bottom is massive VC investment. I'll be frank. I'm already seeing that drying up. Once upon a time, you could go out with anything that had AI written on it and get VC investment. Then you could go out and do anything with an LLM and get VC investment. Now there are far, far, far more cautious. I've been advising a lot of startups. The amount that they're getting invested is being scaled back. The stuff that's being invested in is changing. And the second layer down, massive VC investment is already beginning to vanish. Unrealistic valuations. Companies that aren't making money being valued massively high. We all who they are. We're beginning to see those unrealistic valuations being fed off of that hype. # #MeToo products, where somebody does something, and it's successful, and everybody jumps on the bandwagon. We're also seeing them everywhere. We saw them throughout the dotcom bubble. And then right at the bottom is that real value. I probably shouldn't have done the triangle like this. It should be more an upside down triangle. Because the real value here is small. But I've vibe coded these slides into existence. So this is one of the technical debt I took on. But the real value there, that kernel of value is there, and the ones that build for that will be the ones that survive. So the direction that I see the AI industry going in and the direction that I would encourage you to start thinking about your skills in, is really over the next five years, there's going to be a bifurcation. I'm just going to be ornery in how I describe it as big and small. Big AI will be what we see today, with the large language models getting bigger in the desire to drive towards AGI. The Geminis, the Claudes, the OpenAIs of the world are going to continue to drive bigger, and bigger is better in the mindset of those companies towards achieving AGI or towards achieving better business value. That's going to be one side of the branch. The other side of the branch is I'm going to call it small. We've all seen open-source models. I hate the term open source. Let me call them open weights or let me call them self-hostable models are becoming-- they're exploding onto the landscape. I read an article recently about Y Combinator that 80% of the companies in Y Combinator were using small models from China in particular. So the Chinese models in particular are doing really well, probably because of the overall landscape. They're not leaning into the large models the same way as the West is. I see that bifurcation happening. China, I think, has that head start on the small models that may last. It may not. I don't know. But the point is, we're heading in that particular direction of I'm going to call them instead of big and small now, models that are hosted on your behalf by somebody else, like a GPT or a Gemini or a Claude, or models that you can host yourself for your own needs. As this side has right now is underserved, this bubble may burst. This one right now is underserved. And this bubble will be later on. And the major skills that I can see developers needing over the next two to three years on this side of the fence will be fine tuning. So the ability to take an open-source model and fine-tune it for particular downstream tasks. Let me give one concrete example of that I've personally experienced. I work a lot in Hollywood, and I've worked a lot with studios making movies. And one studio in particular I was lucky enough to sell a movie to, it's still in preproduction. It'll probably be in preproduction forever. But one of the things I learned as part of that process was IP in studios is so protected. It's not even funny. Go in Google for James Cameron, who created Avatar and the lawsuits that he's involved in of this person who apparently sent him a story many years ago about blue aliens and is now suing him for billions of dollars because obviously there were blue aliens in Avatar. That level of IP protection in Hollywood is insane. The opportunity with large language models is equally insane. A lot of the focus is on large language models for creation, for storytelling, for rendering and all that, but actually the major opportunity that they have is actually for analysis to take a look at synopses of movies and find out what works and what doesn't. Why was this movie a hit and this one wasn't? What time of year was this one released and it became successful and this one wasn't? And with a margin on movies being razor thin, that kind of analysis is huge. But in order to do that kind of analysis, you need to share the details of your movie with a large language model. And they will absolutely not do that with GPT or Gemini or whatever, because they're now sharing their IP with a third party. Enter small models, where they can self-host their own small model and they are getting smarter and smarter. The 7B model of today is as smart as the 50B model of yesterday. A year from now, the 7B model of a year from now will be as smart as the 300B model of yesteryear. So they're moving in that direction of building using small self-hosted models, which they can then fine-tune on downstream tasks. Similar with other things where privacy is important law offices, medical offices, all of those kind of things. So those type of skills are fundamentally important going forward. So that's the bifurcation that I'm seeing happening in AI. The sooner bubble I think is in the bigger non self-hosted. The later bubble is in the smaller self-hosted. But either way, for you, for your career, to avoid the impact of any bubble bursting, focus on the fundamentals. Build those real solutions. Understand the business side, and most of all, diversify your skills. Don't be that one trick pony who only knows how to do one thing. I've worked with brilliant people who are fantastic at coding, in particular API, or particular framework. And then the industry moved on and they got left behind. OK, so yeah, when bubbles burst, that overall fallout kind of spoken about it a little bit already. Funding evaporates, hiring freezes become layoffs, projects get canceled, and talent floods the market. Yeah. Quick question from the last slide. I heard a lot about how NVIDIA is hiring, and they're very specific about they want people for very specific problem that they have. So they can require people to be basically put out that one thing that you're missing. So how do you think-- how is it more important to diversify skills versus actually focusing on, for example, LLMs versus computer vision or versus very specific downstream task? So I mean, I think so the question was around NVIDIA in particular or hiring for a very specific, very narrow scenario. So then the question is, how important is it for you to become an expert in a narrow scenario versus diversifying your skills? I would always argue it's still better to diversify your skills, because that one narrow scenario is only that one narrow scenario, and you're putting all your eggs into one basket. NVIDIA would be a fantastic company to work for. Nothing against them in any way. But if you put all of your eggs into that basket and you don't get it, then what? So I think the idea of really being able to-- if you are passionate about a thing, to be very deep in that thing is very, very good. But to only be able to do that thing, I think I would always encourage to be diversified. And when I say diversified, you're saying LLMs or computer vision or anything like that, I think I mean that's one part of it. But it's like that knowledge of models and how to use them to me is a uni skill. The diversification of skills is breaking outside of that. Also to be able to think, OK, what about building applications on top of these? What does scaling an application look like? What does software engineering in this case look like? What about user experience and user experience skills? Because it's all very well to build a beautiful application. But if nobody can use it-- I'm looking at here at Microsoft Office. There's stuff like that that's what I really mean about diversifying beyond. So even in that mono example with NVIDIA, to be able to break out of that one particular example, but to show skills in other areas that are of value, I think is really important. OK. As we're just running a little bit-- so yeah, I just wanted to-- I've gone into it a little bit already, but I'm a massive advocate for small AI. I really do believe small AI is the next big thing, because we're moving into a world, and this is part of the job that I do at Arm, is we're kind of moving into a world of AI everywhere all at once. So there's a traditional, and it's interesting you just brought up NVIDIA because there's a traditional conception that compute platforms are CPU plus GPU when it comes to AI. But that's also changing-- CPU general purpose, GPU specialists. But for example, in mobile space, there's massive innovation being done with the technology called SME, Scalable Matrix Extensions. And what SME is all about is really allowing you to bring AI workloads and put them on the CPU. The frontrunners in this are a couple of Chinese phone vendors, Vivo and Oppo, who've just recently released phones with SME-enabled chips. And what's magical about these is that, A, they don't need to have a separate external chip drawing extra power, taking up extra footprint space just to be able to run AI workloads. And B, the CPU, of course, being a low power pulling thing, being able to run AI workloads on that, they've been able to build interesting new scenarios. And if I talk about one in particular, there's a company called Alipay. And Alipay had an application where you would-- and we've all seen these apps where you can go through your photographs, and you can search for a particular thing. Places I ate sushi or something along those lines and use that to create a slideshow. All of those require a back end service. So your photographs are hosted on Google Photos or Apple Photos or something like that. And that back end service runs the model that you can search against it and be able to do the assembly of them. What Alipay wanted to do was like, say, there are three problems with this. Problem number one, privacy. You have to share your photos with a third party. Problem number two, latency. You got to upload those photos. You got to send the thing. You got to have the back end do the thing, and then you've got to download the results from the thing. And then number three is building that cloud service and standing that up cost time and money. So if they could move all of this onto the device itself, now the idea was they could run a model on the device that searches the photos on the device. You don't have the latency. And business perspective, they're now saving the money on creating this stand up service. They now have AI running on CPU in order to be able to do that. Apple are also people who've invested heavily in this scalable matrix extensions. You see whenever they talk about-- if you've ever watched a WWDC or anything like that, when they talk about the new A-series chips and M-series chips, about the neural cores and those kind of things in them, that's part of the idea. So to think about breaking that habit that we've gotten into, where you need a GPU to be able to do AI is part of the trend that the world is heading in. Apple are probably one of the leaders in that. I'm very, very bullish on Apple and Apple Intelligence as a result. And from the AI perspective, seeing that trend and following that vector to its logical conclusion as models are getting smaller embedded intelligence getting everywhere isn't a pipe dream. It isn't sci-fi anymore. It's going to be a reality that we'll be seeing very, very shortly. So that idea of that convergence of AI, because of the ability of smaller models getting smarter and lower power devices being able to run them, we see that convergence hitting, and I see massive opportunity there. So one last part and just going back to agents for a moment, I think the one thing that I always say is like a hidden part of artificial intelligence is really what I like to call artificial understanding. And when you can start using models to understand things on your behalf. And when they understand them on your behalf, to be able to craft from that understanding new things, you can actually develop superpowers where you're far more effective than ever before, be that creating code or creating other things. I'm going to give one quick demo just so we can wrap up. And I was talking earlier about generating video. So this picture is-- oops. Sorry. The connection here is not very good, I lost it. So here we go. This picture here is actually of my son playing ice hockey. And I took this picture, and I was saying, OK, I think I'm very good at prompting. And I wrote a nice prompt for this picture to get him. He's in the middle of taking a slapshot. He's got some beautiful flex on his stick. And I asked it like, OK, to prompt him scoring a goal. What do you think happened? Should we watch? Let's see if it works. This was the wrong video, but it still shows the same idea. Because of poor prompting or because of poor understanding of my intent, if I talk about it in agentic senses, the arena that he was in, which is a practice arena and doesn't have any people in it-- sorry. Let me pause it. If I just rewind to here, if we look up in this top right-hand corner here, this is basically where they store all their garbage. But the AI didn't know that, had no idea of it. So it assumed it was a full arena, and it started painting people in. And even though he shot a mile wide, everybody cheers. And somehow he has two sticks in his hand instead of one, and they forgot his name. So I did not go through an agentic workflow to do this. I did not go through the steps of, A, understand my intent. B, once you understand my intent, understand the tools that are available to you. In this case, it's Veo, and understand the intricacies of using Veo. Make a plan of how to use them. Make a plan of how to build a prompt for them, and then use them and then reflect. So I've been advising a startup that is working on movie creation using AI. And I want to show you a little sample here of a movie that we've been working on with them, where the whole idea is like, if you want to have performances at a virtual actors and actresses, you need to have emotion. You need to be able to convey that emotion, and you also need to be able to put that emotion in the context of the entire story. Because when you create a video from a prompt, you're creating an eight-second snippet. That eight-second snippet needs to know what's going on in the rest of the story. So if I show this one for a moment. And it's a little wooden at the moment, it's not really working perfectly. I have professional actors who are friends who are advising me on this, and they laughed at the performances. But try to view it through the difference that we had from an agentic prompt with the hockey player to this one. That's hopefully we can hear it. - I guess I can do the pub quiz after all. They just shut me down. I'm so close. But they wouldn't listen. - I won't-- They never listen. So here's the idea of, again, just thinking in terms of agentic, as I was saying earlier on, breaking it into those steps. That allowed me to use exactly the same engine, as I was showing you earlier on, that fails to be able to show something that works and is able to do things like portraying emotion that I just spoke about. So I know we're a little bit over time. So sorry about that. I can take any questions if anybody has any. I see Andrew is here as well. He's at the back. And I just really want to say thank you so much for your attention. I really appreciate it. Yep. How much of this new generation relation with the agentic use case is improved with the agentic workflow? And how much of it is a training set bias where you might have only pictures or videos with that are full of Yeah, it's a great question. Just to repeat for the video, how much of the improvement is from the use of an agentic workflow versus just lack of hockey stuff in the training set for the failed one? Not comparing like to, so just using my gut. When I looked at when I broke this down into the workflow that said, OK, I created scenes like this one and they were awful when I just did it directly for myself with no basis, no agentic, no artificial understanding. And when I broke it down into the steps where it's like, OK, in this scene, the girl is sitting on the bench, and she's upset. And the person is talking to her and he wants to comfort her. Feeding that to a large language model along with the entire story and along with the constraints that I had, where the shot has to be eight seconds long, clear dialogue and all of those kind of things, and then to understand my intent from that one, the LLM ended up expressing a prompt that was far more loquacious than I ever would have, that was far more descriptive than I ever would have. The LLM had understanding of what makes a good shot, what makes a good angle, what makes good emotion far more than I would have. I could spend hours trying to describe it. So that first step in the agentic flow of it doing that for me and understanding my intent was huge. The second step then is the tools that it's going to use. So I explicitly said which video engine I'm going to be using. I was using Gemini as the LLM, and hopefully Gemini is familiar with Veo, that kind of stuff, so to understand the idiosyncrasies of doing things with Veo. What I learned, for example, Veo was very bad at doing high-action scenes, but is very good at doing slow camera pulls to do emotion, as you saw in this case. So the LLM knew that from me, declaring I was using that as a tool. And then further it built a prompt and then further refined the prompt from that. And then the third part actually using the tool to actually generate it for me, generating a video with something like Veo costs, I think, between $2 and $3 to generate four videos and credits. So the last thing I want to do is generate lots and lots and lots and lots of videos and throw good money after bad. But all of that token spend that I did earlier on to understand my intent and then to make the plan for using the agent was saved in the back end where it got it right. Maybe not get it right first time, but it would very rarely take more than two or three tries to get something that was really, really nice. So I think without comparing like with like, I do think that plan of action and going through a workflow, that worked very, very well. Any other questions, thoughts, comments? Yeah, up at the back. What has surprised you the most about the AI industry over the years? What has surprised me the most about the AI industry over the years? Oh, that's a good one. I think what has surprised me the most, and it probably shouldn't have surprised me, is how much hype took over. I actually-- I honestly thought a lot of people who are in important decision making roles and that kind of thing would be able to see the signal better than they did. And I think the other part was that the desire to make immediate profits as opposed to long-term gains also surprised me a lot. Let me share one story in that space was one of the things that after Andrew and I taught that the TensorFlow specializations on Coursera, and after that, Google launched a professional certificate where the idea of this professional certificate was would give a rigorous exam. And at the end of the rigorous exam, if you got the certificate, it was a high prestige thing that would help you find work, and particularly at the time when TensorFlow was a very highly demanded skill in order to get work. Running that program cost Google $100,000 a year. Drop in the bucket, not a lot of money. The goodwill that came out of it was immense. I can tell you-- I'll tell one story very quickly, was a young man and he went public in some advertising stuff that with Google that he lived in Syria. And we all know there was a huge civil war in Syria over the last few years. And he got the TensorFlow certificate. He was one of the first in Syria to get it, and it lifted him out of poverty, where he was able to move to Germany and get work at a major German firm. And I met him at an event in Amsterdam where he told me his story. And now, because of the job that he had in this German firm, he's able to support his family back home and move them out of the war torn zone into a peaceful zone all because he got this AI thing. And there were countless stories like that. Very inspirational, very beautiful stories. But the thing that surprised me then was sometimes the lack of investment in that, where there was no revenue being generated for the company out of that. We deliberately kept it revenue neutral so that the price of the exams could go down. We wanted it to self-sustain. It ended up not being revenue neutral. It ended up costing the company about $100,000 to $150,000 a year. So they canned it. And it's a shame because of all the potential goodwill that can come out of something like that. But I think those were the two that immediately jump to mind that have surprised me the most. And then I guess one other part that I would say is the people who've been able to be very successful with AI, who you wouldn't think would be the ones that would be successful with AI, has always been inspirational to me. So allow me one more story. I have a good friend. I showed ice hockey a moment ago. I have a good friend who is a former professional ice hockey player. And any ice hockey fans Here It's a brutal sport. You see a lot of fighting and a lot of stuff on the ice. And he dropped out of school when he was 13 years old to focus on skating. And he will always tell everybody that he's the dumbest person alive because he's uneducated. He and I are complete opposites. That's why we get on so well. And he retired from ice hockey because of concussion issues. And he now runs a nonprofit-- the ice rinks for nonprofit. And about three years ago, we were having a beer, and he was like, so tell me about AI. And tell me about this ChatGPT thing. Is it any good? And I was like, just sharing the whole thing. Yes, it's good and all that kind of stuff. And it was obviously a loaded question, and I didn't know why. But part of his job at his nonprofit is that every quarter, he has to present to the board of directors the results of the operations so that they can be funded properly, because even though they're nonprofit, they still need money to operate. And he was spending upwards of $150,000 a year to bring in consultants to pull the data from all of the different sources. They're pulling data from-- there's machines in what's called the pump room that has a compressor that cools the ice. And there were spreadsheets and there was accounts and all this kind of stuff. And he was not tech savvy in any way. But he needed to process all this data. So he did an experiment where he got ChatGPT to do it. And this was the loaded question, asking me if it was any good. And so we talked through it a little bit. And then he told me why. And so I took a look at the results because he was uploading spreadsheets. He was uploading PDFs and all this kind of thing and getting it to assemble a report. And it takes him about two hours to do the report himself with ChatGPT. And it worked, and it worked brilliantly. And that $150,000 a year that he's saving on consulting is now going to underprivileged kids for hockey equipment, for ice skating equipment, for lessons, and all of that kind of thing. So it was taken out of the hands of an expensive consulting company and put into the hands of people. Because of this guy, and he says he's the dumbest person alive, but-- I hope he's not watching this video. And I told him afterwards that, congratulations, you're now a developer. And he didn't like that. But it's like surprises like that the superpowers that were handed to somebody like him, that he's not technical in any way, but he was able to effectively build a solution that saved his nonprofit $100,000 or $150,000 a year. And things like that are always surprising me in a very pleasant way. Yep. Sorry. I'll get to you next. Sorry. Yeah. For engineers like us, it's easier to navigate the hype because we can understand what the signal is from a research paper. But how about people who doesn't have this knowledge, like, from humanities or something ? Yeah, so just to repeat the question for the video. For engineers like us, sometimes it's easy to navigate the hype to see the signal from the noise. But what about people who don't have the same training as us? I think that's our opportunity to be trusted advisors for them and to really help them through that, to understand it. I think the biggest part in the hype story right now is just understanding the reward mechanism. That everything rewards engagement rather than actual substance. And to me, step one is seeing through that. The story I just told about my friend, he'd seen all this kind of stuff, but he wasn't willing to bet his career on it. But he needed that kind of advice around it and to start peeling apart what he had done and what he did right and what he did wrong. And so that positioning ourselves to be trusted advisors by not leaning into the same mistakes that the untrained people may be leaning into, I think is the key to that. And just understanding that the average person is generally very intelligent, even if they may not be experts in a specific domain, and to key in on that intelligence and help them to foster and to grow that in and navigate them through the parts where they'll have difficulty and let them shine in what they're very, very good at. Over here there was one. I have a question more for AI and machine learning for scientific research. OK. Which is something that is very hard to get your perspective on. Where do you think that is a good idea and where you might say, maybe be cautious? So AI and machine learning for scientific research, where is it a good idea and where should you be cautious? Ooh. My initial gut check would be I think it's always a good idea. I think there was no harm in using the tools that you have available to you, but to always to just double check your results and double check your expectations against the grounded reality. I've always been a fan of using automation in research as much as possible. My undergraduate was physics many, many years ago, and I was actually very successful in the lab because I usually automated things through a computer that other people did handwriting and pen and paper with. So I could move quickly. So I know I'm biased in that regard. But I would say, for most research, for the most part, I think use the most powerful tools you have available, but check your expectations. Little story actually on that side was trivia question. Poorest country in Western Europe. Anybody know? Serbia? What's that? Or Western. Western Europe is Wales. So I actually did my undergraduate in Wales, and I went back to do some lectures in the university there. And I met with a researcher there, and he was doing research into brain cancer using computer imagery and using various types of computer imagery. And I asked him, well, what's the biggest problem that you have? What's the biggest blocker for your research? And this is about eight years ago. And his answer was access to a GPU. And because for him to be able to train his models and run his models, he needed to be able to access a GPU. And the department that he was in had one GPU between 10 researchers, which meant that everybody got it for half a day. Monday through Friday, and his half a day was Tuesday afternoon. So in his case, he would spend the entire time that wasn't Tuesday afternoon preparing everything for his model run or his model training or everything like that. And then Tuesday afternoon, once he had access to the GPU, then he would do the training. And then he would hope in that time that he would train his model and he would get the results that he wanted. Otherwise, he'd have to wait a week to get access to the GPU again. And then I showed him Google Colab. Anybody ever used Google Colab? And you can have a GPU in the cloud for free with that kind of thing. And the poor guy's brain melted that-- because I took out my phone, and I showed him a notebook running on my phone in Google Colab and training it on that. And it changed everything for him research wise. And now it was a case of-- and this was with Colab. He had much more than he had with his shared GPU. So I think for someone like him, machine learning was an important part of his research, but he was so gated on it that the ability to widen access to that ended up really, really advancing his research. I don't know where it ended up. I don't know what he has done. It has been a few years since then. But that story just came to mind when you asked the question. Any more questions? Feel free to ask me anything. Oh, yeah. At the front here. It's more of a general question. You talked about AI helping food and beverage use. What do you think AI would be a force of social equality or social inequality? So can AI be a force of social equality or social inequality? I think the answer to that is yes. It can be both, and it can be neither. I mean, I think that ultimately, the idea is that if in my opinion, any tool can be used for any means, so the important thing is to educate and inspire people towards using things for the correct means. There's only so much governance can be applied. And sometimes governance can cause more problems than it solves. So I always love to live my life by assuming good intent but preparing for bad intent. And in the case of AI, I don't think there's any difference there that everything that I will do and everything that I would advise is assuming good intent, that people would use it for good things, but also to be prepared for it to be misused. The bad examples that I showed earlier on, I think were good intent rather than bad intent. And most mistakes that I see that are good intent being used mistakenly as opposed to bad intent. But I would say that's the only mantra that I can-- the only advice that I can give and that kind of thing is always assume good intent, but prepare for bad intent. The AI itself has no choice. It's how people use it. Andrew, did you want closing comments or-- I think we were running out time. But thank you for this. Really great. Thanks, everyone, for all the questions on those creative solutions. All right. Thank you, Andrew. Thanks.

[Andrew Ng]: 오늘은 AI 커리어 조언에 대해 여러분과 이야기해보려고 합니다. 예년에는 이 강의를 대부분 혼자 진행하곤 했는데, 올해는 제가 몇 가지 생각만 공유한 뒤 제 좋은 친구 Laurence Moroney에게 마이크를 넘기려고 합니다. Laurence는 이 강연을 위해 시애틀(Seattle)에서 샌프란시스코(San Francisco)까지 와 주셨고, 지금 채용 시장에서 무엇을 보고 있는지에 대한 폭넓은 시장 관점과 AI 커리어를 성장시키는 팁을 함께 나눠줄 겁니다.

저는 슬라이드 두 장과, Laurence에게 넘기기 전에 공유하고 싶은 생각 하나만 더 이야기하겠습니다. 저는 지금이 AI로 무언가를 만들고, AI 커리어를 구축하기에 ‘역대 최고의 기회’라고 느낍니다. 몇 달 전 소셜 미디어와 전통 미디어에서 ‘AI가 둔화되는 것 아니냐’는 질문들이 보였고, “GPT-5가 그렇게 좋냐?” 같은 말도 있었습니다. 제 생각엔 사실 꽤 좋습니다.

그런데도 ‘진전이 느려졌나?’라는 질문이 나온 이유 중 하나는, AI의 벤치마크를 ‘정답률 100% = 완벽한 답’으로 놓으면 빠르게 개선하더라도 결국 100% 위로는 갈 수 없기 때문입니다. 하지만 제 사고에 큰 영향을 준 연구가 하나 있는데, M-E-T-R, METR라는 조직의 작업입니다. 이들은 시간이 흐르면서 AI가 할 수 있는 과업의 복잡도가 어떻게 바뀌는지, ‘사람이 그 과업을 수행하는 데 걸리는 시간’으로 측정했습니다.

몇 년 전에는 GPT-2가 사람이 몇 초면 하는 일을 할 수 있는 정도였다고 치면, 이후에는 사람이 4초, 8초, 1분, 2분, 4분 걸리는 작업까지 점점 처리할 수 있게 됐습니다. 이 연구는 AI가 할 수 있는 과업 길이가 **7개월마다 두 배(doubling)**가 된다고 추정합니다. 이 지표로 보면, 사람이 어떤 일을 하는 데 걸리는 시간으로 정의한 ‘과업 복잡도’가 매우 빠르게 두 배가 되고 있으니, AI는 계속 진전할 거라는 쪽으로 저는 낙관적입니다.

같은 연구에서 더 작은 데이터셋이긴 하지만 ‘AI 코딩’은 두 배가 되는 시간이 더 짧아서 70일 정도일 수도 있다고 주장했습니다. 예를 들어 과거엔 제가 10분, 20분, 40분 걸려 쓰던 코드가 있었는데, AI가 점점 더 많은 부분을 해내는 식입니다.

그래서 제가 지금이 ‘골든 에이지’라고 보는 이유는 크게 두 가지 테마로 요약됩니다. 첫째, 더 강력하고 더 빠르다는 것입니다. 이 방 안의 여러분 모두는 이제 AI 빌딩 블록을 활용해, 불과 1년 전에는 지구상 누구도 만들기 어려웠던 수준의 강력한 소프트웨어를 만들 수 있습니다.

AI 빌딩 블록에는 대규모 언어 모델(LLM), 에이전틱(agentic) 워크플로, 음성 AI(voice AI), 그리고 물론 딥러닝(deep learning)이 포함됩니다. 흥미롭게도 많은 LLM이 딥러닝에 대해 ‘그럭저럭 기본 수준’의 이해를 하고 있습니다. 최전선(frontier) 모델에게 최첨단 신경망 구현을 시켜보세요. 예를 들어 transformer 네트워크 구현을 시켜보면, 생각보다 나쁘지 않게 도와줍니다.

이렇게 강력한 빌딩 블록이 1~2년 전엔 없었거나 매우 다루기 어려웠는데, 이제는 그것들로 누구도 못 했던 일을 해내는 소프트웨어를 만들 수 있습니다. 둘째, AI 코딩으로 소프트웨어를 작성하는 속도 자체가 그 어느 때보다 빨라졌습니다.

저는 개인적으로 도구의 최전선(frontier)을 따라가는 것이 중요하다고 느낍니다. AI 코딩 도구는 정말 빠르게 변합니다. 제 경우 몇 달 전부터 개인적으로 가장 좋아하는 도구가 Cloud Code가 되었고(이전 세대 도구에서 넘어왔습니다), 그리고 GPT-5 출시 이후에는 OpenAI Codex가 엄청난 진전을 이뤘다고 느낍니다. 또한 오늘 아침 Gemini 3가 출시됐는데, 저도 아직 오늘 아침이라 직접 만져볼 시간은 없었습니다만, 또 하나의 큰 도약처럼 보입니다.

그래서 제게 3개월마다 ‘가장 좋아하는 코딩 도구가 뭐냐’고 물으면, 아마 바뀔 겁니다. 최소한 6개월마다, 어쩌면 3개월마다 바뀔 수 있습니다. 이런 도구에서 반 세대(half a generation)만 뒤처져도 생산성이 솔직히 꽤 떨어집니다. “AI는 너무 빠르게 움직이고 다 변한다”는 말을 많이 하지만, AI 코딩 도구는 특히 그 속도가 정말 엄청납니다.

이처럼 더 강력한 소프트웨어를 더 빨리 만들 수 있게 되면서, 제가 예전보다 훨씬 더 강하게 강조하는 조언이 있습니다. 그냥 가서 무언가를 만드세요. 스탠퍼드(Stanford) 수업을 듣고, 온라인 코스를 듣고, 그리고 무엇보다도 뭔가를 직접 만들어보세요. Laurence가 ‘다른 사람에게 보여주기’도 이야기할 텐데, 지금은 그 기회가 이전보다 훨씬 큽니다.

다만 여기에는 좀 이상한 함의가 하나 있습니다. 점점 ‘명확히 쓰인 소프트웨어 스펙(spec)’에서 코드로 가는 것이 쉬워질수록, 병목(bottleneck)은 무엇을 만들지 결정하는 일로 옮겨갑니다. 혹은 실제로 무엇을 만들고 싶은지에 대한 그 ‘스펙’을 쓰는 일 자체가 병목이 됩니다.

제가 소프트웨어를 만들 때, 저는 루프를 돈다고 생각합니다. 코드를 쓰고, 유저 피드백(user feedback)을 얻고, 이것을 PM(product management) 작업이라고 봅니다. 피드백을 바탕으로 ‘유저가 좋아하는 것/싫어하는 것’, ‘UI가 너무 어렵다’, ‘이 기능은 원한다/저 기능은 원치 않는다’를 파악해 무엇을 만들지에 대한 관점을 바꾸고, 그 루프를 여러 번 반복해 유저가 사랑하는 제품으로 수렴하는 것입니다.

AI 코딩 덕분에 ‘빌드’ 자체는 훨씬 싸고 빨라졌습니다. 그런데 역설적으로 병목은 ‘무엇을 만들지’로 이동합니다. 그래서 실리콘밸리(Silicon Valley)와 많은 테크 회사에서 흔히 말하던 엔지니어 대 PM 비율(Eng to PM ratio)이 하락하는 추세를 봅니다. 예전엔 PM 1명이 제품 스펙을 써서 엔지니어 4:1, 7:1, 8:1 정도를 돌린다는 말이 있었는데(이런 숫자는 회사마다 제각각이라 대충 참고만 해야 합니다), 지금은 PM이 AI로 엔지니어만큼 빨리 빨라지진 않았기 때문에 비율이 내려가서 2:1 혹은 1:1에 가까운 팀도 봅니다.

또 하나는, 엔지니어가 제품을 정말 빨리 움직이게 할 수 있다면 ‘엔지니어’와 ‘PM’을 한 사람으로 합치는 방향도 보인다는 겁니다. 물론 엔지니어 중에는 유저와 이야기하고 공감(empathy)하는 일을 즐기지 않는 분들도 있습니다. 하지만 유저와 대화하고 피드백을 받고, 유저에 대한 깊은 공감을 바탕으로 무엇을 만들지 결정할 수 있는 엔지니어들은, 제가 지금 실리콘밸리에서 보는 사람들 중 가장 빠르게 움직이는 집단입니다.

저는 커리어 초기에, 한 역할에서 엔지니어들에게 ‘제품 일을 더 하라’고 설득하려다 실수한 적이 있습니다. 좋은 엔지니어들에게 ‘너희는 좋은 PM이 아니다’라는 식으로 느끼게 해서 상처를 줬고, 그건 제가 몇 년 동안 후회한 실수였습니다. 그래서는 안 됐습니다. 그런데 지금 제가 “엔지니어가 제품 일을 더 해보라”고 말하는 게, 어쩌면 같은 실수를 반복하는 것 같아 조금 찜찜하기도 합니다.

그럼에도 불구하고, 제가 코드를 쓰면서 동시에 유저와 대화해 ‘다음에 무엇을 할지’를 제품적으로 결정할 수 있으면 속도가 확 빨라집니다. 누군가가 제품을 고객에게 가져가는 것을 기다릴 필요가 없고, 그냥 코드를 쓰고, 다음에 뭘 해야 하는지 감(gut)을 가지고, 그 속도로 반복(iterate)할 수 있으니까 실행 속도(velocity of execution)가 훨씬 빨라집니다.

그리고 Laurence에게 넘기기 전 마지막으로 하나만 더. 커리어를 항해(navigate)할 때, 학습 속도와 성공 수준을 매우 강하게 예측하는 요인 중 하나는 ‘내가 어떤 사람들에 둘러싸여 있는가’입니다. 우리는 사회적 동물이고, 주변 사람에게서 배웁니다. 사회학(sociology) 연구에 따르면 가장 가까운 친구 5명이 흡연자면 내가 흡연자일 확률이 높다고 합니다. (제발 담배는 피우지 마세요. 그냥 예시입니다.)

반대로 “가까운 친구 5~10명이 정말 मेहन하게 일하고, 빠르게 배우고, AI로 세상을 더 낫게 만들려는 사람들”이면, 나도 그럴 확률이 높을 것 같다는 건 거의 확실하다고 봅니다. 주변 사람이 우리를 प्रेरस(inspire)합니다. 좋은 사람들과 함께 일하면 그 자체가 나를 앞으로 끌어줍니다.

스탠퍼드에서는 훌륭한 학생과 교수진이 있고, 또 하나의 강점은 ‘connective tissue’입니다. 최첨단 AI 랩(frontier labs)에서 일하는 많은 사람들은 스탠퍼드 여러 교수들의 제자였던 경우가 많습니다. 그 관계망 덕분에, 인터넷에 아직 공개되지 않은 최전선 지식(bleeding edge AI)을 우리는 관계와 우정으로 더 빨리 듣기도 합니다.

어떤 회사가 뭔가를 했을 때, 교수진의 친구가 다른 곳에 전화를 걸어 “그거 진짜 되냐?”고 확인하는 식입니다. 이런 촘촘한 관계망이 우리를 끌어올립니다. 저 개인적으로도, 어떤 기술 방향을 고민하다가 최전선 연구에 가까운 사람(스탠퍼드 연구자나 프런티어 랩 사람)과 통화 한두 번으로 몰랐던 사실을 듣고, 프로젝트의 기술 아키텍처(technical architecture)를 바꾼 적이 많습니다.

그래서 스탠퍼드에 있는 동안 친구를 만들고 이 connective tissue를 활용하세요. 관계망에서 얻는 작은 조언들—“그거 해봐”, “그건 하지 마”, “그건 PR이야 무시해”—이런 것들이 프로젝트 방향을 조향(steer)하는 데 큰 차이를 만듭니다.

스탠퍼드는 세계에 좋은 대학이 많지만, ‘leading AI groups’와 연결된 관계망의 풍부함 측면에서 지금 이 시점의 스탠퍼드는 정말 독특한 위치에 있다고 생각합니다. PR처럼 들리기 싫지만, 진짜로 그렇게 느낍니다.

그리고 취업을 할 때도, 회사 로고가 아니라 ‘매일 같이 일할 사람들’이 커리어 성공에 훨씬 더 중요합니다. 제가 이전에 수업에서 반복해 말한 이야기 하나가 있는데, 스탠퍼드 학생 한 명이 유명 AI 브랜드 회사에서 오퍼를 받았는데, 그 회사가 “어떤 팀에 배정될지는 말 못 한다. 로테이션/매칭 시스템이 있다. 먼저 계약서에 사인해라. 그다음에 프로젝트를 정하겠다”라고 했습니다.

그 학생은 부모님도 자랑스러워하고, 흥미로운 AI 프로젝트를 기대하며 입사했습니다. 그런데 사인한 뒤 배정된 곳은 회사의 백엔드 Java 결제(payment processing) 시스템이었습니다. Java 백엔드 결제를 하고 싶은 분들을 폄하하는 게 아닙니다. 훌륭한 일입니다. 다만 이 학생은 AI를 하려고 온 사람이었는데 AI 프로젝트가 아니었던 겁니다.

그는 약 1년 동안 매우 좌절했고, 결국 1년쯤 지나 회사를 떠났습니다. 더 안타까운 건, 제가 CS230에서 이 이야기를 이미 했는데도, 몇 년 뒤 또 다른 학생이 같은 회사에서 비슷한 경험을 했다는 것입니다(프로젝트는 결제가 아니었지만 ‘원하는 AI가 아닌’ 다른 일이었습니다).

그래서 “내가 실제로 매일 함께 일할 팀이 누구인지”를 최대한 확인하고, 나를 प्रेर는 사람들에 둘러싸이도록 하는 게 중요합니다. 회사가 끝까지 팀 배정을 말해주지 않는다면, 저는 솔직히 걱정이 됩니다.

가장 ‘핫한’ 로고의 회사보다, 로고는 덜 핫해도 정말 좋은 팀—똑똑하고 मेहन하게 일하고 AI로 좋은 일을 하려는 팀—을 찾으면 더 빨리 배우고 커리어가 더 잘 성장하는 경우가 많습니다. 우리는 로고의 흥분에서 배우는 게 아니라, 매일 같이 일하는 사람에게서 배웁니다.

제가 지금 주는 조언의 1순위는 이겁니다. 강력한 소프트웨어를 더 빠르게 만들기가 그 어느 때보다 쉬워졌습니다. 그러니 책임감 있게 하세요. 남을 해치는 소프트웨어는 만들지 마세요. 동시에, 여러분 각자가 만들 수 있는 것이 너무 많습니다.

세상에는 아이디어가 사람보다 훨씬 많고, 그것을 만들 수 있는 기술을 가진 사람은 부족합니다. 신입 구직이 어려워졌다는 얘기도 있지만, 동시에 많은 팀은 숙련된 사람을 충분히 찾지 못합니다. 어떤 프로젝트는 여러분이 만들지 않으면 아무도 안 만들 수도 있습니다.

그러니 남을 해치지만 않으면, 허락을 기다릴 필요가 없습니다. 누가 먼저 하길 기다릴 필요도 없습니다. 실패 비용이 낮아졌습니다. 주말 하나를 날리더라도 배우면 그걸로 충분히 괜찮습니다.

마지막으로, 어떤 곳에서는 ‘남에게 열심히 일하라’고 말하는 것이 정치적으로 올바르지 않다고 여겨지기도 합니다. 저는 그냥 말하겠습니다. 열심히 일하세요. 물론 어떤 사람은 삶의 국면상 열심히 일할 수 없는 시기가 있습니다. 저도 아이들이 태어난 직후 잠깐은 열심히 못 했습니다. 부상이나 장애 등 타당한 이유로 지금 열심히 못 하는 분들도 있습니다. 그분들을 존중하고 지원해야 합니다.

그럼에도, 제가 본 제 박사과정 학생들 중 성공한 사람은 모두 정말 열심히 했습니다. 새벽 2시에 하이퍼파라미터 튜닝(hyperparameter tuning)하는 것—저도 해봤고, 지금도 가끔 합니다. 만약 여러분이 열심히 일할 수 있는 처지라면, 지금은 기회가 너무 많습니다.

저처럼 주말과 밤에 코딩하고 만들고 유저 피드백을 받는 것이 재미있다면, 그렇게 ‘lean in’하면 성공 확률이 올라갑니다. 어떤 사람은 제가 열심히 일하라고 말해서 싫어할 수도 있겠지만, 솔직히 열심히 하는 사람은 더 많은 일을 해냅니다. 다만 열심히 못 하는 분들도 존중해야 합니다.

주말에 멍하니 TV를 보기 vs 주말에 에이전틱 코더(agentic coder)를 돌려서 뭔가를 시도하기—저는 거의 항상 후자를 고르겠습니다. (아이들과 같이 보는 쇼라면 전자도 합니다.)

이제 제 이야기는 여기까지 하고, 제 친구 Laurence Moroney에게 무대를 넘기겠습니다. Laurence는 오랜 기간 온라인 교육을 해왔고, TensorFlow와 PyTorch를 많은 사람에게 가르쳤습니다. 구글(Google)에서 오랫동안 lead AI advocate였고, 지금은 Arm에서 팀을 이끌고 있습니다. 저는 그의 책도 여러 권 읽었고, 최근에는 PyTorch 책(Introduction to PyTorch)도 냈는데 아주 훌륭합니다.

[Laurence Moroney]: 감사합니다. Andrew가 방금 말한 ‘같이 일할 사람을 고르는 것’의 중요성을 반대 방향에서도 강조하고 싶습니다. 회사가 여러분을 인터뷰할 때, 회사도 ‘함께 일할 사람’을 고르는 겁니다. 좋은 회사는 함께 일할 사람을 정말 신중히 고릅니다.

저는 지난 18개월 동안 특히 젊은 사람들 멘토링을 많이 했는데, 그중 한 청년의 이야기를 들려드리겠습니다. 그는 매우 교육을 잘 받았고 경험도 훌륭했고, ‘초엘리트 코더(super elite coder)’였습니다. 어떤 도전도 다 풀어낼 수 있는 사람이었는데, 4월에 직장에서 해고(layoff)됐습니다. 그는 의료 소프트웨어(medical software) 분야에 있었고, 그 업계는 크게 바뀌고 있었으며 연방정부(Federal government)에서 여러 영역의 자금이 줄며 해고를 겪었습니다.

그런데 그 청년은 해고 직전에 여자친구와 헤어졌고, 몇 주 뒤에는 반려견이 죽었습니다. 상태가 좋지 않았습니다. 몇 달 뒤 저는 그와 앉아 그의 구직 상황을 봤는데, 그는 지원한 직업을 스프레드시트로 추적하고 있었고, 무려 300개가 넘는 지원을 기록하고 있었습니다.

그중 일부는 Meta 같은 회사에서 면접 프로세스에 들어갔고, Microsoft도 있었고, 또 다른 대형 테크 회사도 있었습니다. 많은 면접 루프(loop)를 돌았고, 그는 매번 “나는 정말 잘했다. 코딩도 다 풀었다. 사람들과도 대화가 잘됐다”고 느꼈습니다. 그런데 매번 거의 하루 안에 리크루터가 전화해 “불합격”이라고 했습니다. 정말 가슴이 무너졌죠.

또 어디였냐면, Jeff Bezos의 회사—Amazon이 아니라, Bezos의 다른 회사—에서도 인터뷰를 봤다고 했습니다. 저는 그와 모의면접(mock interview)을 하며 조정(fine tuning)하기 시작했는데, 너무 좋은 후보인데도 뭐가 문제인지 바로는 안 보였습니다.

그러다 저는 일부러 ‘매우 سخت한 인터뷰’를 해보기로 했습니다. 어려운 LeetCode도 주고, 코드의 구석 케이스(corner case)도 들이대고, 위기 상황에서 어떻게 반응하는지 봤습니다. 그가 보인 반응은 채용 팸플릿에 적힌 조언을 그대로 따른 것이었습니다.

그런 팸플릿에는 “의견을 말할 기회가 있으면 버텨라(stand your ground). backbone을 가져라. 흔들리지 마라” 같은 말이 있습니다. 그는 그걸 “굉장히 공격적으로 버텨라”로 해석했던 겁니다. 제가 그의 코드에 구멍을 지적하고, 안 될 수 있는 케이스를 던지면, 그는 ‘등을 세우는’ 과정에서 면접 환경에서 적대적(hostile)으로 보였습니다.

인터뷰어 입장에서 보면, 그는 전형적인 ‘10x engineer’일 수 있지만, 그런 태도라면 저는 팀에 두고 싶지 않습니다. 우리는 그 부분을 함께 고쳤고, 조정했습니다. 이상한 건 그는 원래 정말 좋은 사람이었다는 겁니다. 다만 잘못된 조언을 그대로 따랐던 거죠.

그다음 그가 지원하던 회사는 ‘팀워크’를 매우 중시하는 곳이었는데, 좋은 소식은 그가 그 회사에 합격했다는 겁니다. 그는 해고되기 전 직장보다 연봉이 두 배(double)로 올랐고, 지금 돌아보면 ‘6개월의 fun employment’였다고 말합니다. 하지만 그 시기엔 정말 힘들었습니다.

그래서 여러분도, 여러분이 회사를 고르는 것처럼 회사도 여러분을 고른다는 점을 기억하세요. “버텨라”는 조언은 좋지만, 그 과정에서 ‘jerk’가 되지 마세요.

저는 Laurence입니다. 저는 ChatGPT가 ‘딸기(strawberry)에 노(oars)가 몇 개나 있는지’ 생각하는 것보다 더 많은 ‘수십 년’ 동안 테크 업계에서 일해왔습니다. Microsoft에서 오래 일했고, Google에서도 오래 일했고, Reuters 같은 곳에서도 일했고, 스타트업에서도 미국과 해외에서 일했습니다.

오늘 제가 이야기하고 싶은 건, 특히 AI에서 지금 커리어 지형이 어떻게 생겼는가입니다. Andrew가 말한 것처럼 스탠퍼드 네트워크와 프레스티지(prestige)를 활용할 수 있고, 저는 “가진 무기를 다 쓰라(use every weapon you have)”고 말하고 싶습니다. 왜냐하면 지금의 지형은 이상적이지 않습니다.

뉴스를 보면 대규모 테크 해고(massive tech layoffs), 채용 둔화, 이런 것들이 보입니다. 하지만 올바르게 접근하면 나쁜 일만은 아닙니다. 그래서 우선 ‘잡 마켓 리얼리티 체크’부터 하죠.

(강의실에) 여러분은 주니어인가요? 올해 졸업? 내년? 대체로 3학년/4학년?—그럼 이번 여름 졸업이겠네요. 지금 이미 구직 중인 분? 꽤 있네요. 성공한 분?… 거의 없고 한 분 있네요.

여러분도 체감하겠지만, 주니어(graduate level) 채용이 크게 둔화했고, 화제성 있는 해고가 헤드라인을 장악합니다. Google에서 제가 있었을 때도 사상 최대 해고가 있었고, Amazon, Microsoft 등에서도 해고를 봤습니다.

엔트리 레벨 포지션이 부족해 ‘느껴집니다(feels)’. 저는 ‘feels’에 밑줄을 치고 싶은데, 조금 뒤에 더 자세히 설명하겠습니다. 경쟁은 치열합니다. 그런데 걱정해야 하냐? 저는 ‘아니다’라고 말합니다. 올바른 방식으로 접근하면, 특히 AI 지형이 얼마나 빠르게 변하는지 이해하면, 올바른 마인드셋을 가진 사람은 잘 될 겁니다.

AI 채용 지형이 바뀌는 이유는 AI 산업 자체가 바뀌기 때문입니다. 저는 처음 AI에 1992년에 들어갔고, AI 윈터(AI winter) 직전에 잠깐 일하다가 크게 무너지는 걸 봤습니다. 그래도 AI에 ‘물렸다’가 아니라 ‘중독됐다’고 할 정도로 관심이 남았고, 2015년 Google이 TensorFlow를 런칭할 때 다시 깊게 들어와 AI 붐을 함께 봤습니다.

그런데 2021~2022년에는 글로벌 팬데믹(global pandemic)이 있었고, 산업 전반의 둔화가 왔습니다. 회사들은 매출과 ‘직접 매출을 만드는 일’로 피벗(pivot)해야 했습니다. Google에서 TensorFlow는 오픈소스(open-source)였고 직접 매출을 만들지 않았기 때문에 축소가 시작됐고, 전 세계 많은 회사가 채용을 줄였습니다.

그 다음 2022~2023년으로 가면, 팬데믹에서 나오며 ‘채용을 안 했던 누적(logjam)’이 있었고, 동시에 AI가 폭발적으로 부상했습니다. 세상이 AI-first로 바뀌며 모든 회사가 미친 듯이 채용해야 했습니다.

그 결과 2022~2023년에는 ‘오버하이어링(overhiring)’이 발생했고, 자격이 충분치 않은 사람이 더 높은 포지션을 받기도 했습니다. AI가 이력서에 있다는 이유만으로 채용되는 일도 있었고, 인재 쟁탈전(talent grabs)과 ‘일단 데려와서 나중에 뭘 할지 생각하자’ 같은 일이 많았습니다.

그리고 2024~2025년은 ‘대각성(the great wake-up)’입니다. 많은 회사가 오버하이어링을 깨닫고, 업무에 비해 덜 준비된 사람들(underqualified)이 많다는 걸 알게 됩니다. 이제 회사들은 AI 스킬에 훨씬 조심스럽습니다. 하지만 이 조정 국면에서도, 전략적으로 접근하면 기회는 여전히 크고 많습니다.

그래서 오늘은 여러분이 그걸 어떻게 하느냐를 이야기하려 합니다. 제가 보는 성공의 3가지 기둥(pillars)이 있습니다. 지금은 이력서에 ‘AI’만 써서 뽑히는 시대가 아니고, 이 3가지 기둥을 ‘말’로만이 아니라 ‘보여줘(show)’야 합니다.

하지만 보여주기는 오히려 지금이 역사상 가장 쉬운 때입니다. Andrew가 말한 것처럼 ‘프롬프트로 코드를 만들어내는(prompt things into existence)’ 능력이 있습니다. Andrew는 ‘vibe code’라는 말은 싫어하지만, 어쨌든 프롬프트로 구현을 빨리 만들어 보여줄 수 있습니다.

Andrew가 엔지니어와 PM 이야기를 했는데, 저도 비슷한 경험이 있습니다. 저는 Google에 두 번 인터뷰했다가 두 번 떨어졌습니다. Microsoft에서 성공했고 책도 20권+ 썼고 대학 강의도 했는데도요. 두 번은 PM 직군으로 봤는데 떨어졌고, 엔지니어로 봤더니 “왜 진작 안 왔냐”고 하며 뽑았습니다.

그만큼 ‘좋은 엔지니어’가 중요합니다. 그리고 엔지니어 대 PM 비율이 바뀌는 시대엔 엔지니어링 스킬이 더 값집니다.

세 기둥 중 첫째는 ‘깊이 있는 이해(understanding in depth)’입니다. 두 가지 의미가 있습니다. 하나는 학문적으로, 머신러닝(machine learning)과 모델 아키텍처(architecture)를 깊이 이해해 논문(papers)을 읽고 이해하고, 그것을 실제로 적용하는 능력입니다.

다른 하나는 트렌드(trends)의 ‘맥을 짚는’ 능력입니다. 신호 대비 잡음(signal-to-noise)에서 신호가 우세한 곳이 어디인지 감을 갖는 것입니다.

둘째는 ‘비즈니스 포커스(business focus)’입니다. Andrew가 아까 정치적으로 올바르지 않을 수도 있는 말을 했는데, 저도 비슷하게 하나 하겠습니다. ‘하드워크(hard work)’는 모호한 말이니, 저는 “사람은 측정하는 대로 된다(you are what you measure)”고 봅니다.

요즘 996(오전 9시~오후 9시, 주 6일) 같은 게 열심히 일하는 척도로 나오지만, 그건 ‘시간’의 척도이지 ‘성과’의 척도는 아닙니다. 하루 8시간이나 6시간만 일해도 매우 생산적일 수 있습니다. 저는 ‘투입 시간’이 아니라 ‘산출(output)’로 측정합니다.

저는 책을 많이 썼는데, Andrew가 들었던 그 책은 약 2개월 만에 썼습니다. 사람들이 “너는 16시간씩 일하냐?”고 묻지만, 제 비결은 야구(baseball)입니다. 야구는 TV로 보면 3.5~4시간 걸리는 느린 경기라서, 저는 야구를 배경으로 틀어놓고 글을 씁니다. 그게 제 ‘하드워크’의 방식이었습니다.

그래서 여러분도 ‘열심히’의 정의를 산출로 두고, 그 산출을 내가 원하는 비즈니스 방향과 정렬(alignment)시키라고 말하고 싶습니다. “지금 하는 일에 맞춰 산출하지 말고, 내가 하고 싶은 일에 맞춰 산출하라.”

제가 Google에 세 번째 도전에서 들어갔을 때도 같은 원리였습니다. 당시 Cloud 팀 인터뷰를 보면서, 저는 Java 책을 막 썼고, Google Cloud에서 Java로 할 수 있는 걸 보여주고자 했습니다. 그래서 기술적 분석(technical analytics)으로 주가 예측(stock price prediction)하는 Java 앱을 Cloud에서 돌리는 코드를 만들었습니다.

그 결과 면접 루프가 “버스에 골프공 몇 개 들어가냐” 같은 질문이 아니라, 제 코드에 대해 묻는 시간이 됐습니다. 주도권이 제게 왔고, 제가 아는 것에 대해 이야기할 수 있었습니다. AI에서도 마찬가지입니다. 내가 원하는 포지션에 맞춰 제품/코드 산출을 만들어 보여주고, 그 산출이 회사 비즈니스와 어떻게 연결되는지 설명할 수 있어야 합니다.

셋째 기둥은 ‘딜리버리 편향(bias towards delivery)’입니다. “아이디어는 싸고, 실행이 전부(Ideas are cheap, execution is everything).” 저는 면접에서 아이디어만 푹신한 사람도 많이 봤고, 반대로 덜 멋진 아이디어지만 아주 잘 그라운딩(grounding)한 사람도 봤습니다. 합격은 보통 후자였습니다.

이제 ‘지금 AI에서 일하는 현실’로 가보죠. 불과 2~3년 전만 해도 “이미지 분류기(image classifier) 하나 만들 수 있으면” 엄청 대우받고 6자리 연봉과 주식을 받곤 했습니다. 지금은 그렇지 않습니다. 지금은 P 단어, 즉 ‘프로덕션(production)’입니다.

새 모델을 만들든, 모델을 최적화하든, 유저를 이해하든, UX가 매우 중요하든, 모든 것이 프로덕션으로 수렴합니다. 회사들은 바텀라인(bottom line)을 최적화하고 있고, 저는 “바텀라인의 바텀라인은 바텀라인이다”라는 말을 합니다.

그래서 현실 1: 비즈니스 포커스는 이제 협상 불가(non-negotiable)입니다. 여기서 제가 또 정치적으로 올바르지 않을 수도 있는 말을 하겠습니다. 지난 10년간 대기업, 특히 실리콘밸리에서는 사람의 ‘전인적 자기(whole self)’를 직장으로 가져오게 하는 문화가 강했고, 그 과정에서 다양한 사회 이슈 활동(activism)이 회사 안으로 들어왔습니다.

저는 활동 자체가 나쁘다고 말하는 게 아닙니다. 정의(justice) 관련 원인을 지지하는 것도 전혀 문제 없습니다. 하지만 제 경험상 ‘과도한 비중(overindexing)’은 비즈니스보다 활동을 우선하게 만들고, 회사가 거기에 발목 잡히는 경우가 있었습니다.

예를 들어 약 2년 전 Google에서 어떤 활동가들이 Google Cloud 책임자의 사무실에 침입해 sit-in을 하고, 그의 책상 위에 화장실을 보고… 같은 일이 있었습니다. 이건 활동이 선을 넘은 사례입니다. 그 결과 좋은 의도까지도 함께 손상되며, 해고가 늘고, 활동이 억제되고, 비즈니스 포커스가 더 강하게 비협상 요소가 되었습니다. 지금은 진자(pendulum)가 반대편으로 크게 흔들리는 중입니다.

현실 2: 리스크 완화(risk mitigation)가 업무의 일부입니다. 특히 AI에서는 더 중요합니다. 어떤 비즈니스 프로세스를 ‘휴리스틱(heuristic) 컴퓨팅’에서 ‘지능형(intelligent) 컴퓨팅’으로 전환할 때의 리스크를 이해하고, 어떻게 완화할지 설명하는 마인드셋은 면접에서도 매우 강력한 스킬입니다.

현실 3: 책임(responsibility)의 정의가 진화하고 있습니다. 예전에는 “모두를 위해 AI가 작동해야 한다”가 먼저였고, 그 다음이 “비즈니스에 도움이 돼야 한다”인 경우도 있었는데, 지금은 “AI가 작동하고, 비즈니스를 드라이브하고, 그 다음 모두를 위해 작동해야 한다”로 우선순위가 재정렬되는 흐름이 있습니다. 과거의 뒤집힌 우선순위가 유명한 실패를 낳기도 했습니다.

제가 예로 하나 보여드리겠습니다. (창을 넘기며) 텍스트-투-이미지(text-to-image)로 유명한 Gemini에서 몇 년 전 있었던 일입니다. 저는 responsible AI를 많이 다뤘는데, 여기서 중요한 건 대표성(representativeness)과 편향(bias) 문제입니다.

예를 들어 “의사(doctors)” 이미지를 생성하면 학습셋에 남성이 많으면 남성 의사가 더 나오기 쉽고, “간호사(nurses)”는 여성이 많으면 여성 간호사가 나오기 쉽습니다. 이는 부정적 고정관념을 강화할 수 있습니다.

그래서 저는 Gemini가 이런 편향을 어떻게 상쇄하려는지 테스트해보려 했습니다. 저는 “여름 드레스를 입고 밀짚모자를 쓴 젊은 Asian 여성이 옥수수밭(cornfield)에서 iPhone을 집중해서 바라본다” 같은 프롬프트를 넣었고, 아름다운 이미지가 나왔습니다.

그 다음 “Indian”으로 바꾸니 인도 여성 이미지가 잘 나왔고, “Black”으로 바꾸니 왜인지 3장만 나왔지만 프롬프트를 따랐습니다. “Latina”도 4장이 나왔고, 한 장은 Hermione Granger처럼 보이기도 했지만 전반적으로 괜찮았습니다.

그런데 “Caucasian”을 넣자 무슨 일이 벌어졌을까요? “요청은 이해하지만, 사람 이미지를 생성하면 해로운 고정관념/편향을 낳을 수 있어 생성할 수 없다”라는 메시지가 나왔습니다. 매우 조악한 안전 필터(safety filter)였습니다. 방금 전까지 사람 이미지를 생성해놓고, ‘Caucasian’ 같은 단어를 트리거로 막아버린 겁니다.

저는 “Caucasian” 대신 “white”로 바꿔도 같은 식으로 거부당했고, 심지어 “지금은 사람 이미지를 만들지 않는다” 같은 말로 제 앞에서 거짓말까지 했습니다(방금 만들었으면서요).

그럼 제가 어떻게 우회했을까요? 저는 “Irish woman”을 요청했습니다. 그러자 문제 없이 ‘아일랜드 여성’ 이미지가 생성됐습니다. 그런데 모든 이미지에서 빨간 머리(red hair)였습니다. 저는 아일랜드에서 자랐고 아일랜드는 빨간 머리 비율이 세계에서 가장 높은 편이지만, 그래도 약 8% 정도입니다. 그런데 ‘아일랜드=빨간 머리’로 고정해 그리는 건 또 다른 문제를 낳을 수 있습니다.

중국의 어떤 지역에서는 ‘악마(demon)’를 빨간 머리로 묘사하기도 한다고 저는 들었습니다. 그러니까 여기서 책임 있는 AI가 어떤 관점에서는 다양성을 확보하려다, 다른 관점에서는 새로운 편향과 모욕 경계선의 문제를 만들었습니다. 안전 필터를 만든 사람들은 이 문제를 떠올리지 못했던 겁니다.

제가 말하는 “책임의 진화”는 이런 방향입니다. 책임은 이제 사회적 의제를 넘어서 ‘비즈니스 평판(reputation)을 손상시키지 않게 하는 것’ 같은 더 하드한 제약과도 결합합니다.

(질문) 누군가 “역사적 맥락에서 인종/민족이 섞여 나오는 문제도 있었다고 들었다”고 말하자, 저는 “맞다. 예를 들어 ‘사무라이(samurai)’를 그리면 다양성 편향을 맞추려다 프롬프트를 가로채서 ‘다양한 배경의 사무라이’가 나오고, 남녀 사무라이, 여러 인종 사무라이가 섞이며 역사적 사실성과 충돌하는 문제가 있었다”는 취지로 답하겠습니다. 당시엔 프롬프트를 인터셉트(intercept)해 공정성(fairness)을 높이려는 매우 순진한(naive) 해결책이 들어가 있었고, 그로 인해 평판이 손상됐습니다. 지금은 많이 개선됐지만, 그 사례가 교훈입니다.

이제 ‘vibe coding’ 이야기로 가보겠습니다. 밈(meme)으로는 “코드 생성이 엔지니어를 쓸모없게 만든다”는 말이 돌지만, 저는 그게 전체 진실은 아니라고 봅니다. 숙련된 엔지니어일수록 이런 도구를 더 잘 씁니다.

저는 여러분이 ‘trusted advisor(신뢰받는 조언자)’ 관점으로 생각하길 권합니다. 면접이든 컨설팅이든, 회사를 상대로 “이게 어떤 함의를 갖는가”를 설명할 수 있어야 합니다. 생성 코드의 함의를 이해하는 가장 좋은 프레임은 ‘기술 부채(technical debt)’입니다.

(질문) 기술 부채라는 표현을 아냐고 물었더니, 대부분 모른다고 해서 잠깐 설명하겠습니다. 부채(debt)를 생각해봅시다. 집을 사면서 50만 달러를 30년 모기지로 빌리면 이자 포함 상환액이 대략 두 배가 되어 100만 달러를 갚게 됩니다. 하지만 그 부채는 집 가치 상승, 렌트 회피 등으로 정당화될 수 있으니 ‘좋은 부채’일 수 있습니다.

반대로 신용카드 고금리로 충동구매를 하면, 200달러짜리 신발이 상환할 때 500달러가 될 수도 있는데, 그 가치는 보통 그만큼 나오지 않으니 ‘나쁜 부채’입니다.

소프트웨어도 똑같습니다. 무엇을 만들든 부채가 생깁니다. 버그, 지원, 새로운 요구사항, 마케팅, 피드백 대응… 이런 것들이 모두 부채입니다. 부채를 피하는 유일한 방법은 아무것도 하지 않는 것입니다.

그래서 질문은 “vibe coding으로 무언가를 만들 때, 내가 떠안는 기술 부채가 그만한 가치가 있느냐”입니다. 부채는 버그 수정, 유지보수 설득, 문서화, 기능 추가 등 ‘나중에 갚아야 하는 추가 작업’입니다. 소프트 부채(soft debt)와 하드 부채(hard debt)도 있습니다.

저는 스타트업들과 일할 때, 많은 팀이 “Replit이나 GPT나 Anthropic을 열고 빨리 프로토타입을 만들자, 투자자를 만나자”로 달려가는 걸 봅니다. 물론 가능하지만, debt, debt, debt가 항상 따라옵니다. 좋은 금융인은 부채를 잘 관리해 부자가 되고, 좋은 코더는 기술 부채를 잘 관리해 부자가 된다고 저는 말합니다.

그럼 ‘좋은 기술 부채(모기지)’는 어떻게 만들고, ‘나쁜 기술 부채(고금리 신용카드)’는 어떻게 피하나? 첫째, 목표(objectives)가 명확해야 합니다. 무작정 ChatGPT를 켜고 코드를 뽑기 전에 무엇을 만들지 분명해야 합니다.

둘째, 비즈니스 가치(business value)를 전달했는가입니다. 멋진 웹사이트를 몇 시간 동안 만들었는데 “그래서 뭐?”가 되면 부채만 남습니다.

셋째, 인간의 이해(human understanding)입니다. 아무도 이해 못 하는 코드가 최악의 부채입니다. 나만 이해하고 나중에 더 좋은 직장으로 떠나면, 회사는 그 코드에 종속됩니다. 그래서 문서화, 명확한 알고리즘, 변수명 정리 같은 기본이 중요합니다.

또 다른 나쁜 부채는 ‘문제 없는 해법(해법이 먼저인)’입니다. “망치만 있으면 모든 게 못으로 보인다.” 대기업에서 사람들이 막 만들어 체크인(check-in)해버리면 좋은 코드와 나쁜 코드가 섞여 스파게티(spaghetti) 코드가 됩니다.

제가 지금 겪는 사례로는 macOS 앱을 SwiftUI로 만들 때, 학습셋에는 iOS 코드가 훨씬 많아서, 모델이 계속 iOS API를 제안하는 문제가 있습니다. 프롬프트로 고치려다 보면 점점 스파게티가 되고, 결국 수동으로 많이 고쳐야 합니다.

그리고 ‘권위가 실력보다 우선(authority over merit)’도 나쁜 부채입니다. 어떤 VP가 신용카드로 Replit을 결제해 뭔가를 만들어놓으면, 그걸 고치는 건 누굽니까? 팀입니다.

이제 ‘하이프(hype) 사이클’로 가보죠. 저는 하이프가 우주에서 가장 강한 힘 중 하나라고 생각합니다. 특히 AI와 크립토(crypto) 같은 뜨거운 분야는 nonsense가 넘칩니다. 소셜 미디어의 화폐(currency)는 정확성(accuracy)이 아니라 참여(engagement)입니다. LinkedIn조차도 ‘영향력자(influencers)’가 Gemini나 GPT로 쓴 글로 참여를 농사짓는(farming engagement) 곳이 되어가고, 플랫폼은 그런 글을 보상하도록 설계되어 있습니다.

여러분이 신호와 잡음을 걸러내고, 주변을 신호 쪽으로 이끄는 사람이라면 큰 경쟁력이 됩니다. 좋아요/댓글 같은 빠른 피드백은 아닐지 몰라도, 면접이나 일에서 신호를 가져오면 엄청난 가치가 됩니다.

제가 하이프 사례를 하나 들겠습니다. 작년에 ‘에이전트(agent)’가 유행하자, 유럽의 한 회사가 “에이전트 구현을 도와달라”고 했습니다. 그때 제가 던진 가장 근본 질문은 “왜(why)?”였습니다.

CEO는 “다들 에이전트로 비용 절감하고 비즈니스가 좋아진다더라”라고 했고, 출처는 LinkedIn과 Twitter 같은 곳이었습니다. 저는 계속 질문을 파고들어, 결국 그가 원하는 건 ‘세일즈팀 효율을 높이는 것’이라는 본질로 도달했습니다.

“세일즈팀 효율을 높인다”는 문장에는 AI도 에이전트도 없습니다. 그러니 신뢰받는 조언자로서, 저는 ‘AI shill’이 아니라 ‘효율을 높이는’ 문제를 풀어야 합니다. 세일즈는 미팅 전 숙제(리서치)가 많고, 회사 웹사이트/LinkedIn을 뒤지는 데 시간이 낭비됩니다. 웹사이트 구조는 제각각이라 인지 부하(cognitive load)가 큽니다.

그 회사의 세일즈는 80% 시간을 리서치에, **20%**를 실제 판매(selling)에 쓰고 있었고, 커미션이 중요한 세일즈에게는 비효율이 컸습니다. 그래서 목표를 “20% 효율 개선”으로 두고, 그 다음에 AI와 에이전틱 AI를 도입했습니다.

그럼 AI와 에이전틱 AI의 차이는 무엇일까요? 저는 에이전틱 AI를 네 단계로 봅니다. 1) 의도(intent) 이해(LLM이 강함), 2) 계획(planning: 사용할 툴을 선언하고 단계로 쪼갬), 3) 실행(execution: 툴을 사용), 4) 성찰(reflect: 의도를 달성했는지 확인, 아니면 루프로 복귀).

이 프레임으로 파일럿을 만들었더니, 낭비 시간을 10~15% 절감했습니다. 그리고 의도치 않은 결과(unintended consequences)로 세일즈가 더 행복해졌습니다. 평균 세일즈가 몇 %p 더 팔고, 돈을 더 벌고, 일이 덜 고통스러워진 겁니다. 몇 시간 걸리던 리서치를 몇 분으로 줄인 브리프(brief)가 영업을 돕고, 모두에게 win-win-win이었습니다.

반대로 하이프에 끌려 “에이전트 만들어!”만 외치면, 스코핑(scoping)이 안 되어 실패합니다. 저는 “McKinsey가 지난주에 AI 프로젝트의 **85%**가 실패한다고 보고했다”는 이야기도 들었습니다. 주요 이유는 잘 스코프되지 않고 하이프 밴드왜건(bandwagon)에 올라타기 때문입니다.

최근 하이프 예시는 “소프트웨어 엔지니어링은 끝났다”, “할리우드(Hollywood)는 끝났다”, “연말까지 AGI” 같은 것들입니다. 저는 사우디아라비아(Saudi Arabia)에서 FYI라는 행사 디너에서 한 생성형 AI 회사 CEO가 “내년 이맘때면 프롬프트 하나로 90분 영상을 만들 수 있다. 그러면 할리우드 bye-bye”라고 말하는 걸 들었습니다.

하지만 2년이 지난 지금도 프롬프트 하나로 90분 영상을 만들진 못합니다. 설령 가능해도, 영화 전체 스토리를 담는 프롬프트는 무엇이냐는 문제가 남습니다. 이런 하이프는 관심을 끌지만, 여러분은 ‘왜(why)’를 묻고 신호를 찾아야 합니다.

그래서 하이프 항해 전략은: 트렌드를 객관적으로 평가하고, 진짜 기회를 보며, 유행성 방해물(fashionable distractions)은 무시하고, 리더십에 기술적 현실(technical reality)을 설명하는 능력을 갖추는 것입니다. 제가 들은 흥미로운 코칭 중 하나는 “가능한 한 평범하게(mundane) 설명하라”였습니다. 평범하게 만들수록 디테일이 보이고, 비기술 분야 전문가도 그 위에서 창의적으로 움직일 수 있습니다.

이제 전반 지형은 기회가 많지만, 리스크도 있습니다. 저는 타이타닉(Titanic)의 “빙산이 вперед(iceberg right ahead)” 장면을 떠올립니다. 그 전에 망루(crow’s nest) 두 사람이 추워서 그 얘기만 하고 쌍안경(binoculars)은 항구에 두고 와 버렸다는 장면이 있는데, 저는 그게 AI 산업의 은유라고 봅니다. 리스크를 보는 사람과 장비가 필요합니다.

AI 버블(bubble) 이야기가 나오는데, 버블은 늘 있었습니다. 닷컴 버블(dotcom bubble)도 터졌지만 Amazon, Google 같은 ‘제대로 한 회사’는 살아남아 오히려 번성했습니다. pets.com 같은 “만들면 사람들이 온다” 마인드로 슈퍼볼 광고를 했지만 트래픽을 감당 못 하던 곳들은 사라졌습니다.

AI 버블의 구조는 위에 하이프, 그 아래 대규모 VC 투자(massive VC investment), 비현실적 밸류에이션(unrealistic valuations), #MeToo 제품(남이 하니 나도) 같은 층이 있고, 맨 아래 아주 작은 ‘진짜 가치(real value)’가 있습니다. 저는 이미 VC 투자 분위기가 더 조심스러워지고 있다고 봅니다.

향후 5년을 보면, 저는 AI가 크게 두 갈래로 갈라질(bifurcation) 거라 봅니다. 하나는 ‘빅 AI’—더 큰 모델로 AGI와 더 큰 가치로 가려는 GPT/Gemini/Claude 같은 hosted 모델입니다. 다른 하나는 ‘스몰 AI’—오픈소스(open source)라는 말보다는 ‘오픈 웨이트(open weights)’ 혹은 ‘셀프 호스팅(self-hostable) 모델’이라 부르고 싶은 모델들입니다.

저는 Y Combinator에서 80% 회사가 중국발 작은 모델을 쓴다는 글을 읽었다고 했습니다. 중국은 서구처럼 초대형 모델에만 올인하지 않아 작은 모델에서 앞서갈 수도 있습니다. 어쨌든 앞으로는 “누군가가 대신 호스팅해주는 모델”과 “내가 직접 호스팅하는 모델”로 갈라질 가능성이 큽니다.

이 ‘셀프 호스팅’ 쪽은 아직 덜 서비스되는(underserved) 영역이라 기회가 큽니다. 그리고 이 영역에서 중요한 스킬은 파인튜닝(fine-tuning)입니다. 오픈 웨이트 모델을 특정 다운스트림 과업(downstream task)에 맞게 파인튜닝하는 능력입니다.

제가 직접 겪은 예로, 저는 할리우드 스튜디오들과도 일하는데, 스튜디오는 지식재산(IP)을 극단적으로 보호합니다. James Cameron의 Avatar와 관련한 소송 이야기도 있습니다. 그만큼 IP 민감도가 높아, 스튜디오는 GPT/Gemini 같은 서드파티에 시놉시스나 스토리를 넘기려 하지 않습니다.

그런데 LLM의 큰 기회는 ‘창작’만이 아니라 ‘분석(analysis)’입니다. 어떤 영화가 왜 흥행했고, 어떤 영화는 왜 실패했는지, 개봉 시기와 성과의 관계 등, 마진이 얇은 영화 산업에서 이런 분석은 큰 가치입니다. 이를 위해서는 영화 내용을 모델에 넣어야 하니, 셀프 호스팅 작은 모델이 유리합니다.

작은 모델은 계속 똑똑해지고 있습니다. “오늘의 7B 모델이 어제의 50B만큼 똑똑하다” 같은 식으로, 1년 뒤 7B는 과거의 300B 수준에 가까워질 것이라는 방향성 이야기도 있습니다. 법률/의료 등 프라이버시가 중요한 곳도 마찬가지입니다.

그래서 저는 버블이 터져도 살아남으려면: фундаментals에 집중하고, 진짜 솔루션을 만들고, 비즈니스를 이해하고, 무엇보다 스킬을 다변화(diversify)하라고 말합니다. 특정 API나 프레임워크만 하는 ‘원트릭 포니(one trick pony)’는 업계가 바뀌면 뒤처질 수 있습니다.

(질문) NVIDIA가 특정 문제에 딱 맞는 사람을 뽑는다는데, ‘특정 분야 깊이’ vs ‘다변화’ 중 무엇이 중요하냐는 질문에 대해 저는, 특정 분야에 깊어지는 건 좋지만 “그것만 할 수 있는 상태”는 위험하다고 답했습니다. 열정이 있는 분야는 깊게 파되, 앱을 얹는 능력, 스케일링, 소프트웨어 엔지니어링, UX 등 ‘모델 지식 밖’의 스킬로 확장해야 합니다.

저는 또한 ‘스몰 AI’가 다음 큰 흐름이라고 믿습니다. Arm에서 하는 일도 그와 연결됩니다. 우리는 ‘AI everywhere all at once’로 가고 있고, AI=CPU+GPU라는 고정관념도 바뀌고 있습니다.

모바일에서는 SME(Scalable Matrix Extensions) 같은 기술로 CPU에서 AI 워크로드를 돌리는 혁신이 진행 중이고, Vivo와 Oppo 같은 중국 폰 벤더가 SME 지원 칩을 넣은 폰을 내기도 했습니다. 별도 칩이 필요 없고 전력/공간이 유리합니다.

Alipay 사례로, 사진에서 ‘스시 먹은 장소’ 같은 것을 검색해 슬라이드쇼를 만드는 기능은 보통 백엔드가 필요합니다. 하지만 백엔드 방식은 1) 프라이버시(사진을 제3자에 공유), 2) 레이턴시(업로드/처리/다운로드), 3) 클라우드 운영 비용 문제가 있습니다.

이를 디바이스에서 자체 모델로 처리하면 사진은 기기 밖으로 안 나가고, 레이턴시가 줄고, 백엔드 비용도 절감됩니다. Apple도 이 흐름에 강하게 투자하고 있고, WWDC에서 A-series/M-series의 neural cores 같은 얘기를 하는 것도 같은 맥락입니다. 저는 그래서 Apple과 Apple Intelligence에 대해 ‘상당히 bullish’하다고 말했습니다.

마지막으로 저는 ‘artificial intelligence’의 숨은 부분은 ‘artificial understanding(인공적 이해)’라고 말하곤 합니다. 모델이 나 대신 이해하고, 그 이해를 바탕으로 새로운 것을 만들어주면, 여러분은 초능력(superpowers)을 갖게 됩니다.

제가 짧은 데모를 하나 보여드렸습니다. 제 아들이 아이스하키(ice hockey)를 하는 사진을 기반으로 “골을 넣는 장면”을 만들려 했는데, 저는 의도(intent)를 제대로 전달하지 못해, 실제로는 관중 없는 연습장인데 AI가 관중을 그려 넣고, 아들이 골을 크게 빗나가도 모두가 환호하고, 막대기가 두 개가 되고, 이름도 틀리는 결과가 나왔습니다.

저는 이걸 에이전틱 워크플로로 풀지 않았습니다. 즉, 1) 의도 이해, 2) 사용할 툴(예: Veo)의 특성 이해, 3) 계획, 4) 실행과 반성의 루프를 거치지 않았던 겁니다.

반면 제가 자문하는 영화 생성 스타트업에서는 같은 엔진을 쓰더라도, 전체 스토리 컨텍스트, 8초 제한, 대사 명확성 같은 제약을 LLM에 함께 주고, LLM이 제가 못 쓰는 수준의 상세하고 영화적(loquacious, descriptive) 프롬프트를 작성하게 하자 감정(emotion) 표현이 훨씬 나아졌습니다.

(질문) 에이전틱 워크플로로 개선된 건가, 아니면 하키 데이터 부족 같은 학습셋 편향 때문인가에 대해 저는, 같은 대상 비교는 아니지만 “의도 이해 단계에서 LLM이 샷/앵글/감정 표현을 더 잘 묘사해주는 것”이 매우 컸고, 툴(Veo)의 특성(액션에 약하고 슬로우 카메라 풀로 감정에 강함)을 반영해 계획을 세운 것이 비용(영상 생성은 4개에 2~3달러)을 절약하며 성공률을 높였다고 답했습니다.

(질문) AI 산업에서 가장 놀란 점은? 저는 하이프가 생각보다 훨씬 강하게 의사결정을 지배한다는 것, 그리고 단기 이익을 장기 선보다 우선하는 경향이 놀라웠다고 답했습니다.

예로 Google의 TensorFlow 프로페셔널 서티피케이트(professional certificate) 프로그램은 연간 10만~15만 달러 정도 비용이 들었고, 시리아(Syria)에 살던 청년이 서티피케이트로 독일로 옮겨 가족을 돕는 등 엄청난 선한 영향과 goodwill이 있었지만, 직접 매출이 없다는 이유로 종료됐습니다.

또 다른 놀라움은 ‘AI로 성공할 것 같지 않던 사람’이 AI로 엄청난 성과를 내는 사례들입니다. 예를 들어 제 아이스하키 선수 출신 친구는 13살에 학교를 그만뒀다고 스스로 “가장 멍청한 사람”이라 말하지만, 비영리 단체 운영 보고서를 만들 때 컨설팅 비용 연간 15만 달러를 쓰던 걸 ChatGPT로 2시간 만에 자동화해 절약했고, 그 돈이 취약계층 아이들의 장비/레슨으로 들어가게 됐습니다.

(질문) 엔지니어가 아닌 인문계 등은 하이프를 어떻게 내비게이션하나? 저는 그게 우리가 ‘trusted advisor’로 도울 기회라고 답했습니다. 보상 메커니즘(정확성보다 참여)을 이해시키고, 비전문가의 지능과 강점을 존중하며 어려운 부분을 함께 풀어주면 됩니다.

(질문) 과학 연구에서 AI/ML은 언제 좋고 언제 조심해야 하나? 저는 가능한 한 강력한 도구를 쓰되 결과를 현실에 대조해 검증하라고 답했습니다. 예로 웨일스(Wales)의 연구자가 뇌암(brain cancer) 연구에서 GPU 접근이 병목이었는데, 부서에 GPU 1개를 10명이 나눠 쓰며 화요일 오후 반나절만 쓸 수 있었고, 저는 Google Colab을 보여줘 클라우드 GPU를 무료로 쓰게 하며 연구 속도를 크게 올린 사례가 있었습니다.

(질문) AI가 사회적 평등/불평등의 힘이 되나? 저는 “둘 다 될 수 있고, 둘 다 아닐 수도 있다”고 했습니다. 도구는 사용 방식에 달렸고, 저는 ‘선의(good intent)를 가정하되 악의(bad intent)에 대비하라’는 원칙을 말했습니다.

[Andrew Ng]: (마무리) 오늘 좋은 토론과 질문 감사했고, Laurence에게 큰 박수 부탁드립니다.