Transcript
Will: Welcome back to week six of the Business Idiot podcast. This week we're going to be doing something a little bit different. We're not going to be going through the news stories, but instead we're going to be looking at the biggest trends in AI over the last few years and sharing our raw opinions on them as people have spent 60 hours a week trying to make these things work. Our goal here is really to kind of sort the week from the chat here and help you to sort of figure out which of these trends you should be paying attention to. And if you're spending money on the other ones, then you might be the business idiot of the week. So let's dive straight into it. Alex, let's pick up direct LLM calls.
Alex: Right, so this is kind of the foundation of every bit of agentic engineering or modern LLM tricks that exist. It's where you go to open AI, you say, here is some text, give me the bit of text that finishes this text, more or less. Everything from code code to the thing that reads your emails for you is based on this fundamental idea. It's kind of boring. It's kind of nuts and bolts. But like most things in technology, you've got to start somewhere. And it's a good foundation to actually start. If you're just getting in, it's worth poking these things a few times and getting a feel for what you're actually working with.
Jim: Well, and particularly to understand it. I think that's the – because it is the foundation of everything we're going to talk about and everything that happens with anything in our AI. Yeah. And a lot of people don't realize that it is effectively a, you know, not all of it is rest, but it's a rest call to something that's hosting the LLM and it's giving you the next however many words after it.
Alex: And anyone that's kind of going, well, REST, API calls, like what are these? If you've ever used ChatGPT, you've done this. You put in some text and you click send and it sends you something back. Fundamentally, everything that we're going to talk about from here is ways of automating text in and out of ChatGPT or Claude.
Will: Yeah, and probably short history lesson there, there was, for many years, this was the only real way that you could kind of interact with GPT. Yeah, totally. And then there was this kind of step where they said, well, if we actually allowed you to sequence many of these messages back and forth to the API into a thread, then it starts to feel like a conversation and not just like an individual message.
Jim: And that's how when they started to talk about the... context window or the the model having memory all it's doing is remembering or looking at the the last calls that you've done and chaining them all together exactly like you said will and so that that's sort of and that's why we talk about it being the foundation of it all you know is that no matter what there is a whole lot of weights which are effectively numbers that go into a matrix multiplication scenario, sitting on a box somewhere, your call comes in, it does some matrix multiplication, predicts what the next words should be and sends it back. And that's effectively what's going on every single time.
Alex: There's a little bit of black magic in there. But, like, fundamentally, this is – if you've ever sent a letter to someone and got a response back, or particularly to, like, a big organization where they're following some step-by-step process, it's the same thing, just orders of magnitude faster and with a very weird and high-tech way of coming to that process.
Jim: And so is it a fad or is it a fact?
Alex: Oh, absolute fact.
Jim: Absolute fact. Yeah.
Alex: It's as concrete and non-bullshit as you can find in the LLM space.
Jim: The whole ALM world, yeah, is that no matter what, there has to be an LLM call. There has to be some sort of call to a model, I guess, you know, like whether it could, it doesn't have to be an LLM, I guess, yeah.
Alex: Your thoughts?
Will: I think one of the reasons this one's on the list and why we're talking about something that's so elementary and was a, you know, a bit of evolution in this technology so many years ago, is because it's still valuable, even just this most smallest, tiniest piece. One of the examples, I think probably actually Alex, you're introducing me to this, was if you were trying to do some experimentation around getting the AI to get to a certain outcome for you, and you can sit there and you can like prompt, let's say Chachapiti a hundred different times, but that can be a very... onerous and time-consuming task to do and you also get all this extra weight in there around how the model works inside its harness which we'll get to in a second but when you just go to the raw api and you just write a program that says here's 100 different versions of this prompt send it to the api and get the response back and then we can measure the differences between them that's valuable even in 2026 we don't need all of this other agentic engineering to just get raw value from the api but that that
Jim: step alone so many people are skipping it but it's what will actually bring the value to your harness to your context to whatever else you're trying to do in this scenario um it and and again everyone is just missing oh well hang on what is the what are we actually sending as that direct call and coming back you know and what's coming back yeah back on we'll take us to number two then jim prompt engineering Well, so prompt engineering then becomes changing what you ask, what you send in that direct LLM call, you know? And so you word a sentence, you know, the quick brown fox jumped over the... And then see what response it gets is that it very well, again, most of the time, the... the any the quick brown fox jumps over the lazy dog is what you're looking for right is that if you gave it just the quick brown fox jumps is it these days that would never be a question is that every llm would come back with over the lazy dog but go back five years and there's a possibility or come to a smaller model and there's a possibility that it won't Come back with the rest of that sentence, because the prompt you've given it is slightly, slightly changed. So it doesn't quite know what you're trying to what you're trying to ask. And so that's sort of where and it's a bit of a weird example, but it's more just saying the way in which you build your prompt will change. influence the response you're getting because they are probabilistic in nature. They're looking for what is the next sentence, the next step in what you're asking. And so that's effectively what prompt engineering is. You change the way in which you ask the question of the LLM.
Alex: I think it's an interesting topic in that it keeps coming up in research. You know, if you talk to court authoritatively, it won't push back quite so much. But if you act with a lack of confidence around it, it'll start to overrule your decisions. And it's one of the weird things with these things being... so chaotic and so non-linear, these small changes, even with the modern models, can still have that sort of decisive impact.
Jim: And even between models. Even between models, yeah. And there's probably a lot of other influencing factors on that, you know, is that in the way that whichever lab is serving their models and routing your query, but it's still... The way in which you structure the prompt and the way in which you try and put guardrails or whatever into the prompt affects the results you're going to get because it can sway what it focuses on and things like that.
Will: Prompt engineering is an interesting one, I think, because we spoke about it so much years ago. But we don't really talk about it anymore and it's actually not that it's... something that we've stopped doing or even something that we've mastered because i find myself personally even with each new model release revisiting a lot of the ways that i need to try and prompt it to try and get the results that i'm after so i think it's it's interesting because it's it's sort of fallen into that that bucket of like if you think back to kind of google search when you first started using it and when google first first came out. Yeah, yeah. There's like a very specific way that you have to structure the keywords in order to get the result that you want. And over time, the technology developed to be able to understand better what you were actually looking for. And so your ability to put the exact keywords in the right way and with a plus sign or a minus, these things fell away over time.
Alex: Yeah.
Will: I think the AI is getting similarly better in a way where it's getting better at understanding your intent. So your prompt is... kind of in a way like less important to be structurally perfect yeah and my prompts personally are getting lazier but i am also finding now that um because they're focusing so much on trying to get an understanding of what your intent is if you've used the wrong word somewhere or you've been too vague sometimes it can take that and go the wrong direction so now i'm starting to tighten up my prompts a little bit again because it allows me then if i know that it's quite
Jim: well-defined what i'm asking it then i can let it go on its own for longer periods of time without going off the rails and i think it's a really good way of defining it you know like that the models have really folk or the training of the models has really focused on understanding the intent and that's sort of where and like i would say my prompt engineering has become a notice that I structure, even when a new model comes out and I'm refining it, I still structure things as to this is the goal, this is how I think we should be doing it, this is what you're not allowed to do, you know, and make sure you ask any clarifying questions because if you assume anything, you're in trouble. And it's funny how much that is still my final, the final thing I put forward because I started doing it years ago, but that is still something that carries through and still has, I believe, a positive effect on the result I get.
Alex: I think there's levels of prompt engineering too, right? Like there's everything from just tweaking whatever you're putting into chat GPT, you know, press edit and resend the message because it's clearly going the wrong direction. versus you know i've got a bit of code that's dynamically swapping in template entries and my structures are all fixed and my instructions kind of change um we talk about skills uh fairly shortly the sort of idea of uh giving agent skills and and this is fundamentally prompt engineering yeah injecting in bits of prompts as we go um but like i i think it is a bit of a fad i i remember uh
Jim: The way in which they spoke about it, where everyone was going to, or the new job was going to be a prompt engineer.
Alex: I remember a recruiter telling me in 12 months, we're not going to be able to hire enough prompt engineers. And it's going to be the job of the future. They'll be paying quarter million dollar salaries.
Jim: But it comes back to, like we say, a lot of weeks. You can tell who just talks about AI and who actually builds with AI. I think everyone who legitimately thought prompt engineers was going to be a real job never actually built anything.
Alex: Or they spoke to ChatGPT for 30 minutes and didn't try to like...
Jim: fire that in anger in like a live and or we're just looking for putting in generic things looking for generic responses but the moment the moment you get a you want any sort of specificity or any sort of repeatable predictable result It becomes clear that you need more than just the prompt.
Alex: So, yeah, fatal fact.
Jim: Is it, I think, the way in which everyone was speaking about it, you know, where it was going to be, oh, the new job of the new generation, you know, everyone who lost their job to AI was going to become a prompt engineer. I think that was a fad. I think... Prompt engineering as a whole is fact because how you form your prompt and the way in which what you put in and what you don't put in really does affect the result.
Will: So I think it has to be fact based off that. Yeah, I agree. I take a pretty simple approach to this one. It's like the prompt is the input into the machine and the input is a large percentage of the quality of the output.
Jim: So it's always going to be important. And it's the first thing that everyone learns about AI. Garbage in, garbage out.
Will: And I think, like, until... I can still regularly sit down with smart people who don't work in AI and show them how to improve their prompt and get a much better result than it blows their mind. That's actually... So until that has passed, you know, prompt engineering is still very much alive.
Jim: And it's certainly about... Like, I really did like your point of view on that the models are getting very... You know, are trying to make it obsolete. But... I think because, again, because they are probabilistic and are always going to take... the general, the most probable answer, it's going to always be difficult for that to ever be perfect.
Alex: That's actually worth digging into just like very briefly that LLMs, the way they're working is kind of like biasing flips of a coin. They're not guaranteeing that heads come up. They're just making it more likely that the thing that you get out the other side is the thing that you expect. But the noisiness of it is somehow inherent to them.
Jim: Well, and that's sort of where we then flow on to the next couple of trends. Yeah. Is that it's effectively trying to minimize that noise. Yeah, reduce the variance. Yeah.
Alex: So yeah, I think prompts engineering, we can call that a fact, but if it's on your LinkedIn page, maybe change it.
Jim: That's it, yeah. It's past the point where you should have it on your CV.
Will: Yeah, agreed. Vibe coding is next. So I'm really in two minds about vibe coding. So vibe coding effectively is... Now that you can talk in natural language to AI and AI can output code and that code can run applications, you can now have people who have no software background now building software is, I suppose, the promise of it. I think this is fantastic because it's opened up a lot of doors for people who are like, adjacent to or around software being developed and it's been a huge unlock for them and being able to they're also learning very fast about how technology works you know people talking to me now about their Supabase back end and they would have no idea even anything about a database even a few years ago so some of that is like it's quite cool and and we still will regularly use something like Lovable to build a prototype to help sort of work through what a first version of this front end might look like and how the users would want to interact with it and make it work. That's on the one side. On the other side, man, I hate it because it's like so dangerous because there is a very unclear line at the moment between what a vibe-coded app is and what a production-ready, well-engineered app is. The people who typically have the budgets and are wielding them to get a certain output don't understand the distinction between these two things. And so when they see a vibe-coded app, they go, that solves my problem because I can click that button and that result appears. And so I'm happy with that. And this has caused a lot of chaos in my space because a lot of the things I've spent... years learning how to make sure that an application is reliable and that multiple users can use it and can only access the right amount of the right data that they should be using and that it's not going to go down on them, that it's secure. These things are all super important, but they were always like in an appendix somewhere in the buyer's mind. They're like, let the engineers make sure the app works. You guys build it. It's pushed further back into the appendix than I've experienced in history now. And so there's a lot of buyers who are just happy with prototypes now.
Jim: Right up until they want to take it, you know, want to expand it. And that's what I find is that the people who, because I'm definitely vibe coding is a fad. And I think we're getting to the point where we're through it. Is that the moment anyone, like, I think you can spot a vibe coded app online. a mile away i can definitely tell if you've used lovable or Replit or anything anything like that um yes it's great that everyone is starting to understand that oh there's a database and a back end but they don't actually understand they oh i've got a Supabase but they don't actually understand what's going on and that sort of thing and so the I think, yes, it's great in they can prototype something and get a this is an idea. But the fact that everyone is starting or and I think it's more just that it was sold in that way that anyone can now code their own app.
Alex: Well, Andre Carpathy getting on stream and acting like a clown probably didn't help.
Jim: That's the biggest thing for me. Even he's gone back, oh, no, we don't. He called the end of vibe coding months ago and really then got into, no, it's got to be a proper SDLC and it's more agentic engineering now. and but everyone seems to have no one seemed to have picked up on that one all they've done is just kept focusing on the vibe coding and i just think it's more just it's great for prototyping and it's great for getting something quick and i would this work but the get the the chasm between that and something that is repeatable um predictable and and efficient is is so far and so i think that's where i like and again i'm probably coming down on fad just because i would like everyone to stop like it's it's more just if you're vibe coding then i i'm i'm not if you're vibe coding for anything more than a prototype then i'm not interested in what you're doing
Alex: I'm very sorry because I do a lot of vibe coding internally.
Will: For prototypes.
Alex: No, no, no. For production apps. In my defense, I know what the hell I'm doing. I'm cutting corners quite deliberately. I'm doing things inside firewalls. I'm doing things with tight external constraints. They're usually external tools where I don't want to spend any longer than I have to on looks or user experience or anything. Because the person using it is usually someone at not a fantastic salary who's paid to use internal software. It's a massive upgrade on the .NET or legacy thing that they're using that's vaulting directly into a SQL server somewhere. Um, I think, I think vibe coding is used and abused, uh, as a, as a, as a lever. I think when you're selective about like prototypes of one good example, I would highlight internal tooling where you don't give a shit how it looks. Vibe code your way to success.
Jim: And again, if either of you said, hey, I vibe coded this, don't get me wrong. If either of you or anyone else who I know can actually produce something says, oh, I vibe coded this. Oh, yeah, great. But the problem is that the natural tendency of people is to take shortcuts. So much so that I believe that the models have learned that.
Will: How do we actually define vibe coding here, I suppose, because... days of writing with AI. So what are we saying when we say like sometimes a vibe code and sometimes I don't. What do you mean by that Alex?
Alex: So there is a blurry line here between kind of agentic engineering and it kind of comes down to the discipline you apply to the process. You know sometimes I'm very particular about no this must be implemented in this specific way because I know that ten steps down the line I'm going to need this to plug into something else that doesn't exist yet. Sometimes I just go, build me an app to do this and don't stop until it's complete, make it secure, make it good, no bugs. And, you know, let it churn, go off, get a cup of coffee, come back and see what I've got. But I think, you know, we're talking about brainstorming, and I want to take a step away from software engineering to, like, what are we doing when we vibe code? Yeah, yeah. We're sketching. We're brainstorming. That's it. Right? Like, if you're building a consumer-facing app... or something that needs to be secure for whatever reason, yeah, don't be so, you know, what's the word with it, cavalier with what you're doing. Make sure you understand the security concerns that you're buying. Don't confuse the fact that you've got a button that you can push for the fact that it's secure or the fact that it's going to stand up to any sort of scrutiny because hackers will spot you and they will crack your system.
Jim: And also, you know, understand that just because it runs in Claude on your laptop doesn't mean that you can then get it to work for the rest of your business or team or anything else.
Alex: Our old head of legal was very much like that. Oh, but it works on my Claude. I don't understand why I can't have it work on her machine.
Jim: I've had so many, like, again, so many clients show me their vibe-coded app and they're so proud of it and you can't, you know, you then can't go... denigrated in any way yeah but it's just the steps and again they think they miss the fact that it's all static data they miss the fact you know and so anyway it's more but in terms of like you were saying in terms of take it away from yeah take it take vibe coding away from vibe coding and vibe anything right it becomes a really useful tool for brainstorming or for trying to flesh something out but to me that's where that's why it's really good for prototyping because you're fleshing something out you're not over engineering that's it you're kind of you don't want to invest the time in an idea that you haven't really got a feel of yet like the
Alex: The biggest problem of smart engineers is they try and production... They can't see that these things can't be thrown away. So they spend six weeks doing all the production setup and then one week on the prototype instead of just hacking away at it for one week and...
Jim: being ready to... Or web designers really focused on, oh no, this button is three pixels too large. This whole landing page that you wanted to put into this app is all about getting website traffic. The size of the button ain't going to make that big a difference. We want the SEO infrastructure to be perfect. Yes. That's what matters. And so we were doing OKRs a couple of months ago. And it was a really good tool in fleshing out the possibilities in what the key results could be for any objective we were saying. And again, also suggested, it helped us really form up what all the objectives were going to be and what the key results were going to be for those objectives and which should be in what bucket and things like that. And I think that to me is a similar... approach to vibe coding because it we weren't worried about okay it's it's 25 increase or 24 increase we were worried about getting some sort of key result in there that we could then put a measure against and that's putting the measure against is turning it into production you know i think there's um to just dive into this perhaps a
Alex: One of the things I do like about them is because they do a lot of work around how these models fail and yeah they do make it easy to add things like um you know secure practices and stuff that would take that takes some knowledge to hook up unless you're in there well that's there is some benefit but that's where i see vibe code or particularly the vibe coding apps right you have transitioned i think a bit into what are now being called harnesses
Jim: And that was, again, the switch. All the software developers and the software engineers could vibe code and that was, oh, they're getting the greatest use out of AI. But when Claude switched... Claude Code into Claude Cowork, that became vibe coding for everyone else. For office workers. And so to me, that's the thing, is that where it's really just that's effectively what a harness becomes. And so that's sort of why I come down on that vibe coding was a fad and you're now... just doing something in a harness, you know?
Alex: I don't know. Then what about work slop? Because work slop is the office equivalent of vibe coding.
Jim: Well, but that's why I think it has now, you know, it's a past fad because it's now transitioned into a harness where you're trying to minimise the slop because we got so much slop. Like the number of Git... commits and the number of apps that lovable was putting out every single week was just off the charts like there weren't there weren't enough consumers in the world to to have that many e-commerce sites i think even coming back to the way alex described it early on was like the you were used to a discipline in your coding approach um and sometimes you don't always have this
Will: and that's when you consider yourself vibe coding. That's the same thing with using code work and just vibe email. That's it. You create slop in your email. as well.
Jim: Or your article or whatever it is you're creating.
Will: So maybe it's something around having the discipline to build the thing to actually be able to do the outcome that is required without having some massive downside to it like your database is exposed to the internet
Jim: an email that says would you like another three options of this email and so and so that's sort of that's why i'm you know again trying so desperately to make it a fad because i just believe the more discipline everyone brings to their use of ai the better results they're going to get and that and that sort of as we progress on to the actual you know other topic other current trends that are becoming more and more useful, I believe, it's about enforcing some sort of discipline.
Will: Good call. Let's move on to a really useful trend now, Jim. You can take this one. Token maxing. Token maxing.
Jim: Well, again, to me, it's the same vibe. And so token maxing was, or is, because again, there's still some people out there doing it, was... And it was a lot of larger enterprise just saying to whoever was using AI, use as many tokens as possible and your budget is effectively unlimited.
Will: Can I extend on that one a little bit? I think it's a little bit around the goal setting by leadership. So the goal was to improve AI adoption and therefore adoption could be measured by the number of tokens that someone was generating. So we create a leaderboard, look at the people who are at the bottom, and they're the ones who are not adopting the AI. And this was always where AI adoption was being used as a proxy for AI value realisation.
Alex: And this was particularly in software engineering too. But it did show up in other places, but software engineers... With their tools of scaling, could have fairly easily gamed the system.
Jim: And that's where I think, yes, that was the intent, you know, to improve adoption and improve the result. But all effectively it did was... create gaming of the metrics. All that ended up happening was people were just throwing more and more tokens, creating nothing and creating slop. And so I think the moment you start going, okay, well, it's how much you use it is the point of value. That's deemed adoption. That's the error. You need to be looking at the objective and what's the result. And To me, that's why it has absolutely fallen, you know, become a fad and a past fad. Thank God. Because, again, everyone realized that, oh, my God, this just costs millions and millions and, in some cases, billions of dollars. And we're not getting any ROI whatsoever.
Alex: I mean, it's not the first time people have gained metrics, but it was a particularly stupid crime.
Jim: Well, that was what was so, you know, how did they possibly think it was going to create any other result than it did?
Alex: Like, I like the idea of, you know, create space for experimentation, but like... Surely, I don't know, I feel like a little bit of foresight and a little bit of knowledge of the problems you were trying to solve might have led to significantly less gameable metrics that weren't quite so expensive. But, you know... The cynic in me, or rather the tinfoil hat wearer, says a decent amount of these CEOs putting out these mandates would hold shares in these AI companies and therefore mass adoption might be good for their portfolio.
Jim: Absolutely. But that's the thing, is it? Like, just... Running more tests than could ever possibly be necessary. Yeah. That was, again, I can still token max now and get no better result other than just using more tokens.
Alex: Yeah, and I do sit on the fence a little bit on this one because the token budgeting, when you have someone going, oh, you can only use this many in a day. We've got this many for the week. It's like... if I'm using this to make myself multiples more efficient, the cost of the tokens relative to my time are so cheap, you are mad not to pull that lever as many times as you can.
Jim: Well, again, that's why I believe it should be focused on the result and what the actual ROI and then down to the individual. Because if you are just sitting there running too many tests and using up your token budget because you're too scared that... at your performance review someone's going to say oh you you've only ever used 60 of your your token allocation then then that person should they they should have a a reduction in that token allocation whereas the person who's producing results and usable results I believe they should be getting far more tokens than possible because that is exactly what you're saying.
Alex: I also feel like this whole way of thinking has accountants' fingerprints all over it. Yeah, yeah, yeah. It's a very accounting view, you know, like, oh, well, we have budgets and we do this. But it's also what they're...
Jim: tighten it right up you know if you don't use it you're going to lose it and but it's also what the allocations were i feel like so many of them were just plucked out of thin air like yeah you know they have no idea and and again is i'm still lost even for the like my claude max subscription is it i'm getting 50 more to one date and then 30 more of one model and not the other i still don't know how many actual tokens i've got week on week yeah to uh to use through that subscription
Alex: I think token maxing could have been less disastrously stupid if it was a post hoc justification thing. Like you spent $50,000 of tokens this month doing a bunch of stuff. Cool. You've got the receipts for what you were doing and you can explain it and justify it. Like you've got a day to put together a sort of one page on each of these kind of key use cases, explain that chunk of money.
Jim: Even if you got the model to insert the Confluence page. Yeah, shit yeah. Just do it. Do a one page on every test you ran. Get the model to insert it, but at least let's document something. Yeah, shit yeah. And that's the other side of it is that token maxing was – No one learned anything except tokens are expensive. That's it. There was so much usage. And so that's where, again, the conspiracy tinfoil hat wearer in me goes, it was something invented by OpenAI and Claude, Anthropic, just to get more tokens running through, which, again... The corporate equivalent of eating Tide pans. But logs were kept by the company. You know, all the prompts and everything was kept by the company and could be reviewed. And then we understood that, oh, well, all these prompts were just slop. Let's not do those again. Fantastic. Then you get some sort of ROI. But no one, everyone just dumped the tokens.
Alex: There were some companies like... Token maxing is an idea. Dumb. But companies like ClickUp, they actually got immense multiples on what they were doing. And so I think it's the idea of creating a surplus so that there's room for experimentation and looking for people trying to drive extremism.
Jim: Great idea in practice. ClickUp in general, I think, has, again, it's always been, you know, a lesser used product, but always a really good product. Yeah, really. And I think that is representative of their team.
Alex: Yeah.
Jim: And so I think, again, if the team at ClickUp can produce the product that they have, that sure, they don't have the marketing budget or whatever it is that it isn't because it isn't, as widely used as other tools, but it's still such a quality product that they must be a quality team is, again, my simple man thinking on it. And so that's the thing is that I think if you throw token maxing into... a quality team you get a better result because to your earlier point that if you're actually someone who is delivering value then you want to turn the dial up on them and give them as many tokens as possible at least until returns start that's actually pretty high yeah this one sounds like it's an ai trend token maxing but it's really the same thing humans have always been doing is that when there's a lack of leadership then we don't have
Will: teams being pointed towards the right things we measure metrics that should not be measured that then end up being the thing that we optimize towards that don't actually have any real value creation and i think it was pretty clear to anyone from day one of hearing the word token maxing that there's going to be a lot of people and maybe even the majority of people in corporate enterprises are going to go i'm more than happy to optimize towards that outcome because i can It's easy and I don't have to solve any really hard problems and I'm going to get a bonus for it. And that is just such an obvious thing that is going to happen to anyone who actually adopted these. They have to be the biggest business idiots of the year.
Alex: Well, this is why I keep thinking, who's owning stocks in these companies? It's obviously going to happen.
Will: It is so stupid. There must be another reason.
Jim: There must be another reason. But also just that the people who were dictating the goal were definitely stupider than the people who had to fulfill the goal. And so they just went straight around them and went, okay, sure, I'll meet your metric. No problem. Actually, I meet your metric as I go to sleep every night.
Will: Metric gaming. You've got a system that helps you to game the metrics. Anyway, let's move on to the next one. But definitely fad. Definitely fad. I think that's pretty clear by this stage how we all feel about this one. So the next one is agents. I suppose we're talking about agents here. We've talked about these direct LLM calls and you can now then chat your BT where you can have a conversation with it. Agents is really where you move from, I suppose, answering questions with the AI to being able to take actions on your behalf.
Alex: Well, I think to get a little bit technical, we're typically talking about like the react when we talk agents, right? Like something that can... iterate on itself a few times to carry out some more complex goal than... With a lens.
Will: Can see its output, look at it, and then decide whether to continue working, right? And it's the original loop, really, and we'll get to loop soon.
Alex: Under the hood, it's still just API calls, but with some clever prompting, you can convince it to kind of...
Jim: But also... Also because it allows it to go back and...
Alex: to look at the result through a lens to say okay well is this what you know what my objective is you know as as however i've created that agent i think this is where you start to see like multipliers on human time coming back like messages in the chat gpt can be helpful in getting some general information or sandy checking some ideas or something but if you're really talking about um As a non-programmer, because programmers have access to these primitives by default, as a non-programmer, if you're trying to maximize your individual output, agents are kind of the key tool in your toolbox, whether or not it's in one of the cloud products or some agentic platform like Codebots. What's the... There's a few of them at the moment. I haven't played with any of them for a while. The only one that kind of comes to my mind is Area, but
Jim: But again, they sort of, I believe they, again, they don't come to mind purely because everything seemed to move past it a bit, you know?
Alex: Yeah, I think everyone, well, with the exception of the sort of the coding bots, which are becoming the everything bots because of tool use, but the build your N8N agent, you know,
Jim: Again, that's why, again, I'm a little bit on the fence like Will on previous things with this. I'm not sure whether it's fact or fad in terms of the agent and giving something a persona because... Yeah, I think we should tease out these two aspects.
Alex: One is the technical implementation, the sort of primitive, and one is, hey, build your company with...
Jim: Which we have spoken about before. But even the technical, it still is just another area of compute or deterministic process where it's got an objective, sending an LLM call and getting it back and deciding what it is. To me, I'm not sure whether, again, making it an agent or, as we'll move on to next, just defining a skill and making an LLM call. Again, so that's sort of where I go, why does it have to be an agent? I think just having the skill can be enough.
Will: Yeah, there's a lot now of agents being used where it can just be an LLM call, for sure. And that's like massive over-engineering, but it's also... So easy to do.
Jim: But that's why my spidey senses are tingling a little bit because exactly how you just framed it, you know, it could very easily just be an LLM call, but there's so much hype around agents and everyone needs agents and you've got to have agentic and all this, you know, all of these things that all of a sudden my spidey sense starts to going, hang on, all that is is just a few guardrails and an LLM call. Why is it... you know, this over-engineering.
Will: I'll tell you why. And I've done this before. I've built agents where I know that an LLM core could have done the job. It's because I know that two months from now they're going to say, hey, can you add these tools? Can you look up this knowledge base? Can you add all of these different skills into it? And everything that kind of gets wrapped in there around that. And I suppose there's kind of a harness here and separate out agents from harnesses as well. But it's my risk mitigation for the future that I don't have to come back and rebuild that thing into an agent.
Jim: And I understand that. And maybe that's a bit more what I'm feeling here. Because again, I can't really define it properly. And it could be that harnesses have sort of become more of the descriptor to the way I think about things a bit more.
Alex: I'm not sure. I think it depends on where you're using them. Again, if you're buying a specialized tool to manage your agents or something like that, you're potentially over-engineering things that could just be LLM calls. Personally, like I'm not doing anything with tools with my agents. I don't trust the bastards. If I need information, I am looking up that information. I'm force feeding it to it. It's like, you will look at this. You don't have a choice in this.
Jim: And that's sort of, again, is that I come down to is that I do so much with, you know, the graph engineering and the context that... The use of having an agent just seems redundant to me.
Alex: I actually take completely the opposite stance.
Jim: Well, you're wrong. See you next week, guys.
Alex: I don't like being told I'm wrong. Get some fake skin, mate.
Jim: Let's get out.
Alex: For me, particularly around data extraction tasks, like when I was doing stuff with documents, it would always be output into a structured format that if we didn't match, it would force the agent to really look at what it's done and have another crack. Now, that can be handled on the LLM call. But having the agent capable of re-exploring the document and figuring out why it's done something wrong, coming up with a plan around how to handle particular edge cases and then outputting something that passes the spec, that's a key reliability primitive for me.
Jim: Oh, I'll see that.
Alex: Yeah, and so I think it doesn't waste tokens, absolutely, but only in about 5% of cases.
Jim: And that's where I go, you know, yes, okay, I can see the use case, but... Again, is it predictable? Is it repeatable? And is it efficient? Yeah. And a lot of every time, or not every time, but so much of the, oh, well, I've just used my agents in this case, is this non-repeatable, non-predictable, non-efficient scenario. Yeah. Where, yes, there is a great use case, like, you know, for working out why when Doris filled out the handwritten the form and filled it out and went outside the box so much that it couldn't it couldn't ocr it yes i understand that's a great that's a great use case for it it's just yeah in 99.9 percent of the time i just it can just be an llm call i i
Alex: I definitely agree with the spirit of that statement. But as Will said before, if you're using something like land chain or these other agentic SDKs, it's so bloody easy. It saves you writing a shitload of boilerplate. Why not?
Will: That's it. But again... So for me, agents is like the real deal. It's the promise of what AI is meant to be. But once we get there... we realized that it's actually opening like a Pandora's box of other challenges that have to be solved. And I think we almost need to give credit to the AI engineering companies that probably within a year's period, we went from like starting to have agents to having tool use, external context, and evals, right? And some of these like pre-AI, that could have been 10 years of like progress to figure out how do we make these agents actually valuable and useful?
Alex: I disagree now. It's mostly like MLOps tricks repackaged and they're quite expensive. They're inefficient, but like, yeah, they're good, but I don't know.
Will: Sure, but you're talking about like kind of getting some of these concepts from like adjacent areas and going, hey, this would be useful in this case. Like when I first heard about agents in NNN and I went in and had a look at it, it already had placeholders for all of these things. Like I was very impressed and, that they've gone, not only should we put this Chatuby T thread on like a bit of a loop here where it can kind of have this React loop, but also let's give it all these things that solve a lot of the problems that come with it being able to make its own decisions.
Alex: I just think these were inevitable, like particularly evals. It was an inevitability because you can't control this thing well enough. You need some measure.
Will: They are to people like you and I. There's a lot of people who still haven't ever used an eval.
Jim: Yeah, definitely. And that's where – but that's, again, that's why I am tending a little bit towards FAD on – and again, by the sounds of it, you guys are a little bit fact and I might be the only one out of my branch here. But just that I think the whole concept of agent has been a little bit superseded by – a harness to a degree. Even Langchain is sort of talking that they are now a harness rather than a gentic platform sort of thing. Yeah, I think that's fair. And that's sort of why I'm going, okay, yes, I could see where they were going, but I think we've just evolved it a little bit. And that's why I'm... far more agent skills give skills to an LLM call and it works in effectively the same fashion.
Alex: I just don't like that. Again, I don't like the non-determinism when that's buried deep inside another program. Skills are great if you can see them, if you're directly interacting with the agent yourself or whatever LLM call, but deep inside some other system, then it may not use it. It may make a credit determination without consulting the person's credit file, for example. Force feed that thing in, but then still keep the agentic loop so that it can identify problems in its own thinking and come to a better overall result.
Jim: Yeah, and again, I guess a lot of the... Maybe it's just my control, my need for control over everything. It's just the result is just... The result from agents or that agent, you know, decision-making scenario is just like the possibilities are just so rain, so broad that I just mean, to me, it then becomes, you know, okay, that could cost, you know, N number of tokens or Y number of tokens every single time. And it's not repeatable and predictable.
Alex: We're also talking different ends of the spectrum, right? Like you're talking about...
Jim: solving a business problem i'm talking about usually addressing a thin slice of some hard problem yeah and again creating an agent to be your best friend or your psychologist again i think that works really well you know but again i guess maybe i do get a bit too lensed on on you know the business case or the business scenario yeah um but it just seems to be that Again, even if it is my psychology, even if I've created an agent that is my psychologist, I've got no idea what's coming out of it every time. Just what you put into it, mate.
Alex: It's a Rorschach diagram.
Jim: But it's also the most probabilistic response, whereas psychology is very, very...
Will: I would definitely say there's a lot of merit to what you're saying, Jim, because most of the effort that I originally put into making my coding agent a better agent is now making my agentic harness better. So I've definitely shifted that focus. But when I think about a business workflow where there's a component where it's going to go to an LLM to do some work, That's either going to be an LLM call or an agent, and I'm not putting some sophisticated harness in place for that.
Jim: Yeah, yeah. Again, I just believe that all of that sort of operates within what we're going to call a harness. I suspect so. What we're going to call the harness, I think, is going to be effectively the infrastructure in which everywhere... And I think as well, like...
Will: I think a lot about like lovable and vibe coding, like lovable has their own harness. They don't talk about it as a harness. But it was a harness before. Like when Claude Code got a big bump with like Opus 4.5 and they, there was a lot more harness. that came out at that same time. Lovable got that harness bump, but you can still not get the same outcome with Lovable. Like, Lovable can do a lot of things that if you just went into Cloud Code and asked for it, you'd still have to step through. Like, there's clearly a harness over here with a workflow and preferences and integration. So, yeah. No, I'm with you on that one, Jim. All right, well, let's move on to MCPs. I think we can probably cover this one off pretty quick. So, starting with Alex.
Alex: Most common... And MCP is, we talked earlier about prompt engineering kind of having this spectrum, everything from tinkering with words that go into a prompt all the way up to sort of dynamically swapping bits in and out. And MCP, we're kind of at the point now where we're starting to dynamically swap things in and out of our prompts. Essentially what it does, it gives some additional context, some additional keywords to the LLM. that if the llm uses these keywords it can start sending calls across networks and accessing various systems or accessing tools on your machine etc um and it can do so in a way that can be turned on and off and it can do so without uh taking up your entire uh context window to just for that one operation that you use maybe once a month um It's a moderately smart idea, I think. Like, spoilers, I think it's a fad because certainly these days I do a lot more bare REST calls, just tell a thing to make Python calls. But, you know, there's merit to the idea. Certainly the hot swapping aspect and the simplifying things for people and then formatting responses for agents to natively consume them
Jim: I think that's good. And I absolutely agree with that. In principle, it's absolute fact. Yeah. You know, the implementation turned it into a fad.
Alex: Yeah.
Jim: And everyone got the sort of, oh, we have one too syndrome and then didn't flesh it out. No one, yeah, no one actually implemented their entire, their entire API, you know, created the MCP that their entire API could be used. No one, no one, seemed to do any maintenance and so everything was a friggin stateful session this was a bad idea in the 2000s well so that and that's why i'm wondering whether the you know the change that we were talking about last week or the week before where where it's now they they've refactored mcp I'm wondering whether it will come back into being fact, you know, because it actually has a lot better use case. And I guess, you know, the promise can be fulfilled, you know, of what the principle was.
Will: So for me, MCP was, I think it's phenomenal to be able to, plug something into your agent and then start working with it right away like there's some magic to that yeah um the longer you work with that mcp the more that you wish that you were just talking to the apis yeah um but uh for that kind of getting started on something that's quick and small like fantastic um but yeah i also agree with you jim that the the main thing that i hated about mcps was just um What do you call it? Just effectively the reliability of the connection and just making sure that it was just going to work. And it's just amazing how much time you just spend re-authenticating and updating the config. I know there's a new version of the handler or the auth server or...
Alex: What got me was so many of them was like, oh, yeah, by the way, you need to install a new programming language on your machine, a particular version because we don't have any version control around it. We didn't Dockerize it. We don't have any things. We've got one here that's clobbering this other one that you have. That's your problem. It was an absolute blast. I wouldn't call it a gold rush because I don't think many people got rich out of it.
Jim: But that's why MCP and the implementation of MCP just gave me so much PTSD with crypto. Because again, the principle of blockchain and crypto is brilliant. It should be everywhere and it should be used a lot more than it ever was for other than gaming and a whole lot of incel sitting in their basement. But... Personally attacked. Well, again, you know, intentionally.
Will: The hype killed the real adoption.
Jim: No, but the actual implementation, because again, too many people were cutting corners and you'd integrate with their API for their, you know, whatever their crypto project was. And their coding principles and their documentation were all just subpar because it was just some kid in a basement not implementing something properly. And MCP just turned out the same because no one actually implemented it properly or saw the real potential. And that's the thing. And so that's why I'm hoping that, again, the changes to the authentication or the session management for MCP hopefully will bring it back and it will turn back into a fact. But it is absolutely a fad at the moment because, again, why wouldn't you just make an API call?
Alex: Yeah, I kind of get the feeling that if we just did this as open source software that was designed to be running online, we could very easily bundle the whole thing into a .compose instance, handle authentication and everything via end variables. Not have to reinstall my Python every time someone decides to update their MCDP and force update my system Python. Yeah, like I got a huge amount of PTSD trying to juggle a lot of sophisticated capability. And I learned that in the end it wasn't worth it because the more you put in, the less reliable any single point got.
Jim: The number of failed calls to Asana MCP that I get are phenomenal. And you just go, how can this be possible? And deeper into a session, it seems to happen more often. It is. And so, again, one of the tasks for next week is literally just to create my own... integration with the REST API just to, and that's the tool it uses now. Because it's just too frustrating.
Will: Okay, Jim, agent skills.
Jim: Right, well, agent skills, I think this is, so it must have been, was it last year or the year before that Anthropic put forward skills.md? And so it might have been last year.
Alex: No, I think it was late 2024. I think it was because you and I were running at the time.
Jim: Yeah, it could have been. Agent skills are effectively just one file that gets put in and it goes in. And this is sort of where we were talking with prompt engineering that you stack things into or you improve the prompt by including other things. And so and, you know, you can put in. the steps that it needs to take, the responses that it's looking for, what's good, what's bad. You can somewhat put in guardrails to restrict what it can and can't do with the response. And it allows you to get what I would think is a more predictable response each time.
Alex: yeah and i think i think one of the the key attributes here is the model has a sense of what skills it has available and when it should use them but it's not that information about how to use it isn't sitting around the context the whole time it's not um constantly thinking about debugging when it should be thinking about like sequencing work It says, oh, I'm sequencing work. Let's have a look at what is required to sequence work. And at that point there, it's kind of popped into the context and the model can then follow those instructions. Consequently, it's not blurred somewhere right in the back of the context. It's right at the front, right at the point where the model is generating data. Can we talk about context windows? Should we briefly talk about where things sit in a context window?
Jim: Yeah, touch on it. Well, but again, because it does come into this.
Will: That's why we have skills and also deciding what is inside that context window at any given time. ends up being the entire game of context engineering, which we'll talk about next. So let's just weave all those in together.
Alex: We did talk briefly about, with these API calls recursively, putting your previous messages in your chat context, and that's kind of your context. But your model will pay attention to two areas of your context more than anything else. One is right at the beginning. One is right near where the latest message is. In between, things get blurry. So if you tell it halfway through a session, hey, don't do this thing, expect the model to forget it. But if you put it into, say, the system prompt, if you're doing some prompt engineering work or injecting it in via the skill right at the point where the skill is invoked, the model is much more likely to follow those instructions. And this is why we've got the reliability gains that Jim's talking about here. It's actually wired into the maths of how the models generate their stuff in the first place.
Jim: And it's similar to, you know, if you use Claude or there's a Claude.md, which is sort of overarching rules or whatever, or how you want Claude to behave. And again, because that with the skills gets put right in the most recent part of the API call or the LLM call so that it understands exactly what it can and can't do with the response that it gets. And that's where you get these million token context windows. It becomes really, really helpful because that blurry part really becomes blurry.
Alex: They used to refer to this reinjection of the same prompt over and over again as power steering.
Will: Yeah. I've always kind of liked that analogy. So actually, when Skills first came out, I was a total skeptic of it. Almost because it was too simple to just think that it could be that powerful, that you could just take some instructions and put them over to the side and just call them when you need. But I think almost about what you were saying there, Alex, like there's some elegance to that in almost the way that it works, similar to the human mind, where when we're doing a task, like if you're tightening a screw with a screwdriver, you don't need to hold... in your mind at that time your entire knowledge of your entire lifetime you don't even be thinking about how to drive a car or the color of a giraffe you're just thinking about that that thing that you're doing at that time and it actually allows you to have focus and that focus very much narrows down to the skill you're needing the objectives that you're trying to achieve and it turns out that agents work very much the same if you can narrow their focus just down to the task that they're trying to work on and the skills required for that then they don't get this context poisoning going on for all the other things that they're meant to know.
Jim: Well, and that's been the greatest use case for me in particular, is that I found with skills a much... less capable model is able to achieve the same result.
Will: Yes.
Jim: And, and that to me, it was the complete unlock as to why skills were so important. And, and that's right. And I quite love, I quite like your screwdriver analogy, because again, is it, I don't ever need to know, right. You know, righty tighty lefty Lucy until I have the screwdriver in my hand, you know, and it's the same for the models. It doesn't need to know that it can't,
Will: reformat a document until it goes to reformat a document it's like the matrix right where it's like you want a skill and you can that's it you just install like fighting skills and that's it it's really cool to be able to have your agent and it's and it can't do something today and then you you just drag and drop this file in the skills section or you know save it in the folder and then now it can do it It's this really cool unlock for agents.
Alex: I think, yeah, certain skills are really critical to my workflows. But I don't know. I'm sort of leaning to fad here for a couple of reasons. One, if you're not in the loop or you're doing something, again, my typical use case, deep in a stack somewhere solving some gnarly little problem, they're absolutely worthless. All they do is make the result less predictable and less reliable. Um, if you're in the sort of major space, if you're doing like coding or something like that, they can be helpful, um, particularly for like software development life cycles or to do some stuff around, um, analyzing marketing compliance. They can be really useful. But for example, um, I installed some skills. I use a particular framework, uh, temporal for those playing at home. Um, quite a lot and it's got some real quirks to how it handles long-running stateful things you can absolutely tie yourself in knots and so i thought cool let's get the temporal skills built by the temporal people heavily supported by the anthropic people get this into the get this into the um the agents so it can kind of help me debug pipelines a bit more or at least get further along before i need to intervene And it absolutely failed. And this has been my experience with quite a few sets of skills. They sound really nice. You install them. They don't actually move the needle. There are some that do. There are some that don't. And then on the other side of it, if I actually need something to happen in a particular manner, I don't reach for skills anymore. I reach for hooks.
Jim: Yeah, or I create something deterministic. Yeah, exactly. And I think that's another – you know, the other side of this. It becomes – where I can see why you'd call it fad is because it – As much as it narrows the predictability down to, you know, a very small window, it is still... It's still noisy. It's still somewhat, you know, and again, can be quite noisy. I just, again, for things like... email connections. Yeah. You know, I think it becomes perfect because you can go, okay, don't touch these emails, but touch these emails. And it, it, the, the LLM instantly understands it and can stay within its guardrails. Like you say, where you get down into really deep in the stack and it's, and it's something that you're actually trying to, again, to me, even using an agent in that scenario is question, you know, like when it comes to question, it should be a deterministic thing. piece of code you know yeah and so that's sort of where i you know or again my approach to it is i'd um i that's where i see them as as fact because if you are trying to achieve something with
Alex: If you're manually invoking them, they're pretty good. But if you're trying to give the agent the tools to teach itself to fish... Again, my need for control means that I don't really let the agent pick. Yeah, but I'm lazy. I want to go down to the pub and drink beer and tell the boss that I'm still at work.
Will: So where I go with you, Alex, is the deeper I go into the stack, the less skills become useful, unless I've built a specific skill for that depth of that stack, which... you know, rarely, rarely you do. The other one that you sort of mentioned was like, when I go to like the anthropic skills hub or the library, like after the first 15, I'm not even using any of the other 10,000 in there because I just don't trust them. They just create noise in my workflow and I don't find that help. But there is one example where I regularly am connecting to new systems and ingesting the data and kind of building pipelines specifically for that system. And I'm always sending it to the same lake house that I work with. And that lake house has a bunch of, I suppose, opinions about how things should work. And they're not necessarily conventional in the sense that an agent would just infer that right away. So having a skill which says... This is my kind of heuristics for how I look at different types of data in the source database. Is there a timestamp? Should we be snapshotting this data? Then also how I want to use it in the other side. It can now go and research both the source and the destination and come back, look at my heuristics and say, yeah, here is my design for how this should work. And that has like massively sped up my build process.
Alex: Yeah, I agree that that would, but like where I ended up getting to was that was insufficiently strict. I really needed things that were much stricter and with much tighter enforcement because of the...
Will: To, like, run workflows, yeah. I think this is only valuable in this case because there's an element of research and discovery when it's looking at those two systems and forming a plan.
Alex: Well, again, this had that as well, but I just feel like... half the time i'd install a skill for you know let's say as your devops um which you know has some opinions about how things work the agent had just brute force try to use the github command line until i told it stop doing that put a block on that we use as your devops um and then it would just use the command line so rather than try and force it to use the skills i just said, okay, well, let's double down into using the command line.
Will: I've got to give you that one. I actually don't think I use anyone else's skills anymore.
Jim: I absolutely do not use... The only... But it's now a plug-in. I used to use the superpowers skill, but it's now a plug-in, so it doesn't count anymore.
Alex: Yeah, superpowers. A plug-in is just a combination of skills.
Jim: It is. I'm just putting a delineation in there for no reason.
Alex: In terms of... essentials yeah that's one of the ones that absolutely enables my workflows um we were joking before the before we started recording there's one called caveman it makes your model much more terse and much more to the point and occasionally like oonga boonga type type talk but responds with what it needs to respond with and that's it It's faster, it saves you tokens, it actually does a better job. It shouldn't work, but that starts every time I load up cloud code because it's so damn useful. There is gold in them hills.
Jim: And I think that that's the use case. And a lot of it, like what Will was saying, it falls into that research and prototyping and designing phase. But when you actually get into production, the production workflow running time and time again... it becomes a bit more deterministic.
Alex: Yeah, you sort of need to be a bit nastier to the model and a bit more forceful.
Jim: That's it. But again, is it, I've really... I don't use generally anyone else's skills, but I have been creating skills for a lot of things now and getting quite good results.
Alex: I think as a subset of skills to commands, just quick little instructions. Hey, clean up all the crappy Git work trees and Git branches I have all over my system. I've racked up 25 in the last day and I'm lazy. One command, off it goes. Switch to a new tab and keep working.
Will: Yeah. So actually, point of learning here, Jim, because I only figured this out this afternoon, you were mentioning earlier on a skill being one file. I don't know if this is just the Claude desktop app or wider than that, but because you can upload a zip file to that, I was sort of iterating with Claude trying to build a new skill where I gave it the instructions, but I also wanted to give it a bunch of examples for it to work off. But my fear would be that it would then use those examples in the actual outputs. So those zip files, you can actually upload a lot more files than just the one as reference files. Your agent will read the skill. It will decide should it read the reference files or not, the examples, and then go and read just that one. And it's so much more powerful.
Jim: I've got one for a mixture of experts analysis, I guess, or a judgment panel scenario. And, again, is that it randomly picks four of the judges from, again, a folder with – again, skills defining the, or other files defining the judges, and it picks a judge for its panel, picks it, put its panel together, and away it goes. And again, it's just, it's how you build it. But again, I just like to still think about it as one file. Sure, it turns into a file system, but it, yeah.
Alex: Yeah, I'm a software developer. Everything lives in its own file. That's it. It's just one thing. It's some ridiculous naming convention that only I understand. Yeah.
Will: So context engineering then, Alex. We've obviously just spoken a lot about context, but specifically this domain of what is it and is it a fact or fad?
Alex: Yeah. I kind of feel like we've talked about a bunch of things that have faddishness to them. But under the umbrella term of context engineering, I think I'm going to come out the gates and say this is a fact. But there's a lot of bullshit and snake oil when you dig in. Yeah. What is context engineering? It's all these tricks that you use to swap things in and out of your context to get the information to the model before it begins. Be this give it tools, inject in additional information. Even the React loop is a kind of context engineering where you're saying, hey, you have the ability to iterate on yourself if you use this following token sequence. Yeah.
Jim: Well, that's why I like to think about it as the right information going in the LLM call. Yeah. Because, again, if you don't give it any context, the LLM has to use... the whole world or the whole knowledge that it's been trained on.
Alex: Yeah. Which is going to be blurry as hell by the time it gets to inference time.
Jim: And again, because they're probabilistic, it goes for the most probable based on what everyone thinks. Yeah. Whereas the moment you give it... the right context it will give you a far more nuanced response and you get far more use out of it or you move faster you know you you move faster towards your objective and so that's that's why i like to think about it as the right information rather because and and sort of that's where everyone like we're talking earlier about the context gets blurry Everyone just leaves the context. And particularly I find when working with clients that gave everyone Claude subscriptions, everyone just lets it get to a million tokens and wonders why their results denigrate. They just start a new chat interface every morning. And just leave it run. Don't even think to switch when they switch tasks.
Alex: This is kind of, you know, we were talking about having a discussion about context engineering. It didn't include for me even things like compaction and compaction timing, which is where you summarize your conversation history to reduce its footprint. Like, this all kind of falls under the umbrella because, you know, beyond the context, the model's only got its training patterns to rely on.
Jim: And that's where – and particularly – and that's why I, like – again, I have a separate bugbear about – or another theory about how much – they were all originally trained on Reddit and how much they then have a tendency towards certain outcomes. Oh, yeah. And I just wonder how much of an influence that was because of what they were trained on leads to the same outcomes. And so that's why I'm so focused on getting the right context in because I – getting to the common outcomes is most nine times out of 10, not what I'm looking for. You know, I'm looking for something very specific that is predicted, you know, that is predictable and repeatable. And so that's, that's sort of, and I'm more and more, and I'm very enjoying the fact that there are some significant leaders starting to talk more and more about the context is the absolute foundation and, owning your data and getting the right data and having access to it in the right way is what's going to be the unlock the the ceo of palo alto networks has been talking about it quite a lot recently about how they every time someone is using ai in any way he wants a you know some sort of
Alex: analysis on that interaction just so they can improve their context and improve it over time and i love that concept yes so around like how to actually put this into practice we've kind of talked about skills which is a kind of dynamic form um we've talked about prompt engineering which is very deterministic form of of context engineering um there are all these associated techniques and skills, things like tools, things like RAG to kind of assist the generation process. Where do you lean in terms of how to operationalize this best? Is it a bulk dump and a single interaction or is this a thing where you trust the agent to find what it needs?
Jim: I am very much on... Again, trust the agent wholeheartedly, no. But I do believe you've got to have the context organised in a way. And that's, again, I'm quite the advocate for RAG or more graph RAG, which we'll sort of tend into next. But just because I think... Again, getting that right information at the right time. And so, again, if you think about a workflow, you don't need the skill at every time. You don't need all of the research data at every call to the API along the workflow. And so getting... So searching the context in a way that the LLM can do it quite efficiently is, to me, the most important thing. And so that's what, I guess, RAG or Retrieval Augmented Generation is what that stands for. And effectively, it assigns a number to all your data, to all your information. word document will be transferred you know transitioned into a whole lot of vectors which the llm understands a lot more and so then it can do effectively what is you know i i'm looking for this what are the what's all the information that's closest to that and we'll send that as part of my as part of my my prompt or my query and again that significantly improves the result
Alex: Where do you sit on this, Will, around context engineering practice?
Will: Yeah, it's probably been like the biggest part of the last 18 months for me, focusing on context, especially from like, I suppose, running a data business. What we've been trying to do is focus on how we get AI to reliably interact with a data set and really the big unlock for what we've been able to do is take our lake house which is effectively where we store all of our structured and unstructured data from the business and we kind of wrap it in what we now call an AI lake house where we now for all the data in there that we document what the data is, we provide a whole bunch of context around what the business does and its org structure and what its customers do and on all the fields we define what that field is and how it's used and what are the key metrics within the business. Like that is a lot of, I suppose, groundwork doing things that we never really ever did in the past, sort of before AI. And we always recommended that you should have, you know, a good data catalogue, but very few companies did it. It was always homework to keep it up to date. So this AI Lakehouse now we can connect. ai agents to it through the mcp and just say like hey can you help us to forecast where our revenue is going to be next quarter you know and it'll go in and factor in everything like churn and growth rates and upgrades and add-ons and cross-sells, all these things, because it can see all the context about how the business works, what are the product offerings, and pick up that data at the right time in order to support its analytical process to get that answer in the end. So all of that has only been able to be unlocked because of all the context engineering that we've done up front, not only to collect that data, but the key insight was to store the descriptions of how the business works right alongside that data, not as two separate things where the AI has to go reaching, but be able to see the data and what it is all at once.
Jim: Yeah, and it's so important. That's why, like I was sort of talking before, that's why I really like the integration of like a knowledge graph into that where it becomes more of an ontology that maps all those relationships. And I found that sort of graph engineering take us there, Jim. Well, but again, it just means that, again, because you can then effectively map the relationships between pieces of data, is, again, the way I understand it. And so then you can have these ontologies which describe these relationships and you can then just add ontologies for different use cases. Or that's how I found it really, really useful, particularly with AI or agents, however you want to describe it. And so you then have, you know, what are general business requirements on relationships or, you know, the master fishbone, I can't remember what the, what his name was that came up with it. But, you know, it's sort of, you know, will is a data you know a data scientist who runs beyond data you know and that those those three pieces of information are linked in in the graph but only need to be stored once and and so you can then use the ontology to retrieve the data in in a much more efficient manner and and so understanding how business works and how they use their data then you can then design an ontology to suit that and it instantly starts to work or starts to make their AI use a lot more efficient. And then the next department can have a slightly different ontology and you're referencing the same data.
Will: Yeah, so we work with ontologies a bit of beyond data and the reason for that is because the nature of questions that people ask They start out very simple, which is like, oh, can you just tell me, you know, what orders are going out tomorrow? Yeah. And that is just a normal kind of database query. But what typically happens quite quickly because of the way that our businesses are set up is our workflows that people do each day, the tasks that they do, they interact with many different entities within that business. It's not just like... You sit there and you only look at orders and you never look at your products or your customers or locations.
Jim: But in the natural course of a person doing a job, they don't think, they don't realize how much.
Will: They don't see it like that. But that's how brains work. But, you know, the questions very quickly come to be. Can you please tell me what shipments are going out tomorrow for high-priority customers that have paid all of their recent invoices and are in Southeast Asia and won't be affected by the shortage of... of supply that we've got from this particular supplier now a query like that is like extremely hard to do typically and an ai would likely fail on that many times over but when you've got an ontology that underpins that and it already describes the relationships between all those things in your business yeah map to a data model that shows how to join from object to object yeah now the thank you yeah now the ai can just go to that model and go please give me the answer to this and it doesn't have to do any of this inventive creative non-deterministic querying it can just ask that question and get the answer back and it's super powerful well but and and the result particularly the result you get back you know and and so again is it i've really been playing a lot with like news and and things like that and the relationships between
Jim: the relationships between topics, you know, just to get, you know, and seeing what a general RAG query will produce or return compared to having something through an ontology for a specific use case, you know, be it, you know, writing articles or for a podcast or something like that. And all of a sudden you get a much clearer view
Alex: result and it is for the use case i i'm actually going to lean on the side of fad here but not because of any inherent um iconoclasm or or willingness to piss off my friends it's i i say this only because ontologies have been an idea that have been tried repeatedly throughout history i don't think AI as a technology is fundamentally enough of a game changer to solve the problems that you hit with these systems. They definitely have a massive leg over rag. And that's good enough. Let's start with that. And I think that's a little bit...
Jim: the way in which will and i are using them you know is that it is just if everyone everyone because rag sort of turned into a fad i think yeah and still and to me still is but a lot of people affected dbs a lot of but a lot of people still talk like you know they talk a lot and they keep going on and on and on about like it's going to solve all the problems whereas i just yeah and it And that it's the solve for AI memory where you just go, I can't possibly, you know, have you actually ever built a rag and put enough data in it to then start to get to a max? Yeah.
Alex: That's the key there, the density of – sorry, real nerd term. When you're doing these vector graphs, it's kind of like you're building a big cloud. If you've only got 10 or 12 points, it's really easy to say, yep, that's the one I want based on a nearest – looking at the nearest neighbors. But if it's super dense, if you're looking to find out like – I don't even know how to give this a physical analogy. It's a needle in a haystack. If the haystack's pretty small, you can find the needle.
Jim: Yeah, or a particular branch on a particular tree.
Alex: A particular branch on a particular tree, yeah. If it's a small tree, it's easy to find. That's it. What the ontology does is it gives us another lens for doing that reduction first, right? Great. Except you've still got the density problem. These graphs have this nasty property to them. And then on top of that, language is kind of like poorly defined and concepts are inherently fuzzy. I don't know, man. I just, I get the feeling.
Will: I completely agree with you. And I think like any big enterprise, which is using ontologies will not have one ontology for the whole business because that just doesn't work.
Alex: Can I, can I just briefly vent? teams that are building ontologies in enterprises and they do it for like five years and by the time they're done it's completely irrelevant it's all a giant waste of time they all get a promotion and move up and a new team comes and goes what the hell have they built it i think it's more like i like the idea of and again that it that
Jim: Adding the graph element to RAG made it actually useful.
Alex: Yeah, because you're just doing graph operations then, like traversal operations.
Jim: But I'm a very big advocate of version control of the ontology and not leaving it. Whereas I think...
Will: Again, so much... That almost needs to be like the most governed thing in your entire data model.
Jim: But that's the way in which people need to think about it. But they don't. Even with, you know, the Microsoft product, GraphRag, it's not the primary focus. You know, whereas to me, it should be like the version control of the ontology should be the primary focus. And is this still relevant? Is it still giving us what it needs? And...
Alex: I don't think businesses think about ontology in the right way. They think of it as something that, rather than something that drives how they structure operations, it's something that they sort of archaeology out. I don't know. I feel like there's promise in the technology, but the way that we... And it is an upgrade on RAG, but I think it's a fact. I don't think we're going to be disciplined enough to apply it at scale.
Jim: I agree on the discipline and on the understanding, but I'm going to go with fact. Purely because it has been the only reliable, repeatable... way in order to retrieve data that is needed to achieve these things.
Will: Yeah, this one's a definite fact for me. But I do also think a lot of the things that we're talking about today, a lot of the concepts that could be updated in 10 years and we'll say, why are we ever doing it that way? And honestly, there's a lot of limitations to all of these trends that we all kind of see. But as you said, is it a big step forward to what we're doing today? Yeah. Absolutely.
Alex: Yeah, and if nothing else, like, I'll pay that any day.
Will: Yeah, yeah, yeah. Well then, Alex, let's run through a couple here pretty quick. Agent Swarm.
Alex: Let's not bury the lead on this one. Crocker shit. It was an absolute sales ploy. All it did was take a bunch of noisy, chaotic things, put them in a room with other noisy, chaotic things. Sorry, I should define this. No, I'm going to vent first. Anyone that thought that agent swarming your way out of a problem was going to work are damn fools. And that includes a lot of very smart people that I worked with. I told them it was a mistake and it took them three or four months of wasting time to realize it was a mistake.
Will: Agent swarming is like a group project at university. Yeah, exactly. But like you've got like a hundred people in the project and they don't have coordination.
Alex: And none of them have clear instructions because they've been written by other people on the project.
Jim: And one of the people ended up doing the whole project anyway. And that's it. It's the same with the agent swarm. One agent solved the problem, got it done. And 99 were discarded. That's all you needed to do. And so to me, again, it's absolute fad. It's the same as token maxing.
Alex: We should define it. Multiple agents working together. But...
Jim: But almost too many agents. Like that's the thing, is that way too many agents to achieve the task required.
Will: Well, the idea, if I understand correctly, is throwing so many agents at it that the right answer must pop out eventually. Which to me is the same as token maxing. I never saw agent swarms implemented in a way where they said, let's start with two and we'll keep adding an agent as long as we see improvement. I never saw that. I always saw them go, let's start with 50 and see which agent gets it right and then figure out how to help the other 49 converge on that one faster. It's a group project analogy to me.
Alex: There is some validity to this idea. Like you go with an orchestrator dispatching sub-agents. Again, I believe that's different.
Jim: And so I believe Agent Swarm, fad, is an orchestrator with sub-agents, fact. Because again, the orchestrator is creating the agents and tasking them separately in exactly the same way that a human would, whereas the Agent Swarm was just a noisy mess.
Alex: So you heard it here first, Jim says democracy doesn't work.
Jim: Then tell us, Jim, Mixture of Experts. Mixture of Experts. What is it? Absolute fact. It depends on how you utilize it. But in the sense of a model, Mixture of Experts is where they fine-tune parts of the model on a certain topic. And so Kimi K3, for instance, is a Mixture of Experts model. And so it has... massive amount of parameters, but there's only a certain number of parameters that are used at any time. And so as you pass your query to the LLM, that query is then sent to the expert and the responses comes out. And so you can also use a mixture of agents, a mixture of expert agents, where you create... separate agents, all that have a certain expertise, and you then get all of their opinion back and then, I guess, assess all of their opinions and come to a decision. And it sort of works in the same... It's the same idea, but it really does produce a better result because you end up getting the different points of view and the different risks, particularly is where I find it very challenging. or I utilize it a lot that they come back. And then all of a sudden you've thought about something that you haven't thought about because you framed it slightly differently.
Alex: It also depends on what you're feeding in. Like you can, you can get an advantage just by speaking up six chat TPT things and saying, Hey, think about different aspects of the problem. But if you're feeding them different contexts and having them really drill down into sub-aspects of the problem, sort of pre-structuring the mixture, I think you get qualitatively better results. You do see some improvement in the generic agent version, but... I don't know, it feels far more hit and miss and it just tends to reinforce the biases of the model itself.
Jim: Yeah, I quite like using different models as the base of different expert agents and you really, really get a qualitatively better result.
Alex: Well, this is like, it's kind of, if anyone's ever done data science, we recognize this effect. But weirdly, the ancient Greeks knew about this one. If you ask, what was it, two experts the length of a cow or something like that, you get a worse answer than if you just ask 10 random people, as long as the 10 random people can't talk beforehand. Their average guess will give you a better... measurement on the cow than the two expert cow measures. I don't know what the Greek for expert cow measure is. I don't know much Greek at all, in fact.
Jim: It still must be cowboy.
Will: Uh, also fact on mixture of experts, uh, only for strategic kind of discussions or points. Um, if you're throwing a mixture of experts, it is like basic work. You are just token maxing. Yeah. Again, I have made some much, some, some materially better decisions in my life by consulting a panel of different expert agents, which gave me their own lens, uh, you know one from a legal perspective and one from a brand perspective and one from a finance perspective and and what i found great was the first time that i started to use it i would get them to sort of they could see each other's work and it was worse and then i separated them off so that they couldn't see each other's work and then let a better model like Fable just orchestrate or combine it all together and give me the verdict, much better result.
Jim: Yeah, I found that then doing a gaps analysis, once you get all of the... All of the experts having like doing a gaps analysis and then asking them again. Also, and that's where, again, switching from Claude to Chet Jippity.
Will: That'd be a good time to switch then.
Jim: It really does get, you know, and all of a sudden you drill down. And so it really is a planning and design tool. But again, absolute fact.
Alex: We were talking smack about N8N before, but I find this particular use case is where I reach for N8N. Yeah, right. Very good for structuring it and making sure that it unfolds in the way that I kind of want it to. And for these things where it's like one shot, I kind of want to do it once or twice and then kind of. move on to the next set of problems i don't want to spend all day doing it um i find it and that ends a great way to just hook it up and get it going who would have thought we found a use case for n8n in 2026 i do my thing man bring it i'm bringing it back bringing it back so OpenClaw and hermes agent next um so these ones uh kind of took
Will: or maybe 18 months-ish?
Jim: It was actually, it was the end of last year. That's it. It hasn't even been a year yet.
Will: End of last year, wow. And I suppose the breakthrough on these was this idea that you have your agent, but you're going to give it two extra things. The first one is a... file that's called like soul.md which is like you're giving it a personality that is um its own independent view of itself it is not like i'm a task bot here to perform a certain task it is like i have a soul and this is what i do and then the second one being um this idea of a heartbeat which is just effectively a timer which goes off every 5 15 minutes 60 minutes which just wakes it up and says hey is there anything that i should be doing right now should i check some things should i take some action and these two together gave it this really kind of human-like this anthropomorphized feeling that this agent is like a real living thing because it acts like a human in many ways and it kind of it comes to me with things that need to be done and it will work on its own and and you get this like kind of big buzz at first by starting to use this thing and I'm definitely one of the people who spent a lot of time pouring time into this and I think it had a huge leg up originally on the like the open ai and entropic harnesses at the time and so it was fantastic but now it's just like complete fad for me and i just need to find a way to unwind this thing because it still does some workloads for me but everything that i needed to do can be done in the other tools but the other tools are just have gotten way better and now it's just this extra like data security risk for me.
Jim: I got caught up in the OpenClaw Hermes and have run it, have had an agent running the whole time. But again, I just never, ever found the use case. And I really... tried to work out why for a long time. And I think it's because I grew up on, on cron jobs.
Alex: Yeah.
Jim: And because my, you know, knowing or learning Linux, understanding Linux and knowing that I needed a cron job or understanding what that, what that did for me being able to schedule things and organize things. really helped me understand that, okay, well, I can just use a cron job for this. Why do I need the restaurants?
Will: Why don't you understand what's under the covers? It's lost. It's all its magic. And that's it. It's really not that sophisticated.
Jim: And that's why I really like the guy who did... I've forgotten his name now, but again, who got open AI to effectively buy it off him. Yeah. Is a genius. Like all he wanted was AI to be, that he could WhatsApp, you know, and that's how he did it. And he was on holidays at the time and that's what he vibe coded together because he just wanted to be able to send something to AI through WhatsApp.
Will: Yeah.
Jim: And that's how it evolved. And it worked, you know, worked really well for an individual to achieve things like that. Yeah.
Will: Whereas... Yeah, I heard of a senior AI leader in Australia standing up in front of a crowd of like 800 people and saying, I've got a personal digital twin of myself and it does everything that I need to do and it acts just like me. Like, we know now that that is bullshit. Yeah. We never trusted to do that. And if you're an AI leader saying that, that's even more questionable. That's it.
Alex: Well, again, I think there's a lot of grift in the industry. I'm going to... I told you so. I called it very early on that this was a cool little toy. It could do things for you. It couldn't do them super reliable. I don't know. Like Jim, I never saw the killer use case, and I kind of saw that from the beginning that, yeah, it can answer my emails, but I don't trust it to answer high-priority emails. Those emails I wouldn't normally read anyway. I'd skim the subject lines and go, yeah, delete. And the important ones, when I read myself and reply anyway, it might save me.
Will: five ten seconds a day so so so what you just said i think is like the the unsaid thing that's at the heart of the entire ai industry right now and i think we should touch on this even though it's not really about a trend the things that are hard to do the most valuable things typically you would not let your ai do it the things that you would let your ai do were always the low value things anyway yeah so when we talk about there being large amounts of AGI or maybe whatever, you know, are humans going to ever be as good as AI in the future? Like, let's still just be clear how many people are actually using AI to do really important, critical and risky things. But I also think... The answer hopefully is zero.
Jim: No, no, but I think it really shows up in how businesses start to integrate AI because they don't look at where the bottleneck is. They look at all of the easy things and that's where they throw AI really quickly. Can we use AI? Let's do AI. And then all of a sudden the bottleneck gets worse. And they go, AI is terrible. And to me that sort of illustrates what you're saying so clearly is because people aren't willing to throw it at the hard thing. And that's where the bottleneck is. Whereas you should really be breaking the hard thing apart into a whole little things that can assist the hard thing rather than just speeding up the easy things. Exactly.
Will: And when you go and look at those complex problems, at those bottlenecks, tools to be in my arsenal. But I cannot just point AI at this problem. There are really intelligent and competent people who've spent years trying to fix this bottleneck and they still have not.
Alex: I think like Everyone talks about prioritization in business and there's all these things you've got to do and you've got to work through them one by one. I've always felt like this is a fallacy and AI is just making this really pointy because what people are automating with AI are those low-value tasks. Yeah. That then suddenly create a lot more bullshit work for other people in the organization. That's why the bottlenecks getting worse because the capacity to filter through all the shit that everyone's AI is starting to send out. That level of just ambient noise is getting higher and higher. And all those tasks, for the most part, probably shouldn't exist in the first place if you are properly structured and running things like cleanly and efficiently. They're always the patchwork workflows where people half-ask the solution five years ago and you never go back and fix it properly. I get furious about this. But, like, we went looking for a study that was talked about on LinkedIn about a government AI rollout. Yeah. And, okay, we've got this new co-pilot going to government and solving these tasks. What did it end up being used for? Sense checking things, chatting, rewriting emails. Big three use cases of a multimillion-dollar rollout. Which of these things should they be kind of doing in a day? Maybe sense checking things. And even then, what are they fundamentally sense checking? They live in a highly process and regulation driven environment. They should probably just have a process for it. I get so annoyed when people talk about these tools because it's obviously not going to move the needle, and yet there is a pile of use cases that will make the case, that will move the needle, but... but people won't touch it because it's a bit hard and it might involve some serious change and some serious rethinks.
Jim: And I think, and even just back to Hermes and OpenClaw, it's sort of, it was just an over-engineered, or it became this over-engineered solution to, you just needed a cron job to send an LLM call about your, and then here's all the context of the emails.
Alex: Or just read your frigging emails. It's not that fucking hard. Yeah.
Jim: No, it's not. But again...
Alex: I think it's a cool toy. But it was only ever going to be a toy and anyone that spruced it as revolutionary deserves a smack upside down.
Jim: Well, but I just think it illustrates who the real players are. And I think, like, even in what you said earlier, Will, you know, where... You were amazed by it. But as you learnt more and more about what it was actually doing, you just went, okay, well, why do I need this? I think 90% of the open-claw Hermes people aren't questioning that. And all they're doing is just... token maxing through a little buddy.
Alex: It's vibe coding but for the real world.
Will: Yeah, and that's sort of where... I will say there are some good use cases that I have my open core doing but the biggest problem is that they're all very token intensive and so I turn them off. So I would be interested to know if at least the patterns of this... chrome job to wake it up and and check for work and do research and come back when we have the ability to kind of do that work a little bit more freely and a little bit less compute and cost constrained what will that look like um but you know there's also a good argument that ai will only be more expensive in the future so we may never end up in that situation i um i think you could i think if you put a better infrastructure around your your hermes agent yeah right a bit more tightly scoped to what it was you actually wanted it to
Jim: And actually gave it a graph rag of your life. It had data operations that ingested everything into that. I think it would become a lot more Jarvis-like, which is, I think, where the attraction was. Everyone wanted the Iron Man scenario, and Hermes was close enough to that, or OpenCore was close enough to that. Whereas if they just put the proper infrastructure around it, and again, sure, run it on a Mac Mini, get an open weights model so it doesn't cost you as much.
Alex: It doesn't matter if it takes two minutes to generate the prompt because it's happening without you touching it. That's it.
Jim: Because again, you've given it a command, it's gone away and done it, come back. But just give it the context was the biggest thing for me. If you could create a skill for it to search a graph rag, I think you would get significantly better results. And weren't just randomly connecting it to the most important parts of your life yeah i i think you get a better result by stripping out some of the things that you're asking it to do just optimizing around that don't fix a problem that shouldn't exist that's it and so and so even as i say it now i'm just talking about a a you know effectively and infrastructure to run ai yeah so why do i need the hermes agent yeah i can just chat to i can just have a deterministic thing where i send it a chat or i talk to it uh send it a voicemail and it does it all itself i don't necessarily need the hermes agent you know okay guys we've got time for probably one or two more um so take a pick we've got harness engineering
Will: We've got, which is probably the big one I've been waiting to get to. Maybe Alex, you just want to do a quick take on dynamic routing.
Alex: Oh, dynamic routing, it's all a piece of shit. If people are trying to sell it to you, don't buy it.
Jim: Well, but again, I think it's more that if you are relying on dynamic routing, you haven't done the work on what you're asking to do.
Alex: Yeah, if you're trusting an LLM to know which LLM to talk to, it's talking out its butt because it doesn't know any of the other ones that are current. But, you know, there are some things where if you have multiple routers available, you can switch to
Jim: lower power models but it should just be it should just be okay I have I use this lower power model for this task because I know that's all it needs yeah and and
Will: And so dynamic routing is just token maxing wrapped up.
Alex: It is. Putting it into another LLM to say what LLM do you use.
Jim: That's it. It's just a way to create more tokens.
Will: So harness engineering, we've got loops and recursive self-improvement.
Jim: Well, again, I think all of these come together in a way. And they've been talked about. interchangeably a little bit of late. You know, some people are talking about loops where they're really talking about a harness or I believe what a harness is going to be. And I think recursive self-improvement is the overall objective of what these things are doing.
Alex: Yeah, it's loops applied to the actual models themselves and sort of bootstrapping up the intelligence factor.
Jim: And so that's where I've come down on the side that, If you group them all together and then just call that the harness, that becomes fact. Whether recursive self-improvement can actually happen, I don't know, but it's more that... Even within your organization or within your team, at the end of the quarter, you sit around and you do a retro and you go, okay, what did we do well? What did we do poorly? What can we do to improve this next time? Is it all recursive self-improvement is just finding a way to do that better? on a more repeatable, on a more frequent automated basis, more frequent and automated basis. That to me is the same with loops, you know, is that loops are just do this, check whether it's right. Is that, um, I can never remember what the third one is, but then the third, the fourth one is.
Will: optimize or plan plan yeah plan a plan and then do test learn and that's it then then yeah improve right yeah is it it's just speeding it all up you know and so to me this actually i sort of challenge that it's just about speeding it up uh because i think that there's a lot of people teams organizations that never do the learn stage they will just it's like the whole ooda loops right like you know you'll you'll go out and you'll you'll take some action. And it's something like only like what, 2% or 3% there was a study, right, of teams will actually look back at what they learned from the outcomes and then feed that back into their operating procedure for the next time.
Alex: I guarantee you that 75% of those teams claim to be practicing agile and they're probably practicing some flavor of scrum and have a ceremony called a retro and yet they will still fail to do the learn step.
Will: Yeah. So, yes, there was a lot of teams that were doing agile and doing retros. The reason that was so small of teams actually doing it was that part where you feed the learning back in. Oh, yeah. Because that is the part that requires real change in your team and the way that you do things.
Jim: But in real leadership. And this is sort of why I want to group them all into the one thing and just call it a harness. Because it should just be all the one thing. And it shouldn't be defined separately. Like this, we all... Teams across the world know this is what they should be doing. But as you both just said, no one actually does it. Week in, week out, you see people having stand-ups and doing sprint planning and having retros. More than half the call are just sitting there doing nothing.
Alex: I think even the people that are talking half the time, like I remember sitting through some of these planning meetings and it's like, you guys have learned nothing. Yeah. You guys have learned nothing. You have multiple major failures on your record and yet you're going in for another helping. And what are you doing? Like you got yelled out by the CEO recently. Do you really want that to happen again? You don't like...
Will: This is how we work. We are X and then they put their role title in there and that's how that's done. That's it. That's how that's done.
Jim: And so that's why I think if you want for all the, I believe individually, they're a little bit fatty, right? I think if you combine them together, I think you can get to fact where it will actually be something that moves AI forward, helps the integration and actually produces some sort of return on investment. And that's why, again, I've landed on harness just because I think that is the best way to describe where integration with AI is going to end up. It's some sort of environment or infrastructure where your agents, your context, everything can live and interact together. And the objective is to get to recursive self-improvement or some, doesn't have to be even be like recursive, some sort of self-improvement.
Alex: That's it. This is, this is why I get really like rage baited by.
Unknown: Point is well made.
Alex: But yeah, rage baited by people that keep their cloud config on their machines and not in their repos. That's it. Because it's like that there is a shared artifact that you can use to make everyone else's job. If you solve a problem and you solved it twice, you then put it into the config. No one else ever has to deal with it.
Jim: But it also could then also get version control in a repo template. So whenever you create a new repo, all of a sudden, all of the learnings from every other repo can go straight in.
Alex: And you then also- It's the easiest mechanism in the world for basic improvement of your agency framework, and yet no one bloody does it. And again, I think- No, that's not true.
Jim: A decent chunk of people do do it. Well, I don't think it's a decent chunk. I think a small elite percentage.
Alex: A minority of people that have their heads screwed on, that learn the lessons from the retro.
Jim: But again, and I end up sounding like a broken record on this, but I ultimately think this comes down to leadership. Same with the actual implementing the retros, actually taking on the learnings, actually having everyone contribute to a common Claude.MD. These are the things that everyone knows will move the bar forward, even if it's only an inch. But an inch forward every single month, week, day gets actual improvement over time.
Alex: And there's empirical evidence to support this, right? Like we've talked in the past about using like open code or root code, these sort of open source agent harnesses. And the quality of the harness more so than the quality of the models was like a major determining factor for the success of these things. Why is Claude dominant? Well... Anthropic models are pretty good, but Claude Code is a supremely good harness as a product.
Jim: And I feel like they distilled, just because I'm spicy, they distilled from all the other harnesses. Oh, absolutely. And they took all these learnings in and put it into a better harness.
Alex: Hey, look, learning translating to business success.
Will: They're an organization that learns at an incredible rate.
Jim: And then get upset when anyone else tries to do it on them. That to me is, and so I just believe that if you, the better the harness you build and implement, it doesn't have to be for whatever scenario, that's what you're going to get. And it should have an element of looping in it. but it shouldn't be the only thing you focus on and it should have an element of improvement in it. Again, the bars drop from recursive.
Will: So I've built myself a loop in my harness. It is fantastic the level of outcomes that I get from it and the amount of time that it takes, that it saves me to get that high quality outcome. That is way more expensive to do those loops. So I've got it turned off, but yeah,
Jim: access to the number of tokens i need well and that's that's where i see the you know like where ultimately all of everything we've talked about today yes you know some of the things that have we've moved past or we've built on top of then come into this mixture of things, or this combination of all these trends or these fads that become fact together. And that to me is where you then end up with a whole lot of skills using an open weight model then get then enables you to use your loops and get a better result because you're using a tenth of the number of tokens or the cost of the tokens is a tenth of what of what it costs you using fable or opus but you end up with the result because would you have to if if it wasn't costing you your cloud subscription usage would you have turned your loops off?
Will: No.
Jim: Exactly. That would be fantastic. Because again, if it costs you...
Will: This is an area where I'm looking forward to the compute supply rapidly increasing. Yeah. Because just the quality of all of the AI outputs in the world should be able to improve just because we have the surplus to be able to check the work. That's it. And I've just...
Jim: I've gone... Because even bedrock... you know, bedrock can become a bit of a headache sometimes in all, you know, you've got 27 marketplace subscriptions for all the different models and the different versions of the models. And you've got to then manage the subscriptions and all these sorts of things. And that's like open router. You know, again, I just came back to how much I enjoy it again this week, just because for exactly this scenario, I've, I've, come down the path a bit more to, I just want to go, okay, specific tasks, this goes to this model, and we record all the responses and improve them over time.
Alex: I think we've talked about all these ideas kind of growing out from the very fundamental to the... by now they're all encompassing. And I think it's interesting how, as we've kind of evolved our practice over the years, the level at which we kind of operate with our system is always this level higher and this level more abstracted. We started focusing on these sort of things that were almost like purely concrete. And now we're dealing with these purely abstract systems of constraints and restrictions and, sort of abilities of the thing and not so much worrying about you know what are the exact things that I type into this and the technology ecosystems move with this as well and an anecdote I keep telling people because it's kind of true for anyone leveling up in their practices you get to this point where you're thinking at the level of harnesses where you're thinking about How do I connect these things in safely? How do I prevent it from doing things that I don't want it to do? Where what you actually put into the model kind of ceases to matter. And I remember showing someone this at work because they were like, yeah, but okay, how do you prompt? What's the right thing to prompt? It's like, no, don't think about the prompt.
Will: Think about everything else around it. I almost said this in the prompt engineering section. My prompts are pathetic now because my harness is so good. I just open a new session and say continue.
Alex: Yeah, I spat something absolutely garbage, like the most face roll, you know, concave brain thing I could think of into this. And it went off and built this entire feature. And he was like, yeah, but that prompt that you put in, what's so special about it? And it's like, it's really hard to break out of some of these older stories and these older ways of focusing on the thing immediately in front of you and seeing the system as a whole. if we take nothing else from the current AI boom, if the whole thing bubbles out, right? If we take a systemic approach to what we're trying to do and start to think about improving the system iteratively and not just the immediate thing that's in front of us, the whole malinvestment will have been worth it.
Jim: I believe that's going to be the delineation between who really succeeds and who doesn't. Oh, for sure.
Will: In utilising AI. This is a room of systems thinkers advocating for everyone to be systems thinkers.
Jim: But I just think... Well, they should sit on a tack. And, again, I can see the path so clearly, though. Like, that's the thing, is that, yes... We are biased, Will, if you will. I'm not. But it is just so clear now that if you have the process, then it can cost you nothing to achieve what you're trying to achieve. And so you can then do anything with your business. Well said, Jim. Well, that's called that. Who's your business idiot out of all this?
Will: The leaders who set up the KPIs for TokenMax.
Jim: Yeah, I'll give you that.
Alex: Yeah, no, I got nothing better. It's got to be them.
Jim: It's hard to think. I think it's where the three of us really started to clarify what a business idiot was.
Will: Yeah, for sure. It felt like the ape NFT moment for AI.
Alex: I'll be different here. I'll say the CEO who set up a digital twin of himself. If we could give away a cock of the week, that would also go to him.
Will: Off mic, I'll tell you the name of who that was and you'll be astonished.