AI Software Development – What Does The Data Say?

…

• The maximum effective context limits of LLMs… beyond which model outputs become unusably inaccurate is orders of magnitude smaller than advertised limits…
• LLMs cannot distinguish between recent and out-of-date information in the context, and information in the model itself, learned during training (“dominant priors”), can often “outweigh” information we give it…
• Repo-level .md files tend to make model performance worse…
• LLMs struggle with negation…
• LLM inference is more accurate when we give them examples (demonstrations) rather than just describing what we want…
• Large/long-scale industry studies show a clear trend – output is up (more code, more commits, bigger diffs), but outcomes don’t reflect that trend. If anything, the average team is taking longer to ship worse software…
• The psychological and cognitive factors in LLM use are a growing field of serious research. One study found a significant correlation between confidence in AI output and belief in the paranormal…
• Deep neural networks, including LLMs, struggle to learn patterns with long-range dependencies, at any scale of model…
• The energy and compute needed to train an LLM to be an order of magnitude more reliable – e.g., wrong 3% of the time instead of 30% – is 10^20 times what the current frontier models require. Don’t expect significantly more reliable models any time soon….
• …many published benchmarks that do indeed show LLMs getting better and better. But other research finds that we might wish to be more skeptical of benchmark performance… they’re not really like real-world problems…  [and] increasingly models are being “trained to the test”.

…

This is a really great summary of the state of the art in research into the efficacy of AI-driven software engineering, and it parallels with much of my experience. In particular, my personal experience has been that, properly-used, LLMs can be…

• …great at summarisation, which can make them helpful at tasks like parsing a large codebase to “trace” inconsistent logic, or explaining how a bug report represents a condition that is not covered by an automated test. This is most-valuable for unfamiliar codebases, where the developer’s intuition might be slower than the LLM’s token analysis: however this does introduce a risk compared to a more-systematic debugging process that an edge case (or parallel functionality) might be missed.
• …good at extrapolation. Given a solid example of what you’re trying to achieve, it’s relatively simple for an LLM to extrapolate that to other cases throughout your codebase. This can be useful when switching between many cases with slightly-differing logic, or when mirroring front- and back-end functionality in different programming language, where the LLM’s extrapolation capability may outperform the developer’s context-switching capability.
• …mediocre at creation: greenfield development from a “spec” to “code” is something LLMs appear to do pretty well, particularly for common and well-understood concepts (or, for “deep thinking” models, for tasks that they’re able to summarise correctly as a series of well-understood concepts). But when extending existing functionality, or when implementing functionality that is not well-represented by examples in the training data (e.g. truly new concepts, or use of new features of programming languages), LLMs routinely get stuck in flip-flopping loops, produce invalid solutions, or otherwise cause significantly more harm than they do good.

(They’re also okay for rubberducking, but so’s a rubber duck so that’s not really a good comparison.)

Since I first echoed Molly White’s observations that AI isn’t useless… but I’m not certain that they’re “worth it” several years ago, I’ve continued to experiment with different AI-assisted coding methodologies and technologies, but my fundamental position hasn’t changed: there are clearly some things that a coding LLM can help with, as described above, but there are problems when they get used for other tasks (which they routinely do). And I’m still not convinced that they’re “worth it”, and I’m concerned by those whose anthropomorphise GenAI and by the fact that a lot of industry is being built upon technologies whose runway is running out…

Anyway: great summary, Jason; thanks!