The Skills Of AI-Oriented Engineering

A recent Anthropic study established that some AI coding techniques hurt learning and understanding more than others. Unfortunately for us, the patterns that hurt understanding the most (AI delegation) are the ones we’d really like to use. It may well be that there is a sustainable path forward based on AI acting as a multiplier for human productivity (keeping the human engaged with the problem at hand), but techniques like spec-driven development may offer too much value to be passed up.

Let’s imagine for a moment that OpenAnthroMind has released GPT-6 Opus Pro and solved all the problems standing in the way of a delegation-based workflow. GPT-6 Opus Pro can stay on-task for over an hour, achieves high accuracy in identifying when it has completed a specification, and makes reasonable design decisions (within the scope of its tasks). When it encounters an issue it can’t solve in a reasonable amount of time or with a reasonable accuracy, it identifies that and reports back to the user. This seems like a reasonable minimal set of requirements to enable delegation workflows to function (please contact me if you think otherwise, I would like to hear).

What are the key skills of an engineer working with these tools day-to-day?

We need to break projects down into optimal chunks for coding agents to execute on. There are decisions to be made around agent accuracy (smaller tasks = higher chance of success), concurrency potential (many small tasks can be executed in parallel), ordering around dependencies, and giving the agent room to perform refactors (do large tasks give the agent freedom to refactor independently? Is it even capable of that?).

We will want to produce good specs for implementation. This means clearly scoped, not forgetting about key information or design constraints, no inconsistencies, etc. I expect this one to trip people up because identifying good changes to make to the codebase requires a deep understanding of that codebase. If there are two options that both look good on paper, but the application is architected so that one is a two-line change that fits into existing patterns and the other needs you to build a whole new framework, whoever makes the decision needs to know that. How will human engineers develop this knowledge without working in the codebase themselves? Good question.

Anyone wrangling AIs for a living will need to be on top of AI-driven development best practices. A lot of this comes from the existing software engineering discipline, but much is new. Are techniques like Docs In Code worth using? Scouts? Swarms? There is a whole field to develop here.

For the foreseeable future, the AIs are going to do the occasional oopsie-daisy. Human engineers will have to identify failure patterns. Not only bugs - agents today can’t consistently identify when an implementation is buggy even with extensive testing tools, but GPT-6 Opus Pro is no artifact of the present - but spotting when the agent is beginning to write itself into a corner or one of the other AI agent failure patterns.

It would be even better if we can let the AI prevent those issues itself, and that comes from good verification system design. An organization I previously worked in had a dedicated QA team that was on top of us devs. They had automated tests for every possible usage pattern that let me know if something I built wasn’t right or the spec was inconsistent. Simply using TDD myself (I do) wouldn’t be enough here; I am expected to make sure my own tests pass. This is a mechanism to make sure that my tests are testing the right thing. Even if the AIs could write exactly the tests we specify (they don’t) every time, we still need verification that we are specifying the right thing.

Finally, triage and recovery. Things will go horribly wrong. We will discover new problems the likes of which we’ve never even imagined. You will end up with million line repos full of slop. Identifying when an issue has occurred and designing a plan to fix it is not something new to software engineers, but the practicalities will surely change. Inheriting a huge legacy codebase which no one understands is an engineer’s nightmare today, but tomorrow it may be just another task on the list.

Each of these implicitly answer the question of: “How do you maintain competence and control over a system you don’t understand?” It is a shift from deep knowledge of the codebase to a solid idea of what the system should look like and deep knowledge of how to use AI to get there.

Generally, these skills are learnable today, and AI can help to some extent. For example, agents can help iterate on specs to identify missed points or inconsistencies. The engineers who get on top of these skills now are less likely to be caught flat-footed as models improve (they will) and the industry reinvents itself around them.

- omegastick