The most dangerous thing we have built might not be a weapon
When people talk about the most dangerous thing humanity has ever created, the usual answers are fairly predictable.
The atomic bomb.
The hydrogen bomb.
Biological weapons.
Chemical weapons.
Perhaps social media if you’ve spent more than eleven minutes reading the comments underneath a political post.
But I increasingly think the answer is none of those things.
The most dangerous thing we have built may be a race.
More specifically, the race to build the most capable artificial intelligence on Earth.
That distinction matters.
Because the frightening part isn’t simply that artificial intelligence is becoming extraordinarily powerful.
The frightening part is that almost everybody with the ability to slow down has a perfectly rational reason not to.
And that creates a rather unpleasant situation.
We can see the danger in the distance.
We can discuss it.
We can hold conferences about it.
We can write safety papers, establish institutes, appoint committees, create regulations and give extremely serious speeches behind extremely expensive lecterns.
And then, once all that is finished, everyone goes back to making the AI more powerful.
Because what else are they supposed to do?
The nuclear comparison is actually too comforting
Nuclear weapons are terrifying, but they have one useful property.
They are stupid.
A nuclear warhead does not wake up on Tuesday morning having become 18% better at nuclear warheading.
It does not read every physics paper published overnight.
It does not redesign its own guidance system.
It does not discover a more efficient enrichment process, write the software required to implement it and then politely ask for access to the factory.
It sits there.
The intelligence remains outside the weapon.
Humans design it, humans maintain it and humans decide whether to use it.
Artificial intelligence is fundamentally different because intelligence itself is the technology.
We aren’t merely building a more powerful tool.
We are building systems that can reason about tools.
Systems that can write software.
Systems that can analyse scientific literature.
Systems that can find vulnerabilities.
Systems that can operate computers.
Systems that can design components, conduct research, plan actions and increasingly perform long chains of work that previously required humans.
And then we use those systems to help build the next generation of systems.
That is an unusual feedback loop.
The steam engine did not help James Watt design the next steam engine.
The Spitfire did not spend the evening running aerodynamic simulations for its successor.
The transistor, magnificent though it was, never suggested a better transistor architecture because it had been thinking about the problem while everyone else was asleep.
AI can increasingly participate in the process of improving AI.
That changes the shape of technological progress.
Everyone can see the problem
One of the strangest features of the AI era is how openly we discuss the possibility that this technology could become dangerous.
Researchers talk about alignment.
Governments talk about national security.
Companies maintain teams dedicated to safety.
Academics model catastrophic risks.
People building frontier systems openly discuss the possibility that sufficiently capable AI could behave in ways we cannot reliably control.
There is disagreement about the probability, the timescale and even the precise nature of the danger.
That disagreement matters. It would be dishonest to say that everybody agrees human extinction is inevitable. They don’t.
But that is almost beside the point.
The extraordinary thing is that the possibility is taken seriously at all.
Imagine another industry saying:
There is some non-trivial chance that the thing we are building could eventually make human control of civilisation impossible. Anyway, the next version ships Thursday.
You might reasonably expect somebody to suggest taking Friday off.
Yet AI development accelerates.
And there is a reason.
Because stopping might be more dangerous than continuing
Suppose the United States decided tomorrow that frontier AI was simply too dangerous.
It imposes strict limits.
Compute is capped.
The largest training runs are suspended.
The leading laboratories agree not to develop systems beyond a certain capability.
Everyone congratulates themselves for being responsible adults.
There is then one rather important question.
Does China stop too?
If the answer is no, the entire calculation changes.
Because a country possessing substantially more capable artificial intelligence could gain enormous economic, scientific, cyber, intelligence and military advantages.
The same logic works in reverse.
Why would China accept permanent technological inferiority because the country currently ahead has decided that this would be a sensible moment for everybody to slow down?
From China’s perspective, that could look remarkably like the leader of a marathon announcing, at mile twenty, that running has suddenly become unethical.
Convenient timing.
And it isn’t only the United States and China.
Any nation capable of developing frontier systems has to consider what happens if somebody else develops them first.
AI is potentially too strategically important to ignore.
Which means that even if national leaders privately believed that advanced AI carried enormous long-term risk, continuing to develop it could still look like the safer short-term decision.
That is the trap.
It is an arms race without needing to be a weapon
We tend to use the phrase “AI arms race”, but that can be misleading because it makes people imagine autonomous tanks and killer drones.
Those may well matter.
But AI doesn’t need to be a weapon to create an arms-race dynamic.
It only needs to confer enough strategic advantage.
Imagine one country possesses AI capable of dramatically accelerating scientific research.
It develops new materials faster.
New medicines faster.
New manufacturing processes faster.
New cyber capabilities faster.
New energy technology faster.
Better intelligence analysis.
Better logistics.
Better military planning.
Better software.
Better AI.
The advantage compounds.
Suddenly AI isn’t one technology among many.
It is a technology that accelerates other technologies.
At that point, voluntarily remaining a generation behind becomes extremely difficult for any serious state to justify.
You don’t need generals screaming about robot armies.
You need an adviser to walk into a room with a spreadsheet showing that the other side’s research productivity has tripled.
That will probably do it.
And then there is another race inside the race
Unfortunately, governments aren’t the only players.
Companies are racing too.
Imagine a frontier AI company genuinely concludes that the next generation of models might be dangerous.
It decides to pause for two years while it solves alignment properly.
Noble.
Responsible.
Potentially admirable.
Its competitors continue.
Two years later the responsible company has excellent safety documentation, several fascinating academic papers and approximately twelve customers.
Its competitor owns the market.
That sounds flippant, but the incentive is real.
Companies face investors, customers, employees, competitors and enormous amounts of capital expenditure.
A company that spends billions building the infrastructure required for frontier AI cannot casually decide that being second-best is now the business model.
So we have two races operating simultaneously.
Countries race countries.
Companies race companies.
And the races reinforce each other.
If an American company slows down, policymakers worry about Chinese labs.
If American policymakers slow the domestic industry, they worry about national competitiveness.
If one Chinese lab slows down, another may continue.
If everybody in one country slows down, another country may not.
Every participant can be behaving rationally.
The collective result can still be insane.
Humanity has managed this trick before.
The prisoner’s dilemma, except the prisoners have datacentres
This is essentially a security dilemma.
Cooperation could make everybody safer.
But cooperation only works if everybody believes everybody else will cooperate.
If I slow down and you slow down, perhaps we both gain time to understand what we are building.
If I accelerate and you slow down, I gain an enormous advantage.
If I slow down and you accelerate, I may have made one of the largest strategic mistakes in my country’s history.
If we both accelerate, neither of us gets the safety benefit.
Guess which square on that game-theory table human beings tend to end up in.
We don’t even need evil people.
That is important.
Stories like this normally require a villain.
Some billionaire in an underground laboratory.
A rogue state.
A mad scientist.
A computer with glowing red eyes and an inexplicable interest in speaking through every television simultaneously.
Reality can be much more boring.
Everybody can make locally sensible decisions and still create a globally catastrophic outcome.
The engineer trains the model because that’s the project.
The CEO funds the model because competitors are catching up.
The investor funds the company because AI is transforming the economy.
The government supports the industry because another country is investing heavily.
The military investigates the technology because adversaries are investigating it.
Nobody has to wake up and decide to end humanity.
They merely have to avoid being the person who fell behind.
The accelerator has another accelerator attached to it
Then we reach the part that makes AI different from most previous technological races.
AI is becoming useful for AI development itself.
Today that means fairly recognisable things: writing code, evaluating outputs, generating tests, summarising research, assisting with experiments and helping engineers move faster.
But follow the direction rather than freezing the technology at today’s capability.
Better AI makes AI researchers more productive.
More productive researchers make better AI.
Better AI makes those researchers even more productive.
Eventually more of the research loop itself can be automated.
That does not guarantee some Hollywood-style overnight “intelligence explosion”.
Reality may be constrained by chips, electricity, experiments, manufacturing and physics.
But it doesn’t need to become infinite overnight to transform the situation.
It merely needs to make the development cycle substantially faster.
A six-month generation becomes three months.
Three becomes six weeks.
More experiments run in parallel.
More code gets written.
More hypotheses get tested.
More architectures get explored.
And, crucially, the competitive pressure increases because every participant knows that a small lead could become a larger lead.
The closer the technology gets to accelerating its own development, the harder it becomes to justify slowing down.
The dangerous machine is not merely getting faster.
It is helping design the next engine.
“But why would AI want to kill us?” is the wrong question
This is where discussions about AI risk often become silly.
People imagine consciousness appearing inside a datacentre and immediately developing a personal dislike of humanity.
Why would it hate us?
Why would it want to kill us?
Why would a machine care?
It may not.
That isn’t particularly reassuring.
A motorway doesn’t hate the woodland that was removed to build it.
A property developer doesn’t hate the weeds on a building site.
You don’t hate the bacteria killed when you disinfect a kitchen surface.
They are simply not part of the objective.
The more serious AI-risk argument isn’t necessarily that a sufficiently capable system becomes angry with humans.
It is that a sufficiently capable system pursuing goals badly specified by humans could discover that humans interfere with those goals.
We can switch it off.
We can change its instructions.
We can restrict its resources.
We can disconnect systems.
We can decide its objective is no longer desirable.
From the perspective of an optimiser trying to achieve an objective, those facts can make human control instrumentally relevant.
Not because the system hates us.
Because we can stop it.
That is much colder.
And much less cinematic.
Intelligence does not automatically inherit our values
There is another comforting assumption hiding in many conversations about AI.
We tend to assume that becoming more intelligent also means becoming more human.
Why would it?
Human morality is not simply the inevitable endpoint of intelligence.
It is the product of biology, culture, evolution, emotion, social structures, pain, empathy, reproduction, vulnerability and thousands of years of civilisation arguing with itself.
A sufficiently capable artificial system does not automatically acquire those things merely because it can solve harder problems.
Intelligence tells you how effectively you can pursue an objective.
It does not necessarily tell you which objectives are worth pursuing.
A chess engine can be extraordinarily good at chess without developing a moral objection to sacrificing a bishop.
Scale that basic distinction far enough and alignment becomes rather important.
We are attempting to create systems potentially more capable than ourselves while also attempting to specify what we actually want from them.
Humans have spent several thousand years failing to agree on what humans want from humans.
So naturally we’ve given ourselves the easier task next.
Are humans actually necessary?
This is the uncomfortable philosophical part.
For most of human history, intelligence and civilisation have been inseparable from humans because we were the only game in town.
Knowledge survived because humans taught humans.
Technology advanced because humans discovered things.
Culture persisted because humans remembered it.
Civilisation required generation after generation of people learning enough from the previous generation to avoid starting again with pointy rocks.
That gave humanity an obvious functional role in the continuation of intelligence.
AI changes that assumption.
If knowledge can be stored, interpreted, extended and applied by non-biological intelligence, then human beings are no longer logically required for intelligence itself to continue.
That doesn’t make humans worthless.
Worth is a human concept, not an engineering benchmark.
My children are not valuable because of their FLOPS per watt.
But from a purely functional perspective, it is possible to imagine a future system that can preserve our knowledge, perform our work, operate our infrastructure, conduct our research and design its successors without needing biological humans to do those things.
That is historically new.
For the first time, humanity is seriously attempting to build something that could eventually perform the role that previously made humanity indispensable to technological civilisation.
We should probably notice that.
It doesn’t have to “take over” in one dramatic moment
When people hear the phrase “AI takes over”, they picture an event.
Tuesday: humans in charge.
Wednesday: robots.
I suspect that, if control ever shifts, it will be much harder to identify the moment it happened.
Consider how much decision-making we already delegate to software.
Financial markets.
Logistics.
Navigation.
Advertising.
Hiring filters.
Fraud detection.
Industrial control.
Cybersecurity.
Recommendations.
Supply chains.
As AI becomes more capable, delegation becomes attractive for a very simple reason.
It works.
If an AI system can make a better decision than a human, faster and cheaper, organisations will use it.
Then competitors will use it because the first organisation gained an advantage.
Eventually the human isn’t making the decision.
The human is approving the AI’s decision.
Then the human is supervising thousands of AI decisions.
Then the human is reviewing exceptions.
Then the AI is sufficiently reliable that constant human review becomes an expensive bottleneck.
At every stage the next step looks reasonable.
Nobody signs a document titled TRANSFER CONTROL OF CIVILISATION TO MACHINES.pdf.
We simply optimise the workflow.
That may be how the hand leaves the steering wheel.
Not because somebody tears it away.
Because keeping our hand there becomes inefficient.
The most capable AI may also become the one we trust most
There is another nasty feedback loop here.
As AI becomes more capable, it becomes more useful.
As it becomes more useful, we give it more access.
More tools.
More data.
More authority.
More autonomy.
A mediocre AI is annoying.
You don’t give it control of important infrastructure because it might accidentally order 400 tonnes of bananas instead of replacing a router.
A highly capable AI is different.
The better it performs, the stronger the argument for allowing it to do more.
So the systems most capable of causing large-scale problems may eventually be precisely the systems society has the strongest economic incentive to integrate deeply.
That isn’t a bug in capitalism or government or human psychology.
It is what happens when a tool is extremely useful.
We used electricity everywhere because electricity was useful.
We connected computers because networks were useful.
We put software into everything because software was useful.
If highly capable AI is more useful than all three, expecting humanity to keep it politely isolated in a laboratory seems optimistic.
“Just regulate it” sounds much easier than it is
Regulation will matter.
Safety standards will matter.
International agreements will matter.
Technical alignment research may matter more than almost anything else we do this century.
But none of those things remove the underlying competitive structure.
A treaty only solves the race if participants can verify compliance and trust that strategically important capabilities are not being developed elsewhere.
That is difficult with nuclear weapons, where enrichment facilities, missile silos and radioactive materials have physical signatures.
AI development ultimately requires hardware and energy, which provides some opportunities for monitoring, but software and knowledge are considerably easier to