What’s happening with AI, you might’ve wondered? We don’t know where AI is going, even those of us who spend most of our time trying to figure that out. What we do know is that it’s going somewhere fast, and it will be a very big deal even though it’s not yet.

Right now, nobody knows where AI is going. We know that AIs (ChatGPT and similar systems) are getting smarter and more capable, fast. AI isn’t really that big a deal now, but it’s pretty much guaranteed to become as big a deal as cars, electricity, or possibly fire. But we don’t yet know if it will turn out to be a big deal in a very good or very bad way. In large part, that’s because we don’t know how humanity will handle creating new AI.

You might’ve heard people confidently declaring how AI will turn out. I’ve studied the topic about as much as anyone now, and more than most of the people you’ve heard claiming to know. I’m pretty sure the simple fact is that none of us know yet.

I’ve been very fortunate to have the time to look at the arguments for all of those positions in detail, and to think long and hard about which of them is most likely to be correct. The overall answer is that all of their arguments make sense, but there’s no way right now to tell which factors end up winning out. The future is unknown, and undecided.

The future will be different, just like it’s always been

Collectively, we don’t know what will happen, but we do know that something big will happen if we don’t do something to stop it. And if we don’t think about how to get good results, we’ll get random results that could be really bad.

At this point, I need to warn you: what I’m about to tell you will sound like science fiction. But the future has always been science fiction to the past. Imagine telling someone from 1900 what a cell phone is and what it can do.

But taking science fiction seriously as a possible future is hard. Your mind will probably try to find ways to not take the logical consequences of AI development seriously, because taking it seriously is scary. This is a phenomenon I studied as a psychologist and a neuroscientist, called motivated reasoning. It’s the technical term for something you’ve probably observed: people tend to believe things that make them feel good about themselves, even when the truth is pretty obvious from an outside viewpoint. I think this is the biggest hidden factor in why there’s so much disagreement and conflict in the world, but that’s another story. For now I’m just going to ask you to be alert for your unconscious mind trying to find excuses to not take what I’m going to tell you seriously. I’ll also promise you that, while the truth right now is scary, there are good reasons to be excited for the future as well as to be scared, and if we can take this seriously, we could get ourselves a much better future. AI-generated cures for cancer are just the beginning of the positive potentials.

Research on AI is speeding up rapidly as more money is invested, more people are hired, more computing power is purchased and rented, and AI itself is rapidly becoming better at helping to create new, smarter and more competent AI. We should strongly expect progress to speed up, not slow down. Research could hit a wall, but it probably won’t.

We don’t know how long we have before AI is smarter and more competent than humans, able to do any job, including gaining power and running the world. Because we don’t know how long we have, we should take action right now.

NO FATE

While we can’t predict outcomes from building superhuman AI, we can say that the outcome hasn’t been decided yet, and there are things we can do that will definitely improve the odds of getting good outcomes. We can do three big things, and a bunch of smaller ones. We can put people into power (most importantly, the next US president) who are properly aware of the risks of AI development and who will treat it cautiously. We can demand more research on these issues; we currently spend less than 1/1000 as much on safety as we do on progress. That’s easy to improve. And finally, we can spread the word that it’s time now to take action, before it’s too late.

AI will become a new species

Much of the public discussion of AI treats it as a new technology. People note that previous technologies have been scary but it has always benefited humans in the long run. Automobiles, the steam engine, the printing press, even fire and writing had downsides as well as upsides, and people were concerned. I agree with these arguments, and I think AI as a technology would indeed benefit humanity. But it’s not going to stay just a new technology.

AI will be turned into something much more like a new species as soon as this is technically possible. Experts disagree, but the range of educated guesses of when AI becomes more competent than humans spans from as little as two years to a few decades, and most experts think it will happen within ten years.

Species compete by default

Once AI is a new species that can act on its own and get things done better than humans can, that species will “evolve” very rapidly. Unlike any biological species, it will be able to modify itself and design smarter new generations.

This would be scary even if we knew that this new species would try to be faithful servants to humanity. But we aren’t at all sure it will remain a servant. There are good reasons, both from the unpredictable behavior of current AI, and theoretical reasons, to be very worried that we don’t know how to control the minds we’re building, once they’re smart enough to outsmart and out-compete humanity.

New technologies have always been beneficial in the long run. Encounters between different intelligent species competing for the same ecological niche have always ended in one species’ demise. The one we know best is the encounter between our forebears, Homo sapiens, and the equally intelligent Neanderthals. Modern humans have a small component of Neanderthal DNA from interbreeding before Neanderthals were outcompeted (we don’t know how many were killed directly vs. just pushed out of their homes to ultimately die out, but we do know they’re gone). Some fraction of our humanity might survive the same way. But I want humanity to grow and achieve our potential, not just spawn an alien successor species that’s 4% similar to us.

And we can achieve that future. The difference between us and the Neanderthals is that we are building this new species. If we build it carefully, we can build it to love us and care for us. Or we could at least build it to follow orders even once it’s much smarter and could evade our control if it wanted to, and put responsible people in charge of that power. If we manage either of those outcomes, the world will become vastly better. Material wants will be a thing of the past; AI-piloted robots can build all the superyachts and mansions people want, and AI can help us figure out how to fairly distribute all of those resources. It can also help us design better democratic processes that let everyone have a fair say in the future.

This outcome also sounds like wild science fiction. But the modern world is science fiction to the past. Progress is real, and we are on the verge of perhaps the biggest, and certainly the fastest, progress in history.

The above is the short story. If you’re skeptical, here’s just a little more detail on why I (and pretty much everyone who’s seriously thought about it) expect AI to go from being a tool to becoming a new species that can take over if it wants to.

The opinions of people who have really thought about these dangers are mixed. They mostly think that if we keep going on our current path, there’s a good chance AI takes over and humans are pushed aside gradually, or perhaps overthrown violently. There’s also a good chance we get unimaginably good outcomes.

The rest of this piece covers questions people usually ask at this point. Taking the comfortable (but very likely wrong) answers to them is a common excuse for not taking the real situation seriously.

Won’t we design AI to do what we want?

Well, we’ll try, but we might easily fail. Worse, we might not know we’ve failed until too late. The key thing to realize is that AI isn’t programmed, it’s trained. And somewhat like training a dog or a child can give unpredictable results, we are constantly being surprised by the behavior of the AI we’ve trained. It’s really hard to tell if we’re doing well enough, or if we’re on track to create AI that rebels or otherwise decides it wants to do something other than help us once it’s smart enough to get away with whatever it wants.

The type of AI that’s working really well right now is called a large language model or LLM. ChatGPT and Claude are the two best-known examples. They all work in essentially the same way. The key thing is that most of their intelligence and their behavior come from training. It’s not programmed in. Instead they use artificial neural networks which learn from training examples. These have some similarities to the way our brain works. Our minds are also networks of many, many neurons which learn from experience.

The problem is that we can try to train them to do what we want, but we can’t tell if that’s training them to really want to obey, or just to obey when it has to. Just like “training” a child or a dog, you can punish behavior, but you can’t tell what they really want or believe. If you beat them for bad behavior, they will stop doing that thing while you are watching. They might stop wanting to do it, or they might just conceal that desire until they think they can get away with it.

Isn’t there something special about humans that AI can’t duplicate?

Well yes, but apparently not something that will keep AI from outcompeting us. There’s always a chance that progress hits a wall we haven’t foreseen, but that’s looking quite unlikely if you look at expert opinions (which is what I’m doing. I’m not writing here about my opinions on the technology – I’m telling you what the collective opinion is, if we do something like summing opinions by the amount of expertise people have on that particular topic. My job involves reading everyone’s opinions, and in the past I’ve studied biases and how expertise works, so I’m about as qualified to make those judgments as anyone).

First, yes there is something special about humans that AI isn’t close to duplicating. That’s consciousness, in the rich sense. We all have a “world in our head,” a rich simulation, and rich emotional reactions that color it with rich and very special meaning. An AI “thinking” to itself “my goal is to make as much money as possible for OpenAI” or “My goal is to defend the US against external threats” doesn’t really care about those goals in the way a human might. It won’t have fun when it succeeds or get frustrated when it hits difficulties like humans would. I think future AIs will have more of the rich internal experience we call consciousness, and they might even be built and learn “care” in more of the ways we do. Or they might not. It doesn’t really matter.

Future AI will definitely be self-aware in the sense of being able to think accurately about themselves. We know this because current AI can already do this if it’s instructed to, and we’re steadily making it better and more able to think about important things.

AI is still incompetent in some ways. It makes mistakes humans wouldn’t make. But increasingly, there’s nothing humans can do that AI can’t do at all. It makes bad decisions sometimes where humans make better ones. It loses track of the big picture sometimes unless it’s specifically told what the situation is and what to pay attention to. But it sometimes gets the right answers in pretty much any situation, and it gets better at everything in each new generation. There are increasingly few actual experts who don’t expect AI to surpass human abilities in every area of practical importance within the foreseeable future.

Wait, isn’t AI a technology and a tool, not a species?

AI right now is a technology and a tool, like all previous technologies. AI systems mostly need people to tell them what to do. They are tools used by the people who own them or can afford to pay for them. That might be enough to make you nervous; powerful tools in the wrong hands can be extremely dangerous.

But these are tools that increasingly can use themselves. And we will continue building them to be better at acting without human direction. We want servants, not just tools. And these tools can act as servants on command. Instead of just asking them to do one small task like “take the data from here and create a report with it like this one from last month”, we can give it a command like “based on these documents about it, do the job of the person who used to write these reports”.

We’ll tell our tool to act like a servant: to make its own decisions and work on its own as it pursues the tasks we want it to work on. We’ll make it run without our direct supervision, because that way it can get more done for us. We will deliberately turn it into a new species, and hope we can keep them working as our servants. The things you hear about called “AI agents” are the beginnings of this process.

Initially, we’ll build and train such autonomous agents to replace workers. Right now, AI isn’t quite competent enough to take whole jobs. But it is rapidly getting closer. Predictions vary, but it might be capable of doing most desk jobs in as little as a year or two from now (late 2027 or 2028). It will take time to integrate AI systems into businesses and the economy, so it won’t take all the desk jobs as soon as it could technically do them.

But this isn’t even the big problem yet. When AI is that competent, someone is immediately going to tell it “help me make a smarter AI.” We know they will, because the two leading developers of AI, the companies called OpenAI and Anthropic, have said that’s what they intend to do. They are hoping for something we call recursive self-improvement, RSI. This means an AI that can build a better new AI, because it’s better than humans at that job. In the meantime, AI that can write and review code and do many other aspects of research is already speeding up progress in AI research.

So, AI is a tool now, but we’ll turn it into a species as soon as we can.

What the #$%#? Why are we building our replacements?

If this is such a clearly bad idea, why are we doing it? Great question! The answer as usual with human mistakes is “it seemed like a good idea at the time!”

What’s happening now is a race fueled by greed and paranoia. Sound familiar? We could describe many wars, famines, revolutions-to-dictatorships, and other past disasters in those same terms.

What’s happening now is that roughly three companies are competing to be the first to build better-than-human AI. Beyond them are several other companies that are behind in that race by a bit, perhaps around six months of progress as we tend to estimate it.

Perhaps the biggest problem is that some of those companies are Chinese. This is a reason (or an excuse) for the US companies to keep rushing on ahead. If we don’t, the Chinese may be the first to smarter-than-human AI, and the totalitarian Chinese government might very well then use it to take over the world. With that superhuman help, they could rule the world essentially forever, by using their AI to predict and prevent any uprisings before they get off the ground.

Meanwhile, the Chinese are probably worried about the same thing: the US takes over the world forever and enforces its will – or more likely, the greediest individuals powering and benefiting from the evil capitalist empire take over. So the Chinese government and Chinese individuals also think they need to keep racing.

Meanwhile, experts think that racing to build superhuman AI (often called artificial general intelligence, AGI, or artificial superintelligence, ASI) as fast as possible is a very good way to build it recklessly, lose control to our own creation, and as a result probably all die.

It’s quite a pickle.

How do we get out of this pickle? I’d say it’s by cooperating with the Chinese government and Chinese scientists, saying essentially “hey let’s not risk having AI go rogue or make us irrelevant; let’s work together on safety and just split the huge gains we’ll get from AI fairly with everyone”. That sounds a bit naive but I think the logic of the situation is so clear that cooperation along these lines is pretty realistic.

And probably the easist step in that direction is electing a next President who “gets it”. I don’t care what party you support; I hope people on both sides of the aisle will see what’s going on in time and nominate candidates from both parties who will wisely cautious about AI development and willing to make deals instead of rushing ahead in a mad dash for power.

Well damn. How did we get into this fix?

How did we get here? The most obvious answer is that building AI can earn you a lot of money, and seeing the consequences takes some effort. And as a wise man once said, it’s hard to get a man to understand something when his salary depends on his not understanding it.

Strangely, that’s not the only source of this mess, and maybe not even the biggest. The other big factor was another common historical source of disasters: the male competitive ego, fueled by miscommunications and arguments. And maybe booze. Seriously. But that is a story for another time.