A possible AI counter-narrative

Are advances in the field of Artificial Intelligence going to slow down sooner than people think?

---

Introduction

Today, @cmp2020 sent me a message with a quote from an article like this one from Wired, OpenAI’s CEO Says the Age of Giant AI Models Is Already Over. According to this article, Sam Altman, the CEO of OpenAI, is now saying that the technique of just making bigger and bigger Large Language Models (LLMs) has reached a point of diminishing returns, and OpenAI is now looking for new directions to advance. Altman confirms that GPT-5 is not under development. On one hand, given all of the recent hype about the AI apocalypse, this is surprising.

On the other hand, I have already argued (privately) that the rapid growth of AI capabilities might slow down sooner than people think.

Off the top of my head, I can think of three reasons why AI's growth might start to level off after the recent explosion that we have seen. First, by posting AI content all over the Internet, people are polluting the training data for future AI generations; Second, future AI models will only get incremental increases in the amount of available data; and third, as Altman pointed out, the cost of increasing scale may be prohibitively expensive. Two of these have been obvious to me for quite a while, but it is only today that I realized the third obstacle.

Pollution of training data

Here on the Steem blockchain, we have had many conversations about the difficulty of distinguishing AI-generated content from human content. Of course, the AI systems of the future will have the same difficulty disambiguating the two. So, when this generation of AI systems pushes out errors and people publish it on the Internet, those errors will feed back into the training data for the next generation.

The more content the AIs produce, the more they're going to be choking on their own content. Any errors from the current generation will be amplified in the next generation. In the end, I can imagine that it might produce a feedback loop that gums everything up and dramatically slows down future iterations.

The low-hanging fruit has now been plucked

The current generation of LLMs appeared on the scene in an explosive way, but this is because they were able - for the first time, ever - to ingest a sizable percentage of the Internet. In short, these early generations of LLMs got to harvest all of the digitized data from people over the course of more than 30 years, and even centuries if you count ancient books that have been digitized.

Once the Internet has been fully-ingested, however, future data growth can only be incremental. We might not expect to see the same explosion from a year or two's worth of training data that we saw from ingesting data from decades or centuries. This means that future advances must depend far more on algorithmic improvements, and less on increasing data access.

In a metaphor, by ingesting massive amounts of data, the car went from 0 to 200 in 30 seconds, but advancing to 210 or 220 might take another 30 seconds or 60 or 90. And there might even be a maximum speed limit that no car can pass with currently known techniques.

On one hand, AI advocates have argued that people don't sufficiently appreciate the impact of exponential growth - and they're right. But, it may be a mistake to assume that exponential growth in this field will continue indefinitely.

In the book, Future Hype - The Myths of Technology Change, Bob Seidensticker argued for a spotlight & S-Curve model of technology. Basically (as I recall), he suggested that most technologies develop slowly at first, then hit a phase that looks like exponential growth, then finally level off to finish the S-Curve. People can be blind to this because the spotlight tends to shine on the technology during its phase of exponential (seeming) growth.

Dollars and Cents

According to the above-referenced article, OpenAI has already spent $100 million dollars building the current generation of models. Even to a well-funded company, that is a lot of money. It's not clearly stated, but the implication of the article (to me) is that getting to the next level in size is prohibitively expensive. We've been accustomed to technology that improves at the rate of Moore's Law, but who's to say that AI will advance at the same pace?

Certainly, for quantum computing, there has been an argument that Moore's Law does not apply. It's also conceivable that Moore's Law doesn't apply to progress in artificial intelligence.

Conclusion

Like the old quote says, "Making predictions is hard, especially about the future." When the whole world is poised to see the AI revolution unfolding in real time during the next couple of years, I'm not about to be the one who claims that they're all wrong.

I do think, however, that there are some reasons to wonder if the hype might be --- overhyped. To recap: there is a possibility that future AI training might choke on its own pollution; Future data ingestion will switch from a multi-decadal "big bang" to much smaller increments; and the cost of growth may not follow the Moore's Law path that we're all accustomed to where technology is concerned.

One last unrelated point that I might elaborate on in a future article has to do with AI & censorship. I just wanted to jot it down so it doesn't slip my mind. When Internet data is being used to train our AI companions, censorship becomes particularly dangerous. People complain about bias in AI, but there's really no such thing. The bias is in the training data. The problem of so-called AI bias, then, will be compounded by censorship. With all the risks of AI, this is one that I think deserves more awareness.

What do you think? AI rapid growth without bounds, or will things slow down?

---

Thank you for your time and attention.

As a general rule, I up-vote comments that demonstrate "proof of reading".

---

Steve Palmer is an IT professional with three decades of professional experience in data communications and information systems. He holds a bachelor's degree in mathematics, a master's degree in computer science, and a master's degree in information systems and technology management. He has been awarded 3 US patents.

---

^{Pixabay license, source}

Reminder

Visit the /promoted page and #burnsteem25 to support the inflation-fighters who are helping to enable decentralized regulation of Steem token supply growth.