
“Hot turquoise eels doth flabbergast escapees”—this is an example of a sentence that AI would not be able to generate on its own. This is because I am choosing the words based on the message I want to convey. On the other hand, AI chooses the tokens that are most likely to follow the preceding tokens (roughly). Because of this, AI is unable to generate completely original sentences: it must follow established patterns from its sample data.
Furthermore, evidence against this idea can be found in human dialectal variation. People say things in different ways. Some people call it a “soda”, some people call it a “coke”, others call it a “pop”. There is no dialectal variation in ChatGPT and similar applications because it relies on its sample data to produce a response. In other words, it cannot innovate speech patterns (or ideas) as humans do; it can only aggregate human-produced text and make predictions based on it.
LLMs cannot create original thoughts, they can only read off of human thoughts that are already on the internet or in whatever data set they were trained on. If Vincent van Gogh had access to chatGPT or Gemini or whatever “AI” of your choosing, there is no prompt he could feed to it that would make it create Starry Night rather than painting it himself, because it would not have the source material to try to emulate