It explains the LLM/NN. If you want to explain why it emits words in a certain order you need to explain how reality generated the dataset, ie., you need to explain how people communicate (and so on).
There is no mystery why an NN trained on the night sky would generate nightsky-like photos; the mystery is why those photos have those patterns... solving that is called astrophysics.
Why do people, in reasoning through physics problems, write symbols in a certain order? Well, explain physics, reasoning, mathematical notation, and so on. The ordering of the symbols gives rise to a certain utility of immitating that order -- but it isnt explained by that order. That's circular: "LLMs generate text in the order they do, because that's the order of the text they were given"
If the LLM is capable of rewording the MIT license into a set of hard-hitting rap battle lyrics, but the training dataset didn't contain any examples of anyone doing that, is the LLM therefore capable of producing output beyond the limits of its training data set?
Is an LLM inherently constrained to mediocrity? If an LLM were writing a novel, does its design force it to produce cliche characters and predictable plotlines? If applied in science, are they inherently incapable of advancing the boundaries of human knowledge?
Why transformers instead of, say, LSTMs?
Must attention be multi-headed? Why can't the model have a simpler architecture, allowing such implementation details to emerge from the training data?
Must they be so big that leading performance is only in the hands of multi-billion-dollar corporations?
What's going on with language handling? Are facts learned in an abstract enough way that they can cross language barriers? Should a model produce different statements of fact when questioned in different languages? Does France need a French-language LLM?
Is it reasonable to expect models to perform basic arithmetic accurately? What about summarising long documents?
Why is it that I can ask questions with misspellings, but get answers with largely correct spelling? If misspellings were in the training data, why aren't they in the output? Does the cleverness that stops LLMs from learning misspellings from the training data also stop them from learning other common mistakes?
If LLMs can be trained to be polite despite having examples of impoliteness in their training data, can they also be trained to not be racist, despite having examples of racism in their training data?
Can a model learn a fact that is very rarely present in the training data - like an interesting result in an obscure academic paper? Or must a fact be widely known and oft-repeated in order to be learned?
Merely saying "it predicts the next word" doesn't really explain much at all.
Which conditional probability sequences can be exploited for engineering utility cannot be known ahead of time; nor is it explained by the NN. It's explained by investigating how the data was created by people.
Train a NN to generate pictures of the nightsky: which can be used for navigation? Who knows, ahead of time. The only way of knowing is to have an explanation of how the solar system works and then check the pictures are accurate enough.
The NN which generates photos of the nightsky has nothing in it that explains the solar system, nor does any aspect of an NN model the solar system. The photos it was trained on happened to have their pixels arranged in that order.
Why those arrangements occur is explained by astrophysics.
If you want to understand what ChatGPT can do, you need to ask OpenAI for their training data and then perform scientific investigations of its structure and how that structure came to be.
Talking in terms of the NN model is propaganda and pseudoscience: the NN didnt arrange the pixels, gravity did. Likewise, the NN isnt arranging rap lyrics in that order because it's rapping: singers are.
There is no actual mystery here. It's just we are prevented form access to the data by OpenAI, and struggle to explain reality which generated that data -- which requires years of actual science.
It has a lot of things already encoded regarding the solar system, but it cannot really access it, it cannot - as far as I know - run functions on its own internal encoded data, right? If it does something like that, it's because it learned that higher-level pattern based on training data.
The problem with NN arrangements in general is that we don't know if it's actually pulling out some exact training data (or a useful so-far-unseen pattern from the data!) or it's some distorted confabulation. (Clever Hans all again. If I ask ChatGPT to code me a nodeJS IMAP backup program it does, but the package it gleeful imports/require()s is made up.
And while the typical artsy arts have loose rules, where making up new shit based on what people wish for is basically the only one, in other contexts that's a hard no-no.
There is no mystery why an NN trained on the night sky would generate nightsky-like photos; the mystery is why those photos have those patterns... solving that is called astrophysics.
Why do people, in reasoning through physics problems, write symbols in a certain order? Well, explain physics, reasoning, mathematical notation, and so on. The ordering of the symbols gives rise to a certain utility of immitating that order -- but it isnt explained by that order. That's circular: "LLMs generate text in the order they do, because that's the order of the text they were given"