of how AIs "know" how to select and arrange words to create their responses?
I just looked at another article, this one from Microsoft, stating that:
Pretraining is where the model learns the bulk of its knowledge. The model is fed massive amounts of text from the internet — books, articles, code, websites — and learns to predict the next token given all previous tokens. This stage requires enormous compute (thousands of GPUs for weeks or months) and produces a base model.
A base model is essentially a text-completion engine. Given a prompt, it generates plausible continuations based on patterns in the training data.
* * * * *
Your full prompt (system message, conversation history, user input) is converted into tokens and fed into the model. The model processes all input tokens and produces a probability distribution over its vocabulary — predicting which token is most likely to come next.
(From
https://learn.microsoft.com/en-us/agent-framework/journey/llm-fundamentals ).
Or from IBM:
LLMs work as giant statistical prediction machines that repeatedly predict the next word in a sequence. They learn patterns in their text and generate language that follows those patterns.
(
https://www.ibm.com/think/topics/large-language-models)
I apologize if I'm not using all the terms exactly correctly, but the point I'm trying to make is that in the course of an LLM's training, it has somehow registered large and small patterns of how frequently words or phrases do or don't occur together and in what order. It has no understanding of what a vase is or what flowers actually are, but it can assemble the phrase, "a vase of flowers," because it has registered that statistically, vases more often contain flowers than trees or cookies. It's basically a statistical, probabilistic operation.
= new reply since forum marked as read