Foundation Models and Large Language Models

Domain 1 gave you a one-sentence version of a large language model: it predicts the next token, then does it again. That sentence is true, and it is not enough. It does not explain how a model reading "the trophy did not fit in the suitcase because it was too big" works out that "it" means the trophy. Nothing in the immediate neighborhood of "it" says so. The answer sits nine words back, and the model has to reach for it.

How a model reaches back across a sentence is the mechanism that made modern generative AI work at all. This lesson opens up the architecture, then uses it to separate two terms the exam likes to blur: foundation model and large language model.

What a foundation model actually is

The Amazon Bedrock glossary defines a foundation model as "an AI model with a large number of parameters and trained on a massive amount of diverse data. A foundation model can generate a variety of responses for a wide range of use cases. Foundation models can generate text or image, and can also convert input into embeddings."

Read that definition as three claims stacked together. Scale: billions of parameters, not thousands. Breadth: diverse data, not one curated task dataset. Adaptability: many use cases from one model.

The third claim is the one that changed how teams build. A traditional ML model is a single-purpose tool. You train a fraud classifier on labeled transactions, and it detects fraud. It cannot summarize a contract, and no amount of prompting will make it try. A foundation model inverts that. One model summarizes, translates, drafts code, and answers questions, steered by what you type rather than by retraining.

The economics follow from the architecture. Pre-training a foundation model costs millions of dollars and months of compute, which no ordinary team can absorb. But that cost is paid once by the model provider and then amortized across every customer and every task. That is why "pick a model and prompt it" replaced "train a model per problem" as the default starting point, and why the exam guide spends a whole domain on models you did not build.

The problem transformers solved

Before 2017, sequence models read text the way you read a fax coming out of a machine: one word at a time, left to right, carrying a running summary forward. That is a recurrent neural network. AWS puts the contrast plainly: "RNNs process data sequences one element at a time" while transformers "process entire sequences simultaneously," which gives "much faster training times and the ability to handle much longer sequences than RNNs."

Two things go wrong with reading one word at a time. First, it is slow, because step 500 cannot start until step 499 finishes, so you cannot spread the work across thousands of GPUs. Second, and worse, the running summary decays. By the time the RNN reaches "it was too big," the trophy has been squeezed through 9 rounds of compression and is barely present.

The transformer throws out the running summary. It loads the whole sequence at once and lets every token look directly at every other token. Nothing has to survive a relay.

Self-attention: deciding what matters

Self-attention is the mechanism that does the looking. AWS describes it as the mechanism that "enables the model to look at different parts of the sequence all at once and determine which parts are most important."

Think of a project meeting where everyone can hear everyone. When the topic turns to the deployment date, you tune in hard to the release engineer, half-listen to the designer, and tune out the room. You do not compress the meeting into a summary and then react to the summary. You react to the people, weighted by relevance to the current topic. Self-attention works this way: for each token, the model scores every other token for relevance and builds that token's meaning as a weighted blend of the ones that matter. The analogy breaks in one place worth naming: you tune in based on understanding the topic, while the model's relevance scores are learned statistical weights with no comprehension behind them.

Run it on the trophy sentence. When the model processes "it," self-attention scores every earlier token. "Trophy" and "suitcase" both score high because both are candidate nouns. Then "too big" pulls the balance toward "trophy," because the training data contains countless sentences where the thing that does not fit is the thing that is too big. Change one word to "because it was too small" and the same mechanism swings the weight to "suitcase." One architecture, no rules written by hand, and the resolution comes out of learned weights.

Positional encoding: putting order back in

Reading everything at once creates a new problem. If the model sees all tokens simultaneously, "the dog bit the man" and "the man bit the dog" arrive as the same bag of words. Order carries meaning, and parallel reading throws it away.

The fix is positional encoding. As AWS describes it, "positional encoding adds information to each token's embedding to indicate its position in the sequence," because "the model itself doesn't inherently process sequential data in order." Each token's numeric representation gets a position signal mixed in, so token 4 and token 40 are distinguishable even when processed in the same instant.

This is a good example of an architecture trade being paid for elsewhere. Parallelism bought speed and long-range reach, and positional encoding is the bill.

Encoder and decoder

AWS describes the full architecture in two halves. The encoder "reads and processes the entire input data sequence" and transforms it into "a compact mathematical representation." The decoder "takes this summary and, step by step, generates the output sequence."

Not every model uses both halves. Models built for generating text, the GPT family among them, are decoder-only: they take the tokens so far and produce the next one. Models built for producing embeddings lean on the encoder, because their job ends at the compact numeric representation and never becomes text. That split matters for the next lesson, where embedding models earn their own section.

Parameters: what the model learned

During training, a transformer adjusts millions or billions of numbers until its predictions stop improving. AWS names them directly: "weights and biases along with embeddings are known as model parameters. Large transformer-based neural networks can have billions and billions of parameters." AWS cites GPT-3 at 175 billion parameters and Jurassic-1 at 178 billion as examples of the scale.

Here is the misconception worth killing now, because it drives wrong answers throughout this course. Parameters are not a compressed copy of the training data, and the model has no index it can look things up in. Those 175 billion numbers encode statistical regularities: which tokens follow which, which words behave alike, which shapes of sentence are grammatical. When an LLM tells you a fact, it is not retrieving a stored document, it is generating the continuation those weights make most likely. That is the whole mechanism behind hallucination, and it is why the exam keeps pairing foundation models with retrieval augmentation and grounding.

Parameter count is a rough proxy for capability, not a guarantee of it. A larger model generally handles harder reasoning, and it also costs more per token and responds more slowly. Domain 2 comes back to that trade repeatedly.

Pre-training without labels

Traditional supervised learning needs labeled examples: this transaction is fraud, this one is not. Labeling is slow and expensive, and it is the reason traditional ML datasets stay small.

Foundation models sidestep it. AWS describes transformer LLMs as "capable of unsupervised training" through "self-learning," where they "learn to understand basic grammar, languages, and knowledge." The trick is that raw text is its own answer key. Hide a word and ask the model to predict it, and you have a training example with a correct answer, generated for free. Do that across a very large text corpus and grammar, facts, style, and reasoning patterns all fall out of the same objective.

This is why foundation models can be trained on broad, mostly unlabeled data while a fraud classifier cannot. Human labeling does not disappear entirely, but it moves later in the process, into fine-tuning and alignment, where a much smaller curated dataset shapes the model's behavior. The FM lifecycle lesson comes back to exactly this split.

Foundation model, LLM, and traditional model

These three terms sit at different levels, and the exam tests whether you can keep them apart.

The relationship runs one way. Every LLM is a foundation model, but not every foundation model is an LLM. An image generator and a text embedding model are both foundation models, and neither is a large language model. If a question describes a model that turns input into vectors for similarity search, "LLM" is the wrong label even though the model is large and trained on text.

Exam tips

• The words "broad," "diverse," "unlabeled," and "adapted to many tasks" point at foundation model. A model trained on one labeled dataset for one job is traditional ML, no matter how large it is.
• "LLM" is the narrower term. If the scenario mentions images, video, or embeddings as the output, answer foundation model, not LLM. Options that treat the two as interchangeable are usually the trap.
• For transformer questions, the differentiator is parallel processing of the whole sequence, with self-attention weighting relevance and positional encoding restoring order. If an option says a transformer processes tokens one at a time, it is describing an RNN.
• Expect a distractor claiming pre-training needs labeled data. It does not. Self-supervised pre-training on unlabeled text is exactly what makes foundation-model scale possible.
• Do not read parameter count as an accuracy guarantee. Bigger models generally reason better and always cost more per token, and none of them stop hallucinating.

The idea to carry forward: a foundation model is a very large pile of learned weights that predicts plausible continuations, and everything you build on top of it is an attempt to steer those predictions toward something useful and true. To steer them, you first have to know what the model is actually consuming, which is not words. The next lesson breaks the input down into tokens, chunks, and embeddings.