Architecture and Construction of Large Language Models

These notes will take you through some of what goes into constructing and running modern large language models.

Many large language models are proprietary, and the companies that build them release technical details about the models only sparingly.

Fortunately, over the last few years a substantial number of organizations have release open source large language models. These organizations have been more open in sharing information about the architecture of their models and the process they used to train them. These notes will draw on a lot of this open source information.

A diverse array of LLMs

Over the last few years a wide variety of different LLMs have been built. These models vary considerably in their size, their internal architecture, and the datasets used to train them.

The main thing that explains this diversity of models is the fact that LLMs can be used for a wide variety of applications. The applications a particular LLM is designed for will determine many of the details of how that model is built.

Here is a list of common applications for LLMs:

These applications are listed in rough order of complexity, from easiest to hardest. Small LLMs can handle easier tasks readily, while large LLMs are aimed at performing more complex tasks.

Designing the architecture of an LLM

When an organization sets out to design an LLM the first thing they will determine is which potential applications they are planning to target. This choice will determine the scale and the architecture of the LLM, as well as the nature of the data that will be needed to train the LLM.

LLMs vary greatly in size. The standard measure of the complexity of an LLM is the number of trainable parameters in the model. Small LLMs with have something like one billion trainable parameters, while the largest LLMs will feature over one trillion trainable parameters.

The number of trainable parameters is also strongly correlated with the number of training tokens needed to train the model. A model will one billion trainable parameters will typically be trained on up to one trillion tokens of text, and datasets for larger models scale in a proportional way.

How to scale an LLM

The number of trainable parameters in an LLM is determined by a number of broad factors:

Different applications will drive these parameters to different values. For example, if the model only has to be able to process English language text, we can make the embedding dimension smaller. If the model needs to be able to process text in a large number of different languages, the embedding dimension will need to go up. For code generation applications the model many need to be able to generate new code based on a large body of existing code: this will drive the model toward having to process very long sequences. Many of the other design parameters will scale with the overall size of the model, which in turn is driven by the complexity of the tasks the model will have to perform.

Mixture of Experts (MoE)

One recent architectural development in LLMs is the use of the Mixture of Experts, or MoE architecture. A conventional, or dense, transformer layer often ends in one or more additional fully connected layers. In the MoE architecture these final linear layers are broken up into a large number of experts. These experts are designed for a variety of applications. When a sequence of tokens arrives at the MoE layer a router will decide which subset of the full set of experts to route the tokens to and then push the tokens through only those experts. All of the largest LLMs use this architecture to make both training and operation of the model more efficient, since any given input will pass through only a subset of the full model.

Example: gpt-oss

The gpt-oss models are open source models that were released by OpenAI in 2025. There are two models, gpt-oss-20b with 20 billion parameters and gpt-oss-120b with 120 billion parameters. The diagram below shows some of the architectural features of these models.

Training an LLM

Once the designers have made some broad architectural decisions about the model they want to build, the process of training the model starts.

Two training phases: pre-training and fine tuning

Every LLM starts with an initial phase of training, called pre-training. In this phase the model is trained on a wide range of examples of text. At this stage the only task the model is expected to perform is text completion. At this stage we can test the model by giving it some text to start with and ask it to generate more text from that starting text.

The second stage of training is called post-training, or fine tuning. In this stage we train the model to produce outputs that align more closely with the desired applications we want to have the model perform. This is done by training the model on a large number of examples that demonstrate these tasks. For example, if we are training the model to do question answering we will train it on a large number of question/answer pairs. If we are training the model to do code generation we will train it on a large number of prompt/code pairs.

Reasoning models

Another recent development in LLMs is the advent of reasoning models. These models are motivated by an observation that researchers made earlier in the history of LLMs: when we prompt an LLM to show the steps that it used to arrive at some result (called chain-of-thought prompting) it can very often produce better results. Reasoning models are specially trained to solve complex problems by breaking them into steps and working on each step in turn until the problem is solved. Reasoning models can also backtrack and rethink their approach to a problem if they think their current approach won't work out.

RLHF

Reasoning models also require new training methods. One recently developed training method used extensively with reasoning models is reinforcement learning through human feedback, or RLHF.

Here are the key steps in this method:

  1. Developers prepare a set of problems.
  2. For each problem the developers pose the problems to the LLM multiple times in an effort to get the LLM to generate multiple candidate solutions.
  3. The multiple candidate solutions are then given to expert human raters who rank the potential solutions from highest quality to lowest quality.
  4. The ratings from the raters are used to train a critic model that learns to evaluate LLM solutions.
  5. The LLM is trained on the same set of prompts with a custom loss function that runs the LLM's solution through the critic model to produce a rating.
  6. Over time the LLM learns to produce better answers.

Engineering for Scale

The largest LLMs can have anywhere from hundreds of billions to more than multiple trillions of trainable parameters. The sheer scale of these models makes training and running the models a formidable engineering challenge.

One of the most obvious problems with dealing with models at this scale is the fact that the model is so large that it can no longer fit on a single GPU. For example, one of the GPUs commonly used for training is the Nvidia H100 GPU, which has 80 gigabytes of video RAM. In training, the data type used for most model parameters is the bf16 data type, which is a 16 bit floating point data type. With this data type each parameter in the model takes up two bytes of RAM. This means that a single H100 GPU can store up to a maximum of only 40 billion model parameters at once, which is only enough to store a subset of the layers for any large model. To be able to fit the entire model into memory for training, the model has to be broken up and split across multiple H100 GPUs. Those GPUs then have to be networked together to enable information to flow quickly from one layer to the next.

One response to the engineering challenge posed by training and running very large models has been to apply architectural tricks to make training and operation more efficient.

Grouped query attention

One recent architectural trick targets multi-headed attention. In traditional multi-headed attention each transformer head maintains its own distinct set of Q, K, and V matrices. In the grouped query attention variant the multiple heads will share the same K and V matrices. The diagram below illustrates how this works:

This trick allows the transformer layers to reduce the total number of parameters needed while maintaining approximately the same level of performance.

Sliding window attention

Another architectural trick is useful in models that have to process long sequences of tokens. We saw earlier that generating models can make use of masked attention, which forces each token in a transformer layer to only pay attention to the tokens before it in the sequence. A variant of this strategy is to also limit how far back in the token sequence any given token can look.

The diagram below illustrates how this works in practice:

The main advantage of sliding window attention is that it can cut down on the number of calculations needed in the transformer layer.

Sliding window attention is also referred to as banded attention.

Quantization

Another trick that is aimed at making fully trained LLMs easier to run is quantization. In this technique the model is trained at full precision, typically using a data type like bf16, but then the precision of the parameters is reduced to an 8 bit data type, or even smaller data types. This allows more parameters to fit into less space while (hopefully) only marginally impacting the performance of the model.

An even more extreme version of this trick is to train the models using mixed data types that reuse information. For example, the gpt-oss models were trained using the MXFP4 data type, which is an approach that encodes floating point data using a variety of tricks to get the average parameter size down below four bits.

Further Reading

Here are some online resources where you can find more in-depth information on LLMs and how they are built.

  1. Sebastian Raschka has a substack where he writes widely on the topic of LLMs. Here is a detailed post he wrote on different LLM architectures.
  2. Jay Alammar has a nice blog post covering the architecture of the gpt-oss models.
  3. The Hugging Face Blog has a nice article on the MXFP4 data format used by the gpt-oss models.
  4. Hugging Face has published a huge guide detailing how they designed and built the SmolLM3 open source model they recently released.