The majority of users interact with LLMs by purchasing access to the LLM from a commercial provider such as OpenAI, Anthropic, or Google. Since those providers typically provide all of the software needed to run and interact with the LLM users get little real insight into what goes on behind the scenes.
A better way to gain an appreciation for how LLMs work is to set up and run your own LLM. The advent of open source LLMs, along with a new ecosystem of tools designed to support these LLMs, has made it easier for us to "look inside the box" to gain a more detailed appreciation for how LLMs work.
In these notes I am going to introduce you to some of the tools in this ecosystem. Along the way will learn more about how LLMs actually work, and what is needed to make a fully functional LLM system.
To run an LLM you will need specialized software. The first open source LLMs were released with software designed to allow users to download and run those LLMs on their own machines. The first significant family of open source models was the Llama series released by Meta. When the Llama models were released they came along with a software tool called Ollama that would allow users to run the models.
Over time a new ecosystem of software systems designed for running LLMs locally emerged. One of the leaders in that ecosystem is llama.cpp. Here at Lawrence I have used llama.cpp to set up a couple of servers that run the gpt-oss models using llama.cpp.
The llama.cpp server software runs these models and accepts requests to run interactions with the LLM. Each interaction consists of a user request to run a prompt and a response by the LLM to that prompt.
Not only do large models take up a lot of space, but they also require a lot of computation to process their inputs. An number of optimizations have been developed to make computations in LLMs go faster.
One of the most common optimizations is called kv caching. This technique is motivated by the way that LLMs work. LLMs generate one token at a time. To generate a long sequence of tokens we have to make multiple passes through the model, one for each token we want to generate. As we do those multiple passes earlier tokens are going to make multiple trips through the model. When those tokens arrive at a transformer layer their embedding vectors have to be multiplied by Q, K, and V matrices.
If we are going to be making multiple passes in which the same embedding vector is going to repeatedly get multiplied by the same matrices, we can apply an obvious optimization: we can store those products in a cache and then just look the products up in the cache the next time that same embedding vector arrives at the same transformer layer. This caching strategy, called kv caching, trades a little bit of additional storage space off for a substantial savings in computational effort. (Here the k stands for 'key' and v stands for 'value'. In most applications the key vector is an embedding vector, and the value is the result of multiplying that embedding vector by one of the Q, K, or V matrices.)
kv caching is available as an option in llama.cpp. When users are setting up an LLM they need to decide whether or not to invoke the option, because the kv cache can take up a considerable amount of space on a GPU.
Systems like llama.cpp are not designed for end users to interact with directly. End users will instead interact with the LLM via one of two broad classes of intermediaries. One of these classes are specialized client applications that provide a user with a graphical user interface through which they can interact with the LLM. These client applications also provide services beyond simply providing a GUI.
One essential service that LLM client applications have to provide is conversation state memory. LLM servers like llama.cpp are designed to be stateless. This means that if you send a llama.cpp server a prompt and get back a reply, the server will not maintain a record of either the prompt or the reply it sent back. This means that when you interact directly with the LLM server you will find it impossible to carry on a conversation. To enable things like conversations with multiple rounds, client applications maintain a transcript of the conversation for you. Each time that user types a new query the client software will send both the transcript and the new query to the LLM server, so the LLM can consult the transcript when formulating a reply to the most recent query. This creates the illusion that LLM is successfully carrying on a conversation with the user.
In the lecture I am going to show two examples of GUI LLM clients, AnythingLLM and LM Studio.
Another way to interact with an LLM is to use a framework that allows you to construct applications that communicate with the LLM.
One such framework is langchain/langgraph. I am currently teaching a course on AI agents in which students learn to write applications in this framework that interact with one or more LLMs to get useful work done.
In the lecture I going to show some example code to illustrate how this works.
If you have done any work with LLMs you know that to get useful information from an LLM you have to prepare a text prompt that tells the LLM what you want.
There is much more going in those interactions than simply the prompt that you provide. In this section of these notes we are going to go behind the scenes to take a closer look at what goes on when you prompt an LLM.
When you send a prompt to an LLM, the LLM actually sees more than just the prompt you provided. Here is what the LLM is going to get in a typical interaction:
Experience has shown that constructing an effective prompt for an LLM is sometimes hard to do. In fact, an entire subfield, called prompt engineering, has sprung up to address this problem.
To give you a sense for what prompt engineering entails, here is a link to a getting started guide from Google.
When you ask an LLM a question, you have to hope that it can generate the answer from its built-in knowledge and/or the context you provided in your prompt.
In some cases there will be gaps in the LLM's knowledge, and it will not be able to answer some questions without additional help.
Here are some instances where gaps may occur:
To fill some of these knowledge gaps, most LLMs are designed to make use of additional information resources, called tools. Most LLMs gain access to tools through something that looks a lot like a function call mechanism. If you send an LLM a query and offer it a list of tools to call if it needs extra help answering, the LLM can arrange to call an appropriate tool to help it. For example, if you want an LLM to reason about things happening in time you may want to offer it a tool it can call to determine the current time. If you can anticipate that an LLM may have to do some arithmetic to answer a question, you can offer it a calculator tool it can invoke to do arithmetic on demand. If you need to give an LLM access to private data, you can offer it a tool that it can use to query that private data on demand.
In the lecture I am going to show an example from the AI agents class that demonstrates how we can offer an LLM tools in an application.
Another technique we can use when prompting an LLM is proactively inserting additional context into the prompt that the LLM can use to answer a question and then specifying in the prompt that the LLM should only use the context information provided to answer the question.
The technique has evolved into an entire subfield of AI agent building called retrieval augmented generation, or RAG. In this technique developers who write RAG applications collect a large body of local information and supplement that information with a search mechanism that can find relevant documents or parts of documents from that knowledge base that might be useful in answering a user query. The application will then send the user's query along with a set of documents that the application thinks might be useful to the LLM along with instructions that the LLM should only refer to the supplied documents when answering the user's query.
In the AI agents course we spent quite a few weeks exploring a wide range of RAG techniques.
A more extreme alternative to building RAG applications is embedding custom knowledge directly into the LLM you are using. The process for doing this is called fine tuning. In this process we freeze most of the weights in the LLM and then supplement those weights with a smaller set of trainable parameters. We then train the LLM on our own collection of custom data.
There are several toolkits available for fine tuning. Hugging Face offers the PEFT library along with extensive documentation on how to use it to fine tune models. Unsloth offers their own toolkit and guide for fine tuning.