How to Run an LLM Locally: Offline AI Setup Guide

In short: You run an LLM locally by downloading an open-weight model once and then running it offline on your own computer or server with Ollama, llama.cpp, LM Studio or vLLM. So yes, offline AI is possible. An 8-billion-parameter model takes about 5 GB with 4-bit quantization and fits in 8 GB of VRAM. Models with 14 billion parameters need 12 GB at 4 bits.

For teams that do not want their data to leave the company, the answer is usually a local LLM: you download the model once, and from then on it runs on your own machine with no network connection, which is what offline AI means in practice. Ollama's Linux install guide gets the setup down to a single command. The real work is figuring out which GPU the model fits on and choosing the right tool.

Yes, AI can run without an internet connection. Cloud services such as ChatGPT send your request to the provider's servers. A local LLM (a large language model on your own hardware) means downloading the model's weight file to your machine and running it there. As long as the model sits on disk, you need no internet for chat, document summaries or coding help.

The vendors' own documentation says the same. According to LM Studio's Offline Operation page, chatting with a downloaded model, chatting with documents (RAG) and the local server work without the internet. A connection is needed to search for and download models, download runtimes and check for updates. The Ollama FAQ also states that Ollama does not see your prompts or data when it runs locally. Ollama goes online to download models, to fetch updates in its macOS and Windows versions, and for optional cloud features.

So offline AI has two stages. First you download the model once. After that the model runs even when the machine is cut off from the network.

A local LLM is a copy of an open-weight language model (one whose weights are published) running on your own hardware. With a cloud API, the model stays with the provider and you pay for the tokens you use. With a local setup, hardware, maintenance and updates are your job, but prompts and documents never leave your infrastructure. We explain the large language model itself and how to use it through an API in our guide to what an LLM is.

For an enterprise, three differences stand out. We cover how a company should choose between the two in Local LLM vs ChatGPT.

• Data flow: Prompts and responses stay on your own network. That is the starting point for any data residency discussion. We cover the broader question of who controls the data, the model and the infrastructure in our sovereign AI guide.
• Cost structure: Instead of a per-token fee, you pay for GPUs, servers and operations. Which one is cheaper depends on your usage volume.
• Model choice: You can only run models whose weights you can download, and every model has its own license. For example, the Qwen3-8B model card lists the model as Apache-2.0 licensed. We compare license terms in our post on open-weight model licenses.

The shortest route is Ollama. The steps below are for a Linux server, and Ollama has installers for Windows and macOS.

Ollama supports Windows 10 22H2 and later according to the Ollama Windows page, and macOS Sonoma (14) and later according to the Ollama macOS page. Macs with Apple M-series chips use the CPU and GPU together, while Intel Macs use only the CPU. vLLM's NVIDIA and AMD GPU support is Linux only. According to the vLLM GPU installation guide, you can use it on Windows through WSL.

1. Install Ollama. The official command from the Ollama Linux install guide is:

curl -fsSL https://ollama.com/install.sh | sh

2. If you use an NVIDIA card, confirm that the driver is installed with nvidia-smi.
3. Download and run the model. The example uses the 8-billion-parameter version of Qwen3.

ollama pull qwen3:8b
ollama run qwen3:8b

4. Check whether the model fits on the GPU. According to the Ollama FAQ, the PROCESSOR column in the ollama ps output shows "100% GPU", "100% CPU" or a split such as "48%/52% CPU/GPU".
5. Connect your applications. According to the Ollama OpenAI compatibility docs, Ollama serves a subset of the OpenAI API at http://localhost:11434/v1/. An application that uses an OpenAI client can be pointed at the local model just by changing the address.

In a fully isolated environment, you prepare the model on a machine with internet access and carry it over as files.

• Ollama: You can download and unpack the install package with the manual install steps in the Linux guide. For the model, copy over a GGUF file, write a Modelfile with a FROM /path/model.gguf line and import it with ollama create, as the Ollama import docs describe. You can switch off the cloud features with the OLLAMA_NO_CLOUD=1 setting from the FAQ.
• vLLM: According to the vLLM Engine Arguments docs, the --model parameter accepts a Hugging Face model name or a local folder path. According to the huggingface_hub environment variables docs, setting HF_HUB_OFFLINE=1 means no HTTP request goes to the Hugging Face Hub and only cached files are used. vLLM collects anonymous usage statistics by default. The vLLM usage stats docs show how to turn this off with VLLM_NO_USAGE_STATS=1 or DO_NOT_TRACK=1.
• LM Studio: Model files obtained outside the app can be sideloaded, as the Offline Operation page describes.

The VRAM you need depends on the model's parameter count, its quantization level and the context length, and you can roughly calculate it yourself.

The rule of thumb: VRAM ≈ parameter count × bytes per weight + KV cache + runtime overhead.

Hugging Face's Optimizing LLMs for Speed and Memory guide says that loading the weights of a model with X billion parameters takes roughly 4 × X GB of VRAM in float32 and roughly 2 × X GB in bfloat16 or float16. Quantization lowers that number. In the table measured on Llama-3.1-8B in the llama.cpp quantize README, Q4_K_M uses 4.89 bits per weight, Q8_0 uses 8.50 bits and F16 uses 16.00 bits.

Here is the rule on a real model. The Qwen3-8B model card gives 8.2 billion parameters. In this post, GB means 10⁹ bytes.

• At 16 bits: 8.2 × 2 = about 16.4 GB. The qwen3:8b-fp16 tag in the Ollama library is 16 GB.
• At Q4_K_M: 8.2 × 4.89 ÷ 8 = about 5.0 GB. Among the Ollama qwen3 tags, qwen3:8b-q4_K_M is 5.2 GB.

In both cases the rule lands close to the real file size. Quantization has a price, though. The Hugging Face guide describes it as a trade-off between memory efficiency and accuracy, and llama.cpp notes that quantization can cost some accuracy. We explain the details in our quantization explainer.

While the model generates text, it keeps a key vector and a value vector in memory for every token. This is the KV cache, and it grows linearly with context length. By the formula in the Hugging Face guide, the number of stored values is 2 × sequence length × number of layers × number of heads × head dimension. In models that use grouped-query attention (GQA), the smaller number of KV heads takes the place of the number of heads, and the memory need drops.

Qwen3-8B's configuration file lists 36 layers, 8 KV heads and a head dimension of 128. At 16 bits that is 2 × 36 × 8 × 128 × 2 bytes = 147,456 bytes per token. That gives the following figures.

Two settings change this number. According to the Ollama FAQ, the default context is 4,096 tokens and you change it with OLLAMA_CONTEXT_LENGTH. If you raise OLLAMA_NUM_PARALLEL to answer several requests at once, the memory needed grows with the number of parallel requests × context length. With Flash Attention on, setting the KV cache type to q8_0 with OLLAMA_KV_CACHE_TYPE cuts this memory roughly in half.

The table below adds the KV cache for the default 4,096-token context to the Qwen3 file sizes in the Ollama library. The totals are estimates and leave out the runtime's own memory overhead, so check real usage with ollama ps.

Parameter and layer details come from the Qwen3-14B and Qwen3-32B model cards. Here is the practical answer from this table.

• 8 GB VRAM: runs an 8-billion-class model comfortably at 4-bit quantization, with the default context and a single user. The Q8_0 version (8.9 GB) is over 8 GB on its own. If you allow four parallel requests, the KV cache grows to 2.4 GB and even the 4-bit model reaches the limit. For smaller cards, our Liquid AI LFM3 post covers which tasks models in the 2-billion-parameter class can handle.
• 12 GB VRAM: runs the 8-billion model at 8 bits, or at 4 bits with a long context (about 10 GB at 32,768 tokens), and the 14-billion model at 4 bits. For long-context work such as document summarization, 12 GB is the safer choice.
• 24 GB and above: needed to run the 32-billion class at 4 bits on a single card, or to serve a small model to several users at once.

A model that does not fit in VRAM still runs, but part of it is processed on the CPU. According to the llama.cpp README, mixed CPU+GPU inference exists to partially accelerate models larger than the VRAM. Be careful with mixture-of-experts (MoE) models. The qwen3:30b-a3b tag in Ollama is 19 GB at 4 bits: the active parameter count is small, but all the weights are loaded into memory. Our Mistral Large 4 guide explains why MoE models are fast to compute but expensive to keep in memory.

• Ollama: According to the Ollama hardware support page, NVIDIA cards need compute capability 5.0 or higher and driver 550 or later. Cards with compute capability from 5.0 to 6.2 need driver 570 or later. The list includes the RTX 3060 (8.6), RTX 4060 (8.9) and GTX 1080 (6.1). AMD Radeon cards are supported through ROCm v7, Apple GPUs through Metal and some other GPUs through Vulkan.
• vLLM: According to the vLLM GPU installation guide, NVIDIA cards need compute capability 7.5 or higher (T4, the RTX 20 series, A100, L4, H100 and similar). On AMD, the MI200, MI300 and MI350 and the Radeon RX 7900 and RX 9000 series are supported. So an older GTX 1080 works with Ollama but not with vLLM.
• LM Studio: According to the LM Studio system requirements page, Windows needs a CPU with AVX2 support, and at least 16 GB of RAM and at least 4 GB of dedicated VRAM are recommended. On a Mac it needs Apple Silicon and macOS 14. Intel Macs are not supported.

Ollama is built for a quick start on a single machine, vLLM for a multi-user production service. The choice usually comes down to how many people you will serve at the same time.

Ollama installs with a single command, comes with a ready model library and provides an OpenAI-compatible API. According to the Ollama FAQ, with default settings a model handles one request at a time (OLLAMA_NUM_PARALLEL defaults to 1), at most 512 requests wait in the queue, and a full queue returns a 503 error. It suits personal use, pilot projects and small teams.

vLLM is an inference engine built for high-concurrency serving. The vLLM documentation highlights efficient KV cache memory management with PagedAttention, continuous batching of incoming requests, tensor and pipeline parallelism, and an OpenAI-compatible API server. According to the vLLM Quickstart, the server starts at http://localhost:8000 by default and hosts one model at a time. It reserves most of the GPU memory up front and uses what the weights leave over for the KV cache. The gpu-memory-utilization setting in the Engine Arguments docs sets that share. vLLM supports quantization formats such as FP8, INT8, INT4, GPTQ and AWQ. The vLLM GGUF docs, however, describe GGUF support as highly experimental.

On licensing, Ollama and llama.cpp are MIT licensed and vLLM is Apache-2.0 licensed. The license texts are in the Ollama, llama.cpp and vLLM repositories.

llama.cpp is the C/C++ library that runs the GGUF format, which Ollama can also import. According to its README, it needs no dependencies, runs on Apple Silicon, x86 CPUs, NVIDIA (CUDA) and AMD (HIP), and supports quantization from 1.5 bits to 8 bits. The bundled server, according to the llama-server README, offers OpenAI-compatible endpoints, parallel decoding and continuous batching. LM Studio is a desktop app. Model downloads, chat and a local server sit in one interface, and it works offline with downloaded models.

The last section of the Running Local LLMs with Ollama course covers when Ollama stops being enough and it is time to move to vLLM. On the vLLM side, we cover memory and batching settings, splitting a model across several GPUs and monitoring KV cache usage in the LLM Serving with vLLM: Production Inference Deployment course.

Besides general models such as Qwen3, open models trained for specific Turkish tasks run offline in the same way. Three models that RuneLab has published on Hugging Face under the Apache 2.0 license are examples.

• Hati Glint SQL: a 9-billion-parameter model based on Qwen3.5-9B that turns Turkish questions into SQL and can call tools. According to the Hati Glint SQL model card, its bf16 weights need about 16.8 GB of GPU memory, and the model was tested with vLLM 0.31.0. So at 16 bits it does not fit on a 16 GB card and needs a card in the 24 GB class.
• Vör-LateEmbed-TR: a 0.6-billion-parameter late-interaction embedding model trained for Turkish document search. In an offline RAG setup, this is the part that finds the documents.
• Kenning Turkish Legal: a 0.3-billion-parameter Turkish legal embedding model adapted for searching contracts, case law and legislation.

Embedding models do not write answers. They find the relevant documents. In an internal document assistant, you run the search with Vör-LateEmbed-TR or Kenning Turkish Legal and the answer with a local LLM, and no document leaves the network. Our Turkish RAG evaluation guide shows how to measure whether such a setup finds the right document.

If you want to build this pipeline yourself, the Local LLM & On-Premise AI Training program follows the same order, from the model and VRAM plan to a local embedding model, a vector database and internal RAG. If you would rather use it ready-made, RuneLab's RuneDesk assistant runs on your company network and gives every answer with the document name and page.

Installing is easy, but running it takes more care. Add these points to the checklist of your procurement and IT teams.

• Network exposure: According to the Ollama FAQ, Ollama listens on 127.0.0.1:11434 by default, and you change the address with OLLAMA_HOST. If you open the API to the network, put a reverse proxy with authentication and access logging in front of it. RuneLab's RuneGate provides this layer ready-made: it puts models behind a single OpenAI-compatible endpoint, stops a team before its budget is exceeded and logs every request.
• Outbound connections: Turn off Ollama's cloud features with OLLAMA_NO_CLOUD=1 and vLLM's usage statistics with VLLM_NO_USAGE_STATS=1. On machines that need to download models, set the proxy with HTTPS_PROXY.
• Storage: According to the Ollama macOS page, model files can take tens and sometimes hundreds of GB. You can change their location with OLLAMA_MODELS.
• Capacity: As the number of users grows, the bottleneck becomes the KV cache, not the weights. Estimate the number of concurrent users and the average context length up front, and choose VRAM to match. Our GPU Capacity Planning and Hardware Sizing for LLM Inference course runs this calculation end to end, from the weight and KV cache formulas to a concurrency target derived from your user profile. Our Mistral Large 4 on-premise guide shows where this math leads for a trillion-parameter MoE model.
• Hardware: If you do not want to build the server yourself, RuneLab's RuneBox infrastructure is installed in your organization's data center with NVIDIA-based GPUs.
• Performance testing: Speed varies a lot with the card, model, quantization and context. Measure with your own workload on the target hardware before you buy.

Yes. Desktop apps such as LM Studio run chat, chat with documents and a local server without the internet once a model has been downloaded, and Ollama also runs models locally after the download. You need the internet for tasks such as downloading models, searching for them and installing updates.

No. Ollama and llama.cpp can run a model on the CPU alone, and if a model does not fit on the GPU they hand part of it to the CPU. A GPU still makes a clear difference, especially with many users, and vLLM needs an NVIDIA GPU with compute capability 7.5 or higher.

Yes, for an 8-billion-parameter model with 4-bit quantization. The 4-bit Ollama version of Qwen3-8B is 5.2 GB, and the KV cache for the default 4,096-token context is about 0.6 GB. The 8-bit version, or 14-billion-parameter models at 4 bits, need 12 GB.

Ollama is an MIT-licensed tool that downloads open-weight language models with a single command and runs them on your own computer or server. It runs on Linux, Windows and macOS and provides an OpenAI-compatible API.

Quantization stores model weights in fewer bits, which lowers memory needs at the cost of some possible accuracy loss. GGUF is the model file format that llama.cpp uses and Ollama can import. Q4_K_M, for example, uses about 4.9 bits per weight.

Yes. RuneLab's Apache 2.0 licensed Hati Glint SQL model turns Turkish questions into SQL and calls tools, and its bf16 weights need about 16.8 GB of GPU memory. The embedding models Vör-LateEmbed-TR for Turkish document search and Kenning Turkish Legal for legal texts also run without the internet.

A single number would mislead, because the result depends on the hardware, the model and the number of concurrent users. vLLM is designed for multi-user serving and Ollama for easy setup on a single machine. Measure with your own workload before you decide.

• Ollama, Ollama FAQ: privacy, environment variables, context, parallel requests, the queue and the KV cache type
• Ollama, Ollama hardware support page, Ollama Linux install guide, Ollama Windows page and Ollama macOS page
• Ollama, Ollama import docs, Ollama OpenAI compatibility docs and Ollama qwen3 tags (file sizes)
• vLLM, vLLM documentation, vLLM GPU installation guide, vLLM Quickstart, vLLM Engine Arguments docs, vLLM GGUF docs and vLLM usage stats docs
• ggml-org, llama.cpp README, llama.cpp quantize README (Llama-3.1-8B measurement table) and llama-server README
• LM Studio, Offline Operation page and LM Studio system requirements page
• Hugging Face, Optimizing LLMs for Speed and Memory and huggingface_hub environment variables docs
• RuneLab, Hati Glint SQL model card: base model, license, bf16 memory needs and the vLLM test
• Qwen, Qwen3-8B model card and configuration file, Qwen3-14B and Qwen3-32B model cards

• What is an LLM?
• What is LLM quantization?
• What is sovereign AI?
• Mistral Large 4 on premise: MoE memory and GPU math
• Courses: Running Local LLMs with Ollama, LLM Serving with vLLM: Production Inference Deployment, Running Local LLMs on Desktop and Workstations: llama.cpp, GGUF and LM Studio, Local LLM & On-Premise AI Training