llama.cpp illustration: a sealed stone cartridge sliding into its socket on a local workbench, beside a rack of differently shaped stones.
Blog

What Is llama.cpp? How It Differs from Ollama

The GGUF format, llama-server's OpenAI-compatible API, choosing between llama.cpp and vLLM at a company, and an open-weight model for Turkish work.

13 min readBy Engin Deniz Alpman

In short: llama.cpp is an MIT-licensed, open-source C/C++ inference library and server for running large language models (LLMs) on your own computer or server. It runs models in the GGUF file format and includes an OpenAI-compatible API server and a built-in chat interface. The project moved to v1.0.0 on November 13, 2026. Ollama and LM Studio are separate applications from llama.cpp.

Most people who want to run an open model on their own computer start with Ollama or LM Studio. All three, llama.cpp included, do inference, producing answers from a trained model, on your own hardware. How llama.cpp differs from Ollama comes down to control: it hands you every setting, the C API and the server to run yourself. Installing it takes a one-line script, winget on Windows, Homebrew on a Mac, or the desktop app called Llama. Three things decide which one to start with: how much control you want, how many people the model will serve, and what hardware it will run on.

What is llama.cpp, and what is it for?

#

llama.cpp is a C/C++ inference project for running LLMs and multimodal models with minimal setup on a wide range of hardware, locally and in the cloud. That is how the project's llama.cpp README defines its main goal, and it adds that the project is built on ggml, a tensor library for machine learning.

The official llama.cpp v1.0.0 announcement on Hugging Face says llama.cpp runs multimodal LLMs on hardware that ranges from consumer-grade machines to high-end enterprise servers. It does this through quantization, optimized backends and a portable C/C++ implementation. According to the same announcement, the project supports more than 150 architecture variants as well as all mainstream speculative decoding methods. Our speculative decoding explainer shows how a small draft model speeds up a big one.

Which model format does llama.cpp use?

#

llama.cpp requires the model to be stored in the GGUF file format. According to the repository's models document, models in other formats are converted to GGUF with the convert_*.py scripts in the repository, and Hugging Face hosts thousands of GGUF models compatible with llama.cpp. Quantization stores weights in fewer bits, which cuts the memory a model needs. Our quantization guide explains how a big model fits in fewer bits.

Can llama.cpp run as an API server?

#

Yes. After installation there are two main commands: llama cli runs a model in the terminal, and llama serve starts an OpenAI-compatible API server. According to the llama-server README, the server runs F16 and quantized models on GPU and CPU. Its features also include OpenAI-compatible chat completions, responses and embeddings routes, an Anthropic Messages API compatible route, parallel decoding with multi-user support, continuous batching, function calling and a built-in chat app called Llama Chat. An application that works with the OpenAI API can connect to a model on this server by changing its address.

Is llama.cpp free, and what is its license?

#

llama.cpp is free and MIT licensed. The license grants permission, free of charge, to any person obtaining a copy, to deal in the software without restriction, including to use, copy, modify, merge, publish, distribute, sublicense and sell copies, on condition that the copyright and permission notice is included in all copies or substantial portions. The software is provided "AS IS", without warranty, and the authors and copyright holders are not liable for any claim or damages.

This license covers the tool itself. The model file you run with llama.cpp has its own license. We compare the licenses and commercial terms of open models in our open-weight model licenses guide.

How does llama.cpp differ from Ollama and LM Studio?

#

llama.cpp is a C/C++ inference library and server. Ollama and LM Studio are separate applications that make it easier to download and run a model. Their official documents describe the relationship as follows. The Ollama README lists llama.cpp under "Supported backends". Ollama's Importing a Model document says Ollama does not quantize GGUF models during import, and recommends preparing them first with a GGUF tool such as llama.cpp's llama-quantize. The LM Studio documentation says LM Studio runs models on Mac, Windows and Linux using llama.cpp, and also using Apple's MLX on Apple Silicon Macs.

llama.cpp, Ollama and LM Studio compared (official documentation, November 18, 2026)
Topic llama.cpp Ollama LM Studio
Official descriptionC/C++ inference for LLMs and multimodal modelsBuilding with open models, locally or in the cloudRunning LLMs such as Llama, DeepSeek and Qwen locally
InstallInstall script, Llama app, package managers, Docker, build from sourceScript or installer for macOS, Windows and Linux, Docker imageInstaller for macOS, Windows and Linux
Interfacellama cli, llama serve and the built-in Llama Chatollama command lineDesktop chat interface and the lms command line
APIOpenAI and Anthropic Messages compatible routesIts own REST API and a subset of the OpenAI APIIts own REST API, OpenAI and Anthropic compatible endpoints
Model formatGGUF required, other formats convertedGGUF or Safetensors import with a Modelfilellama.cpp (GGUF) and MLX on Apple Silicon
Relationship to llama.cppThe project itselfREADME lists llama.cpp under "Supported backends"Docs say it runs models using llama.cpp
LicenseMITMITOwn terms of use (proprietary)

The license row matters for business use. Ollama's license file is MIT, like llama.cpp's. LM Studio is not open source: the LM Studio App Terms of Service grant a non-exclusive, non-transferable license to use the app solely for personal or internal business purposes. On the API side, Ollama's OpenAI compatibility document says it supports a subset of the OpenAI API, and the LM Studio Developer Docs list OpenAI and Anthropic compatible endpoints alongside its own REST API.

The rule that follows from the official descriptions is simple. Choose llama.cpp if you want every setting, want to connect through the C API from your own application, or want to run the server yourself. Choose Ollama if you want to pull a model with one command and connect it to coding agents such as Claude Code or Codex. Choose LM Studio if you want to search for, download and chat with models in a graphical interface. For teams taking the Ollama route, RuneLab offers the Running Local LLMs with Ollama course.

What changed in llama.cpp v1.0.0?

#

On November 13, 2026, llama.cpp and ggml both moved to v1.0.0, and the version number now signals API compatibility. The v1.0.0 release page went up on GitHub and the announcement on Hugging Face the same day. Under the announcement's rules, a major-version (MAJOR) bump means a breaking change to the public C API in ./include. MINOR covers a backward-compatible feature, backend update, model support or API addition, and PATCH is a bug fix with no API change.

The user-facing items in the release notes are these:

  • The deprecated public API was removed, and the NextN and speculative decoding API was promoted to llama.h.
  • The KV cache, where attention layers keep their key and value vectors in memory, can now be stored in the OCP FP8 format.
  • A host-memory MoE expert cache with a configurable CPU share was added.
  • llama-server received fixes for SSRF, CORS and DNS rebinding.
  • A new OpenAI-compatible /v1/audio/speech endpoint was added.
  • The web UI was redesigned with chat groups and a markdown editor.

The server now cancels MCP tool calls when a client disconnects or times out, and notifies the MCP server. Our What is MCP article explains this protocol for connecting models to outside tools.

Can you download llama.cpp from the v1.0.0 release page?

#

No, the pre-built binaries are published as nightly builds. As of November 18, 2026, the v1.0.0 page holds only nightly-tag.txt. The tag in that file belongs to the b12359 nightly build, published the same day as v1.0.0. That build carries 38 pre-built files for backends such as CPU, CUDA, Vulkan, ROCm and SYCL. The nightly builds continue, and b12503, published on November 18, also came with 38 files.

The repository's release process document also says that, for now, only nightly and development builds are published to GitHub releases. Releases reach users through the install script, package managers and building from source. So to install v1.0.0, use one of the official install paths below.

How do you install llama.cpp?

#

The quickest way to install llama.cpp is the install script in the README. The README lists five paths: the install script, the llama.app page, Docker, pre-built binaries from the GitHub releases page, and building from source. On Linux and macOS, run the script like this:

curl -LsSf https://llama.app/install.sh | sh

On Windows, the PowerShell line does the same job:

irm https://llama.app/install.ps1 | iex

How do you install llama.cpp with a package manager on Windows and Mac?

#

On Windows, besides the PowerShell script, the install document gives winget and conda-forge:

winget install llama.cpp

On a Mac, the options are Homebrew, MacPorts, conda-forge and Nix. With Homebrew it is one line:

brew install llama.cpp

According to the document, the winget package and the Homebrew formula update automatically with new llama.cpp releases. If you would rather skip the command line, llama.app offers a desktop app called Llama for Mac and Windows 11. The Hugging Face announcement presents Llama as the desktop app from the team behind llama.cpp and Hugging Face.

For Docker, the official images are published under ghcr.io/ggml-org/llama.cpp with the full, light and server tags. According to the Docker document, the server image contains only the llama-server program, and there are separate tags for accelerators such as CUDA and ROCm.

How do you run your first model after installing?

#

The README's example downloads a model from Hugging Face and runs it directly. The second command serves the same model as an API:

# Download a model from Hugging Face and run it in the terminal
llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF

# Serve the same model as an OpenAI-compatible API
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF

Which model fits your computer, and how to size hardware and memory, is covered in our guide to running an LLM locally. If your team wants hands-on practice with llama.cpp, GGUF and LM Studio, RuneLab's Running Local LLMs on Desktop and Workstations course covers k-quant choice, CPU and GPU layer offloading, Metal and CUDA acceleration and OpenAI-compatible APIs.

When should a company use llama.cpp, and when vLLM?

#

If the model will run on a single machine, a workstation or a mobile device, llama.cpp is the better starting point, and if it will serve many users across several GPUs, vLLM is. The vLLM README describes the project as a fast and easy-to-use library for LLM inference and serving. The features it highlights are high serving throughput, attention memory management with PagedAttention, continuous batching, and tensor, pipeline, data, expert and context parallelism for distributed inference. vLLM supports NVIDIA, AMD and Intel GPUs and x86, ARM and PowerPC CPUs, and GGUF is among its supported quantization formats.

llama.cpp, for its part, highlights minimal setup and a wide range of hardware. According to the distribution table in the Hugging Face announcement, its pre-built binaries are published for Android and iOS as well as Linux, macOS and Windows. llama-server also offers parallel decoding and continuous batching.

The rule that follows from these two descriptions is this. Start with llama.cpp if the model will run on a laptop, a workstation, a CPU-only server or a mobile device, and you want few dependencies with a single GGUF file. Look at vLLM if you will serve many users on GPU servers and split the model across several GPUs with tensor or pipeline parallelism. Make the decision by measuring on your own hardware, with your own prompts and number of concurrent users. RuneLab's LLM Serving with vLLM course covers serving a model in production with vLLM.

Which work belongs in the cloud and which in-house is the subject of our local LLM vs ChatGPT article. Once you decide to run the model in-house, there are two paths. If your own team will run the installation, RuneLab's Local LLM Training teaches running open models on your own servers, from model selection and GPU planning to setup with vLLM and Ollama, and on to in-house RAG and chatbots.

If you would rather not assemble hardware, models and software piece by piece, RuneLab's RuneBox is a ready on-premise LLM installation on your company network. Models, RAG and agent flows run on an NVIDIA GPU box in a closed network with no outside connection. Questions, documents and answers never leave. Because data is processed inside the company boundary, your BDDK and KVKK assessments become easier. Compliance is your organization's own assessment.

Which open model should you choose for Turkish work?

#

For Turkish work, the model you run shapes the result more than the inference tool does, so choose the model by the language and the type of task. RuneLab's Hati Glint SQL is one example: a 9-billion-parameter, Apache 2.0 licensed, open-weight Turkish text-to-SQL model fine-tuned from Qwen3.5-9B. It turns Turkish questions into SQL, also works as a general Turkish assistant and can call tools.

According to the model's Hugging Face card, a 9B model runs on a single GPU in your own infrastructure. The model is published in the transformers format, and the card gives its run instructions for the transformers library.

Frequently asked questions

#

What is llama.cpp?

#

llama.cpp is an open-source C/C++ inference library and server that runs LLMs and multimodal models with minimal setup on a wide range of hardware, locally or in the cloud. It is built on the ggml tensor library. It runs models in the GGUF format and includes an OpenAI-compatible API server.

What is the difference between llama.cpp and Ollama?

#

llama.cpp is a C/C++ inference library and server, while Ollama is a separate application that downloads and runs a model with one command. Ollama's GitHub README lists llama.cpp among its supported backends. If you want every setting and the C API, llama.cpp fits better, and if you want to start with one command, Ollama does.

Is llama.cpp free, and what is its license?

#

llama.cpp is MIT licensed and free. The license permits use, modification, distribution and sale, on condition that the copyright and permission notice is included in copies. It gives no warranty and holds the authors free of liability. The model you run has its own license.

What did v1.0.0 change in llama.cpp?

#

On November 13, 2026, llama.cpp and ggml both moved to v1.0.0, and from now on a major-version bump means a breaking change to the public C API. The release removes the deprecated API, adds FP8 KV caches and a MoE expert cache, hardens llama-server's security and redesigns the web UI.

How do you install llama.cpp?

#

You can use the README's install script, the Llama app from llama.app, winget on Windows, Homebrew on a Mac, Docker, or build from source. As of November 18, 2026, the v1.0.0 release page has no platform files, and the pre-built binaries are published as nightly builds.

Does llama.cpp run without a GPU?

#

Yes. According to the llama-server documentation, llama.cpp runs F16 and quantized models on both GPU and CPU, and the nightly builds include CPU-only packages for Windows and Linux. Speed depends on the size of the model and the hardware.

Should a company use llama.cpp or vLLM?

#

llama.cpp highlights minimal setup and broad hardware support, while vLLM highlights high serving throughput and distributed, multi-GPU inference. vLLM's README lists x86, ARM and PowerPC CPUs alongside GPUs. Start with llama.cpp for a single machine, workstation or mobile device and with vLLM for multi-user, multi-GPU serving, then decide by measuring on your own hardware.

Sources

#
#