Course Objective & Scope
Data residency rules, confidential enterprise data and the need for predictable cost at scale lead many organisations to run open-weight models on their own infrastructure. This course covers the Hugging Face ecosystem, model licences (Apache 2.0, MIT, custom community licences), VRAM and GPU capacity sizing, production inference engines such as vLLM and SGLang, and exposing an OpenAI-compatible internal API.
Participants deploy a quantized open-weight model, integrate it into application code and benchmark throughput and latency under concurrent load.
Who Should Attend?
- AI, platform and MLOps engineers setting up on-premises or private-cloud LLM services
- Backend developers and system architects reducing external API lock-in
- Technical teams in regulated industries with strict data residency and KVKK requirements
- Prerequisite: Linux/container fundamentals and experience building APIs in Python
Key Takeaways
- Evaluate open-weight models on licensing, hardware footprint and task/Turkish quality
- Size GPU VRAM for model weights, KV cache and target concurrency
- Configure PagedAttention, continuous batching and prefix caching in vLLM or SGLang
- Measure the speed and quality impact of FP16/BF16, FP8, AWQ and GPTQ quantization
- Connect applications to self-hosted models via OpenAI-compatible endpoints and fallbacks
Curriculum
The Open-Weight Ecosystem and Licensing
- Open-source (OSI) vs open-weight; Apache 2.0, MIT and custom model licences
- Enterprise assessment of the Llama, Qwen, Mistral, Gemma and DeepSeek families
GPU Sizing and VRAM Mathematics
- Practical formulas for weight memory, activations and KV cache
- How context length and concurrency drive GPU selection
Production Inference Engines: vLLM, SGLang and Ollama
- Local development (Ollama/llama.cpp) vs production serving (vLLM, SGLang)
- Continuous batching, PagedAttention and RadixAttention explained
Quantization and Performance Tuning
- Comparing FP8, INT8, AWQ and GPTQ formats
- Configuring tensor parallelism and context-length limits
Application Integration and Enterprise Security
- Using OpenAI-compatible endpoints for SDKs, tool calling and structured outputs
- Air-gapped weight distribution, checksum verification, authentication and quotas
Lab: Self-Hosted LLM Service and Load Testing
- Deploying a containerised open-weight model and integrating it with an app
- Measuring TTFT, throughput (tok/s) and error rates under concurrent load
Other Courses in This Category
-
Engineering techniques to cut token spend and p95 latency in production LLM applications: prompt and semantic caching, Batch APIs, output length control, tiered models and unit economics.
- Live Course
- 6 modules
-
Hands-on programme on building modular LLM applications with modern LangChain (LCEL, ChatModel abstractions, prompt templates, structured outputs, tool calling, retrievers and LangSmith tracing).
- Live Course
- 6 modules
-
Building type-safe LLM, RAG, tool-calling and MCP integrations directly in enterprise Java (Spring AI, LangChain4j) and .NET (Microsoft.Extensions.AI, Semantic Kernel) stacks without switching to Python.
- Live Course
- 6 modules
-
Hands-on programme for building enterprise generative AI solutions on AWS using the Amazon Bedrock Converse API, Knowledge Bases, Agents, Guardrails, IAM/VPC security and cost controls.
- Live Course
- 6 modules
-
Building enterprise AI applications on Azure AI Foundry and Azure OpenAI Service with GPT and reasoning models, Azure AI Search, Entra ID authentication, Private Endpoints, Content Safety and PTU/quota management.
- Live Course
- 6 modules
-
Hands-on programme for building enterprise AI applications on Google Cloud Vertex AI using Gemini models, the Google Gen AI SDK, Context Caching, Grounding, Vertex AI Search, BigQuery and VPC-SC security.
- Live Course
- 6 modules
Get in touch
