🔒 Air-Gapped Deployment
Air-Gapped LLM Deployment Guide
Deploy open-source LLMs on classified, isolated, or air-gapped SOC infrastructure with zero internet dependency. Covers Ollama, vLLM, llama.cpp — from hardware to first query.
⚠️ Who this guide is for: Defence SOC teams, government network security operations, critical infrastructure operators, and any organisation where security policy prohibits sending log data or queries to external LLM APIs. This covers fully offline deployment — no cloud, no internet, no data leaving your network perimeter.
🏗️ Architecture of an Air-Gapped LLM SOC
An air-gapped LLM setup replaces cloud API calls with local inference. The model runs on your hardware, inside your perimeter. Analysts query it over your internal network. No query, log, or alert ever leaves the building.
Core components
| Component | Cloud equivalent | Air-gapped replacement |
|---|---|---|
| LLM inference | Anthropic API / OpenAI API | Ollama or vLLM running locally on GPU server |
| Model | Claude / GPT-4o | Llama 3.1 70B / Mistral 7B / Phi-3 Medium |
| Knowledge base (RAG) | Cloud vector DB | ChromaDB or FAISS running on local server |
| Embeddings | OpenAI text-embedding-3 | nomic-embed-text or all-MiniLM (local) |
| Analyst interface | ChatGPT / Claude.ai | Open WebUI or custom Python app on intranet |
What works well offline vs what degrades
| Task | 70B model quality | 7-13B model quality | Notes |
|---|---|---|---|
| Log explanation in plain English | Excellent | Good | Best offline use case |
| MITRE ATT&CK mapping | Excellent | Acceptable | May miss sub-techniques |
| CERT-In report drafting | Good | Acceptable | Needs good system prompt |
| Alert triage verdict | Good | Acceptable | JSON output less reliable on 7B |
| Complex multi-step reasoning | Acceptable | Limited | Significant quality gap vs cloud |
| Tool-calling / function calling | Acceptable | Unreliable | Only llama3.1 and mistral support this |
💻 Hardware Requirements
Rule of thumb: The model must fit entirely in GPU VRAM for fast inference. If it doesn't fit in VRAM, it spills to RAM (CPU inference) which is 10-50x slower. Always verify VRAM before purchasing.
GPU VRAM requirements by model and quantisation
| Model | Full (FP16) | Q8 (8-bit) | Q4 (4-bit) | Recommended for |
|---|---|---|---|---|
| Phi-3 Mini 3.8B | 8 GB | 4 GB | 2.5 GB | Low-resource / embedded systems |
| Mistral 7B | 14 GB | 8 GB | 4.5 GB | Single consumer GPU (RTX 4090) |
| Llama 3.1 8B | 16 GB | 9 GB | 5 GB | Entry-level GPU server |
| Phi-3 Medium 14B | 28 GB | 15 GB | 9 GB | RTX 4090 (Q4) or A100 40GB |
| Mistral Nemo 12B | 24 GB | 13 GB | 8 GB | Single A100 40GB |
| Llama 3.1 70B | 140 GB | 75 GB | 43 GB | 2× A100 80GB or 4× RTX 4090 |
| Qwen 2.5 72B | 144 GB | 76 GB | 45 GB | 2× A100 80GB — best quality/size |
Recommended hardware configurations for Indian govt / defence SOC
| Tier | Hardware | Model recommendation | Concurrent users |
|---|---|---|---|
| Minimal (POC) | 1× NVIDIA RTX 4090 (24GB VRAM), 64GB RAM, 2TB NVMe | Mistral 7B Q8 or Llama 3.1 8B Q8 | 1-2 analysts |
| Small SOC | 1× NVIDIA A100 40GB or 2× RTX 4090, 128GB RAM, 4TB NVMe | Phi-3 Medium Q8 or Llama 3.1 8B FP16 | 3-5 analysts |
| Medium SOC | 1× NVIDIA A100 80GB, 256GB RAM, 8TB NVMe RAID | Llama 3.1 70B Q4 or Qwen 2.5 72B Q4 | 5-10 analysts |
| Enterprise | 2× NVIDIA A100 80GB or H100 80GB, 512GB RAM, NVMe RAID | Llama 3.1 70B Q8 or Qwen 2.5 72B Q8 | 10-20 analysts |
CPU-only fallback (no GPU)
If GPU procurement is blocked, llama.cpp can run models on CPU-only. Quality is identical but speed is 10-50× slower. Acceptable for batch analysis (not real-time triage). Recommendation: Phi-3 Mini Q4 on a 64-core server with 256GB RAM gives usable response times (15-30 seconds per query).
# Minimum CPU-only server spec for Phi-3 Mini Q4
CPU: 64-core / 128-thread (AMD EPYC 7543 or Intel Xeon Platinum)
RAM: 128GB DDR4 minimum (256GB recommended)
Disk: 500GB NVMe for model + 2TB for data
OS: Ubuntu 22.04 LTS
# Expected performance on CPU-only:
# Phi-3 Mini Q4 (2.5GB): ~15-25 tokens/second
# Llama 3.1 8B Q4 (5GB): ~5-10 tokens/second
# Llama 3.1 70B Q4 (43GB): NOT recommended on CPU
⚙️ Runtime Choice: Ollama vs vLLM vs llama.cpp
| Runtime | Best for | Setup complexity | API compatibility | Windows support |
|---|---|---|---|---|
| Ollama | Single-server SOC, easy deployment, analyst-facing | Low | OpenAI-compatible REST API | Yes |
| vLLM | High-throughput, many concurrent analysts, production | Medium | OpenAI-compatible REST API | No (Linux only) |
| llama.cpp | CPU-only, minimal dependencies, embedded systems | Medium | OpenAI-compatible REST API | Yes |
| LM Studio | Single analyst workstation, GUI, evaluation | Very Low | OpenAI-compatible REST API | Yes |
Recommendation for Indian govt/defence SOC
Start with Ollama. It is the simplest to deploy, has the lowest operational overhead, supports all major models, and exposes an OpenAI-compatible API that works with any agent framework. Move to vLLM only when you need to serve 10+ concurrent analysts with strict latency requirements.
Ollama quick reference
# Install on Ubuntu 22.04 (run on air-gapped server AFTER transferring installer)
curl -fsSL https://ollama.com/install.sh | sh
# OR for air-gapped: use the .deb package transferred via USB
# Configure Ollama to listen on network (not just localhost)
# Edit /etc/systemd/system/ollama.service.d/override.conf
[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"
systemctl daemon-reload && systemctl restart ollama
# Load a model (from locally transferred GGUF file)
ollama create llama3-airgapped -f /mnt/transfer/Modelfile
# Test
curl http://localhost:11434/api/generate \
-d '{"model":"llama3-airgapped","prompt":"Explain Event ID 4625","stream":false}'
vLLM for production deployments
# Install vLLM (on internet-connected staging → transfer to air-gapped)
pip install vllm
# Serve model (after offline transfer)
python -m vllm.entrypoints.openai.api_server \
--model /opt/models/Meta-Llama-3.1-70B-Instruct-GPTQ \
--quantization gptq \
--tensor-parallel-size 2 \
--host 0.0.0.0 \
--port 8000 \
--api-key "your-internal-api-key"
# vLLM advantage: batching multiple analyst requests efficiently
# Benchmark: 10 concurrent analysts on A100 80GB with 70B Q4
# → average latency ~8-12 seconds per query
📦 Offline Model Transfer Procedure
⚠️ Critical: The transfer procedure is as important as the deployment. A compromised model file is a supply chain attack. Follow every verification step. Do not skip hash verification.
1
Download on internet-connected staging system
Use a clean, dedicated staging system that is NOT connected to your air-gapped network. Download model files only from official sources: Hugging Face official model pages (verify the publisher is the original organisation — Meta, Mistral AI, Microsoft). Never use third-party mirrors.
# Download Llama 3.1 8B GGUF (Ollama-compatible format)
pip install huggingface_hub
huggingface-cli download \
bartowski/Meta-Llama-3.1-8B-Instruct-GGUF \
Meta-Llama-3.1-8B-Instruct-Q8_0.gguf \
--local-dir /staging/models/
# Immediately hash the download
sha256sum /staging/models/Meta-Llama-3.1-8B-Instruct-Q8_0.gguf \
> /staging/models/llama31-8b-q8.sha256
cat /staging/models/llama31-8b-q8.sha256
2
Verify hash against official published values
Compare your computed SHA256 against the hash published on the official Hugging Face model page. These must match exactly. A single character difference means the file is corrupted or tampered. Document the verified hash in your change management system.
# Compare against Hugging Face published hash
# Find hash on: https://huggingface.co/bartowski/Meta-Llama-3.1-8B-Instruct-GGUF
# Under "Files and versions" → click filename → "Copy SHA256"
PUBLISHED_HASH=""
DOWNLOADED_HASH=$(sha256sum /staging/models/Meta-Llama-3.1-8B-Instruct-Q8_0.gguf | cut -d' ' -f1)
if [ "$PUBLISHED_HASH" = "$DOWNLOADED_HASH" ]; then
echo "✓ HASH VERIFIED — safe to transfer"
else
echo "✗ HASH MISMATCH — DO NOT TRANSFER — possible tampering"
exit 1
fi
3
Encrypt before transfer
Encrypt the model file before writing to transfer media. This protects against interception during physical transfer and ensures the media itself cannot be used if lost or stolen.
# Encrypt model file with AES-256 using a pre-shared key
# Key must be communicated via separate, secure channel — not with the media
gpg --symmetric \
--cipher-algo AES256 \
--compress-algo none \
--output /staging/transfer/llama31-8b-q8.gguf.gpg \
/staging/models/Meta-Llama-3.1-8B-Instruct-Q8_0.gguf
# Also encrypt the hash file
gpg --symmetric --cipher-algo AES256 \
--output /staging/transfer/llama31-8b-q8.sha256.gpg \
/staging/models/llama31-8b-q8.sha256
# Write to encrypted USB or optical media
# Log: date, model name, hash, operator name, recipient system
4
Physical transfer with chain of custody
Physical media must be transferred following your organisation's classified media transfer procedures. Document: transfer date/time, operator name, media serial number, destination system, and receiving officer signature. Treat model files as sensitive intellectual property — a 70B model contains hundreds of billions of learned parameters.
5
Decrypt and verify on destination system
# On the air-gapped server:
gpg --decrypt /media/transfer/llama31-8b-q8.gguf.gpg \
> /opt/models/Meta-Llama-3.1-8B-Instruct-Q8_0.gguf
gpg --decrypt /media/transfer/llama31-8b-q8.sha256.gpg \
> /opt/models/llama31-8b-q8.sha256
# Re-verify hash on destination
EXPECTED=$(cat /opt/models/llama31-8b-q8.sha256 | cut -d' ' -f1)
ACTUAL=$(sha256sum /opt/models/Meta-Llama-3.1-8B-Instruct-Q8_0.gguf | cut -d' ' -f1)
if [ "$EXPECTED" = "$ACTUAL" ]; then
echo "✓ DESTINATION HASH VERIFIED — ready to load"
else
echo "✗ HASH MISMATCH ON DESTINATION — transfer corrupted or tampered"
fi
# Set permissions — only ollama/vllm service account can read
chown ollama:ollama /opt/models/Meta-Llama-3.1-8B-Instruct-Q8_0.gguf
chmod 640 /opt/models/Meta-Llama-3.1-8B-Instruct-Q8_0.gguf
🔧 Full Installation Walkthrough (Ubuntu 22.04 + Ollama)
This assumes all packages have been downloaded on an internet-connected system and transferred to the air-gapped server via approved media.
Step 1 — Prepare offline package bundles (on internet-connected staging)
# Download Ollama installer
wget https://ollama.com/install.sh -O ollama-install.sh
# OR download the .deb for Ubuntu:
wget https://github.com/ollama/ollama/releases/latest/download/ollama-linux-amd64.deb
# Download CUDA libraries if using NVIDIA GPU
# (required for GPU acceleration on Ubuntu 22.04)
apt-get download \
nvidia-cuda-toolkit \
libcuda1 \
libnvidia-compute-545
# Download Open WebUI (optional analyst interface)
pip download open-webui --dest /staging/pip-packages/
# Hash everything
find /staging/ -type f | xargs sha256sum > /staging/TRANSFER_MANIFEST.sha256
Step 2 — Install on air-gapped server
# Install NVIDIA drivers (if GPU — done offline via .deb packages)
dpkg -i /media/transfer/nvidia-*.deb
apt-get install -f --no-download # resolve dependencies from local cache
# Install Ollama from offline package
dpkg -i /media/transfer/ollama-linux-amd64.deb
# OR run installer in offline mode:
OLLAMA_SKIP_UPDATE=1 bash /media/transfer/ollama-install.sh
# Verify Ollama installed
ollama --version
# Configure Ollama service
mkdir -p /etc/systemd/system/ollama.service.d/
cat > /etc/systemd/system/ollama.service.d/override.conf << 'EOF'
[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"
Environment="OLLAMA_MODELS=/opt/models"
Environment="OLLAMA_NUM_PARALLEL=4"
EOF
systemctl daemon-reload
systemctl enable ollama
systemctl start ollama
Step 3 — Register model with Ollama
# Create a Modelfile pointing to the transferred GGUF
cat > /opt/models/Modelfile << 'EOF'
FROM /opt/models/Meta-Llama-3.1-8B-Instruct-Q8_0.gguf
# Context window — 8192 is safe, use 4096 if memory constrained
PARAMETER num_ctx 8192
# System prompt for SOC use
SYSTEM """You are a SOC analyst assistant operating on a classified network.
You ONLY use information provided in the conversation.
You do NOT have internet access. You do NOT have access to external threat intelligence.
Always note when your answer requires current threat intelligence that you may not have.
Be precise and security-focused in all responses."""
EOF
# Import into Ollama
ollama create llama31-soc -f /opt/models/Modelfile
# Verify
ollama list
ollama run llama31-soc "Explain what Event ID 4624 LogonType 3 means"
Step 4 — Install Open WebUI for analyst access (optional)
# Install from offline pip packages
pip install --no-index --find-links=/media/transfer/pip-packages/ open-webui
# Configure to connect to local Ollama (no internet)
export OLLAMA_BASE_URL=http://localhost:11434
export WEBUI_AUTH=True # Enable user authentication
export WEBUI_SECRET_KEY="$(openssl rand -hex 32)"
# Start Open WebUI
open-webui serve --host 0.0.0.0 --port 8080
# Analysts access via: http://[server-ip]:8080
# Add firewall rule to allow only analyst subnet
ufw allow from 10.0.1.0/24 to any port 8080
ufw deny 8080
🛡️ Operational Security & Maintenance
Access control
# Create dedicated service account for Ollama
useradd -r -s /bin/false -d /opt/models ollama
# Analysts access via API key (add to Ollama config)
cat >> /etc/systemd/system/ollama.service.d/override.conf << 'EOF'
Environment="OLLAMA_ORIGINS=http://10.0.1.0/24"
EOF
# Log every query — add nginx reverse proxy with access logging
apt-get install nginx # from offline packages
# Configure nginx to proxy port 11434 with logging:
# access_log /var/log/nginx/ollama_access.log combined;
# All analyst queries logged: timestamp, source IP, endpoint, response size
Monitoring the inference server
# Check Ollama service status
systemctl status ollama
journalctl -u ollama -f # live logs
# GPU utilisation monitoring (NVIDIA)
nvidia-smi --query-gpu=name,memory.used,memory.free,utilization.gpu \
--format=csv -l 5 # refresh every 5 seconds
# Check active model sessions
curl http://localhost:11434/api/ps
# Memory usage
curl http://localhost:11434/api/ps | python3 -c \
"import json,sys; data=json.load(sys.stdin);
[print(f\"{m['name']}: {m['size_vram']//1024//1024}MB VRAM\") for m in data.get('models',[])]"
Updating models offline
When a better model version is released: download and verify on staging, transfer via approved procedure, register as a new Ollama model name (keep the old one available), test with your standard prompt set, then switch analysts to the new model. Never delete the old model until the new one is validated.
Key operational differences from cloud AI
| Aspect | Cloud LLM | Air-gapped local LLM |
|---|---|---|
| Model knowledge cutoff | Updated periodically by provider | Fixed at training cutoff — use RAG for recent TI |
| Latency | 1-3 seconds (shared compute) | 2-15 seconds depending on hardware and model size |
| Downtime | Provider-managed (rare) | Your responsibility — plan for GPU failures |
| Model updates | Transparent, automatic | Manual transfer procedure required |
| Context window | 200K+ tokens (Claude) | Typically 8K-128K depending on model |
| Cost per query | $0.001-0.01 per query | Electricity + amortised hardware (~$0.0001) |