Run LLM Offline from a USB Drive

Introduction

Running a large language model (LLM) without an internet connection opens up a range of privacy‑first, edge‑computing scenarios – from field diagnostics to secure offline assistants. In this post we’ll walk through the entire pipeline: picking a suitable model, quantizing it, fitting it on a USB stick, and executing inference on a modest laptop or single‑board computer.

Why Run an LLM Offline?

Data sovereignty – No data ever leaves the device.

Zero‑latency – Inference is limited only by CPU/GPU speed, not network round‑trip.

Deployability – A USB stick can be handed to non‑technical users and works on any compatible host.

Prerequisites

Choosing a Model

For offline use we need a model that balances size, quality, and hardware compatibility. Popular choices:

Llama‑2‑7B‑Chat (quantized to 4‑bit) – ~5 GB on disk.

Mistral‑7B‑Instruct – similar footprint.

Phi‑2 – ~2 GB, good for very constrained devices.

Quantization

Quantization reduces model size and speeds up inference. We’ll use llama.cpp which supports GGML‑format models and 4‑bit/8‑bit quantization.

The resulting *.ggmlv3.bin file is ready for offline execution.

Preparing the USB Stick

Format the drive to exFAT (cross‑platform) and label it LLM-USB.

Create a folder structure:

Copy the compiled llama.cpp binary (./main) into scripts/.

Running the Model

The following Bash script demonstrates a simple REPL that reads a prompt from stdin and prints the model’s response.

Make it executable (chmod +x scripts/run_llm.sh).

Usage example (on the host machine):

The script runs entirely locally; no network traffic is generated.

Performance Tuning

Controls output length. Lower values reduce latency.
| -temp (temperature) | Adjusts randomness; 0.0 = deterministic.
| -ctx (context size) | Larger context consumes more RAM. Keep ≤ 2048 for 8 GB RAM.
| -ngl (GPU layers) | If the host has a supported GPU, add -ngl 16 to offload 16 layers.

Experiment with -ngl on laptops with an NVIDIA GPU and the llama.cpp CUDA build.

Security Considerations

Encrypted USB – Use VeraCrypt or BitLocker to protect the model file.

Sandbox Execution – Run the binary inside a container (docker run --rm -v /mnt/usb:/data alpine:latest /data/scripts/run_llm.sh …).

Prompt Filtering – Add a simple keyword filter in the Bash wrapper to block disallowed queries.

Conclusion

Deploying an LLM on a USB pendrive is entirely feasible with modern quantization tools. By converting a HuggingFace checkpoint to GGML, storing the ~5 GB binary on a flash drive, and using a lightweight inference engine like llama.cpp, you can achieve offline, privacy‑preserving AI on virtually any laptop or edge device.

Feel free to fork the repository, experiment with other models, and share your own USB‑LLM projects!