How-To: Run a Local Coding Model as a Rootless Podman Service

Omarchy’s proposed local-models recipe runs the model server under rootless Podman. That’s a pattern you can use on any Linux box today. Here’s a generic version with llama.cpp’s server image.

1. Grab a model

Put a GGUF model you trust in ~/models. Pick a size that fits your RAM or VRAM.

2. Test it by hand

podman run --rm -p 127.0.0.1:8080:8080 -v ~/models:/models:Z \
  ghcr.io/ggml-org/llama.cpp:server \
  -m /models/your-model.gguf --host 0.0.0.0 --port 8080 -c 32768
curl -s http://127.0.0.1:8080/v1/models

Note the 127.0.0.1 publish. That keeps the model off your network. For a GPU, try the Vulkan image and pass --device /dev/dri.

3. Make it a user service with Quadlet

Create ~/.config/containers/systemd/llm.container:

[Container]
Image=ghcr.io/ggml-org/llama.cpp:server
PublishPort=127.0.0.1:8080:8080
Volume=%h/models:/models:Z
Exec=-m /models/your-model.gguf --host 0.0.0.0 --port 8080 -c 32768

[Install]
WantedBy=default.target

systemctl --user daemon-reload
systemctl --user start llm
loginctl enable-linger $USER   # keep it running after logout

4. Point your agent at it

Most coding agents accept an OpenAI-compatible base URL. Use http://127.0.0.1:8080/v1.

Use the Source!

References & Further Reading

• Podman documentation
• podman-systemd.unit (Quadlet) docs
• llama.cpp Docker docs
• llama.cpp server
• Omarchy PR #16590: local models in the agents panel