The runner sidecar

Hermiq runs one ExApp sidecar: a container that holds the things Hermiq must not do inside PHP — executing vendor LLM CLIs, and running speech models. Hermiq itself stays the governed engine; the sidecar is transport and inference only.

Guardrails, per-tool approval, grants, redaction, evals, model policy and budgets all remain on Hermiq's side of the wire. Nothing in this container decides what an agent is allowed to do.

What it serves

Port	Service	API
9000	LLM CLI transport	POST /run — one fully-assembled turn
8000	Speech	OpenAI-compatible /v1/audio/transcriptions, /v1/audio/speech

The speech half serves both directions: faster-whisper for speech-to-text and Kokoro for text-to-speech, behind one OpenAI-compatible API.

How it is built

exapp/llm-runner/Dockerfile.combined, based on the Speaches image with Node added.

The direction is deliberate. Adding Node to a Python/ML base is a package install. Adding torch to a Node base means owning the ML dependency tree ourselves — which is precisely the maintenance avoided by not writing a bespoke speech sidecar in the first place.

FROM ghcr.io/speaches-ai/speaches:latest-cpu@sha256:21e3df06…
  + NodeSource Node 22
  + @anthropic-ai/claude-code, @openai/codex
  + /app (the runner's src/ and deploy/)

Roughly 3.2 GB. Grok is deliberately absent: xAI ships no verified official CLI on npm, and installing a guessed package would be wrong — a deployer mounts one and sets RUNNER_GROK_BIN.

Why images are pinned by digest

Neither speech project ships a vendor-official image, because none exists: OpenAI publishes the Whisper model, hexgrad publishes Kokoro weights, and neither publishes a service. The images we use are the best-maintained community wrappers.

That is exactly why the base is pinned by digest, not tag. A floating tag lets a poisoned upstream rebuild propagate on the next docker compose pull, and a community image is a softer target than a vendor one.

Design decisions

Two runtimes, one container

Speech and the LLM transport were briefly two sidecars. They are now one, and the arguments for splitting them did not survive scrutiny:

• "A model server needs egress, the runner is jailed." Wrong, and it proved too much — Ollama downloads models too and lives happily behind a locked-down runtime. The distinction missed was provisioning-time vs runtime: weights are fetched once by an operator; afterwards the service needs no route at all.
• "One OOM in transcription kills the LLM transport." An orchestration concern, answered by mem_limit, not by a second image.
• "A torch CVE forces a rebuild of the transport." True, and the accepted price of one container.

What remains is that the image carries two language runtimes. That is image hygiene, not architecture.

Speech models are not run on Ollama

Ollama cannot host them. It has no audio input path, and neither Whisper nor Kokoro is a GGUF model — there is nothing to pull. Ollama remains the LLM inference server; this container is CLI transport plus speech.

The entrypoint exits if either service dies

deploy/combined-entrypoint.sh brings the whole container down when either process exits. This is the point, not a nicety.

The obvious serverA & serverB & wait leaves PID 1 a shell that outlives both children. The container then stays up — and stays healthy, because a probe only hits one port — while half the sidecar is dead. Agents keep working, dictation silently stops, and the restart policy never fires because nothing ever exited. That is the failure nobody notices for a week.

It polls rather than using wait -n: dash does not implement wait -n, and the failure mode would be silent (it would wait for both).

The speech server must start from its own directory

speaches.main:create_app mounts StaticFiles(directory="realtime-console/dist") — a relative path. Starting it from the runner's /app working directory raises Directory 'realtime-console/dist' does not exist and the server dies at import, presenting as an immediate unexplained exit. The entrypoint cds first.

Security

Governed egress, for the LLM half

The runner has no default route. Its network is declared internal: true, so Docker installs no gateway, and the only way out is a CONNECT proxy that asks Hermiq's PDP (POST /api/egress/authorize) about every single connection.

Two complementary layers, not redundant ones:

• the per-agent MCP grant governs what the agent is authorised to do;
• this network backstop governs what the container can reach — including traffic the agent never asked for: a CLI auto-update check, a built-in fetch a future flag fails to disable, a compromised dependency.

The proxy is run-scoped: it requires a run token in Proxy-Authorization, forwards it to the PDP as a bearer token, and denies with no_run_token otherwise. Only CONNECT is served — plain HTTP proxying would let the enforcement point read request bodies.

Stated plainly: CONNECT gives host:port granularity, not URL granularity.

No egress at all, for the speech half

Once models are primed, the speech server needs no network whatsoever. It does not use the egress proxy, and that is deliberate rather than an oversight: model fetching is operator provisioning, not an agent run. There is no run to attribute it to, and minting a token to satisfy the proxy would make the audit trail lie about who reached HuggingFace and why.

So provisioning happens through a short-lived primer with egress, and the serving container then runs with no route. This is a stronger posture than the LLM half has — that one needs its proxy continuously.

HF_HUB_OFFLINE=1 is required, not optional

Cached weights alone are not enough. Without this variable, huggingface_hub still attempts a network check when a model loads and blocks until it times out — so an isolated container looks broken rather than offline.

Measured both ways: network detached and the variable unset, a synthesis call timed out; the same call with it set returned audio from cache.

A timeout is not evidence about the network

While proving the above, a detached transcription timed out and looked like a network block. It was not — the container log showed Processing audio with duration …, the VAD filter and segment processing. It had loaded from cache and was simply slow on CPU while the client gave up. Read the container log before concluding "blocked".

Runtime hardening

cap_drop: ALL, no-new-privileges, non-root (ubuntu, uid 1000), mem_limit so one model load cannot take the host down.

Letting an agent change code in a repository

An agent in a governed run can check out a repository, change files, commit and push. It never gets a shell for this. The runner's denylist stays as it is. The agent calls named workspace tools, and Nextcloud does the git work on its own side.

What you set

• workspace_forge_base_url: the forge the agent may use, for example https://github.com. The agent names a repository as owner/name and never sees a URL.
• workspace_forge_provider: the credential provider of a push (default github).
• Each person who lets an agent push stores a personal, inject-only forge credential in the credential broker, with hermiq in its allowed apps.

What you grant, per agent

Reading, editing and pushing are three separate grants in the agent's tool list:

• read: hermiq.workspaceOpen, hermiq.workspaceStatus, hermiq.workspaceDiff, hermiq.workspaceLog, hermiq.workspaceListFiles, hermiq.workspaceReadFile
• edit: hermiq.workspaceWriteFile, hermiq.workspaceDeleteFile, hermiq.workspaceApplyPatch, hermiq.workspaceCreateBranch, hermiq.workspaceCheckoutBranch, hermiq.workspaceCommit
• push: hermiq.workspacePush?repository=owner/name&branch=in:feature-a,feature-b

The push grant must pin one repository and name the branches. A bare hermiq.workspacePush grant does nothing. The tool catalogue then shows it as not granted, so you see the mistake before a run needs it.

What a person approves

The first edit in a run asks for approval. The request goes to the agent's owner, or to the admin group when the agent has none. It names the repository and the starting branch. Once approved, it covers the rest of that run and no other run. A refusal stops every edit in that run. No setting skips this step.

Who a commit and a push belong to

A commit is authored by the person who owns the run. The agent cannot pass a name or e-mail address. A push uses that person's own forge credential. Nextcloud refuses the push when the run's model credential belongs to someone else. A run without an owner can read but cannot commit or push.

What is recorded

Every edit, commit and push leaves a workspace-write entry in the agent's audit trail. The entry names the run, the tool, the owner, the credential a push used, the approval, the arguments and the outcome. A refused call is recorded with its reason. File contents, secrets and server paths never enter the record.

Bringing one up

1. Build

cd exapp/llm-runner
docker build -f Dockerfile.combined -t hermiq-runner:combined .

2. Prime the speech models

Speaches does not lazily download models — it answers 404 Model '<id>' is not installed locally until each one is installed. This is the only step that needs egress.

cd exapp/speech-runner/deploy
docker compose --profile prime run --rm speech-primer

Or against a running container — through docker exec, because there is no host port:

docker exec hermiq-speech curl -X POST \
  http://localhost:8000/v1/models/deepdml/faster-whisper-large-v3-turbo-ct2
docker exec hermiq-speech curl -X POST \
  http://localhost:8000/v1/models/speaches-ai/Kokoro-82M-v1.0-ONNX

curl http://127.0.0.1:8000/… never worked, and said nothing

This page used to give the commands above without docker exec. Docker cannot NAT a published port into a container attached only to an internal: true network, and it does not complain: docker ps shows a bare 8000/tcp, docker port hermiq-speech prints nothing at all, and no process listens on the host. The compose file's ports: entry and its network jail contradicted each other, and the network won, silently. The ports: entry has been removed rather than left there implying a reachability that never existed.

Use a fresh volume

Speaches runs as ubuntu (uid 1000). The single-purpose Whisper/Kokoro images run as root and cache under a different path, so a reused volume leaves root-owned directories Speaches cannot write. The failure surfaces as a bare 500 Internal Server Error on model install, with the real PermissionError visible only in the container log.

Model ids fail identically whether wrong or missing

Systran/faster-whisper-large-v3-turbo does not exist — Systran publishes a large-v3 conversion but not a turbo one. HuggingFace answers a non-existent repo with 401 Unauthorized, not 404, because it will not leak whether a repo exists. So a wrong id does not read as a typo; it reads as an auth problem, while the container crash-loops on LocalEntryNotFoundError and the network is provably fine.

GET /v1/registry?task=text-to-speech lists valid ids.

3. Run

cd exapp/speech-runner/deploy
docker compose up -d

3a. Let Nextcloud reach it — the step that was missing

The sidecar is on a jailed network of its own, so Nextcloud cannot reach it until you join that network. Attach the Nextcloud container, rather than publishing a port or giving the sidecar a route:

docker network connect <compose-project>_speech-jailed <nextcloud-container>
docker exec -u www-data <nextcloud-container> \
  php occ config:app:set hermiq speech_base_url --value="http://hermiq-speech:8000"

This costs nothing in posture, and it is worth being precise about why: the sidecar still gets no gateway. Verified after doing it — /proc/net/route inside hermiq-speech holds exactly one entry, its own subnet, with no 00000000 (default) destination, and an outbound call to huggingface.co still fails, while the Nextcloud container resolves hermiq-speech and gets 200 from /health.

Skipping this makes Nextcloud advertise speech it cannot perform

registerTaskProcessingProvider() succeeds whether or not the sidecar is reachable, so GET /ocs/v2.php/taskprocessing/tasktypes offers core:audio2text and core:text2speech to Assistant, Talk and every other consumer regardless. Measured on the dev instance 2026-08-20: both task types advertised, getent hosts hermiq-speech failing from the Nextcloud container, and zero core:audio2text tasks ever scheduled — the feature looked present and had never once run.

4. Verify — round trip, not health endpoints

A /health 200 proves the process is listening, not that inference works. Synthesise and transcribe back:

docker exec hermiq-speech curl -X POST http://localhost:8000/v1/audio/speech \
  -H 'Content-Type: application/json' \
  -d '{"model":"speaches-ai/Kokoro-82M-v1.0-ONNX","input":"Round trip.","voice":"af_heart","response_format":"wav"}' \
  -o /tmp/rt.wav

docker exec hermiq-speech curl -X POST http://localhost:8000/v1/audio/transcriptions \
  -F file=@/tmp/rt.wav \
  -F model=deepdml/faster-whisper-large-v3-turbo-ct2 -F language=en

The transcript should match the input text.

That round trip does not exercise Nextcloud at all

It proves the sidecar works. It says nothing about the provider in between, and the provider is where this broke: AudioToTextProvider::process() accepted raw bytes and stream resources, while Nextcloud hands an OCP\Files\File node, so every task failed with No audio was supplied to transcribe. before a single byte reached the sidecar. The unit suite was green throughout — it asserted the two shapes nobody sends.

Verify through the layer that ships:

# upload a clip, then schedule a real task against its file id
curl -u user:pass -T clip.wav \
  "http://localhost:8080/remote.php/dav/files/user/clip.wav"
curl -u user:pass -H 'OCS-APIRequest: true' -H 'Content-Type: application/json' \
  -X POST "http://localhost:8080/ocs/v2.php/taskprocessing/schedule" \
  -d '{"type":"core:audio2text","appId":"hermiq","customId":"probe","input":{"input":<fileId>,"language":"nl"}}'

occ taskprocessing:worker --once --taskTypes=core:audio2text
curl -u user:pass -H 'OCS-APIRequest: true' \
  "http://localhost:8080/ocs/v2.php/taskprocessing/task/<id>"

STATUS_SUCCESSFUL with text in output is the only evidence that counts.

4a. Pick a model that matches the hardware

The default is Systran/faster-whisper-base — a CPU model, and that is deliberate. It used to be deepdml/faster-whisper-large-v3-turbo-ct2, which on CPU is roughly 12–18× realtime: not "slow", unusable. A five-second sentence took over a minute and the caller timed out long before the transcript existed, so the feature read as broken rather than as misconfigured.

A GPU host should raise it deliberately:

occ config:app:set hermiq speech_stt_model \
  --value="deepdml/faster-whisper-large-v3-turbo-ct2"

The primer installs both, so that switch needs no second priming round.

Measured 2026-08-20 on a Ryzen 7 5800H (14 vCPU, CPU-only), same 4.6s Dutch clip, model already resident, timed through the Nextcloud path (occ taskprocessing:worker → provider → sidecar) unless noted:

speech_stt_model	compute type	warm latency
large-v3-turbo	default (float)	81s (sidecar-direct)
large-v3-turbo	int8	55s (sidecar-direct)
Systran/faster-whisper-small	int8	17–23s
Systran/faster-whisper-base	int8	3.1–3.3s (4.3s cold)

Kokoro synthesis, for comparison: 9.3s for a short sentence through the same path.

A CPU deployment needs no action — faster-whisper-base is the default. The row that needs a decision is the GPU one, above.

WHISPER__COMPUTE_TYPE=int8 is set in the compose file and is worth 1.5× on its own; the model choice is worth another 6×. Raising WHISPER__CPU_THREADS is not worth anything measurable — 8 threads gave 17.5s and 23.0s against 19.4s at the default, variance larger than the effect.

⚠️ Nothing in the table is evidence about accuracy. The probe clip was an English Kokoro voice reading Dutch, so every transcript in that experiment was mangled by the input. Judge model quality on real speech from a real microphone, not on synthesised audio.

5. Confirm the jail is real

Configuration is not enforcement until it is observed:

docker exec hermiq-speech python3 -c \
  "import socket; socket.create_connection(('huggingface.co',443),timeout=6)"

This must fail with a DNS error. If it succeeds, the container is not on the internal network and "no audio leaves the instance" is an assertion rather than a guarantee.

6. Point the governed MCP endpoint at the CONTAINER-facing origin

Required for any tool-using turn on executionMode: cli. Skipping it is the one misconfiguration on this page that produces no error anywhere.

occ config:app:set hermiq mcp_run_base_url --value="http://nextcloud"

Use whatever host resolves to Nextcloud from inside the runner container — usually the compose service name, matching AppAPI's daemon nextcloud_url. Confirm it before believing it:

docker exec hermiq-llm-runner curl -s -o /dev/null -w '%{http_code}\n' \
  --max-time 8 http://nextcloud/status.php     # expect 200

Why this is not optional. ProviderFactory::buildGovernedMcpConfig() resolves the endpoint with linkToRouteAbsolute(), which returns the URL Nextcloud publishes to browsers. On a stock dev instance that is http://localhost:8080 — and inside the runner container, localhost is the container itself.

What that looked like when it happened (2026-08-15, hours to diagnose): the CLI connected to nothing, tools/list never served Hermiq's tools, and the run exited 0 with an empty stderr. The model answered confidently, describing its own built-in CLI tools, while every Hermiq tool was silently absent. Nothing in the log said so.

Since #318 the runner preflights the endpoint and refuses to spawn when it is unreachable, naming this config key in the error — a governed turn that cannot reach its governance is not degraded, it is ungoverned. The preflight accepts any HTTP answer including 401/403, because it carries no bearer token: it asks whether a connection can be made, not whether the run is authorised.

Bringing up the LLM half on its own

The steps above bring up the speech half. An instance that only wants Claude on a subscription needs the LLM half, and nothing else. Do this instead, in order.

1. Build the LLM image

cd exapp/llm-runner
docker build -t ghcr.io/conductionnl/hermiq-llm-runner:latest .
docker build -t ghcr.io/conductionnl/hermiq-egress-proxy:latest deploy/egress-proxy

Dockerfile, not Dockerfile.combined. The combined image adds the speech stack, which a CLI turn never touches.

2. Start the runner and its proxy

cd exapp/llm-runner/deploy
printf 'APP_SECRET=placeholder\nEGRESS_PDP_URL=http://nextcloud/apps/hermiq/api/egress/authorize\n' > .env
chmod 600 .env
docker compose -p hermiq-runner up -d

Start before you register. AppAPI heartbeats the container during registration, and the registration fails when nothing answers. The placeholder secret is replaced in step 5.

3. Join the jailed network

docker network connect hermiq-runner_jailed <nextcloud-container>
docker exec <nextcloud-container> curl -s http://hermiq-llm-runner:9000/heartbeat

The runner keeps no gateway. Nextcloud comes to it, the same way it does for speech.

4. Register a manual-install daemon

AppAPI deploys nothing here, because you run the container yourself. Tell it so:

occ app_api:daemon:register hermiq_manual "Hermiq manual install" \
    manual-install http "hermiq-llm-runner:9000" "http://nextcloud"

AppAPI strips the port from the host for a manual-install daemon and takes the ExApp's port from the registration instead. Give it the port there, in step 5.

5. Register the ExApp, then hand it the secret

occ app_api:app:register hermiq-llm-runner hermiq_manual --wait-finish --json-info \
  '{"id":"hermiq-llm-runner","name":"hermiq-llm-runner","version":"0.1.0","port":9000,
    "routes":[{"url":"run","verb":"POST","access_level":2,"headers_to_exclude":[]},
              {"url":"heartbeat","verb":"GET","access_level":0,"headers_to_exclude":[]}]}'

--json-info rather than --info-xml, for one reason: the manifest carries no port, so AppAPI picks a free one in the 23000 range and then heartbeats a port nothing listens on. Note that the JSON branch takes access_level as a NUMBER (PUBLIC 0, USER 1, ADMIN 2). The XML branch is the one that maps the names.

Nextcloud mints APP_SECRET during this command and stores it in oc_ex_apps. Copy that value into deploy/.env and recreate the container, then enable the app:

docker compose -p hermiq-runner up -d
occ app_api:app:enable hermiq-llm-runner

Enabling calls PUT /enabled on the runner with AppAPI's signature. A container still holding the placeholder logs AUTHORIZATION-APP-API secret does not match and stays disabled, which is the one symptom this ordering produces.

6. Verify the turn, not the heartbeat

GET /heartbeat proves a process is listening. Ask AppAPI to make the same call Hermiq makes, from inside the Nextcloud container:

$pf = \OC::$server->get(\OCA\AppAPI\PublicFunctions::class);
$r = $pf->exAppRequest('hermiq-llm-runner', '/run', 'admin', 'POST', [
    'provider' => 'anthropic',
    'model' => 'claude-opus-5',
    'messages' => [['role' => 'user', 'content' => 'Reply with the single word PONG.']],
], ['http_errors' => false]);

Without a credential the CLI exits 1 and the runner answers 502 runner execution failed. That is a pass for the transport: the signature, the runner's own check and the CLI spawn all happened. The runner log shows the turn it accepted.

7. Confirm the jail, and the one route out

# no default route, and no way to Anthropic directly
docker exec hermiq-llm-runner cat /proc/net/route
docker exec hermiq-llm-runner node -e "require('net').connect({host:'api.anthropic.com',port:443,timeout:8000}).on('error',e=>console.log(e.code))"

# the proxy answers, asks the PDP, and denies an unknown run token
docker logs hermiq-egress-proxy

/proc/net/route must hold exactly one entry, its own subnet, with no 00000000 destination. The direct socket must fail. A CONNECT carrying a bogus token must come back DENY … (pdp_rejected), which also proves the proxy reached Hermiq over the jailed network.

A run token minted on the command line will not verify

RunTokenService stores tokens in the distributed cache. With memcache.distributed unset, Nextcloud falls back to the local cache, and APCu gives a CLI process its own segment. A token minted by an occ command or a CLI script is therefore invisible to the web tier, and the PDP answers 401 invalid_token. Mint inside a real turn, or configure a distributed cache. Apache prefork shares one APCu segment across its workers, so an ordinary turn is unaffected.

Add the container-facing host to trusted_domains before a tool-using turn

Step 6 above points mcp_run_base_url at http://nextcloud. Nextcloud refuses a GET on an untrusted host with a 400 page, so add that name: occ config:system:set trusted_domains 1 --value=nextcloud. POSTs are unaffected, which is why the egress PDP works without it and a governed MCP session can still surprise you.

Known limits

• CONNECT proxying is host:port, not URL. The PDP authorises a host, not a path.
• CPU inference is slow. large-v3-turbo on a short clip can take minutes on a constrained CPU. Use the GPU image variants where a GPU exists.
• Whisper turbo cannot translate. Its fine-tuning excludes the translation task, so "transcribe this Polish call into English" is two steps. That is arguably a feature: the translation becomes visible and auditable rather than folded invisibly into transcription.
• Per-language quality varies enormously. "99+ languages, ~12% WER" is an average over a distribution that runs from ~4% to 25–35%. See the speech-services change for the measured, per-direction language matrix — and note that TTS coverage is far narrower than STT coverage.