Staff page. LabOps AI is owner-only and is not part of any student lab. Nothing on
this page is issued to students.
DigitalRCC LabOps AI is an assistant inside the LabOps console that reads a support
request (or a question typed straight into the console), works in an isolated container,
asks permission before every action, and returns a written triage answer with its activity
timeline, token usage and cost.
| Item | Value |
|---|---|
| Console | https://labops.drcc.digitalrcc.com/labops |
| Direct Chat | https://labops.drcc.digitalrcc.com/admin/labops/chat |
| Access | Owner-only for this release (eddie.barlow@tcecure.com), enforced server-side on every page and API route |
| Host | drcc-labops-01 — pve2, VMID 100, 192.168.1.65/24, Ubuntu 24.04, service on port 3100 |
| Edge | crc-proxy-gateway-01 (192.168.1.55) terminates TLS and proxies to 192.168.1.65:3100 |
| Database | DRCC production Supabase, ai_* tables |
| Not affected | crc.ai (192.168.1.61), the student portal, the training tracker, Guacamole, DNS |
crc.ai is untouched and stays as it is. LabOps AI runs its own private, headless runtime
on drcc-labops-01; no stock vendor chat UI is exposed publicly, and the console shows
DigitalRCC branding only.
Open a support request in the LabOps console → Investigate. The sanitized ticket
conversation is the agent's starting brief.
Ask LabOps AI on the LabOps dashboard, or /admin/labops/chat directly. Type a
question with no ticket at all, converse with follow-ups in the same live conversation,
and finish or stop it when done.
The page carries: conversation-history sidebar (paginated), new conversation, transcript
rendered as safe Markdown, multiline composer (12,000 characters per message, text only —
no attachments), Running / Ready / Awaiting-confirmation status, Stop and Finish,
a collapsible activity and tool timeline, the Allow/Refuse control, and live token count,
estimated cost and elapsed time.
How a direct conversation behaves:
ai_runs row is the conversation thread; user and assistant messages live in theai_messages table. No fake support ticket is ever created for a directpaused state, shown as Ready; arunning and reuses the same live container.succeeded, Stop cancels it, and a timeout or budget exhaustion| Control | State |
|---|---|
| Gateway / agent secret separation | /etc/labops/{gateway,agent,model-proxy}.env, root-owned 0640 |
| Supabase service-role key, AWX token, Wiki token in the agent container | None — the agent holds no gateway credential |
| Provider key | Only in /etc/labops/model-proxy.env; the gateway holds a via-model-proxy sentinel and never sees the key |
| Workspace | One container + volume per investigation, created on demand and destroyed at the end |
| Agent egress | forward/output default-deny; the agent reaches the run-scoped model proxy and nothing else |
| Cross-investigation traffic | Denied (tested container-to-container) |
| Docker socket in the agent | Not mounted |
| Agent policy | Always-confirm — every action waits for Allow/Refuse, and both are audited |
| Persisted output | Redacted; internal staff notes are excluded from the brief |
| Database writes | Service-role only; anon/authenticated hold no privilege on ai_* |
| Restart recovery | A gateway restart ends any in-flight run and reaps its workspace (it has already reaped a live orphan) |
Write switches — all five are false: global, awx, github, wikijs,
support_notes. The approval/execution broker schema exists, but the schema existing is
not approval for any integration: AWX, GitHub and Wiki.js writes each still need their own
implementation, credentials, validation and controlled production test. support_notes is
the first capability queued for a pilot, and gets switched back off afterwards.
Wazuh, mailbox intake, DNS changes and student-facing lab changes are deliberately outside
this work.
There is no staging environment for this project. The Supabase project DRCC-staging
(cudbheihfdvbetwtcfdi) is legacy and unavailable as a deployment target: it is a
partial structure-only copy that predates the support-ticket work and is missing
support_messages, support_attachments, integration_events and audit_events. Do not
plan work that depends on "apply it to staging first", and do not treat it as a
pre-production gate.
The procedure of record (owner-set, 2026-08-29):
raise so every row it wrote rolls back.This is acceptable because LabOps AI is not mission-critical and is not in student use.
It is not a licence to change the portal, the tracker or crc.ai in production.
| Task | How |
|---|---|
| Releases | /opt/labops/app/releases/<timestamp>, with /opt/labops/app/current symlinked at the live one |
| Deploy check | platform/labops-ai/scripts/verify-deployment.sh — every check must pass |
| Isolation checks | platform/labops-ai/scripts/test-*.sh (secrets, workspace, egress, cross-run denial) |
| Rollback (app only, preferred) | ln -sfn /opt/labops/app/releases/<previous> /opt/labops/app/current && systemctl restart labops-gateway |
| Schema rollback | The drop-based script in the matching apply log under docs/labops-ai/ |
| Logs | journalctl -u labops-gateway; run history and audit rows live in Supabase |
| Backup | PBS snapshot of VMID 100 is still outstanding — the host holds no authoritative data, but in-flight runs and /etc/labops secrets are not yet covered |
Restarting the gateway ends any in-flight conversation by design, so do not deploy while
someone is mid-run.
Model spend is billed to the DigitalRCC OpenAI account. A pilot triage run cost roughly
$0.05–$0.10; the gateway caps spend per run and records usage per turn in
ai_model_usage.
svc-drcc-labops-ai-ro) and PBS| Page | Purpose |
|---|---|
| Architecture Overview | Where this host sits in the estate |
| VM Inventory | Host record |
| DigitalRCC Portal Integration | Portal and tracker integration |
| AWX Automation | Automation the broker will eventually call |
| Backlog & Known Issues | Outstanding work |