Single-node Ubuntu Server 24.04 LTS · Hostinger VPS

Production-style infrastructure,
built and audited in public.

A containerized edge/application stack behind Traefik, a host-native AI agent under systemd, layered host hardening, and monitoring proven by a deliberate failure test. Most of the engineering effort went into auditing and verification, not installation — and the parts that are unfinished say so.

Read the full case study See the architecture
Public attack surface
22 · 80 · 443

No application port published on the host.

Edge
Traefik v3

TLS termination, ACME, HTTP → HTTPS redirect.

Agent privilege
No sudo, no Docker

Blast radius bounded by Unix permissions.

Method
INSPECT → PLAN → CHANGE → VERIFY → DOCUMENT

AI-assisted engineering with human validation.

Scope note. Encrypted off-site backup is implemented, automated and validated by a real data restore. A full-VPS rebuild drill and a measured recovery time are still pending. Nothing on this page is claimed before it is true — see current status.

01 — System

Architecture

Every request enters through the same door. Application containers publish nothing on the host; they are reachable exclusively through the reverse proxy over an internal Docker network.

  • One door, one owner of TLS

    Adding a service means adding Traefik labels, not opening a host port. The Traefik dashboard is not publicly exposed and insecure is off.

  • Read-only Docker socket

    Traefik only needs to read container metadata. Read-only mounting limits — but does not eliminate — the blast radius; socket access is still privileged and treated as such.

  • The agent sits beside Docker

    Not inside it. It declares no listener, holds no admin SSH keys, and is not in the docker group.

02 — Workloads

What is running

Four components, each with a stated job and a stated boundary.

Traefik v3

Operational

Reverse proxy · TLS · routing

Terminates TLS with ACME certificates, redirects HTTP to HTTPS, and routes to containers by Docker label. The only container publishing ports on the host.

  • Docker provider
  • ACME
  • no-new-privileges

nginx

Operational

Static landing page

The public entry point, and the workload used for the deliberate failure test that validated the alerting chain end to end.

  • static
  • proxy network only

n8n

Operational

Automation orchestration

Holds real credentials, so it is the stateful component that drives the backup design. Monitored on its /healthz/readiness endpoint — usability, not "the process exists".

  • SQLite (WAL)
  • encryption key in volume

Uptime Kuma

Operational

Health monitoring & alerting

HTTP checks against the public endpoints plus a push monitor for the host-native agent. Alerts go to a chat channel via a bot deliberately separate from the agent's.

  • SQLite
  • push + HTTP monitors

Hermes agent

Operational

Host-native AI agent under systemd, with constrained Unix privileges

Runs as its own dedicated user managed by systemd — not as root and not inside Docker. It has no sudo rights, no docker group membership, no access to the Docker socket, no copy of the administrator's SSH keys, and no publicly exposed port. Its liveness is proven by an external heartbeat rather than self-reporting: a systemd timer checks every 60 seconds whether the gateway unit is actually active, and pushes only when it is.

  • dedicated user
  • no sudo
  • no Docker socket
  • no listener
  • systemd-managed

03 — Hardening

Security model

Layered, and each layer validated independently rather than assumed. The controls below were confirmed active against the running system, not merely written into files.

01 Perimeter

  • Provider firewall in front of the host
  • UFW: default deny inbound, default deny forward
  • IPv6 handling explicitly enabled, so IPv6 is not an unexamined blind spot
  • Only 22, 80 and 443 reachable from the internet

02 Access

  • SSH public-key authentication only
  • Root login, password auth, keyboard-interactive and empty passwords disabled
  • Reduced MaxAuthTries and LoginGraceTime
  • Fail2ban jail active on the SSH service

03 Host

  • AppArmor and auditd enabled and active
  • journald with bounded retention
  • unattended-upgrades for security patching
  • sysctl network and kernel hardening, observed at runtime

04 Privilege

  • No passwordless sudo, for any user
  • Admin user deliberately not in the docker group — group membership is effectively passwordless root
  • Docker administration through interactive sudo docker …, so every privileged action is explicit
  • The AI agent has neither sudo nor Docker access

05 Application

  • Services isolated on an internal Docker network, IPv4-only, no host port publication
  • no-new-privileges:true on edge containers
  • Image tags pinned, never latest
  • Secret-bearing files held at restrictive modes and excluded from Git by policy and by .gitignore

06 Stated openly

  • The Docker socket is privileged even when mounted read-only
  • Image tags are pinned but not yet resolved to immutable digests
  • Monitoring shares a host with its subject
  • No restore test has been performed

04 — Reliability

Monitoring & alerting

Two complementary signals, and one blind spot that is documented rather than hidden.

Signal 1

HTTP checks

Uptime Kuma polls the public endpoints through the same path a user takes. Alerts route to a chat channel using a bot deliberately separate from the AI agent's bot, so a failure of the agent cannot silence alerting.

n8n is checked on /healthz/readiness rather than /healthz: all three health paths return 200, and readiness is the one that reflects usability.

Signal 2

Push heartbeat

The host-native agent exposes no HTTP endpoint, so it cannot be polled. A systemd timer fires every 60 seconds, a oneshot unit checks whether the gateway service is genuinely active, and pushes to Uptime Kuma only when it is. A missing push raises an incident.

Verified live: timer enabled and waiting, last invocation exiting 0/SUCCESS, next firing scheduled, script present at restrictive ownership and mode — contents deliberately left unread, because it is secret-bearing.

Validated

The alert was tested, not just configured

  1. 1 Landing container deliberately stopped
  2. 2 Uptime Kuma detected DOWN
  3. 3 Alert delivered to the chat channel
  4. 4 Container restarted
  5. 5 Recovery alert delivered

The whole chain is validated against a real workload rather than against a config file. Traefik's /ping was deliberately not enabled: application endpoints already give a stronger signal, and enabling it would add surface for no gain.

05 — Recoverability

Backup & Disaster Recovery

Encrypted off-site backup is implemented, automated and verified by a real restore. The honesty of this section is still the point: the data-restore path is proven, a full-VPS rebuild has not been drilled, and no recovery time is claimed until it is.

Implemented

  • Data classified before anything was backed up — critical mutable data, secret-bearing recovery material, reproducible configuration, operational docs, disposable runtime data
  • Path-level backup inventory complete — and now the executed scope of a running pipeline, not a plan
  • Six live SQLite databases captured through the engine's own online-backup API instead of a file copy; every snapshot integrity-checked, 6/6, before and after restore
  • Restic archives, encrypted and deduplicated, pushed off-site to a private Backblaze B2 bucket under separate custody — with the repository password and access credentials also held outside the server they protect
  • Daily automation under systemd: a oneshot service and a persistent timer that catches up after downtime, serialized by a lock, with retention of 7 daily, 4 weekly and 6 monthly snapshots
  • Weekly maintenance on its own timer — prune plus a full repository integrity check — sharing that same lock so it can never overlap a backup
  • A real restore executed from off-site storage: all six databases passed their integrity checks and every recovery-critical file was present, including the automation platform's encryption-key file
  • Reproducible configuration under Git — Compose files, Traefik config, systemd units, host hardening, runbooks — plus a clean-host bootstrap sequence with exact version pinning and a local agent patch preserved with its base commit and a verified SHA-256

Still open

  • Not drilled — a full provision-to-operational rebuild: new host, security baseline, container platform, restore, service recreation, DNS and TLS validation
  • Not measured — the recovery time for that rebuild. Restoring the data is proven; rebuilding the whole system is not, so no number is published
  • Missing — external backup-failure alerting. Failures are recorded locally, but nothing off-host reports a missed run or a failed integrity check, and an alarm living on the VPS cannot announce the loss of that VPS
  • Partial — the DR runbook covers the proven data-restore path; the full rebuild sequence stays provisional until it has been drilled
  • Undecided — what role provider-side snapshots play in the recovery policy. A possible additional layer, not a replacement for encrypted off-site backup

Backup pipeline, as it runs

Validated
  1. SourceLive SQLitesix databases · n8n, Uptime Kuma, agent state
  2. CaptureConsistent snapshotonline-backup API, WAL-aware, integrity-checked
  3. StageStaging directoryroot-only, rebuilt and removed on every run
  4. ArchiveResticencrypted, deduplicated, retained 7/4/6
  5. Off-siteBackblaze B2private bucket, separate custody
  6. GateRestore testpassed — restored, integrity-checked, verified

06 — Method

AI-assisted engineering, with human validation

This project was built with heavy AI assistance and deliberate human control. Both halves are true, and stating only one of them would be dishonest.

Model output is an engineering aid or a hypothesis. It is never ground truth.

The operating rule the workflow exists to enforce — so that the system, not the model, gets the last word.

What the AI did

  • Investigation and coverage across a large surface
  • Command drafting
  • Configuration review
  • Risk identification and architecture discussion
  • Troubleshooting and documentation drafting

What the human did

  • Approved every privileged command
  • Executed sudo interactively
  • Validated actual system state and checked results
  • Decided whether to apply changes, and tested behaviour
  • Decided what counts as proof

Privilege boundaries enforced by the operating system, not by prompt instructions

  • No passwordless sudo, any user
  • Admin user outside the docker group
  • Agent user: no sudo
  • Agent user: no Docker socket
  • No admin SSH keys in the agent account
  • No publicly exposed agent port
  • Workspace-scoped by default
  • No Git remote or push without specific approval

The design rule: an assistant's blast radius should be bounded by Unix permissions, not by its own good behaviour. Model behaviour is a probability distribution; file permissions are not.

07 — Evidence

Four times investigation changed the answer

Each of these started as a plausible conclusion and ended somewhere else, because the system was consulted instead of assumed.

Configuration is a hypothesis

The SSH config said two different things

The base sshd_config shipped with a late PermitRootLogin yes, while an Included drop-in set PermitRootLogin no. OpenSSH takes the first obtained value, so the drop-in wins — but that is a claim about parsing order, not evidence. It was settled by reading the effective configuration. 

sudo sshd -t                          # syntax check before any reload
sudo sshd -T | grep '^permitrootlogin '
# → permitrootlogin no

Outcome: on-disk configuration is a hypothesis; sshd -T is the evidence. Had the reasoning been wrong, the difference between "we discussed it" and "we checked it" would have been remote root login.

Anomaly, not vulnerability

net.ipv4.ip_forward=1 looked wrong, and was not

IP forwarding was enabled at runtime with no assignment anywhere in /etc/sysctl.conf, /etc/sysctl.d/, /usr/lib/sysctl.d/ or /lib/sysctl.d/. Correlating with the live Docker iptables chains and a DROP IPv4 FORWARD policy showed it to be runtime state owned by Docker bridge networking.

Outcome: recorded as an inference, not a fact — no persistence file proves ownership. The operational conclusion followed from the uncertainty: leave it alone, and do not add a duplicate persistent sysctl entry asserting an ownership nobody established.

Reconciliation, not correction

A 302 that was not a bug

An unprivileged audit fetched the monitoring root without following redirects and got 302 → /dashboard, appearing to contradict an earlier "final 200" record. Both were correct — they measured different points in one flow (GET / → 302 → /dashboard → 200).

Outcome: the real finding was a hidden dependency — the monitor is only correct while it follows redirects. Accepting 200–299 without following would turn a healthy service into a permanent false alarm.

Storage engines have opinions

WAL behaviour redesigned the backup

Both application databases run SQLite in WAL mode, and at inspection each -wal file was larger than its main database — nearly three times larger for the automation platform. A naïve copy of the .db during writes would have omitted most of the live state.

Outcome: the pipeline was designed around an application-consistent snapshot before any backup code was written — and the audit also surfaced the encryption-key file that a restore would silently need.

Other checks performed against the running system

  • Host services confirmed enabled and active: UFW, Fail2ban, AppArmor, auditd, unattended-upgrades, journald retention
  • Runtime sysctl state observed directly — reverse-path filtering, syncookies, ICMP redirect handling, kernel pointer and dmesg restrictions, hardlink/symlink protection
  • Unexplained loopback listeners investigated to completion and classified expected / benign / localhost-only, with evidence
  • IPv6 exposure audited separately from IPv4: Docker networks confirmed IPv6-disabled, the shared proxy network confirmed IPv4-only
  • Application health endpoints probed and compared before a monitor path was chosen
  • A privileged read-only inventory pass that closed gaps an earlier unprivileged audit had explicitly flagged as unverifiable, instead of filling them with assumptions

08 — Roadmap

Current status

Where something is unfinished, it says so. Where a conclusion is an inference rather than a measurement, it says that too.

Status of each area of the infrastructure project
AreaStatus
Edge routing, TLS, HTTP → HTTPS redirectOperational
Application stacks behind the proxyOperational
Host hardening baselineOperational, audited
Agent under systemd with bounded privilegesOperational
Monitoring + alerting, validated by a real failure testOperational
Path-level backup inventoryImplemented
SQLite consistency mechanism, six databasesValidated
Restic + Backblaze B2, encrypted off-siteOperational
Automated daily backup, retention and weekly maintenanceOperational
Restore test, dataValidated
Full-VPS rebuild drill and measured RTONot performed
DR runbookPartial
External backup-failure alertingPlanned
Independent external monitoringPlanned
Image tags resolved to immutable digestsOpen

Backup is no longer the open item: it runs on a schedule, it is verified, and a real restore has been performed. What is still open — a timed full-VPS rebuild drill and off-host failure alerting — is written here as open rather than folded quietly into the word recoverable.

09 — Source

Read the documentation

The public repository contains documentation and sanitized examples only. No hostnames, IP addresses, credentials, tokens, keys, database files or production paths appear in it — every example configuration is an educational reconstruction using placeholder values.

Relevant to

AI Automation Engineer · Automation Engineer · Platform / Infrastructure Engineer · DevOps-adjacent · Solutions Engineer — containerized service delivery behind a reverse proxy, host hardening and privilege design, monitoring validated by real failure, backup reasoning grounded in storage-engine behaviour, and a documented method for working with AI assistants in privileged environments.

Open the repository