Load Testing - Unnecessary but Interesting

I ran a load test for four minutes. The box was unresponsive for twelve.

That wasn’t the plan. The plan was to find the ceiling before I shared the site publicly, not after. I found the ceiling. Then I ran one more internal test to see what the actual failure mode looked like, and discovered something more interesting: a self-reinforcing aftermath where the kernel cleanup outlasted the attack by a factor of three.

Stack under test:

• Traefik for TLS termination + routing + CrowdSec bouncer
• nginx serving the static site (cpu_shares 256, 128 MB limit)
• WireGuard tunnel to the VPS for the external path
• Critical services sharing CPU under the same cgroup scheduler: DSM, Authentik, Grafana, Immich, Pi-hole
• NAS CPU: Synology R1600, 2 cores / 4 threads. wrk was run with -t4 throughout.

Saturation#

Internal path first. I resolved gread.uk to 192.168.1.16 via a Pi-hole local DNS record to hit Traefik directly rather than round-tripping through the VPS.

wrk -t4 -c50 -d60s --latency https://gread.uk

At 50 concurrent connections: 437 RPS, P50 ~60ms. Host CPU hit ~90% peak. Memory didn’t move. Traefik alone was at 96% CPU — not nginx, not the NAS as a whole.

Doubled to 100 concurrent:

RPS flatlined. Latency doubled. That’s the saturation curve: Traefik has hit a ceiling and more connections just queue more requests without increasing throughput.

Ruling out the usual suspects#

I went through the obvious levers:

Nothing moved the needle. The CrowdSec result surprised me — I expected the bouncer to add meaningful per-request overhead. It didn’t.

Subtler finding: with the bouncer removed, the CrowdSec container CPU still climbed to ~77% during the flood. That wasn’t the bouncer. The log-parsing pipeline was chewing through Traefik access logs in real time — 440 RPS is a lot of lines per second to tail, tokenise, and evaluate. Per-request cost: negligible. Log-ingestion cost: not.

Isolating Traefik from TLS#

Direct nginx on plain HTTP gave 703 RPS vs Traefik’s 439 — looked like Traefik was costing 37% of throughput. It was an unfair comparison. Traefik terminates TLS; the direct test didn’t. TLS handshakes are expensive and any proxy pays them.

To isolate Traefik’s actual overhead I spun up a throwaway Caddy sidecar doing TLS termination + reverse proxy to the same nginx backend, self-signed cert, port 9443. Same backend, both doing TLS:

Caddy averaged ~383 RPS, Traefik ~355 RPS. Traefik’s overhead above bare TLS termination — middleware chain, CrowdSec bouncer, log shipping, routing — is around 7–8%. Not 37%.

Two things from the data worth flagging: Traefik had better tail latency than Caddy (P99 ~1.0s vs ~1.2s), and Traefik’s RPS was dead stable run-to-run while Caddy varied by ~9%. The Caddy sidecar had no resource limits; Traefik runs at cpu_shares 1024. Unconstrained proxy, more variance. Worth knowing if you ever benchmark “bypass” comparisons — the bypass path needs to be tuned comparably or the comparison is meaningless.

The ceiling isn’t “Traefik-specific.” It’s TLS termination on an R1600 at ~400 RPS. Switching to Caddy buys single-digit percent, not a different ceiling.

Blast radius#

The real question: does DSM become unusable when the site gets hammered? The cpu_shares system is supposed to let SSH, DSM, and Authentik keep their share even when the site is eating most of it. Time to verify rather than assume.

I ran a probe script throughout — curl every 5 seconds, each request in a backgrounded subshell so a slow response doesn’t drift the next interval, separate output files per service to prevent write interleaving:

#!/bin/bash
TS=$(date +%Y%m%d_%H%M%S)
INTERVAL=5

probe() {
    local service=$1
    local url=$2
    local out="probe_${service}_${TS}.csv"
    echo "timestamp,response_time_seconds,http_code" > "$out"
    while true; do
        local start=$(date +%s)
        local ts=$(date +%H:%M:%S)
        (
            local result=$(curl -sk -o /dev/null -w "%{time_total},%{http_code}" --max-time 10 "$url")
            echo "$ts,$result" >> "$out"
        ) &
        local elapsed=$(( $(date +%s) - start ))
        local remaining=$(( INTERVAL - elapsed ))
        [ $remaining -gt 0 ] && sleep $remaining
    done
}

probe "dsm" "https://nas.gread.uk/" &
probe "authentik" "https://auth.gread.uk/" &

trap "echo 'Stopping probes...'; kill 0" SIGINT
wait

Baseline vs 100-concurrent load:

Both stayed alive. Every probe returned 200/302. Authentik is more load-sensitive (session validation, OAuth flows) and had one isolated spike to 1.17s. DSM is mostly static and held better.

Extended to 5 minutes at 100 concurrent: 125,103 requests, 417 RPS sustained, 1 timeout. DSM hit a 6.2s spike, Authentik 5.2s — isolated, immediate recovery once wrk stopped. Zero failures.

The cpu_shares system is doing its job.

Through the tunnel#

External test: wrk from the VPS through the WireGuard tunnel. I wrote a small Lua script to rotate across real pages (/, /about, /writing, individual post slugs) rather than hammering the homepage — cache hit rates, page sizes, and per-route middleware all differ, so a single-route test isn’t representative.

Higher RPS than the internal path because the Lua script rotates through lighter pages on average. At 500 concurrent the external path started falling over. Timeouts, degraded latency, no bans.

CrowdSec didn’t fire#

I checked the banned-IP list expecting to find myself there. Nothing. Every active scenario was pattern-based: bad user agents, CVE probes, sensitive file paths, XSS signatures. None of them care about request volume. 900 RPS from a single IP, if it looks like legitimate browsing, passes right through.

CrowdSec was doing exactly what it was configured to detect — the ban log showed Bucklog SARL probing .env files, Censys scanning with bad user agents, Azure IPs doing WordPress scans. All working correctly. The gap was volumetric, and I hadn’t configured anything for that.

The failure mode was at 500 concurrent connections, not high RPS. A connection-count problem, not a rate problem. The fix lives at a different layer.

Connlimit at the VPS#

The VPS runs nginx as a TCP stream proxy — no HTTP-level rate limiting, and stream_limit_conn isn’t compiled into the default Ubuntu 24.04 nginx build. The right place to cap concurrent connections per IP is the firewall, before nginx sees a packet.

UFW doesn’t expose connlimit directly, so it goes into /etc/ufw/before.rules, before the COMMIT marker (rules after COMMIT are invalid iptables syntax and ufw-init will fail):

python3 -c "
with open('/etc/ufw/before.rules', 'r') as f:
    content = f.read()
if 'connlimit' not in content:
    rules = '''# Simultaneous connection limit per IP
-A ufw-before-input -p tcp --dport 443 -m connlimit --connlimit-above 100 -j REJECT --reject-with tcp-reset
-A ufw-before-input -p tcp --dport 80 -m connlimit --connlimit-above 100 -j REJECT --reject-with tcp-reset
'''
    content = content.replace(\"# don't delete the 'COMMIT'\", rules + \"# don't delete the 'COMMIT'\")
    with open('/etc/ufw/before.rules', 'w') as f:
        f.write(content)
"
ufw --force enable

One gotcha: Ubuntu 24.04 ships iptables as a shim over iptables-nft. After applying the rule, iptables -L ufw-before-input -n | grep connlimit returned nothing — looked like it hadn’t loaded. It had, just into nftables:

nft list ruleset | grep connlimit

Counters at 0 just meant nothing had hit the limit yet.

Same 500-concurrent flood with the rule in place:

wrk tried to open 500 concurrent connections. The VPS rejected 331,667 attempts with TCP resets before a single packet reached the WireGuard tunnel. The ~100 that got through behaved normally.

What actually broke#

The external gate was solid. That should have been the end of it.

Then I ran an internal bypass test — resolving gread.uk to the NAS IP via Pi-hole, bypassing the VPS entirely — to see the actual failure mode if something got through. A ladder:

The three tests took under four minutes including cooldowns. Recovery took twelve.

From 14:45 to 14:57, every route behind Traefik was dark: the site, DSM, Authentik, Grafana, Pi-hole, Portainer, Immich, Homepage. Traefik wasn’t crashed. It was alive the whole time — 29-hour process uptime, 206 MB RSS, nowhere near OOM. It simply couldn’t drain its own queue faster than monitoring probes refilled it.

top during the outage: load average 9.36 on a 4-thread machine (2.3x oversubscribed). CPU 79.5% in sy (kernel), 6.4% us.

node_exporter recorded the whole thing:

The number that tells the whole story: user CPU peaked at 14:40, during the active attack. System CPU peaked at 14:51, eleven minutes later, during the cleanup. The attack was brief. The kernel cleanup was the outage.

conntrack hit 8,663. The limit on this machine is 262,144 — nowhere near overflow. The bookkeeping burden itself was the problem. Every packet from every dying connection still traversed nftables, softirq handlers, Docker bridge NAT, and conntrack lookup. Raising the limit would have done nothing. Lowering the TIME_WAIT timeout would have shortened the drain.

Why it outlives the attack#

Once the stack entered this state, every actor made recovery slower:

• Kernel cleaning up dead sockets at 120s TCP TIME_WAIT per entry
• Traefik’s Go runtime with thousands of goroutines parked on net.Conn close signals, queued behind softirq work
• Docker healthchecks probing Traefik every 30s, Prometheus scraping every 15s, Uptime Kuma running monitors, Homepage polling widgets

Every new connection added a conntrack entry and an accept-queue slot. Monitoring was prolonging the outage it was measuring.

Prometheus recorded up{job="traefik"} = 0 from 14:46:53 to 14:55:53 (9 minutes 30 seconds). User-facing outage was 12 minutes. The extra 2.5 minutes: the kernel still had ~1,500 conntrack entries to drain after Traefik was scrape-healthy again.

A manual docker restart traefik short-circuited the loop. Traefik exits, kernel closes all its sockets with RST immediately, 200 MB heap and tens of thousands of goroutines gone in one shot. Fresh Traefik starts with an empty accept queue, other containers get their CPU share back, healthchecks go green and stop probing. Full recovery in ~30 seconds.

Closing the gaps#

“SSH in and restart Traefik” is an escape hatch, not a solution.

The external moat is already there. The connlimit means no external IP can reach the concurrency needed to trigger this. The ladder required bypassing the VPS entirely. For any realistic external threat, this state is already unreachable.

Traefik inFlightReq middleware. Caps concurrent in-flight requests per IP; excess gets a fast 429 instead of entering the accept queue.

# dynamic config (file provider)
http:
  middlewares:
    global-inflight-limit:
      inFlightReq:
        amount: 400

# static config
entryPoints:
  websecure:
    http:
      middlewares:
        - global-inflight-limit@file

Real browsers use 6 to 10 concurrent connections. 400 is far above normal behaviour and still prevents the wedge. Same 500-concurrent attack with the middleware live:

Peak load was similar — 500 concurrent still generates real CPU work regardless of what the response body is. What changed was the aftermath. 429s are short-lived and don’t leave half-completed TLS handshakes sitting in the kernel, so conntrack stayed 68% lower and drained on its normal schedule. Traefik never disappeared from Prometheus.

One gotcha: inFlightReq 429s don’t appear in Traefik’s Prometheus counters. Middleware rejections at the entrypoint short-circuit before the metrics middleware wraps them — so traefik_entrypoint_requests_total, traefik_router_requests_total, and traefik_service_requests_total all showed zero 4xx during the attack. This is a known Traefik limitation. The fix is to alert on the access log instead:

sum by (_router) (
  rate({job="traefik-access"}
    | json
    | _status >= 400 [5m])
) > 5

Promtail was already shipping Traefik’s JSON access log to Loki with _status extracted. Ground truth, per router.

autoheal sidecar. Docker’s default is to mark a container unhealthy and do nothing. docker ps showed “Up 29 hours (unhealthy)” for ten minutes before I stepped in. autoheal restarts unhealthy containers after a grace period:

services:
  autoheal:
    image: willfarrell/autoheal:latest
    restart: always
    environment:
      AUTOHEAL_CONTAINER_LABEL: autoheal
      AUTOHEAL_INTERVAL: 30
      AUTOHEAL_START_PERIOD: 60
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
  traefik:
    labels:
      - "autoheal=true"

Wired up but not yet tested against a real wedge. The next organic unhealthy event is the real test.

Optional kernel smoothing: net.ipv4.tcp_fin_timeout=15 (default 60) and net.ipv4.tcp_abort_on_overflow=1 (default 0) halve the drain window when it does happen. Doesn’t prevent the wedge, just speeds cleanup.

What I didn’t test#

Distributed attacks. Everything here was single-IP. connlimit is per-IP — useless against traffic from many sources. A proper DDoS would sail straight through, and that’s not a problem a homelab is going to solve on its own.

nginx caching and Varnish. Both are plausible levers for moving the Traefik ceiling. The current ~400 RPS ceiling is enough headroom for actual traffic here, so I’m not paying the operational cost of another caching layer yet.

Whether autoheal catches a real wedge. It’s configured. The next organic failure is the real test.

Summary#