What it is

Predictive Inbox is WolfStack's unified surface for forward-looking infrastructure findings — problems the system catches before they become incidents. Each finding carries the evidence used to produce it; nothing executes until you choose to apply it. Snooze, dismiss with a reason, or acknowledge as intentional — operator intent is sticky and survives across analyzer runs.

Click the 🔮 icon in the top nav (between Issues and Apps & Tools). On a healthy cluster, "No predictions right now" is the correct answer.

What it watches

Every cluster node samples these every five minutes. Findings aggregate across the whole cluster into one inbox via /api/proposals/cluster (cached 30 s server-side; per-peer fetch failures show as a yellow “Inbox may be incomplete” banner — never silent gaps).

Capacity & trend

  • Host filesystem fill — linear-fit ETA per real mount (pseudo-filesystems filtered). Flags when a disk will hit 95% based on the last 30+ minutes of growth. Critical at <6h ETA, High at <48h, Warn at <7d.
  • Docker container storage-fill — per running container, runtime-specific finding type so acks scope precisely. Container-ID rotation guard so a recreated container doesn't fit a regression line through samples from two different instances.
  • LXC container storage-fill — same shared verdict logic.
  • VM disk-fill — qcow2 sparse-file size vs. allocated, per WolfStack-managed VM.

Behaviour

  • Docker restart-loops — flags containers whose RestartCount climbs ≥3 in 10 minutes (or ≥5 = Critical). State == “restarting” bumps severity one tier.
  • Backup freshness — per enabled schedule. Warn at 2× the configured interval missed, High at 4×, Critical at 7× (or never-ran).
  • Certificate expiry — Let's Encrypt certs (via certbot) and WolfStack-managed TLS files in /etc/wolfstack/tls/. Warn at <14d, High at <7d, Critical at <3d or already-expired.

Thresholds

  • Host CPU / memory / disk-free / swap / load / failed systemd units — the unified threshold analyzer that replaces the legacy duplicate in alerting.rs + Issues page. CPU Warn at 75%, Critical at 90%; memory Warn at 80%, Critical at 90%; load Warn at 1× CPU count, Critical at 2×.
  • Container memory pressure — Docker + LXC, runtime-specific finding type. Warn at ≥80% of cgroup limit, High at ≥90%, Critical at ≥95%. Replaces the legacy per-2s threshold dispatch.

WolfNet & routing health

  • WolfNet peer reachability — pings every WolfNet peer configured in /etc/wolfnet/config.toml. Three attempts (1s timeout each) per tick gives free hysteresis against a single dropped packet. Unreachable peers emit a wolfnet_peer_unreachable finding; auto-clears the moment ping succeeds again. When the peer's endpoint is RFC1918 and this node has a public IP, the title surfaces the actual diagnosis ("peer advertises a private endpoint — unreachable from this node") instead of generic "handshake stuck" advice that sends you chasing kernel ghosts.
  • Missing WolfNet subnet route — cross-checks every remote peer's workload_subnets against this node's WolfRouter subnet_routes. Catches the case where peers are reachable (you can ping their WolfNet IP) but VMs / LXCs / Dockers behind them aren't (no route for their workload subnet). Each finding names the exact CIDR + gateway to add via WolfRouter.
  • WolfNet DHCP collision — flags multiple nodes handing out DHCP on overlapping ranges, which silently corrupts container IP assignment.
  • Docker / WolfNet bridge collision — Docker's default 172.17.0.0/16 bridge often overlaps with a WolfNet subnet. The analyzer detects the collision and proposes either renumbering the Docker bridge or adjusting WolfNet's CIDR.

Compromise indicators

  • Tamper detection — tracks SHA-256 of every WolfStack-managed config file in /etc/wolfstack/ plus key system files. Out-of-band edits surface as a finding with the diff context so the operator knows what changed and when.
  • OSV vulnerability scanner — consults the OSV.dev database for every installed package across the cluster. Severity-filtered by default (low advisories suppressed) so the inbox isn't drowned in CVE-noise.
  • Threat-intel match — recent connections / failed-login sources are checked against the active threat-intel feeds (FireHOL, Spamhaus DROP, Emerging Threats). A match becomes a finding with the originating feed cited so you can validate it.
  • Unused package detector — flags packages that haven't been touched in >90 days & aren't referenced by any active service. Reduces the host's attack surface; never auto-removes — operator confirms.

Security posture

  • Service bound on a publicly-reachable interface — first analyzer to use the unified NetworkReachability classifier. Severity scales with where the bind is reachable from:
    Service classPublicInternetLocalNetworkOverlayOnly (WolfNet/WireGuard)LoopbackOnly
    Docker API plain (port 2375)CriticalHighInfosilent
    Other risky port (Redis, Mongo, MySQL, Postgres, etc.)HighWarnInfosilent
    The same MariaDB on 0.0.0.0 finding that's Critical on a public VPS is correctly downgraded to Warn on a host whose only non-loopback IP is RFC1918. No more crying wolf for intentional LAN-bound services.
  • sshd PermitRootLogin yes — severity scales with sshd's actual reachability. Public sshd + root-login = Critical; LAN-only = High; overlay-only = Warn.
  • sshd PasswordAuthentication yes — same model. Public sshd + password-auth = High; LAN = Warn.

Operator actions

Every finding card carries four buttons:

  • ✓ Mark applied — you ran the suggested commands; logs the action and clears the proposal. v1 proposals are Manual — the dashboard shows the commands, you run them yourself. v2 will introduce OneClick proposals that dispatch to a strict allowlist of handlers.
  • ⏰ Snooze — presets for 4 hours, 24 hours, 1 week, 30 days. Snoozed findings hide from the inbox and don't re-fire notifications until the snooze expires.
  • ✗ Dismiss — requires a reason (logged for audit and feeds back to the analyzer). Dismissed proposals never re-fire for the same scope until manually cleared.
  • 🛡 Ack as intentional — permanent (180-day default expiry) acknowledgement that this specific (finding-type, scope) is expected. Future findings of the same shape are silenced. Re-prompts at expiry to keep stale acks from rotting.

How notifications work

When a Critical or High finding first appears in the inbox, WolfStack dispatches via the existing notification channels (Discord, Slack, Telegram, email — configured under Alerting & Notifications). One notification per finding, not one every refresh; auto-resolved findings never page; snoozed findings never page.

This is the convergence work that retired the legacy threshold dispatch in cached_status_bg. Operators no longer get double-paged for a CPU spike — the predictive layer is the single source of truth for threshold notifications.

Auto-resolve

When the analyzer no longer sees the condition that triggered a proposal — e.g. /var/lib/docker got pruned and is now at 30%, or a restart-looping container has been steady for 10 minutes — the proposal is automatically marked as condition_cleared in the audit trail. Distinct from operator-applied so the closed-loop signal stays honest.

Critical safety property: a data-source failure (e.g. a hung NFS mount that empties df) does not auto-resolve every disk-fill proposal. The orchestrator distinguishes “the analyzer evaluated this scope and found nothing” from “the analyzer couldn't see this scope at all”.

Cluster aggregation

The Inbox is a global view — it appears in the top icon row, not per-cluster. Multi-cluster operators get a cluster filter dropdown above the list, plus per-cluster grouped headers when no filter is active. Each finding card shows cluster · node · resource for instant context.

If a peer is unreachable, a yellow banner above the list names the missing node(s) and which cluster they belong to (“K of N nodes responded — Inbox may be incomplete”). The dashboard never silently shows partial data.

Why this exists

Before this convergence, threshold conditions could fire from three places: the Issues page, the threshold dispatch in cached_status_bg, and (for security posture) security.rs. Operators received duplicate notifications, with different severity scales and inconsistent suppression semantics. The Predictive Inbox is the single canonical surface; the legacy paths are retired or marked for retirement.

Active-attack scans (SSH brute-force detection, crypto miner detection, /tmp binary detection, suspicious outbound) remain in the System Check page for now — they're event-detection at a different cadence and the convergence shape is non-trivial. Tracked for a future iteration.

Where to find it

Top icon row of the dashboard, between Issues and the new ⊞ Apps & Tools drawer. The badge on the icon shows the count of pending Critical/High findings across the cluster.

Esc