Detect intrusions from Traefik and sshd logs with CrowdSec

Decision

Run CrowdSec on every node against the sshd auth log and the Traefik access log. When a log line matches a scenario, CrowdSec bans the source IP at two points: an nftables firewall bouncer, and the CrowdSec bouncer plugin as Traefik middleware in stream mode. Each node also pulls CrowdSec's community blocklist, which bans IPs reported by other CrowdSec users before they reach us.

Each node runs its own local API on loopback. All of them store alerts and decisions in one Postgres database on the Patroni cluster, so a ban raised on one node applies on every node.

Why

The cluster exposes SSH and Traefik to the internet and gets scanned constantly. Almost all request traffic is TLS, and Traefik terminates it, so the plaintext request exists only inside Traefik and in the lines it logs. A detector that reads logs sees what a packet inspector on the wire cannot.

We ran Suricata first. It saw only encrypted payloads, raised false positives from TCP stream reassembly, used heavy resources and needed constant rule tuning.

Log-based detection is reactive. The first request, or the first few that trip a scenario, get through. For this app that is acceptable: a probe that gets an error back is reconnaissance, and the community blocklist stops most known scanners in advance.

Rejected alternatives

• Suricata. It cannot see inside TLS, which is most of the traffic, and it cost more to run and tune than it caught.
• CrowdSec's inline AppSec WAF. Every legitimate request pays an extra hop through it, and it adds a component that can fail, for protection a pre-launch GraphQL API does not need yet. We revisit this if the app handles payments or other sensitive data, or draws targeted attacks.
• One shared local API for all nodes. It is a single point of failure for banning. Per-node APIs over a shared, replicated database keep banning up when a node dies.

Consequences

Detection adds no latency to requests. CrowdSec sees only what sshd and Traefik log, so a payload that never reaches a log line goes unnoticed. The decision store depends on the Patroni primary: a node cannot register its bouncers while the primary is unreachable.