auditd: actually catching what changed on your server

"watch what your server does" is advice everyone gives and nobody explains. watch it how? the answer that's been sitting in the kernel all along is auditd — the linux audit daemon. it's old, boring, installed by default on most distros, and almost nobody turns it on. here's the small setup that actually catches things.

what it is

auditd hooks into the kernel and records events: syscalls, file opens, program executions, logins. because it's kernel-level, it doesn't matter whether the action came from a shell, a cron job, or a script hiding in /tmp — everything passes through the same door and gets logged.

it's also the one detection tool that still works when attackers use your own tools instead of malware. curl being run on your server is invisible to antivirus — there's no "bad file" to match. auditd doesn't care about files. it logs who ran what, when, and from where.

the setup

on debian/ubuntu it's usually already there, otherwise:

sudo apt install auditd audispd-plugins
sudo systemctl enable --now auditd

the core idea is rules: you tell it what to watch, it logs matches to /var/log/audit/audit.log. the syntax is famously ugly, but three rules get you a decent baseline:

# watch who executes what (every curl, every ssh, logged)
sudo auditctl -a always,exit -F arch=b64 -S execve -k cmd-watch

# watch the cron directories (the classic persistence spot)
sudo auditctl -w /etc/crontab -p wa -k cron-watch
sudo auditctl -w /etc/cron.d/ -p wa -k cron-watch

# watch ssh authorized_keys (the quiet backdoor)
sudo auditctl -w /home/ -p wa -k home-watch

quick decode: -S execve = the exec syscall (a program starting), -w path -p wa = watch that path for writes and attribute changes, and -k is just a label so you can find these events again later.

for reading the log

raw audit logs are brutal, so search by the keys you defined:

# who ran curl recently
sudo ausearch -k cmd-watch --start recent | grep curl

# every touch of the cron files
sudo ausearch -k cron-watch

# a summary instead of raw lines
sudo aureport --exec

entries look like garbage until you spot the useful fields: exe= (which binary), auid= (the original user, even after they sudo'd), and key=. that auid trick is genuinely good — it ties actions back to the actual human, even through privilege changes

the moment it pays off

say something gets a shell on your box and wants to stay. the standard moves are: spawn a shell, add a cron entry for persistence, drop an ssh key into authorized_keys. all three now have receipts:

execve shows a bash process with a weird parent — like the web server user spawning shells — and it's recorded, not guessed
the cron rule catches the persistence entry being written, with a timestamp and the user who did it
authorized_keys gains a key and you see it happen, instead of finding out three weeks later

signatures catch none of this. egress filtering might not either, if the attacker came in through a legit channel. but the behavior leaves kernel-level logs, and that's the only layer they can't skip

keeping it sane

two honest warnings:

logging execve on everything is loud. on a busy server that's a lot of log. scope it if needed — auditctl supports conditions like -F auid>=1000 to skip system users
rules set with auditctl don't survive reboot. put them in /etc/audit/rules.d/ instead — that's the persistent way