Boot Up the Behind

Another important component to think carefully about is what happens when the newer init system, systemd, starts up a modern day Linux after a reboot. The command

$ systemd-analyze
Startup finished in 1.026s (kernel) + 5.925s (userspace) = 6.952s

shows you the slowest services to load when you boot up your system, so you can name and shame those services (and then fine tune them afterward). Just under six seconds of userland applications are slowing down the system. The next command shows the top five worst offenders:

$ systemd-analyze blame | head -5
          2.084s fail2ban.service
          1.958s cloud-init.service
          1.851s cloud-init-local.service
          1.597s apache2.service
          1.134s postfix.service

As you can see, during the last reboot, the fail2ban rules took a whopping two seconds. Be aware these values change with each boot depending on a number of events, so don’t focus entirely on these metrics, just the slower packages. Remember that systemd also can load its services using parallelism.

What’s this got to do with auditing, you might ask? Quite a lot, actually.

It’s very important to log services during the boot process with auditd and not just let those services do whatever they like (e.g., think about trojans, keyloggers, and boot-level malware) without generating any logging that can then be referenced later.

Out of the box, auditd needs a little help. I suspect this might not be the default (on Red Hat machines, anyway) because of performance-related concerns. If you have a multitude of badly organized rules, you can incur a slight kernel performance hit as each bit of activity is checked then logged before being executed. You should do more research if you’re using a shipload of rules in a production environment.

Assuming you’re using the GRUB bootloader, you can append a simple audit=1 to the kernel or linux line within your GRUB config, depending on your Linux flavor.

In Debian, I add the line

GRUB_CMDLINE_LINUX="audit=1"

to the /etc/default/grub file, which then is added to /boot/grub/grub.cfg after the update command:

$ grub-update

This is slightly different on Red Hat derivatives (which, strictly speaking, create a new config file; potentially, there might also be an update equivalent that you can use instead).

$ grub2-mkconfig > /boot/grub2/grub.cfg

As you can probably guess, it’s now just a case of rebooting the system and looking at the start of the logfile to see which services are now also to be logged to the auditd logfile /var/log/audit/audit.log , as well.

Talkin’ Loud And Sayin’ Nothing

All this log output isn’t very useful if you’ve no idea what it means. Interpreting the audit.log file is clearly very important, and I won’t claim it’s always easy, but with some exposure you’ll get the hang of it eventually.

Bear in mind that many of the logs are event based and the timestamps, which are written something like

msg=audit(1513507481.075:145)

are how the system clock records each event: a timestamp (the longer number) followed by a colon and then the ID applied to it (the shorter, trailing number). If a number of events happen at once (which you’d expect on modern systems with high-power threaded CPUs), many entries could have the same time and ID. Using this knowledge, you can check when and what happened in what order.

If you’ve used Linux for a while, then fields such as pid (process ID) and ppid (parent process ID) should make some sense. A handy entry to look out for is auid , which is the “actual user ID” of the user that generated this activity (even if they became root as UID 0, the original user is logged).

With time, this becomes easier to fathom, so don’t despair if it initially induces eye strain; just refine searches with queries per event across short time periods.

Immutability

One important thing to note is the “immutability” feature in auditd . As mentioned, auditd makes its configuration and logfiles accessible only to the root user for very good reasons. Imagine if an attacker got into a system and wanted to affect an application (e.g., sshd ) to hide a second SSH port for backdoor logins. If they knew auditd was dutifully logging all their actions, they would clearly want to disable some, if not all, of the SSH ruleset to hide their actions.

Fear not. Simply append this tiny additional config parameter to the very end of your rules.d/audit.rules file:

-e 2

The result of adding this line to your config is that the only way to alter the auditd rules on a running system after it has been booted up with that setting in place is to rewrite the rules in the config file (i.e., edit, then rerun augenrules , and perform another reboot). As a result, of course, you need to pay much more attention to rebooting systems and treat random reboots as suspicious activity, which you should be doing, anyway.

One gotcha is being mindful of the fact that the system is using the immutability flag in its configuration.

You can tell if immutability is enabled by running the --load option with augenrules :

$ augenrules --load
/sbin/augenrules: No change

In this case, the audit system is in immutable mode, so no rule changes are allowed.