Checkout SlayerLabs.com!
Networks Engineered to Exploit.
- Windows/UNIX - Domains/Subnets - Initial/Post/Lateral - Low Cost VPN Ranges -
Apache HTTPD Custom Logs
There are plenty of reasons you may want to customize your apache server logs. You may want to create a separate log with more lax permissions for a service account to read, or your servers might be behind a load balancer. Whatever is it, custom logs are fairly straight forward to setup.
Apache HTTPD has a custom log module called mod_log_config that should be enabled by default. The Apache HTTP Server project has excellent documentation. Checkout this modules man page here:
http://httpd.apache.org/docs/current/mod/mod_log_config.html#customlog
I’ll briefly be covering two directives provided by this module - LogFormat & CustomLog.
Formatting and Customizing
To get an idea on what needs to be done:
- Define a new custom log format, setting whatever format strings you’d like (referrer, IP, bytes, etc) using the LogFormat directive.
- Then use the CustomLog directive along with a built-in Linux binary to rotate the log on a predefined basis.
Understand you logging goals - do you want them to be more complex than the default format? Do you want them simple and clean, or just in a different order for your service account to ingest? The man page(previous link) goes through plenty of different variables/format strings you can test with.
Fixing the Load Balancer Blues
If your servers are behind a load balancer, you may notice all requests are coming from the load balancers IP. This is a big issue if you’re monitoring traffic for security issues, block ip’s from certain geographical locations, etc.
If you have access to the load balacner, you can configure logs and original IP’s here. Although if you don’t have access to the load balancer luckily there’s a directive for this - %{X-Forwarded-For}
Putting it All Together
Once you have an idea on your formatting needs, you’ll need to define this in your HTTPD config file. It’s best to leave the original httpd.conf alone, and create a custom config file - in our case /etc/httpd/conf.d/prod.conf
Note - Anything in the /conf.d directory ending in .conf will be loaded into Apache upon startup/reload.
Now, define the LogFormat inside OR outside of you VirtualHost block(s). If inside you’ll need to define this LogFormat again if you use within another VirtualHost block. After the Directives are in place, add a custom name to this format. In my example it’s shell.
LogFormat " %{X-Forwarded-For}i %h %l %u %t \"%r\" %>s \"Referer}i\" \"%{User-Agent}i\"" shell
Now that we’ve defined our format, we’ll need to put it to use. With the CustomLog directive we can simply define the new log file to create. Although this will get very large without any log rotation.
As mentioned there are a few Linux binary’s that will rotate the logs for us. In this case we’ll use rotatelogs - which is located in /sbin on most Linux distros.
CustomLog allows for a pipe to be used, time in seconds to rotate, along with a few other options. We’ll set CustomLog to pipe the Linux binary rotatelogs, then define our new log location and name, set how many seconds it should execute rotatelogs, then set our custom LogFormat (shell) - which looks something like this:
CustomLog "|/sbin/rotatelogs /var/log/prod.%Y%m%d 86400" shell
Also if you’d like to make a custom error log you can use the same basic format as above. Altering the original Apache error log is NOT recommended. Simply create a custom error log to avoid any headaches. Here’s my example:
ErrorLog "|/sbin/rotatelogs /var/log/prod_error.%Y%m%d shell
Once saved, Apache is reloaded/restarted the a new prod access and error log will be created in /var/log every day or 86400 seconds.
Log Security
Locking down your logs is very important. Not just because of the sensitive information in the logs, but the possibility of File Inclusion. I’ve create a few File Inclusion posts if you’d like to learn more about this vulnerability.
LFI and RFI Basics: https://liberty-shell.com/sec/2018/04/04/lfi/
Example if LFI to FRI via Log Poisoning: https://liberty-shell.com/sec/2018/05/19/poisoning/
Automated LFI tool - FI Cyberspace Scan https://liberty-shell.com/sec/2018/09/02/cyberscan/
Besides gaining useful information from the logs (users, hashed passwords, login routines, modules, etc) an attacker can inject code into the logs then have the server process their malicious code. This can be very bad news.
Also be sure to keep the custom logs in a safe area such as /var/log. In most cases SELinux won’t write to any logs outside of /var/log, unless otherwise defined. Be sure to keep these factors in mind when creating your custom logs.