Well, who knew Cloudwatch would be so much fun to tinker with?! Not me!
I have been slowly refining my Cloudwatch dashboard: adding new alarms, expanding the scope of the log watch, all that good stuff. It is very satisfying. Over the last week or so I have also set-up fail2ban because (according to my audit log via Cloudwatch ?) sshd was getting hammered. As previously mentioned, this box is not well resourced, so I wanted to nip that in the bud. But does the cost of running fail2ban outweigh the benefits? Hard to say!
Anyway, I am getting quite a lot of email from fail2ban. This is good because I know it is working but I’d rather not have the email and still be able to easily check it was working… so Cloudwatch!
I added the fail2ban log to the config and used the Logs Insights tool to explore. This is typical line:
2021-04-30 17:58:06.631, "2021-04-30 18:58:06,208 fail2ban.filter : INFO [sshd] Found 18.104.22.168 - 2021-04-30 18:58:05"
We could use the date/time a few more times, right? I decided this was the time to jump into the parse command in the CloudWatch Logs Insights query language (rolls off the tongue that). I knew I was going to need another regex within about 10 seconds. But, damn, if the examples aren’t thin on the ground. I googled and found virtually nothing although this post did help a bit.
So, to regex101.com I went. I exported a few lines from the log to test and I must be getting quite a lot better because I got the basics working pretty quickly:
\[sshd\]\ (?<action>[a-zA-z]*)\ (?<ip_address>[\d\.]*)
Then this query in Cloudwatch Logs Insights did the job:
parse @message /\[sshd\]\ (?<action>[a-zA-z]*)\ (?<ip_address>[\d\.]*)/ | display @timestamp, action, ip_address | limit 200
Unfortunately, I find reading timestamp pretty hard so a bit more tinkering:
parse @message /(?<date>\d\d\d\d-\d\d-\d\d)\ (?<time>\d\d:\d\d:\d\d).*\[sshd\]\ (?<action>[a-zA-z]*)\ (?<ip_address>[\d\.]*)/ | display date, time, action, ip_address | limit 200
Excellent! It was running for about 5 minutes and it suddenly produced a blank line. Of course, [sshd] in the log refers to the jail. I have several set up so…
parse @message /(?<date>\d\d\d\d-\d\d-\d\d)\ (?<time>\d\d:\d\d:\d\d).*\[(?<jail>sshd|recidive|mysqld-auth)\]\ (?<action>[a-zA-z]*)\ (?<ip_address>[\d\.]*)/ | display date, time, jail, action, ip_address | limit 200
And that does the job nicely at the moment. You can find an explanation of the regex on regex101.
Once I am a bit more confident, I’ll start filtering on the action, so I can just see bans and unbans:
| filter action = "Ban" or action ="Unban"
Thanks for this post. Helped me get my fail2ban/cloudwatch logging working and able to add to a dashboard so my boss could see the top attempts