Configuring an endlessh honeypot with rsyslog email notifications
Security is all about defence in depth, so I'm always looking for ways to better secure my home network. For example, I have cluster management traffic running over a Wireguard mesh VPN. Now, I'm turning my attention to the rest of my network.
To this end, while I have a guest network with wireless isolation enabled, I do not currently have a way to detect unauthorised devices connecting to my home WiFi network, or fake WiFi networks with the same name, etc. Detecting this is my next focus. While I've seen nzyme recently and it looks fantastic, it also looks more complicated to setup.
While I look into the documentation for nzyme, inspired by this reddit post I decided to setup a honeypot on my home network.
The goal of a honeypot is to detect threats moving around in a network. In my case, I want to detect if someone has connected to my network who shouldn't have done. Honeypots achieve this by pretending to be a popular service, but in reality they are there to collect information about potential threats.
To set one up, I found endlessh, which pretends to be an SSH server - but instead slowly sends an endless banner to the client, keeping the connection open as long as possible. It can also connection attempts to syslog, which allows us to detect connections and send an alert.
Implementing this comes in 2 steps. First, we setup endlessh and configure it to log connection attempts. Then, we reconfigure rsyslog to send email alerts.
Setting up endlessh
I'm working on one of the Raspberry Pis running Raspberry Pi OS in my network, but this should with with other machines too.
If you're following along to implement this yourself, make sure you've moved SSH to another port number before you continue, as we'll be configuring endlessh to listen on port 22 - the default port for ssh, as this is the port I imagine that an automated network scanner might attempt to connect to by default if it were looking for ssh servers to attempt to crack.
Conveniently, endlessh has a package in the default Debian repositories:
sudo apt install endlessh...adjust this for your own package manager if you aren't on an apt-based system.
endlessh has a configuration file at /etc/endlessh/config by default. Open it up for editing, and make it look something like this:
# The port on which to listen for new SSH connections.
Port 22
# Set the detail level for the log.
# 0 = Quiet
# 1 = Standard, useful log messages
# 2 = Very noisy debugging information
LogLevel 1Beforee we can start the endlessh service, we need to reconfigure it to allow it to listen on port 22, as this is a privileged port number. Doing this requires 2 steps. First, allow the binary to listen on privileged ports:
sudo setcap CAP_NET_BIND_SERVICE=+eip "$(which "endlessh")";Then, if you are running systemd (most distributions do by default), execute the following command:
sudo systemctl edit endlessh.serviceThis will allow you to append some additional directives to the service definition for endlessh, without editing the original apt-managed systemd service file. Add the following, and then save and quit:
[Service]
AmbientCapabilities=CAP_NET_BIND_SERVICE
PrivateUsers=falseFinally, we can restart the endlessh service:
sudo systemctl restart endlessh
sudo systemctl enable --now endlesshThat completes the setup of endlessh!
Configuring rsyslog to send email alerts
The second part of this process is to send automatic alerts whenever anyone connects to our endlessh service. Since endlessh forwards logs to syslog by default, reconfiguring rsyslog to send the alerts seems like the logical choice. In my case, I'm going to send email alerts - but other ways of sending alerts do exist - I just haven't looked into them yet.
To do this requires that you have either a working email server (I followed the Ars Technica taking email back series, but whatever you do it's not for the faint for heart! Command line experience is definitely required - if you're looking for a nice first project to try, a web server instead), or an email account you can use. Note that I do not recommend using your own personal email account, as you'll have to store the password in plain text!
In my case, I have my own email server, and I have forwarded port 25 down an SSH tunnel so that I can use it to send emails (in the future I want to configure a proper smart host that listen on port 25 and forwards emails by authenticating against my server properly, but that's for another time as I have yet to find a relay-only MTA that also listens on port 25).
In a previous post, implemented centralised logging - so I'm going to be reconfiguring my main centralised rsyslog instance.
To do this, open up /etc/rsyslog.d/10-endlessh.conf for editing, and paste in something like this:
template (name="mailSubjectEndlessh" type="string" string="[HONEYPOT] endlessh connection on %hostname%")
if ( ($programname == 'endlessh') and (($msg contains "ACCEPT") or ($msg contains "CLOSE")) ) then {
action(type="ommail" server="localhost" port="20205"
mailfrom="sender@example.com"
mailto=["bill@billsboosters.net"]
subject.template="mailSubjectEndlessh"
action.execonlyonceeveryinterval="3600"
)
}...where:
[HONEYPOT] endlessh connection on %hostname%is the subject name, and%hostname%is substituted for the actual hostname the honeypot is running onsender@example.comis the address that you want to send the alert FROMbill@billsboosters.netis the address that you want to send the alert TO3600is the minimum interval between emails, in seconds. Log lines are not collected up - only 1 log line is sent at a time, and others logged in-between are ignored and handled as if the above email directive doesn't exist until the given number of seconds expires - at which point it will then email for the next log line that comes through, and the cycle then repeats. If anyone knows how to change that, please leave a command below.
Note that the template line is outside the if statement. This is important - I got a syntax error if I put it inside the if statement.
The if statement specifically looks for log messages with a tag of endlessh that contain either the substring ACCEPT or CLOSE. Only if those conditions are true will it send an email.
I have yet to learn how to configure rsyslog to authenticate while sending emails. I would suspect though that the easiest way of achieving this is to setup a local SMTP relay-only MTA (Mail Transfer Agent) that rsyslog can connect to and send emails, and then the relay will authenticate against the real server and send the email on rsyslog's behalf. I have yet to find such an MTA however other than Postfix - which, while great, can be hugely complicated to setup. Other alternatives I've tried include:
- nullmailer
- msmtp
- dma - The Dragonfly Mail Agent
....but they all implement sendmail and while that's useful they do not listen on port 25 (or any other port for that matter) as far as I can tell.
Anyway, the other file you need to edit is /etc/rsyslog.conf. Open it up for editing, and put this near the top:
module(load="ommail")...this loads the mail output plugin that sends the emails.
Now that we've reconfigured rsyslog, we need to restart it:
sudo systemctl restart rsyslogrsyslog is picky about it's config file syntax, so make sure to check it's status for error messages:
sudo systemctl status rsyslogYou can also use lnav analyse your logs and find any error messages there too.
Conclusion
We've setup endlessh as a honeypot, and then reconfigured rsyslog to send email alerts. Test the system like so on your local machine:
ssh -vvv -p 22 someuser@yourserver...and watch your inbox for the email alert that will follow shortly!
While this system isn't particularly useful on it's own, it's a small part of a larger strategy for securing my network. It's also been a testing ground for me to configure rsyslog to send email alerts - something I may want to configure my centralised rsyslog logging system to do for other things in the future.
If you've found this post useful or you have some suggestions, please leave a comment below!