Technology / Software /
11 Jul 2022
Setting up Logstash on Debian 'Bullseye'
ELK Stack Ports - 9600 TCP = Logstash API listening port - 10514 TCP = Logstack rsyslog input listening port
Installing Elasticsearch
Guide: https://www.elastic.co/guide/en/elasticsearch/reference/current/deb.html
- Seems straightforward… nope.
- Looks like we have a hard stop: there’s no i686 version of Elasticsearch available.
- To run on Linux, you must have a 64-bit kernel and 64b Java installed.
- Switching to a 64b kernel would probably require OS reinstallation on the home server.
- No thanks. That’ll have to go onto a different machine, or wait for an upgrade of the server to a 64b kernel.
But… let’s see about installing Logstash, just for the fun of it. That’s supposed to be the most complex part of the ELK stack.
Installing Logstash
Didn’t do anything interesting/exotic. Just added Elastic Co.’s APT repos and installed via sudo apt install logstash and it worked…
Configuring Logstash
This is where things got weird. For some reason the configuration on Debian is different in reality than in the documentation.
I got a lot of permissions errors when trying to run logstash on the command line to test the config file.
First I went through and made sure that everything in /etc/logstash/data was chown logstash:logstash, and there were a couple of files in there owned by root (for some reason) that were causing errors.
But then it started to complain about not being able to read the logstash.yml master config file…
Apparently on Debian, the easiest way to get all the permissions stuff right is to sudo as the logstash user.
This command seems to work to test the config:
sudo -Hu logstash /usr/share/logstash/bin/logstash --path.settings=/etc/logstash -t
That runs it as the logstash user (good) and explicitly specifies the config directory, which is (apparently) necessary.
This is an even more explicit version that provides the config file to test, also:
sudo -Hu logstash /usr/share/logstash/bin/logstash --path.settings=/etc/logstash -f /etc/logstash/conf.d/50-receive-rsyslog.conf -t
Next step is to run without test mode and see what actually happens, if we can get a port binding at least…
sudo -Hu logstash /usr/share/logstash/bin/logstash --path.settings=/etc/logstash -f /etc/logstash/conf.d/50-receive-rsyslog.conf
It is SUPER slow to start up (at least 1-2 minutes) each time logstash is run… not sure why as there isn’t much output during this time. It doesn’t seem to say directly what port it’s listening on for incoming logs… which is pretty unfortunate. But we can use netstat in another window and see that it’s listening on 10514, and in fact there’s a connection from the pihole that’s sending rsyslog output.
$ sudo netstat -tapen | grep java
tcp6 0 0 :::10514 :::* LISTEN 998 368382 16811/java
tcp6 0 0 127.0.0.1:9600 :::* LISTEN 998 368394 16811/java
tcp6 0 0 192.168.1.106:10514 192.168.1.10:54454 ESTABLISHED 998 374954 16811/java
And when run this way, we can clearly see log messages from the pihole (192.168.1.10) appear on the terminal, so we know that the two systems are now talking to each other.
Since we apparently can’t install Elasticsearch on this machine, I disabled the log transmission (with rsyslog) on the Pihole, and then also stopped and disabled the Logstash service on the server.
In theory, if I had a separate Elasticsearch instance to receive the data, I could just enable and start it up. But that’s a project for another day.
– 2022-07-13