ELK stack is also known as the Elastic stack, consists of Elasticsearch, Logstash, and Kibana. It helps you to have all of your logs stored in one place and analyze the issues by correlating the events at a particular time.
This guide helps you to install ELK stack on CentOS 7 / RHEL 7.
Components
Logstash – It does the processing (Collect, enrich and send it to Elasticsearch) of incoming logs sent by beats (forwarder).
Elasticsearch – It stores incoming logs from Logstash and provides an ability to search the logs/data in a real-time
Kibana – Provides visualization of logs.
Sentinl – Sentinl extends Siren Investigate and Kibana with Alerting and Reporting functionality to monitor, notify and report on data series changes using standard queries, programmable validators and a variety of configurable actions – Think of it as a free an independent “Watcher” which also has scheduled “Reporting” capabilities (PNG/PDFs snapshots).
SENTINL is also designed to simplify the process of creating and managing alerts and reports in Siren Investigate/Kibana 6.xvia its native App Interface, or by using native watcher tools in Kibana 6.x+.
Beats – Installed on client machines, send logs to Logstash through beats protocol.
Environment
To have a full-featured ELK stack, we would need two machines to test the collection of logs.
ELK Stack
|
1 2 3 |
Operating system : CentOS 7 Minimal IP Address : 192.168.1.10 HostName : server.lintel.local |
Filebeat
|
1 2 3 |
Operating System : CentOS 7 Minimal IP Address : 192.168.1.20 HostName : client.lintel.local |
Prerequisites
Install Java
Since Elasticsearch is based on Java, make sure you have either OpenJDK or Oracle JDK is installed on your machine.
Here, I am using OpenJDK 1.8.
|
1 |
yum -y install java-1.8.0 wget |
Verify the Java version.
|
1 |
java -version |
Output:
|
1 2 3 |
java version "1.8.0_181" Java(TM) SE Runtime Environment (build 1.8.0_181-b13) Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode) |
Configure ELK repository
Import the Elastic signing key.
|
1 |
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch |
Setup the Elasticsearch repository and install it.
|
1 |
vi /etc/yum.repos.d/elk.repo |
Add the below content to the elk.repo file.
|
1 2 3 4 5 6 7 8 |
[elasticsearch-6.x] name=Elasticsearch repository for 6.x packages baseurl=https://artifacts.elastic.co/packages/6.x/yum gpgcheck=1 gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch enabled=1 autorefresh=1 type=rpm-md |
Install Elasticsearch
Elasticsearch is an open source search engine, offers a real-time distributed search and analytics with the RESTful web interface. Elasticsearch stores all the data are sent by the Logstash and displays through the web interface (Kibana) on users request.
Install Elasticsearch.
|
1 |
yum install -y elasticsearch |
Configure Elasticsearch to start during system startup.
|
1 2 3 |
systemctl daemon-reload systemctl enable elasticsearch systemctl start elasticsearch |
Use CURL to check whether the Elasticsearch is responding to the queries or not.
|
1 |
curl -X GET http://localhost:9200 |
Output:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
{ "name" : "1DwGO86", "cluster_name" : "elasticsearch", "cluster_uuid" : "xboS_6K5Q2OO1XA-QJ9GIQ", "version" : { "number" : "6.4.0", "build_flavor" : "default", "build_type" : "rpm", "build_hash" : "595516e", "build_date" : "2018-08-17T23:18:47.308994Z", "build_snapshot" : false, "lucene_version" : "7.4.0", "minimum_wire_compatibility_version" : "5.6.0", "minimum_index_compatibility_version" : "5.0.0" }, "tagline" : "You Know, for Search" } |
Install Logstash
Logstash is an open source tool for managing events and logs, it collects the logs, parse them and store them on Elasticsearch for searching. Over 160+ plugins are available for Logstash which provides the capability of processing the different type of events with no extra work.
Install the Logstash package.
|
1 |
yum -y install logstash |
Create SSL certificate (Optional)
Filebeat (Logstash Forwarder) are normally installed on client servers, and they use SSL certificate to validate the identity of Logstash server for secure communication.
Create SSL certificate either with the hostname or IP SAN.
(Hostname FQDN)
If you use the Logstash server hostname in the beats (forwarder) configuration, make sure you have A record for Logstash server and also ensure that client machine can resolve the hostname of the Logstash server.
Go to the OpenSSL directory.
|
1 |
cd /etc/pki/tls/ |
Now, create the SSL certificate. Replace green one with the hostname of your real Logstash server.
|
1 2 |
openssl req -x509 -nodes -newkey rsa:2048 -days 365 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt -subj /CN=<strong>server.lintel.local </strong> |
Configure Logstash
Logstash configuration can be found in /etc/logstash/conf.d/. Logstash configuration file consists of three sections input, filter, and the output. All three sections can be found either in a single file or separate files end with .conf.
I recommend you to use a single file for placing input, filter and output sections.
|
1 |
vi /etc/logstash/conf.d/logstash.conf |
In the first section, we will put an entry for input configuration. The following configuration sets Logstash to listen on port 5044 for incoming logs from the beats (forwarder) that sit on client machines.
Also, add the SSL certificate details in the input section for secure communication – Optional.
|
1 2 3 4 5 6 7 8 9 10 11 12 |
input { beats { port => 5044 # Set to False if you do not use SSL ssl => true <strong># Delete below linesif you do not use SSL</strong> ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt" ssl_key => "/etc/pki/tls/private/logstash-forwarder.key" } } |
In the filter section. We will use Grok to parse the logs ahead of sending it to Elasticsearch. The following grok filter will look for the syslog labeled logs and tries to parse them to make a structured index.
|
1 2 3 4 5 6 7 8 9 10 11 12 |
filter { if [type] == "syslog" { grok { match => { "message" => "%{SYSLOGLINE}" } } date { match => [ "timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ] } } } |
For more filter patterns, take a look at grokdebugger page.
In the output section, we will define the location where the logs to get stored; obviously, it should be Elasticsearch.
|
1 2 3 4 5 6 7 8 9 |
output { elasticsearch { hosts => localhost index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}" } stdout { codec => rubydebug } } |
Now start and enable the Logstash service.
|
1 2 |
systemctl start logstash systemctl enable logstash |
You can troubleshoot any issues by looking at Logstash logs.
|
1 |
cat /var/log/logstash/logstash-plain.log |
Install & Configure Kibana
Kibana provides visualization of logs stored on the Elasticsearch. Install the Kibana using the following command.
|
1 2 3 |
cd /opt wget https://artifacts.elastic.co/downloads/kibana/kibana-6.2.4-x86_64.rpm rpm -i kibana-6.2.4-x86_64.rpm |
Edit the kibana.yml file.
|
1 |
vi /etc/kibana/kibana.yml |
By default, Kibana listens on localhost which means you can not access Kibana interface from external machines. To allow it, edit the below line with your machine IP.
|
1 |
server.host: "<strong>192.168.1.10</strong>" |
Uncomment the following line and update it with the Elasticsearch instance URL. In my case, it is localhost.
|
1 |
elasticsearch.url: "<strong>http://localhost:9200</strong>" |
Start and enable kibana on system startup.
|
1 2 |
systemctl start kibana systemctl enable kibana |
Install Sentinl plugin:
|
1 2 |
/usr/share/kibana/bin/kibana-plugin install https://github.com/sirensolutions/sentinl/releases/download/tag-6.2.4/sentinl-v6.2.4.zip systemctl restart kibana |
Install and Configure Filebeat
There are four beats clients available
- Packetbeat – Analyze network packet data.
- Filebeat – Real-time insight into log data.
- Topbeat – Get insights from infrastructure data.
- Metricbeat – Ship metrics to Elasticsearch.
To analyze the system logs of the client machine (Ex. client.lintel.local), we need to install filebeat. Create beats.repo file.
|
1 |
vi /etc/yum.repos.d/beats.repo |
Add the below content to the above repo file.
|
1 2 3 4 5 6 7 8 |
[elasticsearch-6.x] name=Elasticsearch repository for 6.x packages baseurl=https://artifacts.elastic.co/packages/6.x/yum gpgcheck=1 gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch enabled=1 autorefresh=1 type=rpm-md |
Now, install Filebeat using the following command.
|
1 |
yum -y install filebeat |
Set up a host entry on the client machine in case your environment does not have DNS server.
|
1 |
vi /etc/hosts |
Make an host entry like below on the client machine.
|
1 |
192.168.1.10 server.lintel.local server |
Filebeat (beats) uses SSL certificate for validating Logstash server identity, so copy the logstash-forwarder.crt from the Logstash server to the client.
Skip this step, in case you are not using SSL in Logstash.
|
1 |
scp -pr <strong>root@server.lintel.local</strong>:/etc/pki/tls/certs/logstash-forwarder.crt /etc/ssl/certs/ |
Filebeat configuration file is in YAML format, which means indentation is very important. Make sure you use the same number of spaces used in the guide.
Open up the filebeat configuration file.
|
1 |
vi /etc/filebeat/filebeat.yml |
On top, you would see the prospectors section. Here, you need to specify which logs should be sent to Logstash and how they should be handled. Each prospector starts with – character.
For testing purpose, we will configure filebeat to send /var/log/messages to Logstash server. To do that, modify the existing prospector under paths section.
Comment out the – /var/log/*.log to avoid sending all .log files present in that directory to Logstash.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
filebeat.inputs: - type: <strong>log</strong> <strong># Change to true to enable this input configuration.</strong> enabled: <strong>true</strong> <strong># Paths that should be crawled and fetched.</strong> paths: <strong> - /var/log/messages </strong> # - /var/log/*.log . . . |
Comment out the section output.elasticsearch: as we are not going to store logs directly to Elasticsearch.
Now, find the line output.logstash and modify the entries like below. This section defines filebeat to send logs to Logstash server server.lintel.local on port 5044 and mention the path where the copied SSL certificate is placed
Replace server.lintel.local with IP address in case if you are using IP SAN.
|
1 2 3 4 5 6 7 8 9 10 11 |
. . . output.logstash: <strong># The Logstash Server</strong> hosts: ["<strong>server.lintel.local</strong>:5044"] <strong># Comment out this line if Logstash is not using SSL</strong> ssl.certificate_authorities: ["<strong>/etc/ssl/certs/logstash-forwarder.crt</strong>"] . . . |
Restart the service.
|
1 |
systemctl restart filebeat |
Beats logs are typically found syslog file.
|
1 |
cat /var/log/filebeat/filebeat |
Access Kibana
Access the Kibana using the following URL.
You would get the Kibana’s home page.
On your first login, you have to map the filebeat index. Go to Management >> Index Patterns.
Type the following in the Index pattern box.
|
1 |
filebeat-* |
You should see at least one filebeat index something like above. Click Next step.
Select @timestamp and then click on Create.
|
1 |
@timestamp |
Verify your index patterns and its mappings.
Now, click Discover to view the incoming logs and perform search queries.
You can see sentinl plugin here
That’s All.
Reference list:
https://github.com/sirensolutions/sentinl
https://www.itzgeek.com
