Installing ELK Stack(Elasticsearch,Logstash,Kibana) on CentOS with Sentinl plugin

ELK stack is also known as the Elastic stack, consists of Elasticsearch, Logstash, and Kibana. It helps you to have all of your logs stored in one place and analyze the issues by correlating the events at a particular time.

This guide helps you to install ELK stack on CentOS 7 / RHEL 7.

Components

Logstash – It does the processing (Collect, enrich and send it to Elasticsearch) of incoming logs sent by beats (forwarder).

Elasticsearch – It stores incoming logs from Logstash and provides an ability to search the logs/data in a real-time

Kibana – Provides visualization of logs.

Sentinl –  Sentinl extends Siren Investigate and Kibana with Alerting and Reporting functionality to monitor, notify and report on data series changes using standard queries, programmable validators and a variety of configurable actions – Think of it as a free an independent “Watcher” which also has scheduled “Reporting” capabilities (PNG/PDFs snapshots).

SENTINL is also designed to simplify the process of creating and managing alerts and reports in Siren Investigate/Kibana 6.xvia its native App Interface, or by using native watcher tools in Kibana 6.x+.

 

Beats – Installed on client machines, send logs to Logstash through beats protocol.

Environment

To have a full-featured ELK stack, we would need two machines to test the collection of logs.

ELK Stack

Filebeat

Prerequisites

Install Java

Since Elasticsearch is based on Java, make sure you have either OpenJDK or Oracle JDK is installed on your machine.

Here, I am using OpenJDK 1.8.

Verify the Java version.

Output:

Configure ELK repository

Import the Elastic signing key.

Setup the Elasticsearch repository and install it.

Add the below content to the elk.repo file.

Install Elasticsearch

Elasticsearch is an open source search engine, offers a real-time distributed search and analytics with the RESTful web interface. Elasticsearch stores all the data are sent by the Logstash and displays through the web interface (Kibana) on users request.

Install Elasticsearch.

Configure Elasticsearch to start during system startup.

Use CURL to check whether the Elasticsearch is responding to the queries or not.

Output:

Install Logstash

Logstash is an open source tool for managing events and logs, it collects the logs, parse them and store them on Elasticsearch for searching. Over 160+ plugins are available for Logstash which provides the capability of processing the different type of events with no extra work.

Install the Logstash package.

Create SSL certificate (Optional)

Filebeat (Logstash Forwarder) are normally installed on client servers, and they use SSL certificate to validate the identity of Logstash server for secure communication.

Create SSL certificate either with the hostname or IP SAN.

(Hostname FQDN)

If you use the Logstash server hostname in the beats (forwarder) configuration, make sure you have A record for Logstash server and also ensure that client machine can resolve the hostname of the Logstash server.

Go to the OpenSSL directory.

Now, create the SSL certificate. Replace green one with the hostname of your real Logstash server.

Configure Logstash

Logstash configuration can be found in /etc/logstash/conf.d/. Logstash configuration file consists of three sections input, filter, and the output. All three sections can be found either in a single file or separate files end with .conf.

I recommend you to use a single file for placing input, filter and output sections.

In the first section, we will put an entry for input configuration. The following configuration sets Logstash to listen on port 5044 for incoming logs from the beats (forwarder) that sit on client machines.

Also, add the SSL certificate details in the input section for secure communication – Optional.

In the filter section. We will use Grok to parse the logs ahead of sending it to Elasticsearch. The following grok filter will look for the syslog labeled logs and tries to parse them to make a structured index.

For more filter patterns, take a look at grokdebugger page.

In the output section, we will define the location where the logs to get stored; obviously, it should be Elasticsearch.

Now start and enable the Logstash service.

You can troubleshoot any issues by looking at Logstash logs.

Install & Configure Kibana

Kibana provides visualization of logs stored on the Elasticsearch. Install the Kibana using the following command.

Edit the kibana.yml file.

By default, Kibana listens on localhost which means you can not access Kibana interface from external machines. To allow it, edit the below line with your machine IP.

Uncomment the following line and update it with the Elasticsearch instance URL. In my case, it is localhost.

Start and enable kibana on system startup.

Install Sentinl plugin:

Install and Configure Filebeat

There are four beats clients available

  1. Packetbeat – Analyze network packet data.
  2. Filebeat – Real-time insight into log data.
  3. Topbeat – Get insights from infrastructure data.
  4. Metricbeat – Ship metrics to Elasticsearch.

To analyze the system logs of the client machine (Ex. client.lintel.local), we need to install filebeat. Create beats.repo file.

Add the below content to the above repo file.

Now, install Filebeat using the following command.

Set up a host entry on the client machine in case your environment does not have DNS server.

Make an host entry like below on the client machine.

Filebeat (beats) uses SSL certificate for validating Logstash server identity, so copy the logstash-forwarder.crt from the Logstash server to the client.

Skip this step, in case you are not using SSL in Logstash.

Filebeat configuration file is in YAML format, which means indentation is very important. Make sure you use the same number of spaces used in the guide.

Open up the filebeat configuration file.

On top, you would see the prospectors section. Here, you need to specify which logs should be sent to Logstash and how they should be handled. Each prospector starts with – character.

For testing purpose, we will configure filebeat to send /var/log/messages to Logstash server. To do that, modify the existing prospector under paths section.

Comment out the – /var/log/*.log to avoid sending all .log files present in that directory to Logstash.

Comment out the section output.elasticsearch: as we are not going to store logs directly to Elasticsearch.

Now, find the line output.logstash and modify the entries like below. This section defines filebeat to send logs to Logstash server server.lintel.local on port 5044 and mention the path where the copied SSL certificate is placed

Replace server.lintel.local with IP address in case if you are using IP SAN.

Restart the service.

Beats logs are typically found syslog file.

Access Kibana

Access the Kibana using the following URL.

http://your-ip-address:5601/

You would get the Kibana’s home page.

Install Elasticsearch, Logstash, and Kibana (ELK Stack) on CentOS 7 - Kibana Starting Page
Install Elasticsearch, Logstash, and Kibana (ELK Stack) on CentOS 7 – Kibana Starting Page

On your first login, you have to map the filebeat index. Go to Management >> Index Patterns.

Install Elasticsearch, Logstash, and Kibana (ELK Stack) on CentOS 7 - Management
Install Elasticsearch, Logstash, and Kibana (ELK Stack) on CentOS 7 – Management

Type the following in the Index pattern box.

Install Elasticsearch, Logstash, and Kibana (ELK Stack) on CentOS 7 - Create Index Pattren
Install Elasticsearch, Logstash, and Kibana (ELK Stack) on CentOS 7 – Create Index Pattern

You should see at least one filebeat index something like above. Click Next step.

Select @timestamp and then click on Create.

Install Elasticsearch, Logstash, and Kibana (ELK Stack) on CentOS 7 - Configure Timestamp
Install Elasticsearch, Logstash, and Kibana (ELK Stack) on CentOS 7 – Configure Timestamp

Verify your index patterns and its mappings.

Install Elasticsearch, Logstash, and Kibana (ELK Stack) on CentOS 7 - Index Mappings
Install Elasticsearch, Logstash, and Kibana (ELK Stack) on CentOS 7 – Index Mappings

Now, click Discover to view the incoming logs and perform search queries.

Install Elasticsearch, Logstash, and Kibana (ELK Stack) on CentOS 7 - Discover Logs
Install Elasticsearch, Logstash, and Kibana (ELK Stack) on CentOS 7 – Discover Logs

You can see sentinl plugin here

sentinl_annotation

That’s All.

 

Reference list:

https://github.com/sirensolutions/sentinl

https://www.itzgeek.com

How to Setup Redis Cluster from Source

What is redis

Redis is an open source in-memory database. It stores data in key-value format. Because of residing in memory, redis is an excellent tool for caching. Redis provides a rich set of data types. This gives redis upper hand over Memcached. Apart from caching, redis can be used as distributed message broker.

Redis Cluster and Sentinels

To achieve high availability, redis can be deployed in cluster along with Sentinels. Sentinel is a feature of redis. Multiple sentinels are deployed across redis clusters for monitoring purpose. When redis master goes down, sentinels elect a new master from slaves. When old master comes up again, it is added as slave.

Another use case of clustering is a distribution of load. In high load environment, we can send write requests to master and read request to slaves.

This tutorial is specifically focused on Redis Cluster Master Slave model. We will not cover data sharding across cluster here. In data sharding, keys are distributed across multiple redis nodes.

Setup for tutorial

For this tutorial, we will use 3 (virtual) servers. On one server Redis master will reside while other two servers will be used for slaves. Standard redis port is 6379. To differentiate easily, we will run master on 6379 port and slaves on
6380 and 6381 ports. Same will be applied for sentinel services. Master sentinel will listen on 16379 port while slave sentinels will be on 16380 and 16381.

Lets put this easy way.

This tutorial is tested on CentOS 6.9. For CentOS 7.X, check below Notes Section.

Installation

We will follow same installation steps for setting up of all servers. Only difference will be in configurations.

  • Step 1: Grab redis source, make and install
  • Step 2: Setup required directories
  • Step 3: Configure redis master
  • Step 4: Configure redis master sentinel
  • Step 5: Add low privileged user to run redis
  • Step 6: Setup init scripts
  • Step 7: Start service

Server 1 (Redis Master)


Install Redis

Setup required directories

Configure redis master

Edit config file /etc/redis/6379.conf in your favorite editor and change below options.

Configure redis master sentinel

Add config file for sentinel at /etc/redis/sentinel_6379.conf. Open a file and add below content

Add non-privileged user

Setup init scripts

You can find sample init scripts in Notes section below.

Start service

Server 2 (Redis Slave 1)


Install Redis

Setup required directories

Configure redis slave 1

Edit config file /etc/redis/6380.conf in your favorite editor and change below options.

Configure redis slave 1 sentinel

Add config file for sentinel at /etc/redis/sentinel_6380.conf. Open a file and add below content

Add non-privileged user

Setup init scripts

You can find sample init scripts in Notes section below. Change $HOST and $PORT values accordingly

Start service

Server 3 (Redis Slave 2)


Install Redis

Setup required directories

Configure redis slave 2

Edit config file /etc/redis/6381.conf in your favorite editor and change below options.

Configure redis slave 2 sentinel

Add config file for sentinel at /etc/redis/sentinel_6381.conf. Open a file and add below content

Add non-privileged user

Setup init scripts

You can find sample init scripts in Notes section below. Change $HOST and $PORT values accordingly

Start service

Sentinel Testing

Redis Fail-over Testing

For fail-over testing, we can take down redis-master either using init script or below command.

Also we can force sentinel to run fail over using below command

Sample init scripts

Redis Init Script

Sentinel Init Script

Notes

Security

  • NEVER EVER run redis on public interface
  • If redis is deployed in cloud environment like AWS, set up security groups/firewalls carefully. Most of times, cloud providers use ephemeral ips. Because of ephermal ips, even redis is bound to private ip, it can be accessed over public interface.
  • For more security, dangerous commands can be disabled(renamed). But be careful with them in cluster environment.
  • Redis also provides simple authentication mechanism. It is not covered here because of scope.

Sentinel management

  • During redis fail-over, config files are rewritten by sentinel program. So when restarting redis-cluster, be careful.

Sources

  • https://redis.io/topics/cluster-tutorial
  • https://redis.io/topics/security
  • https://redis.io/commands/debug-segfault

What is DBF file? How to read it in linux and python?

What is DBF files ?

A DBF file is a standard database file used by dBASE, a database management system application. It organises data into multiple records with fields stored in an array data type. DBF files are also compatible with other “xBase” database programs, which became an important feature because of the file format’s popularity.

Tools which can read or open DBF files

Below are list of program which can read and open dbf file.

  • Windows
    1. dBase
    2. Microsoft Access
    3. Microsoft Excel
    4. Visual Foxpro
    5. Apache OpenOffice
    6. dbfview
    7. dbf Viewer Plus
  • Linux
    1. Apache OpenOffice
    2. GTK DBF Editor

How to read file in linux ?

“dbview” command available in linux, which can read dbf files.

Below code snippet show how to use dbview command.

 How to read it using python ?

dbfread” is the library available in python to read dbf files. This library reads DBF files and returns the data as native Python data types for further processing.

dbfread requires python 3.2 or 2.7.  dbfread is a pure python module, so doesn’t depend on any packages outside the standard library.

You can install library by the command below.

The below code snippet can read dbf file and retrieve data as python dictionary.

You can also use the with statement:

By default the records are streamed directly from the file.  If you have enough memory you can load them into a list instead. This allows random access

 How to Write content in DBF file using python ?

dbfpy is a python-only module for reading and writing DBF-files.  dbfpy can read and write simple DBF-files.

You can install it by using below command

The below example shows how to create dbf files and write records in to it.

Also you can update a dbf file record using dbf module.

The below example shows how to update a record in a .dbf file.

 

What is milter?

Every one gets tons of email these days. This includes emails about super duper offers from amazon to princess and wealthy businessmen trying to offer their money to you from some African country that you have never heard of. In all these emails in your inbox there lies one or two valuable emails either from your friends, bank alerts, work related stuff. Spam is a problem that email service providers are battling for ages. There are a few opensource spam fighting tools available like SpamAssasin or SpamBayes.

What is milter ?

Simply put – milter is mail filtering technology. Its designed by sendmail project. Now available in other MTAs also. People historically used all kinds of solutions for filtering mails on servers using procmail or MTA specific methods. The current scene seems to be moving forward to sieve. But there is a huge difference between milter and sieve. Sieve comes in to picture when mail is already accepted by MTA and had been handed over to MDA. On the other hand milter springs into action in the mail receiving part of MTA. When a new connection is made by remote server to your MTA, your MTA will give you an opportunity to accept of reject the mail every step of the way from new connection, reception of each header, and reception of body.

milter stages
milter protocol various stages

The above picture depicts simplified version of milter protocol working. Full details of milter protocol can be found here https://github.com/avar/sendmail-pmilter/blob/master/doc/milter-protocol.txt  . Not only filtering; using milter, you can also modify message or change headers.

HOW DO I GET STARTED WITH CODING MILTER PROGRAM ?

If you want to get started in C you can use libmilter.  For Python you have couple of options:

  1. pymilter –  https://pythonhosted.org/milter/
  2. txmilter – https://github.com/flaviogrossi/txmilter

Postfix supports milter protocol. You can find every thing related to postfix’s milter support in here – http://www.postfix.org/MILTER_README.html

WHY NOT SIEVE WHY MILTER ?

I found sieve to be rather limited. It doesn’t offer too many options to implement complex logic. It was purposefully made like that. Also sieve starts at the end of mail reception process after mail is already accepted by MTA.

Coding milter program in your favorite programming language gives you full power and allows you to implement complex , creative stuff.

WATCHOUT!!!

When writing milter programs take proper care to return a reply to MTA quickly. Don’t do long running tasks in milter program when the MTA is waiting for reply. This will have crazy side effects like remote parties submitting same mail multiple time filling up your inbox.

Compile C program using gcc in Linux

“This post explains about using gcc to compile C program on Linux”

Compile  C program using gcc:

  • What is a compiler:

Compiler is just like translator between programing language and machine language. It converts source written in programing language to executable instructions file for computer. For different programing languages different compilers are available. Compilers differs from operating system to operating system.

  • Open text editor:

Compiling C program start with a text editor to write our C program like VI . It is generally inbuilt in Linux operating systems. By opening terminal in our system we start from there.screenshot-from-2016-12-05-13-14-11

 

 

 

 

  • Write code:

In terminal type: vi sample.c  and Enter key. Then we enter in to vi text editor with our filename given. Now type a or i key to go in to insert mode. Then type our C program in it. After typing C program press Esc key,colon(:),w,q and Enter key  respectively to save and exit from VI. Here is figure showing source code.

screenshot-from-2016-12-05-13-18-02

  • Compile using gcc command:

Now we are in terminal again.Here we type ls command to see our saved file in the list. Then type gcc sample.c -o sample and enter key. Now the gcc compiler compiles our C file and gives the output as filename we given that is executable. screenshot-from-2016-12-05-13-20-29

  • Execute output file:

If there are any mistakes or errors in the program the compiler gives warnings and error messages with line number to find out them easily, after correcting them compile once again. If it compiles successfully it gives executable file.  To check that file we use ls command and see if it is. If it is, now type ./sample in terminal to execute it. we see the result on terminal.screenshot-from-2016-12-05-13-22-54

Actually in compiling process preprocessor adds the necessary files those are included in C libraries. That we are listed in first of our program like <stdio.h> and some other files also generated by compiler . One of  those file is object file.

How to send email command line in linux

We all know that email plays a vital role in our communication system. We will see how to send mail from command line in this article.

Assuming that you have postfix instaled on your machine. You can send the mail from command line by connecting to your local mail server. You will get the utility sendmail with postfix.

send email using sendmail

Send email using telnet

We all now about telnet command. It is very helpful to debug and understand the protocols. Here is the simple mail sent using telnet

In the above example we connected to the SMTP server as a client which is running on localhost. We gave commands to the server to send a mail. Please note that  the data ends with   <CR><LF>.<CR><LF> . to indicated it is the end of data. As you can see  . (dot) we mentioned  at the end.

Once we connected, we pass the set commands to server to write and send a mail. Here are the few SMTP commands

SMTP Commands

  •  HELO
  • MAIL
  • RCPT
  • DATA
  • RSET
  • VRFY
  • EXPN
  • HELP
  • NOOP
  • QUIT

We used MAIL, RCPT and QUIT commands in the above example. To understand these commands go through rfc . You may not need to understand low level protocol concepts of SMTP unless if you are really interested or you are working with SMTP.

References:

[1] http://tools.ietf.org/html/rfc5321

How to truncate empty lines using sed

Sed command is very helpful  for text processing. It’s just matter of regular expression. Here is how you can  truncate empty lines  in the file using command sed.

The above command will give you the output by deleting empty lines. If you want to modify the original file itself. You use the option -i

Example:

I have a file with following data and named as numbers

Now see, the way to truncate empty lines using sed

The above command won’t change the original file. If you want to change original use option -i

Explanation:

Sed interprets ‘/^s*$/d’  and giving us the desired output. Here  text between slashes(/regular expression/) represent the regular expression followed by sed command “d”  means delete.  So command sed matches the lines with given regular expression. If it finds the match it applies the command.

 

How to change hostname of the linux machine

Host name is the name or label which is assigned to your computer that is used to identify and communicate. Usually you can choose your hostname while installation of operating system. If you take a server you may get machine with pre labeled hostname. It is always possible to change the hostname.

Note: You need root or sudo access to change the hostname of the machine.

There are different ways to changes the hostname of the computer. you  canchoose one of them. There is a volatile way and not volatile way. If you want change host name of the computer permanently you need to write your hostname in couple of files. If you want to change instantly (temporarily) you can change using couple of commands, but this change is volatile you will loose your hostname after reboot.

Using command hostname

Now, we will see  the way to change hostname instantly. This is not a persistent change. You will loose after reboot

The command hostname without arguments will display the current hostname of the system

Where you can give qualified hostname in the place of lintel.in. If you are using desktop linux(ubuntu) you can observer your change on (new session)shell, ie, user@hostname

Using sysctl

sysctl is the command that we use to configure kernel parameter at runtime. We can change hostname using this command as well. You can query the hostname using sysctl  as

To change hostname,

Changes made using either way are volatile. To make permanent change. Follow the below steps.

Edit hostname file

You have to edit the file /etc/hostname to make persistent change. This file contains only the hostname. To update hostname,

Including the above change you may need to edit /etc/hosts file. Change the 127.0.0.1   old_hostname  section. Replace old hostname with new hostname if you find.

it could be something like

Restart the machine to get your changes applied. You can combine both ways to get new hostname now and then after reboot.

Using hostnamectl

The linux distributions are also providing the command called hostnamectl to manage and set the hostname.

hostamectl with out any arguments will give the system hostname and other misc details

Check hostname

Set the hostname using hostnamectl

Chaging hostname using nmcli

Oh yeah, you can change the hostname using command nmcli as well. This command can be used to set the static hostname.

Query the hostname

Change the hostname

 

How to find width and height of the terminal in linux

You may wonder how do I find the number of characters will fit in a line in the terminal or size of the window. There are several ways to figure out the size of the window. Let’s go through the different ways to find size of terminal

First lets see different ways that you can use to figure out width and height of terminal in terms of characters

  1. Using command tput
  2. Using command stty
  3. Using environment variables

    Using command tput

The command tput is very helpful to query the terminal information and to do some other simple operations like placing cursor at required random position.

Here is how you can find the width and height of terminal. You may call it as number of rows and columns

Using command stty

This is also one of the command which useful to query,change and print terminal line settings.

The simple output of stty command regarding information of terminal

Command to get lines and columns count in characters

🙂

Using environment variables

On you terminal always you can query the size of the terminal using the environment variables $COLUMNS

Values of those environment variables will change if size of the terminal window changes.

How to add delete routes in linux

There are two commands which are useful either to add or delete route, those are  route and ip. We will see how to change route using command route.

Route Synopsis

Adding route

Deleting route

A quick way to add default route

A  quick way to delete defualt route