How does email work ?

Most of people in this generation would spend their time daily to send and receive emails. E-mail plays a vital role in our daily activities. As technology evolved email became the one of major communication tool. It may not cost you to send or receive a mail but there is an intricate process involved behind. You just need to press  send button to send your message, your message goes through the complex machanism to get received by recipient.

The invention of email was started in 1961. There were more than couple of people to list as the inventors of email.

The evolution of email started from messaging users on the same computer, then message transition between computer then message among multi user and multi computer, finally email. Email has became revolutionary tool for communication. Now a days it is part of our daily life. We will see how email really works.

Terminology

There are different components involved in the email system. There are few abbreviated terms. Having idea about terminology and abbreviations  will help you to better understand the system.

MSA Mail Submission Agent
MTA Mail Transfer Agent
MDA Mail Delivery Agent
MX Mail Exchange
DNS Domain Name System

MUA is the software that user will use either to send or retrieve  the mail(message) from the server.

MSA is the piece of software which is installed on mail server. Where it is responsible to transfer the message to destination or mail server called MTA (Mail Transfer Agent)

MTA  is the Mail Transfer Agent. It is the piece software on the server which is responsible to route the mail to destination mail server. So we called it as mail router, mail server etc.                                                                        Here are few popular  MTA(Mail Server) softwares : postfixqmailCourier Mail and sendmail.

postfix is the one which is widely used, and it comes with many linux distributions.

You can find the exhaustive mail server softwares list here

Protocols involved

There are different protocol involved in email system. All of them are required to get email delivered to the recipient. They are building blocks of email system. Those protocols are,

SMTP Simple Mail Transport Protocol
IMAP Internet Message Access Protocol
POP Post Office Protocol
DNS Domain Name System (Protocol)

How does email work?

This sections will get you the idea about how these protocols work together to deliver the mail to recipient. Here is the abstract overview of email system.

How does email work abstract
How does email work abstract

The above figure will give you simple abstract overview of email system.  As described in the figure. SMTP is the protocol which is used to push or send an email to server by sender. IMAP and POP are the protocols which are used to check or retrieve message from server by recipient.  The recipient MUA is configured to use either IMAP or POP or BOTH (IMAP & POP). The protocol IMPA is bidirectional where POP is unidirectional. We will see more about POP and IMAP in later section.

        As per the above figure(abstract overview) you can sense that, the sender email client(MUA) will send message using SMPT protocol to mail server (MTA). The mail server will check for the destination if it finds the one it will connect to destination mail server, that is other MTA/MDA and will pass the message(mail) using SMTP protocol. The recipient server will store the received mail locally. Later if recipient will check the mail by using his mail client by connecting to his mail server.

Now let’s see flow of email in detail with following figure

Here Alice is sending  email to Bob(bob@domain.com) by using her email client. Where she will push her message to server using SMTP to send Bob. The mail server will determine the destination by getting the MX record for the destination server. The Alice mail server will check for the domain after in the to(recipient)  mail address to get the MX record. Once Alice(sender) mail server received the result from DNS for the DNS query to get MX record, this server will connect to destination mail server and will delivery the mail (message) using SMTP protocol. The destination mail server will store the message locally. Bob will check the newly arrived mails by using his mail client.

I would like to illustrate this process even more by using following figure.

In the above figure there are new components introduced, those are MSA, MDA and extra MX servers.

       MSA is the Mail Submission agent where it is piece of software it will receive the message from MUA. It uses the same protocol SMTP and port 25. Practically most MTAs perform the function of MSA so you may assume MTA as MSA.

       The MDA means mail delivery agent or message delivery agent  is a computer software component that is responsible for the delivery of e-mail messages to a local recipient’s mailbox. You may not need to get confused, most of MTAs perform the MDA functionality as well though there are softwares which are only designed to work as MDA.

Here, you can see that practically there will be more than 1 mail server(MX SERVER)s. All other MX servers are for backup purpose. When sender query DNS for MX record it may get more that one MX server with priorities. Here is the sample output of DNS query for gmail MX records. Gmail is having 5 MX servers with different priorities.

The above result is the out put of command line utility dig. Where we use this utility to query the DNS. In the output you would see something like below in the right side of  ANSWER SECTION.

Here you would see some numbers in front of domains. Those numbers will decide the priority of MX record. The domain which is having very lowest number associated will have highest priority. Here gmail-smtp-in.l.google.com MX record having highest priority cause it having lowest number associated, that is 5. So the sender MTA will try to connect with most prioritized MX first. If it is down or so, it will try to connect with next MX (Mail Exchange server). As a result mail will be delivered with no down time.

Once the mail is received by the destination mail server, it will store the message in the mail store. There are two types of mail stores used by various mail servers(softwares). Those are,

  • Mail DIR
  • Mail Box

Locally stored mail will be accessed(fetched) by the recipient client using either POP or IMAP.

Protocols POP and IMAP

POP and IMAP are application layer protocols, as described above POP and IMAP are the protocols which are used to fetch or access the mails by recipient email client (MUA). Both protocols are different, they server for different purpose.

The protocol POP means Post Office Protocol and POP3 is it’s  version 3.  As the name describes, this protocol is used to download the message from the server by the client. Once message is downloaded form the server, it will be removed unless you set the flag leave a copy on server just how post card is delivered to destination. If you are only the one who will access mailbox from one location the POP3 suites well. This will also save some memory on server. So, if use POP3 you can’t access the mail using different clients.

Unlike POP3 the IMAP won’t download the message by deleting it on server. It will just access the message like browser does webpages. So it is handy if you use multiple clients from different locations.

 

References:

[1] https://tools.ietf.org/html/rfc3501
[2] https://www.ietf.org/rfc/rfc1939.txt
[3] https://tools.ietf.org/html/rfc5321

Email template

Coding HTML for email can be tough. If you included video, flash, javascript or image chances it won’t show up you wanted it to.

You may have had some issues with your in email clients like Gmail or Apple Mail and Outlook.

Some comman problems are :-

  • Some email client not supported all CSS
  • Syntax improperly in tables or missing tags
  • Fonts are not displaying correctly

Troubleshoot your problem :-

If you’re having issues with columns, or rows not appearing in a table, add a border to the table to identify problems and better visualize the table structure.

If you’re seeing centered text when you prefer it left-aligned, make sure you’ve included properties like valign and align in <td> cells and <p> tags.

You may have some issues with your HTML emails looking great in email clients like Gmail or Apple Mail but rendering poorly in Outlook.

Outlook does not support many features :-

  • Animated GIF.
  • Background Images.
  • Padding for <div>, <p>, and <a> tags. Outlook ignores it. Use hspace or vspace attributes on images or tables to format your content.
  • Width for <div> and <p> tags. Outlook ignores it. Use hspace or vspace attributes on images or tables to format your content.

Email Client CSS Support:-

Open below link and check which CSS support in browser, desktop and mobile in various email client’s like outlook, yahoo, gmail.

https://templates.mailchimp.com/resources/email-client-css-support/

Implementing Webhook Handler in Python.

What is Webhook ?

Webhook is an asynchronous HTTP callback on an event occurrence. It is a simple server to server communication for reporting a specific event occurred on a server. The server on which event occurred will fire a HTTP POST request to another server on a URL which is provided by receiving server.

For example, whenever your colleague pushes code commits to github, an event has occurred on github’s server. Now if a webhook URL is provided in github settings, a webhook will be fired to that URL. This webhook will be a HTTP POST request with commit details inside the body in a specified format.  More details on github webhook can be found here.

In this post, I will share my experience of implementing webhook handler in python. For the readers, basic knowledge on implementing web application in python would be better.

Webhook Handler

A Webhook can be handled by simply providing a URL endpoint in a web application. Following is an example using Django. Add webhook url in urls.py

Now create view function in views.py which will parse the data and process it.  In most of the cases, webhook data is sent in JSON format. So lets load the webhook data and sent the data to process_webhook function.

Most of the web applications accept POST request after verifying CSRF token, but here we need to exempt it from this check. So put @csrf_token decorator above the view function. Also put an @require_post decorator to ensure the request is only POST.

The above implementation of URL endpoint will remain different for various other python web framework like Flask, tornado, twisted. But the below code  process_webhook function implementation will remain same irrespective of any framework.

Processing event

There may be different type events we need to handle. So, before proceeding to implement process_webhook function, lets create a python module named webhook_events.py, which will contain a single function for each type of event wherein will be the logic for that particular event. In other words, we are going to map event name with its function, which will handle the logic for that particular type of webhook event.

There are many ways to implement process_webhook function and how we map a webhook event with its function. We are going to discuss different implementation of process_webhook based on extendability. Most basic version of that is below.

A Better way

Now suppose, there are 10s of webhook to be served. We certainly don’t want to write repetitive code. So below is a better way of implementing process_webhook. Here we just replace dot in event name with underscore, so that we get the function name written in webhook_events.py for that event. If the function is not found that means event is not registered (not being served). In this way, no matter the number webhook to be served, just write the function to handle it, in webhook_events.py

Decorators

More robust and pythonic way of implementing process_webhook is by using decorators. Lets define a decorator in webhook_events.py which will map the event_name to its function. Here the EVENT_MAP is dictionary inside a setting module, which will contain event name as key and event function as its value.

In this case, the process_webhook will look like below:

This is the way which I prefer to implement webhook handler in python. How would you prefer ? Please feel free to comment below.

How to assign IP to network interface command line in linux

This article objective is provide you the way to assign the IP address to the network interface from the command line.

You can see the available network interfaces on your machine by using the simple command ifconfig

You can find the interfaces and their names and you can identify the name which is  preceded by colon(:) in the left side. Something like eth0, lo and wlan0 etc.  The network interface lo interface is the special interface where we called it as localhost and special IP assigned to it 127.0.0.1 also called loopback IP.

Do you know? you can assign multiple IPs to single network interface. This is pretty useful when you need multiple IP addresses but you have only one network card.

We created new interface alias with IP assigned. We can do it simply by giving colon(:) and alias number.

Assign IP using ifconfig

This command will assign the specified IP to the give network interface. It is not a persistent change. The would change after reboot.

 

Get IP using dhclient

The program dhclient will get you the IP to the given interface using the Dynamic Host Configuration Protocol (DHCP). The IP which will be assigned to the given network interface which will be provided by gateway or router. If you won’t have any preference of having specific IP assigned to the network card, this can be used

 

Assign static IP

The IP we assigned above is not a persistent or static IP. Means you will lose that IP and will get an other different IP assigned after reboot. To make this IP permanent we have to edit the configuration files.

Ubuntu /etc/network/interfaces

 

CentOS /etc/sysconfig/network-scripts/ifcfg-eth0

Unlike the distribution Debian, CentOS maintains the configuration in a separate file for each interface. The  file path would be something like  /etc/sysconfig/network-scripts/ifcfg-<interface_name>.