Tag Archives: email

What is milter?

Every one gets tons of email these days. This includes emails about super duper offers from amazon to princess and wealthy businessmen trying to offer their money to you from some African country that you have never heard of. In all these emails in your inbox there lies one or two valuable emails either from your friends, bank alerts, work related stuff. Spam is a problem that email service providers are battling for ages. There are a few opensource spam fighting tools available like SpamAssasin or SpamBayes.

What is milter ?

Simply put – milter is mail filtering technology. Its designed by sendmail project. Now available in other MTAs also. People historically used all kinds of solutions for filtering mails on servers using procmail or MTA specific methods. The current scene seems to be moving forward to sieve. But there is a huge difference between milter and sieve. Sieve comes in to picture when mail is already accepted by MTA and had been handed over to MDA. On the other hand milter springs into action in the mail receiving part of MTA. When a new connection is made by remote server to your MTA, your MTA will give you an opportunity to accept of reject the mail every step of the way from new connection, reception of each header, and reception of body.

milter stages
milter protocol various stages

The above picture depicts simplified version of milter protocol working. Full details of milter protocol can be found here https://github.com/avar/sendmail-pmilter/blob/master/doc/milter-protocol.txt  . Not only filtering; using milter, you can also modify message or change headers.

HOW DO I GET STARTED WITH CODING MILTER PROGRAM ?

If you want to get started in C you can use libmilter.  For Python you have couple of options:

  1. pymilter –  https://pythonhosted.org/milter/
  2. txmilter – https://github.com/flaviogrossi/txmilter

Postfix supports milter protocol. You can find every thing related to postfix’s milter support in here – http://www.postfix.org/MILTER_README.html

WHY NOT SIEVE WHY MILTER ?

I found sieve to be rather limited. It doesn’t offer too many options to implement complex logic. It was purposefully made like that. Also sieve starts at the end of mail reception process after mail is already accepted by MTA.

Coding milter program in your favorite programming language gives you full power and allows you to implement complex , creative stuff.

WATCHOUT!!!

When writing milter programs take proper care to return a reply to MTA quickly. Don’t do long running tasks in milter program when the MTA is waiting for reply. This will have crazy side effects like remote parties submitting same mail multiple time filling up your inbox.

How does email work

How does email work ?

Most of people in this generation would spend their time daily to send and receive emails. E-mail plays a vital role in our daily activities. As technology evolved email became the one of major communication tool. It may not cost you to send or receive a mail but there is an intricate process involved behind. You just need to press  send button to send your message, your message goes through the complex machanism to get received by recipient.

The invention of email was started in 1961. There were more than couple of people to list as the inventors of email.

The evolution of email started from messaging users on the same computer, then message transition between computer then message among multi user and multi computer, finally email. Email has became revolutionary tool for communication. Now a days it is part of our daily life. We will see how email really works.

Terminology

There are different components involved in the email system. There are few abbreviated terms. Having idea about terminology and abbreviations  will help you to better understand the system.

MSA Mail Submission Agent
MTA Mail Transfer Agent
MDA Mail Delivery Agent
MX Mail Exchange
DNS Domain Name System

MUA is the software that user will use either to send or retrieve  the mail(message) from the server.

MSA is the piece of software which is installed on mail server. Where it is responsible to transfer the message to destination or mail server called MTA (Mail Transfer Agent)

MTA  is the Mail Transfer Agent. It is the piece software on the server which is responsible to route the mail to destination mail server. So we called it as mail router, mail server etc.                                                                        Here are few popular  MTA(Mail Server) softwares : postfixqmailCourier Mail and sendmail.

postfix is the one which is widely used, and it comes with many linux distributions.

You can find the exhaustive mail server softwares list here

Protocols involved

There are different protocol involved in email system. All of them are required to get email delivered to the recipient. They are building blocks of email system. Those protocols are,

SMTP Simple Mail Transport Protocol
IMAP Internet Message Access Protocol
POP Post Office Protocol
DNS Domain Name System (Protocol)

How does email work?

This sections will get you the idea about how these protocols work together to deliver the mail to recipient. Here is the abstract overview of email system.

How does email work abstract
How does email work abstract

The above figure will give you simple abstract overview of email system.  As described in the figure. SMTP is the protocol which is used to push or send an email to server by sender. IMAP and POP are the protocols which are used to check or retrieve message from server by recipient.  The recipient MUA is configured to use either IMAP or POP or BOTH (IMAP & POP). The protocol IMPA is bidirectional where POP is unidirectional. We will see more about POP and IMAP in later section.

        As per the above figure(abstract overview) you can sense that, the sender email client(MUA) will send message using SMPT protocol to mail server (MTA). The mail server will check for the destination if it finds the one it will connect to destination mail server, that is other MTA/MDA and will pass the message(mail) using SMTP protocol. The recipient server will store the received mail locally. Later if recipient will check the mail by using his mail client by connecting to his mail server.

Now let’s see flow of email in detail with following figure

Here Alice is sending  email to Bob(bob@domain.com) by using her email client. Where she will push her message to server using SMTP to send Bob. The mail server will determine the destination by getting the MX record for the destination server. The Alice mail server will check for the domain after in the to(recipient)  mail address to get the MX record. Once Alice(sender) mail server received the result from DNS for the DNS query to get MX record, this server will connect to destination mail server and will delivery the mail (message) using SMTP protocol. The destination mail server will store the message locally. Bob will check the newly arrived mails by using his mail client.

I would like to illustrate this process even more by using following figure.

In the above figure there are new components introduced, those are MSA, MDA and extra MX servers.

       MSA is the Mail Submission agent where it is piece of software it will receive the message from MUA. It uses the same protocol SMTP and port 25. Practically most MTAs perform the function of MSA so you may assume MTA as MSA.

       The MDA means mail delivery agent or message delivery agent  is a computer software component that is responsible for the delivery of e-mail messages to a local recipient’s mailbox. You may not need to get confused, most of MTAs perform the MDA functionality as well though there are softwares which are only designed to work as MDA.

Here, you can see that practically there will be more than 1 mail server(MX SERVER)s. All other MX servers are for backup purpose. When sender query DNS for MX record it may get more that one MX server with priorities. Here is the sample output of DNS query for gmail MX records. Gmail is having 5 MX servers with different priorities.

$ dig gmail.com  MX

; <<>> DiG 9.8.3-P1 <<>> gmail.com MX
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 7760
;; flags: qr rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;gmail.com.			IN	MX

;; ANSWER SECTION:
gmail.com.		1206	IN	MX	20 alt2.gmail-smtp-in.l.google.com.
gmail.com.		1206	IN	MX	5 gmail-smtp-in.l.google.com.
gmail.com.		1206	IN	MX	10 alt1.gmail-smtp-in.l.google.com.
gmail.com.		1206	IN	MX	40 alt4.gmail-smtp-in.l.google.com.
gmail.com.		1206	IN	MX	30 alt3.gmail-smtp-in.l.google.com.

;; Query time: 18 msec
;; SERVER: 192.168.4.1#53(192.168.4.1)
;; WHEN: Sun Feb 14 09:42:23 2016
;; MSG SIZE  rcvd: 150

The above result is the out put of command line utility dig. Where we use this utility to query the DNS. In the output you would see something like below in the right side of  ANSWER SECTION.

20 alt2.gmail-smtp-in.l.google.com.
5 gmail-smtp-in.l.google.com.
40 alt4.gmail-smtp-in.l.google.com.
10 alt1.gmail-smtp-in.l.google.com.
30 alt3.gmail-smtp-in.l.google.com.

Here you would see some numbers in front of domains. Those numbers will decide the priority of MX record. The domain which is having very lowest number associated will have highest priority. Here gmail-smtp-in.l.google.com MX record having highest priority cause it having lowest number associated, that is 5. So the sender MTA will try to connect with most prioritized MX first. If it is down or so, it will try to connect with next MX (Mail Exchange server). As a result mail will be delivered with no down time.

Once the mail is received by the destination mail server, it will store the message in the mail store. There are two types of mail stores used by various mail servers(softwares). Those are,

  • Mail DIR
  • Mail Box

Locally stored mail will be accessed(fetched) by the recipient client using either POP or IMAP.

Protocols POP and IMAP

POP and IMAP are application layer protocols, as described above POP and IMAP are the protocols which are used to fetch or access the mails by recipient email client (MUA). Both protocols are different, they server for different purpose.

The protocol POP means Post Office Protocol and POP3 is it’s  version 3.  As the name describes, this protocol is used to download the message from the server by the client. Once message is downloaded form the server, it will be removed unless you set the flag leave a copy on server just how post card is delivered to destination. If you are only the one who will access mailbox from one location the POP3 suites well. This will also save some memory on server. So, if use POP3 you can’t access the mail using different clients.

Unlike POP3 the IMAP won’t download the message by deleting it on server. It will just access the message like browser does webpages. So it is handy if you use multiple clients from different locations.

 

References:

[1] https://tools.ietf.org/html/rfc3501
[2] https://www.ietf.org/rfc/rfc1939.txt
[3] https://tools.ietf.org/html/rfc5321

Email template

Coding HTML for email can be tough. If you included video, flash, javascript or image chances it won’t show up you wanted it to.

You may have had some issues with your in email clients like Gmail or Apple Mail and Outlook.

Some comman problems are :-

  • Some email client not supported all CSS
  • Syntax improperly in tables or missing tags
  • Fonts are not displaying correctly

Troubleshoot your problem :-

If you’re having issues with columns, or rows not appearing in a table, add a border to the table to identify problems and better visualize the table structure.

If you’re seeing centered text when you prefer it left-aligned, make sure you’ve included properties like valign and align in <td> cells and <p> tags.

You may have some issues with your HTML emails looking great in email clients like Gmail or Apple Mail but rendering poorly in Outlook.

Outlook does not support many features :-

  • Animated GIF.
  • Background Images.
  • Padding for <div>, <p>, and <a> tags. Outlook ignores it. Use hspace or vspace attributes on images or tables to format your content.
  • Width for <div> and <p> tags. Outlook ignores it. Use hspace or vspace attributes on images or tables to format your content.

Email Client CSS Support:-

Open below link and check which CSS support in browser, desktop and mobile in various email client’s like outlook, yahoo, gmail.

https://templates.mailchimp.com/resources/email-client-css-support/

Twisted Matrix

Sending emails asynchronously using Twisted – Part 2

In Part 1 of article, we saw how to send blocking emails using ‘smtplib’ module & non-blocking emails using Twisted framework. In this part, we will see how to send asynchronous emails to multiple recipients using Twisted

  • Sending multiple emails

    Refer following script.This script sends emails to given recipients asynchronously. Here we have used twisted.internet.defer.DeferredList API. This API is very useful in some scenarios. Suppose you have to finish multiple task asynchronously and then you have to finish one final task. For examples, your program is connected to 4 different clients & and before shutting it down, you have to make sure that all connections are closed properly. In such cases, DeferredList API is used. Create deferrands of each task & make their list. Pass this list to ‘DeferredList‘ API which will return you another deferrand. This final deferrand will be fired when all deferrands in list will be fired.

        
    
            #!/usr/bin/env python2.7
            __author__ = 'Rohit Chormale'
    
            """
            In this tutorial, same email will be sent to multiple recipients using `twisted.internet.defer.DeferredList` API.
            For each recipient, one defer will be created and will be added in `DeferredList`.
            `DeferredList` API will be fired only when all deferrands in `DeferredList` will be fired.
            """
    
    
            from StringIO import StringIO
            from email.mime.text import MIMEText
            from email.mime.multipart import MIMEMultipart
            from email.utils import formatdate
            from email.header import Header
    
            from twisted.mail.smtp import ESMTPSenderFactory
            from twisted.internet import reactor, defer
            from twisted.internet.ssl import ClientContextFactory
    
    
            # enter email content
            CONTENT = """
            """
            # enter email subject
            SUBJECT = ""
            # enter sender email
            FROM_EMAIL = ""
            # list of multiple recipients
            RECIPIENTS = [['test1@example.com', 'test2@example.com'],
                          ['test3@example.com',],
                          ['test4@example.com', 'test4@example.com'],
                        ]
    
    
            # enter username of your email account
            USERNAME = "your-email"
            # enter password of your email account
            PASSWORD = "your-password"
            # enter smtp host of your email provider
            SMTP_HOST = "smtp-host"
            # enter smtp port of your email provider
            SMTP_PORT = 587
    
    
            def success(result, recipients):
                print 'Email sent successfully | recipients - %s' % recipients
                print result
    
    
            def failure(error, recipients):
                print 'Failed to send email | recipients - %s' % recipients
                print error
    
    
            def send_email(to_emails):
                mime_text = MIMEText(CONTENT, 'html', 'utf-8')
                mime_msg = MIMEMultipart('alternative')
                mime_msg['Subject'] = "%s" % Header(SUBJECT, 'utf-8')
                mime_msg['To'] = ",".join(to_emails)
                mime_msg['From'] = FROM_EMAIL
                mime_msg['Date'] = formatdate(localtime=True)
                mime_msg.attach(mime_text)
                mime_msg = StringIO(mime_msg.as_string())
                df = defer.Deferred()
                f = ESMTPSenderFactory(USERNAME, PASSWORD, FROM_EMAIL, to_emails, mime_msg,
                                       df, retries=2, contextFactory=ClientContextFactory(), requireTransportSecurity=True)
                reactor.connectTCP(SMTP_HOST, SMTP_PORT, f)
                return df
    
    
            def final_success(result):
                print 'All emails processed successfully'
                print result
                reactor.stop()
    
    
            def final_failure(error):
                print 'Failed to process all emails'
                print error
                reactor.stop()
    
    
            def send_multiple_emails(recipients):
                df_list = []
                for r in recipients:
                    df = send_email(r)
                    df.addCallback(success, recipients=r)
                    df.addErrback(failure, recipients=r)
                    df_list.append(df)
                final_df = defer.DeferredList(df_list, consumeErrors=0)
                return final_df
    
    
            def main():
                df = send_multiple_emails(RECIPIENTS)
                df.addCallback(final_success)
                df.addErrback(final_failure)
    
    
            reactor.callLater(1, main)
    
    
            if __name__ == '__main__':
                reactor.run()
    
    

     

  • Sending multiple emails using coiterator

    Though above script runs fine, there is one problem. Here, recipients number is very small. But suppose you have to send emails to millions recipients then will this code work ?. Refer function ‘send_multiple_emails’.

        
            def send_multiple_emails(recipients):
                df_list = []
                for r in recipients:
                    df = send_email(r)
                    df.addCallback(success, recipients=r)
                    df.addErrback(failure, recipients=r)
                    df_list.append(df)
                final_df = defer.DeferredList(df_list, consumeErrors=0)
                return final_df
    

    Here we have used ‘for’ loop which is blocking. So until this ‘for’ loop is iterated, program will not move to next line of code. For 3 recipients iteration will not take much time however for millions of recipients, it will not work.
    So lets modify our code to work like generators.

        
    #!/usr/bin/env python2.7
    __author__ = 'Rohit Chormale'
    
    
    """
    In this tutorial, same email will be sent to multiple recipients using `twisted.internet.task` API.
    Each email recipient will be yielded using `twisted.internet.task.coiterate` API.
    """
    
    
    from StringIO import StringIO
    from email.mime.text import MIMEText
    from email.mime.multipart import MIMEMultipart
    from email.utils import formatdate
    from email.header import Header
    
    from twisted.mail.smtp import ESMTPSenderFactory
    from twisted.internet import reactor, defer, task
    from twisted.internet.ssl import ClientContextFactory
    
    
    # enter email content
    CONTENT = """
    """
    # enter email subject
    SUBJECT = ""
    # enter sender email
    FROM_EMAIL = ""
    # list of multiple recipients
    RECIPIENTS = [['test1@example.com', 'test2@example.com'],
                  ['test3@example.com',],
                  ['test4@example.com', 'test4@example.com'],
                ]
    
    
    # enter username of your email account
    USERNAME = "your-email"
    # enter password of your email account
    PASSWORD = "your-password"
    # enter smtp host of your email provider
    SMTP_HOST = "your-mail-provider=host"
    # enter smtp port of your email provider
    SMTP_PORT = 587
    
    
    def success(result, recipients):
        print 'Email sent successfully | recipients - %s' % recipients
        print result
    
    
    def failure(error, recipients):
        print 'Failed to send email | recipients - %s' % recipients
        print error
    
    
    def send_email(to_emails):
        mime_text = MIMEText(CONTENT, 'html', 'utf-8')
        mime_msg = MIMEMultipart('alternative')
        mime_msg['Subject'] = "%s" % Header(SUBJECT, 'utf-8')
        mime_msg['To'] = ",".join(to_emails)
        mime_msg['From'] = FROM_EMAIL
        mime_msg['Date'] = formatdate(localtime=True)
        mime_msg.attach(mime_text)
        mime_msg = StringIO(mime_msg.as_string())
        df = defer.Deferred()
        f = ESMTPSenderFactory(USERNAME, PASSWORD, FROM_EMAIL, to_emails, mime_msg,
                               df, retries=2, contextFactory=ClientContextFactory(), requireTransportSecurity=True)
        reactor.connectTCP(SMTP_HOST, SMTP_PORT, f)
        return df
    
    
    def final_success(result):
        print 'All emails processed successfully'
        print result
        reactor.stop()
    
    
    def final_failure(error):
        print 'Failed to process all emails'
        print error
        reactor.stop()
    
    
    def yield_recipients(recipients):
        for r in recipients:
            df = send_email(r)
            df.addCallback(success, recipients=r)
            df.addErrback(failure, recipients=r)
            yield df
    
    
    def send_multiple_emails(recipients):
        final_df = task.coiterate(yield_recipients(recipients))
        return final_df
    
    
    def main():
        df = send_multiple_emails(RECIPIENTS)
        df.addCallback(final_success)
        df.addErrback(final_failure)
    
    
    reactor.callLater(1, main)
    
    
    if __name__ == '__main__':
        reactor.run()
    

    Here, we have used twisted.internet.task.coiterate API. This API iterates over iterator by dividing reactor runtime between all iterators. Thus we can send millions of emails asynchronously.

Sending emails asynchronously using Twisted – Part 1

  • Using ‘smtplib‘ module

It is very easy to send emails using ‘smtplib‘ module of python. Check following recipe.

#!/usr/bin/env python2.7
__author__ = 'Rohit Chormale'


from smtplib import SMTP


# enter email content
CONTENT = """
"""
# enter email subject
SUBJECT = """
"""
# enter recipients
TO_EMAILS = ["", ""]
# enter sender's emails
FROM_EMAIL = ""
# enter username of your email account
USERNAME = ""
# enter password of your email account
PASSWORD = ""
# enter smtp host of your email provider
SMTP_HOST = ""
# enter smtp host of your email provider
SMTP_PORT = 0


mailer = SMTP(SMTP_HOST, SMTP_PORT)
mailer.starttls()
mailer.login(USERNAME, PASSWORD)
mailer.sendmail(FROM_EMAIL, TO_EMAILS, CONTENT)
mailer.quit()

But ‘smtplib’ module sends emails synchronously. So code execution is blocked until email is sent. To overcome this, lets try to send email asynchornously.

  • Using Twisted

For this tutorial we are going to use Twisted framework. Twisted is event-driven networking engine. It uses reactor-pattern. Twisted uses deferred objects to address waiting IOs. Deferred is more like subset of promises. Check following recipe to send asynchronously MIME message using Twisted.

#!/usr/bin/env python2.7
__author__ = 'Rohit Chormale'


from StringIO import StringIO
from email.mime.text import MIMEText
from email.mime.multipart import MIMEMultipart
from email.utils import formatdate
from email.header import Header

from twisted.mail.smtp import ESMTPSenderFactory
from twisted.internet import reactor, defer
from twisted.internet.ssl import ClientContextFactory

# enter email content
CONTENT = """
"""
# enter email subject
SUBJECT = ""
# enter recipients
TO_EMAILS = ["", ""]
# enter sender email
FROM_EMAIL = ""
# enter username of your email account
USERNAME = ""
# enter password of your email account
PASSWORD = ""
# enter smtp host of your email provider
SMTP_HOST = ""
# enter smtp port of your email provider
SMTP_PORT = 0


def success(result):
    print 'Email sent successfully'
    print result
    reactor.stop()


def failure(error):
    print 'Failed to send email'
    print error
    reactor.stop()


def send_email():
    mime_text = MIMEText(CONTENT, 'html', 'utf-8')
    mime_msg = MIMEMultipart('alternative')
    mime_msg['Subject'] = "%s" % Header(SUBJECT, 'utf-8')
    mime_msg['To'] = ",".join(TO_EMAILS)
    mime_msg['From'] = FROM_EMAIL
    mime_msg['Date'] = formatdate(localtime=True)
    mime_msg.attach(mime_text)
    mime_msg = StringIO(mime_msg.as_string())
    df = defer.Deferred()
    f = ESMTPSenderFactory(USERNAME, PASSWORD, FROM_EMAIL, TO_EMAILS, mime_msg,
                           df, retries=2, contextFactory=ClientContextFactory(), requireTransportSecurity=True)
    reactor.connectTCP(SMTP_HOST, SMTP_PORT, f)
    return df


def main():
    df = send_email()
    df.addCallback(success)
    df.addErrback(failure)


reactor.callLater(1, main)


if __name__ == '__main__':
    reactor.run()