Twisted Matrix

Sending emails asynchronously using Twisted – Part 2

In Part 1 of article, we saw how to send blocking emails using ‘smtplib’ module & non-blocking emails using Twisted framework. In this part, we will see how to send asynchronous emails to multiple recipients using Twisted

  • Sending multiple emails

    Refer following script.This script sends emails to given recipients asynchronously. Here we have used twisted.internet.defer.DeferredList API. This API is very useful in some scenarios. Suppose you have to finish multiple task asynchronously and then you have to finish one final task. For examples, your program is connected to 4 different clients & and before shutting it down, you have to make sure that all connections are closed properly. In such cases, DeferredList API is used. Create deferrands of each task & make their list. Pass this list to ‘DeferredList‘ API which will return you another deferrand. This final deferrand will be fired when all deferrands in list will be fired.

        
    
            #!/usr/bin/env python2.7
            __author__ = 'Rohit Chormale'
    
            """
            In this tutorial, same email will be sent to multiple recipients using `twisted.internet.defer.DeferredList` API.
            For each recipient, one defer will be created and will be added in `DeferredList`.
            `DeferredList` API will be fired only when all deferrands in `DeferredList` will be fired.
            """
    
    
            from StringIO import StringIO
            from email.mime.text import MIMEText
            from email.mime.multipart import MIMEMultipart
            from email.utils import formatdate
            from email.header import Header
    
            from twisted.mail.smtp import ESMTPSenderFactory
            from twisted.internet import reactor, defer
            from twisted.internet.ssl import ClientContextFactory
    
    
            # enter email content
            CONTENT = """
            """
            # enter email subject
            SUBJECT = ""
            # enter sender email
            FROM_EMAIL = ""
            # list of multiple recipients
            RECIPIENTS = [['test1@example.com', 'test2@example.com'],
                          ['test3@example.com',],
                          ['test4@example.com', 'test4@example.com'],
                        ]
    
    
            # enter username of your email account
            USERNAME = "your-email"
            # enter password of your email account
            PASSWORD = "your-password"
            # enter smtp host of your email provider
            SMTP_HOST = "smtp-host"
            # enter smtp port of your email provider
            SMTP_PORT = 587
    
    
            def success(result, recipients):
                print 'Email sent successfully | recipients - %s' % recipients
                print result
    
    
            def failure(error, recipients):
                print 'Failed to send email | recipients - %s' % recipients
                print error
    
    
            def send_email(to_emails):
                mime_text = MIMEText(CONTENT, 'html', 'utf-8')
                mime_msg = MIMEMultipart('alternative')
                mime_msg['Subject'] = "%s" % Header(SUBJECT, 'utf-8')
                mime_msg['To'] = ",".join(to_emails)
                mime_msg['From'] = FROM_EMAIL
                mime_msg['Date'] = formatdate(localtime=True)
                mime_msg.attach(mime_text)
                mime_msg = StringIO(mime_msg.as_string())
                df = defer.Deferred()
                f = ESMTPSenderFactory(USERNAME, PASSWORD, FROM_EMAIL, to_emails, mime_msg,
                                       df, retries=2, contextFactory=ClientContextFactory(), requireTransportSecurity=True)
                reactor.connectTCP(SMTP_HOST, SMTP_PORT, f)
                return df
    
    
            def final_success(result):
                print 'All emails processed successfully'
                print result
                reactor.stop()
    
    
            def final_failure(error):
                print 'Failed to process all emails'
                print error
                reactor.stop()
    
    
            def send_multiple_emails(recipients):
                df_list = []
                for r in recipients:
                    df = send_email(r)
                    df.addCallback(success, recipients=r)
                    df.addErrback(failure, recipients=r)
                    df_list.append(df)
                final_df = defer.DeferredList(df_list, consumeErrors=0)
                return final_df
    
    
            def main():
                df = send_multiple_emails(RECIPIENTS)
                df.addCallback(final_success)
                df.addErrback(final_failure)
    
    
            reactor.callLater(1, main)
    
    
            if __name__ == '__main__':
                reactor.run()
    
    

     

  • Sending multiple emails using coiterator

    Though above script runs fine, there is one problem. Here, recipients number is very small. But suppose you have to send emails to millions recipients then will this code work ?. Refer function ‘send_multiple_emails’.

        
            def send_multiple_emails(recipients):
                df_list = []
                for r in recipients:
                    df = send_email(r)
                    df.addCallback(success, recipients=r)
                    df.addErrback(failure, recipients=r)
                    df_list.append(df)
                final_df = defer.DeferredList(df_list, consumeErrors=0)
                return final_df
    

    Here we have used ‘for’ loop which is blocking. So until this ‘for’ loop is iterated, program will not move to next line of code. For 3 recipients iteration will not take much time however for millions of recipients, it will not work.
    So lets modify our code to work like generators.

        
    #!/usr/bin/env python2.7
    __author__ = 'Rohit Chormale'
    
    
    """
    In this tutorial, same email will be sent to multiple recipients using `twisted.internet.task` API.
    Each email recipient will be yielded using `twisted.internet.task.coiterate` API.
    """
    
    
    from StringIO import StringIO
    from email.mime.text import MIMEText
    from email.mime.multipart import MIMEMultipart
    from email.utils import formatdate
    from email.header import Header
    
    from twisted.mail.smtp import ESMTPSenderFactory
    from twisted.internet import reactor, defer, task
    from twisted.internet.ssl import ClientContextFactory
    
    
    # enter email content
    CONTENT = """
    """
    # enter email subject
    SUBJECT = ""
    # enter sender email
    FROM_EMAIL = ""
    # list of multiple recipients
    RECIPIENTS = [['test1@example.com', 'test2@example.com'],
                  ['test3@example.com',],
                  ['test4@example.com', 'test4@example.com'],
                ]
    
    
    # enter username of your email account
    USERNAME = "your-email"
    # enter password of your email account
    PASSWORD = "your-password"
    # enter smtp host of your email provider
    SMTP_HOST = "your-mail-provider=host"
    # enter smtp port of your email provider
    SMTP_PORT = 587
    
    
    def success(result, recipients):
        print 'Email sent successfully | recipients - %s' % recipients
        print result
    
    
    def failure(error, recipients):
        print 'Failed to send email | recipients - %s' % recipients
        print error
    
    
    def send_email(to_emails):
        mime_text = MIMEText(CONTENT, 'html', 'utf-8')
        mime_msg = MIMEMultipart('alternative')
        mime_msg['Subject'] = "%s" % Header(SUBJECT, 'utf-8')
        mime_msg['To'] = ",".join(to_emails)
        mime_msg['From'] = FROM_EMAIL
        mime_msg['Date'] = formatdate(localtime=True)
        mime_msg.attach(mime_text)
        mime_msg = StringIO(mime_msg.as_string())
        df = defer.Deferred()
        f = ESMTPSenderFactory(USERNAME, PASSWORD, FROM_EMAIL, to_emails, mime_msg,
                               df, retries=2, contextFactory=ClientContextFactory(), requireTransportSecurity=True)
        reactor.connectTCP(SMTP_HOST, SMTP_PORT, f)
        return df
    
    
    def final_success(result):
        print 'All emails processed successfully'
        print result
        reactor.stop()
    
    
    def final_failure(error):
        print 'Failed to process all emails'
        print error
        reactor.stop()
    
    
    def yield_recipients(recipients):
        for r in recipients:
            df = send_email(r)
            df.addCallback(success, recipients=r)
            df.addErrback(failure, recipients=r)
            yield df
    
    
    def send_multiple_emails(recipients):
        final_df = task.coiterate(yield_recipients(recipients))
        return final_df
    
    
    def main():
        df = send_multiple_emails(RECIPIENTS)
        df.addCallback(final_success)
        df.addErrback(final_failure)
    
    
    reactor.callLater(1, main)
    
    
    if __name__ == '__main__':
        reactor.run()
    

    Here, we have used twisted.internet.task.coiterate API. This API iterates over iterator by dividing reactor runtime between all iterators. Thus we can send millions of emails asynchronously.

About Rohit Chormale

I am software engineer from Hyderabad, India with primary interests in distributed systems. In my leisure time, I love to read technical papers and hacking emacs.

Leave a Reply

Your email address will not be published. Required fields are marked *