Sending emails asynchronously using Twisted – Part 1

  • Using ‘smtplib‘ module

It is very easy to send emails using ‘smtplib‘ module of python. Check following recipe.

But ‘smtplib’ module sends emails synchronously. So code execution is blocked until email is sent. To overcome this, lets try to send email asynchornously.

  • Using Twisted

For this tutorial we are going to use Twisted framework. Twisted is event-driven networking engine. It uses reactor-pattern. Twisted uses deferred objects to address waiting IOs. Deferred is more like subset of promises. Check following recipe to send asynchronously MIME message using Twisted.

 

 

Building hello world using Python Tkinter

Continuing our series of desktop GUIs, in this post we are going to go over the topic of building a small hello world program using Python Tkinter.

What is Tkinter ?

Tkinter is python wrapper/binding to Tk library. Tk was developed as GUI library for Tcl language by John Ousterhout. For many other high level language authors coming to programming scene in 1990s this seemed like easy tool to capitalize on in bringing GUI library to their language. That’s why you can find many different language bindings to Tk. Tk brought easy GUI building to programming masses.

Often times many people write off Tk or Tkinter as ugly old GUI library. Their claims are not completely unfounded. But, Tk in recent times made strides to bring modern look by implementing Tile, themeing engine. Tile is also called as ttk. With ttk in place look and feel issues of Tk are addressed.

ttk xp blue theme demo
Tile xp blue theme demo

But, Tkinter still lags behind in terms available collection of default widgets compared to other libraries like PySide or wxPython . However, Tkinter is still a good candidate if you quickly want to dish out a GUI without having programming styles imposed on you. (You will see these style restrictions in our next posts about wxPython and PySide)

Hello world

Now, we will go over building a simple hello world program using Tkinter. In this program we will create a simple window with “Hello world” text in it. And a button that say “Kill Me”. Clicking that button will cause the python program to exit.

tkinter hello world

That import * line saves your some typing.

This line instantiates Tk and creates main window of the program. Next we create a Label (a widget which displays text ) and call it’s pack method. pack is one three available layout methods provided by Tkinter (rest of the two being grid and place). The widget will not show up until you call pack.  Next, we create a Button widget and pack it, similar to that of Label. The interesting point to note while creating button widget is, we provide command argument to it with a reference of sys.exit function. A newbie mistake that is often made is, passing function with () next to it. That will cause the target function to be invoked immediately. You are only supposed to pass the reference of function and Tkinter will call it when button is pressed. So, when this “Kill Me” button is pressed sys.exit will be called and program exits. One, last important line to notice is root.mainloop() . This line sets the event monitoring loop in motion and Tkinter take control of the program from here.

 

How to create ascii art using VIM plugin DrawIT

Installing VIM plugin DrawIT

Install the plugin if it is no available in your machine

This script/plugin is distributed as a vimball

This can be downloaded here

To download source go to /usr/local/src Create a directory VIM. go to the directory /usr/local/src/VIM (NOTE: it isn’t manadetory to keep source in this location, it’s upto you)

Dowload the source with following wget

wget http://www.vim.org/scripts/download_script.php?src_id=21108 -O DrawIt.vba.gz

To install follow the instructions

Open file vim DrawIT.vba.gz In command mode enter :so %

quit :q

Quick start with DrawIt

That’s all you have to do to install a plug in. Now to test, open a new file

and type the following in command mode

\di

If you see [DrawIt], the plugin is enabled an you can draw you art.

Here is how you will start and stop DrawIt

\di start DrawIt \ds stop DrawIt

User your arrow keys to draw whatever you want

To get help on drawit and to find handy key to draw, type the following in command mode in VIM :help drawit.

 

 

 

Common constructs of GUI programs

In this post we will go over some of the basic and common constructs that you come across while building GUI programs almost in any language or library. The following are some of the common items that I can think of

  1. Loop
  2. Widgets
  3. Events
  4. Layout

Loop

Loop or Mainloop as it is called in some GUI libraries is one of the most common element of all GUI programs. In a GUI program this is usually called at the at end of constructing the all the items (widgets) in program. When you call the mainloop, it takes over the control and keeps running until you terminate the GUI.

The main purpose of loop is to poll for user input (mouse and keyboard actions) and fire events which your program can handle in asynchronous way. Following diagram explains loop in simple terms.

GUI main loop
GUI main loop

As you can see from the above diagram the GUI loop keeps polling for user input. If there is any event via mouse or keyboard or other input devices (joysticks, game pads ) when input is available loop gives control to callback functions that are bound to input events.  Loop keeps running until the main window (also called parent window) exits. Once the main window is destroyed the loop exits and gives control back to your code. In most cases end of the mainloop would be end of many GUI programs.

Widgets

GUIs are collection of widgets. Example of widgets are button, scroll bar, check box etc. Widget is an element of user interaction. Their main purpose is to display or collect information from user. A label displays text to user. A text input widget collects text from user. One way to decide the richness of GUI tool kits is to look at how many widgets they offer. The more widgets a library offers you the better it is for you. So, that you don’t have to write extra code to implement custom widgets.  Widgets usually are sub classes of a Widget parent class. So, most of the widgets inherit properties of widget class in respective GUI libraries. In some GUI libraries main window also has properties of a widget. gui window

Events

When ever user presses a button or enters text using keyboard those actions generate events in GUI libraries. GUI libraries provide a way to hook into these events so that your program can respond to user actions. The event hooking/binding style differs from library to library. But, the aim is same across all libs, that is to provide programmer a way to respond to user input.

Layout

Layout facilities provided by GUI libs helps to make programmer’s life easier when placing widgets in main window. Layouts are simple rules you use to convey to GUI library on how you would like your widgets to be laid out on screen. Instead of manually placing widgets on screen by specifying distance units and placement pixel positions, once you construct your widgets you hand them over to certain layout scheme to let GUI library handle the placement. The benefits are, layouts will take care of expansions, size proportions and other nitty-gritty detail of placing your widgets in orderly way.

The layout features provided by libraries vary greatly. You often have to pick the one that suits your needs. These layouts can be mixed in order to achieve the placement and look you want.

 

Desktop GUI libraries in Python

If you want to build desktop GUIs, Python is right candidate to pick. It offers you plethora of GUI libraries compared to any other language. So, there are high number of chances that, some library in there fits your bill which is readily available, in whole of lot of GUI libs python has to offer. Following is a list of some of the popular GUI libs

  1. Tkinter 
  2. wxPython
  3. PyQt / PySide
  4. PyGTK

(Complete list of available libraries can be found on python wiki page.)

All of the above mentioned ones are cross-platform capable libraries. So, that you don’t have to rewrite any part of your GUI and gladly run it across all the main contenders Windows, Linux and Mac . There are other platform specific ones like win32gui , WinForms using IronPython. But, it helps to stick with cross-platform capable libs to avoid extra work when building your app for other platforms.

Over the next few posts I’ll present you with examples of simple hello world programs using above mentioned list of libraries.

Usually the best way to build your GUI application is to write compute intensive part (if there is any ) of your business logic in low level languages like C/C++ and then write GUI code using the above mentioned Python libs. This saves you lot of time and energy. As, compared to low level languages, in Python you get the benefit of automatic memory management and you only have to write less number of lines of code.

Desktop GUIs are on decline with the advent of web. More and more programs are being coded as services using web technologies ( HTML, CSS, JavaScript ). It has gotten so far, now UI/GUI by default means web UI.  Each passing day web technologies are filling up  the gap in terms of number of features (Dynamic content, Media capabilities ) . But, desktop GUIs still have their place in certain areas and there are numerous programs already written using desktop GUI technologies that needs to be managed.

How to implement PayPal payment gateway

The PayPal REST APIs are supported in two environments. Use the Sandbox environment for testing purposes, then move to the live environment for production processing.

The following endpoints address are two environments:

A complete REST operation is formed by combining an HTTP method with the full URI to the resource you’re addressing. For example, here is the operation to create a new payment:

The PayPal REST sdk can be obtained through pip

OAuth Request / Response

Client Id and Secret Id can be obtained from the application created in the paypal account.

For the following each API call, you’ll need to set request headers, including the access token.

Creating a WebProfile:

name:

The name of the web experience profile which should be unique among the profiles for a given merchant.

presentation:

It contains the parameters for style and presentation.

input_fields:

Parameters for input fields customization:

  1. allow_note : It enables the buyer to enter a note to the merchant on the paypal page during checkout.

  2. no_shipping : Determines whether or not PayPal displays shipping address fields on the experience pages.

    • 0 – PayPal displays the shipping address on the PayPal pages.
    • 1 – PayPal does not display shipping address fields whatsoever.
    • 2 – If you do not pass the shipping address, PayPal obtains it from the buyer’s account profile.
  3. address_override : Determines if the PayPal pages should display the shipping address supplied in this call, rather than the shipping address on file with PayPal for this buyer.

    • 0 – PayPal pages should display the address on file
    • 1 – PayPal pages should display the addresses supplied in this call instead of the address from the buyer’s PayPal account.
flow_config:

Parameters for flow configuration

  1. landing_page_type : Type of PayPal page to be displayed when a user lands on the PayPal site for checkout.

    • Billing – The Non-PayPal account landing page is used
    • Login – The paypal account login page is used.

Creating a Payment:


 

intent:

Payment intent. Allowed values are:

  • “sale” – For immediate payment
  • “authorize” – To authorize a payment for capture later
  • “order” – To create an order
experience_profile_id:

Id that will be obtained from the response of web profile request

payer:

Source of the funds for this payment represented by a PayPal account or a credit card.

  • payment_method : Payment method used. Must be either credit_card or paypal.
  • funding_instruments : A list of funding instruments for the current payment
  • payer_info : Information related to the payer
  • status : Status of the payer’s PayPal account. VERIFIED or UNVERIFIED
transactions:

Transactional details including the amount and item details.

redirect_urls:

Set of redirect URLs you provide only for PayPal-based payments.

  • return_url : The payer is redirected to this URL after approving the payment.
  • cancel_url : The payer is redirected to this URL after canceling the payment.

Execute an approved PayPal payment:

Use this call to complete a PayPal payment that has been approved by the payer.

The payment_id and payer_id are passed in the return_url. Once the payment is executed,it returns an array of payment object.

In the response state of the payment is obtained as : created approved failed, canceled, expired or pending.

The transaction details in the response contains state of the sale which is obtained as: pending, completed, refunded or partially_refunded

If the payment state is approved and the sale state is completed, the payment is successfully executed.

How to chat securely using Pidgin and OTR

These days surveillance news are coming out frequently. After Snowden’s revelation if you’r suffering from paranoia and want to secure your digital presence, follow this tutorial to communicate securely.

  1. Install Pidgin chat client.
    • for ubuntu –
    • for arch linux –
    • also you can download it manually from here & then install it as per instructions
  2. Install OTR plugin of Pidgin.
    • for ubuntu –
    • for arch linux –
  3. Now start Pidgin. It will show ‘Accounts’ Popup. Click on ‘Add’ button. add
  4. Now you will get ‘Add Account’ popup.
    add-account-0
  5. Now configure new account as follows.
    • Tab ‘Basic’
      • Login Options
        • Protocol : XMPP (don’t use ‘Facebook XMPP’)
        • Username: (don’t use a username which will somehow connected to real you)
        • jabber.rayservers.com
        • leave blank
        • enter password you want
      • User Options
        • Local alias : Leave blank
        • New mail notifications : DO NOT check
        • Use this buddy icon : check if you want
        • Create this new account on the server : MUST checkadd-account-1
    • Tab ‘Advanced’
      • Connection security : Require Encryption
      • Allow plaintext auth over unencrypted streams : DO NOT check
      • Connect port : 5222
      • Show Custom Smileys : check
      • Create this new account on the server : MUST check add-account-2
    • Tab ‘Proxy’
      • Proxy type : Use Global Proxy Settings
      • Create this new account on the server : MUST check add-account-3
    • Tab ‘Voice and Video’
      • Use silence suppression : leave default
      • Create this new account on the server : MUST check add-account-4
  6. Now to add this new account, click on ‘Add’
  7. Wait for few seconds. Popup will come for ‘SSL Certificate Verification’. Click on ‘Accept’. Cross-check ‘Certificate Information’. Then ‘Accept’ cetificate. cert-1 cert-2
  8. Now you will get popup saying ‘Register New XMPP Account’. Click on ‘Register’.
  9. From top menus select
    Tools -> Plugins -> Off-The-Record Messaging (MUST checked)
  10. Click on ‘Configure Plugin’
  11. Configure ‘Off-the-record Messaging’ popup as follows, otr-1
    • My Private Keys –
      • Click on ‘Generate’ if you get message ‘No key present’. Generating keys takes time. When keys are getting generated try to do some CPU intensive work to add more entropy.
    • Default OTR Settings – Check all
      • Enable private messaging – check
      • Automatically initiate private messaging – check
      • Require private messaging – check
      • Don’t log OTR conversations – check
    • OTR UI Options
      • Show OTR button in toolbar – check
  12. Once key is generated, click on ‘Close’ to close popup windows of ‘Off-the-record Messaging’ & ‘Configure plugin’.
    otr-2
  13. Now in top window ‘Buddy List’ ,
    Tools -> Preferences -> Logging -> Do NOT check any options here & close popup. otr-4
  14. Now enable your new account as,
    Accounts -> Enable account -> Select newly created account
  15. Now to add buddy,
    Buddies -> Add buddy
  16. In ‘Add Buddy’ popup, enter ‘Buddy’s username’ – something like ‘foo@bar.qux.com’.  Optionally you can add ‘Alias name’ for ease. Here important thing to remember is, once you add buddy it will appear in your buddy list only after authorization of your buddy.
  17. Now if you add buddy successfully and he is online & double-click on buddy and start to chat.
  18. Here you will get another private messaging window. Don’t forget to ‘Start OTR’ here & to Authenticate your buddy.
  19. Further, you can use Tor with Pidgin to circumvent IP address.

How to blink browser tab

Have you ever wished to blink browser tab on certain activity to get user’s attention?

Here is a scenario: Suppose there is a chat application installed on your website. Now what if the user of your website has multiple tabs opened in his browser and there is an incoming message in your website? How will you make your user realize that there is an unread message? Of course you can play some music, but it has some limitations. For instance, user may be listening to some songs or speakers could be muted. If none of them is applicable then also he has to go through each and every single tab in his browser and find out the one which is playing that music.

Visual blinking is the best way to get user’s attention and the easiest way to achieve that is to keep blinking the title of your website until user returns.

Here’s a JavaScript function to do so:

Just execute the function like below and browser tab will start blinking:

How to measure the duration of a function call or code block in python

The simple way to measure the duration of function call in python using context management.

Some times we may encounter the situation where we need to know total time taken by the function call. Here is the code which is pretty much handy and simple to measure the duration of function call  or  time taken by the function(call) in python

Here is how you apply the above code to get the time taken by the function call


 

Output would look like as follows,


 

Introduction to Riak

Riak is a distributed database designed to deliver maximum data availability by distributing data across multiple servers. Riak is an open-source, distributed key/value database for high availability, fault-tolerance, and near-linear scalability.

Riak Components

Riak is a Key/Value (KV) database, built from the ground up to safely distribute data across a cluster of physical servers, called nodes.

Riak functions similarly to a very large hash space. Depending on your background, you may call it hashtable, a map, a dictionary, or an object. But the idea is the same: you store a value with an immutable key, and retrieve it later.

1) Key and value

2) Buckets

Key and Value

Key/value is the most basic construct in all of computerdom.

Buckets

Buckets in Riak  provide logical namespaces so that identical keys in different buckets will not conflict.

Buckets are so useful in Riak that all keys must belong to a bucket. There is no global namespace. The true definition of a unique key in Riak is actually        bucket + key.

For convenience, we call a bucket/key + value pair as an object, sparing ourselves the verbosity of “X key in the Y bucket and its value”.

Replication and Partitions

Distributing data across several nodes is how Riak is able to remain highly available, tolerating out-ages and  partitioning. Riak combines two styles of distribution to achieve this: replication and partitions.

Replication

Replication is the act of duplicating data across multiple nodes. Riak replicates by default.

The obvious benefit  of replication is that if one node goes down, nodes that contain replicated data remain available to serve requests. In other words, the system remains available with no down time.

The downside with replication is that you are multiplying the amount of storage required for every duplicate. There is also some network overhead with this approach, since values must also be routed to all replicated nodes on write.

Partitions

A partition is how we divide a set of keys onto separate physical servers. Rather than duplicate values, we pick one server to exclusively host a range of keys, and the other servers to host remaining non-overlapping ranges.

With partitioning, our total capacity can increase without any big expensive hardware, just lots of cheap commodity servers. If we decided to partition our database into 1000 parts across 1000 nodes, we have (hypothetically) reduced the amount of work any particular server must do to 1/1000th.

There’s also another downside. Unlike replication, simple partitioning of data actually decreases uptime.

If one node goes down, that entire partition of data is unavailable. This is why Riak uses both replication and partitioning.

Replication+Partitions

Since partitions allow us to increase capacity, and replication improves availability, Riak combines them. We partition data across multiple nodes, as well as replicate that data into multiple nodes.

n_val

The Riak team suggests a minimum of 5 nodes for a Riak cluster, and replicating to 3 nodes (this setting is called n_val, for the number of nodes on which to replicate each object).

The Ring

Riak applies consistent hashing to map objects along the edge of a circle (the ring).

Riak partitions are not mapped alphabetically (as we used in the examples above), but instead a partition marks a range of key hashes (SHA-1 function applied to a key). The maximum hash value is 2160 , and divided into some number of partitions—64 partitions by default (the Riak config setting isring_creation_size).

The Ring is more than just a circular array of hash partitions. It’s also a system of metadata that gets copied to every node. Each node is aware of every other node in the cluster, which nodes own which vnodes, and other system data.

N/R/W Values

N

With our 5 node cluster, having an n_val=3 means values will eventually replicate to 3 nodes, as we’ve discussed above. This is the N value. You can set other values (R,W) to equal the n_val number with the shorthand all.

R

Reading involves similar tradeoffs. To ensure you have the most recent value, you can read from all 3 nodes containing objects (r=all). Even if only 1 of 3 nodes has the most recent value, we can compare all nodes against each other and choose the latest one, thus ensuring some consistency. Remember when I mentioned that RDBMS databases were write consistent? This is close to read consistency. Just like w=all,however, the read will fail unless 3 nodes are available to be read. Finally, if you only want to quickly read any value, r=1 has low latency, and is likely consistent if w=all.

W

But you may not wish to wait for all nodes to be written to before returning. You can choose to wait for all 3 to finish writing (w=3 or w=all), which means my values are more likely to be consistent. Or you could choose to wait for only 1 complete write (w=1), and allow the remaining 2 nodes to write asynchronously, which returns a response quicker but increases the odds of reading an inconsistent value in the short term. This is the W value

Since Riak is a KV database, the most basic commands are setting and getting values. We’ll use the HTTP interface, via curl, but we could just as easily use Erlang, Ruby, Java, or any other supported language. The basic structure of a Riak request is setting a value, reading it, and maybe eventually deleting it. The actions are related to HTTP methods (PUT, GET, POST, DELETE).


 

PUT

The simplest write command in Riak is putting a value. It requires a key, value, and a bucket. In curl, all HTTP methods are prefixed with -X. Putting the value pizza into the key favorite under the food bucket is done like this:

The -d flag denotes the next string will be the value. Declaring it as text with the proceeding line -H ‘Content-Type:text/plain’

This declines the HTTP MIME type of this value as plain text. We could have set any value at all, be it XML or JSON—even an image or a video. Riak does not care at all what data is uploaded, so long as the object size doesn’t get much larger than 4MB.

GET

The next command reads the value pizza under the bucket/key food/favorite.

This is the simplest form of read, responding with only the value. Riak contains much more information, which you can access if you read the entire response, including the HTTP header. In curl you can access a full response by way of the -i flag.

POST

Similar to PUT, POST will save a value. But with POST a key is optional. All it requires is a bucket name, and it will generate a key for you.

Let’s add a JSON value to represent a person under the people bucket. The response header is where a POST will return the key it generated for you.

You can extract this key from the Location value. Other than not being pretty, this key is treated the same as if you defined your own key via PUT.

Body

You may note that no body was returned with the response. For any kind of write, you can add the returnbody=true parameter to force a value to return, along with value-related headers like X-Riak-Vclock and ETag.

DELETE

The Final basic operation is deleting keys, which is similar to getting a value, but sending the DELETE method to the url/bucket/key.

A deleted object in Riak is internally marked as deleted, by writing a marker known as a tombstone. Unless configured otherwise, another process called a reaper will later finish deleting the marked objects.

read/write ratios.

Lists

Riak provides two kinds of lists. The first lists all buckets in your cluster, while the second lists all keys under a specific bucket. Both of these actions are called in the same way, and come in two varieties.

The following will give us all of our buckets as a JSON object.

And this will give us all of our keys under the food bucket.

Adjusting N/R/W to our needs

N is the number of total nodes that a value should be replicated to, defaulting to 3. But we can set this n_val to less than the total number of nodes.

Any bucket property, including n_val, can be set by sending a props value as a JSON object to the bucket URL. Let’s set the n_val to 5 nodes, meaning that objects written to cart will be replicated to 5 nodes.

Symbolic Values

A quorum is one more than half of all the total replicated nodes (floor(N/2) + 1). This figure is important, since if more than half of all nodes are written to, and more than half of all nodes are read from, then you will get the most recent value (under normal circumstances).

Hooks

Another utility of buckets are their ability to enforce behaviors on writes by way of hooks. You can attach functions to run either before, or after, a value is committed to a bucket.

Functions that run before a write is called precommit, and has the ability to cancel a write altogether if the incoming data is considered bad in some way. A simple precommit hook is to check if a value exists at all.

I put my custom Erlang code files under the riak installation ./custom/my_validators.erl.


 

Then compile the file.(You need to install erlang before installing Riak)

erlc my_validators.erl

Install the file by informing the Riak installation of your new code via app.config (restart Riak).

Then you need to do set the Erlang module (my_validators) and function (value_exists) as a JSON value to the bucket’s precommit array {“mod”:”my_validators”,”fun”:”value_exists”}.

If you try and post to the cart bucket without a value, you should expect a failure.

Siblings

Siblings occur when you have conflicting values, with no clear way for Riak to know which value is correct. Riak will try to resolve these conflicts itself if the allow_mult parameter is configured to false, but you can instead ask Riak to retain siblings to be resolved by the client if you set allow_mult to true.

Siblings arise in a couple cases.

We used the second scenario to manufacture a conflict in the previous chapter when we introduced the concept of vector clocks, and we’ll do so again here.

Resolving Conflicts

When we have conflicting writes, we want to resolve them. Since that problem is typically use-case specific, Riak defers it to us, and our application must decide how to proceed.

For our example, let’s merge the values into a single result set, taking the larger count if the item is the same. When done, write the new results back to Riak with the vclock of the multipart object, so Riak knows you’re resolving the conflict, and you’ll get back a new vector clock.

Successive reads will receive a single (merged) result.

Will share more on this arctile soon.