How to shuffle lines in the file in linux

We can shuffle lines in the file in linux using following commands

  • shuf
  • sed and sort
  • awk
  • python

As an example we will take a file shuffle_mylines.txt  having numbers till 10 each digit in a new line.

Create a file using following command

$ seq 10  > shuffle_mylines.txt

Command shuf

This command is light wight and straight forward. You just need to call this command with file name as an argument.

Shuffle lines using sed

You may have already know about command sed(Stream Editor). It is one of the command widely used for text processing in unix/linux. We can’t shuffle line using single sed command, but we will do by combining other commands. Let’s take a look at following command,

How does it work?

Breakdown of above command,

Commands we have used in the above example are,

  • cat
  • while loop
  • $RANDOM   environment variable
  • soft
  • tail
  • sed

 

Now, lets come to see how this command work. First command cat will read the file content and will pipe it to shell while loop

Where, while loop will read the piped input into variable x and will iterate over all lines to generate  output  <random_number>:<line> as you can see $RANDOM:$x. Where $RANDOM is the environment variable, each time you query this variable you will get random number. Which is useful for to shuffle lines.

Then, we will sort output of above while loop using sort command

Out put of this command will always be randomly shuffled lines. It’s because $RANDOM.

Output here would look like,

To remove preceded random values we will use sed.

That’s it. On every execution of this command you will get shuffled lines. You can redirect output to new file if you want to store using (>) or (>>).

 

Shuffle lines using awk

The awk is the programming language which is specially designed for text processing. We will use it to shuffle lines.

 

Another example using awk. It’s is similar to sed and sort example.

Shuffle lines in file using python

Python is popular scripting language widely used today from big projects to small scripts. We will see, how you can shuffle lines using python.

Python Example 1

In this example, we are passing file name as command line argument. Reading it and shuffling the lines of file and printing them on terminal.

The output can be redirected to  a file using redirect operator (> or >>)

Conclusion:

If you are looking for a quick shuffle command shuf is best choice or you can have a fun of using other ways to shuffle lines in the file.

 

 

How to write port-forwarding program using Twisted

Recently I was faced with an issue where a long running process is listening on loop back IP (127.0.0.1) on port 8080 on one of our servers and client programs on other machines are trying to access it on server’s local IP 10.91.20.66.  We ended up at this situation when we have updated server configuration and restarted the server program and forgot to change IP binding info in config file from loop back to local IP. Server got busy with it’s work, with lots of customer’s connections already, by the time we have discovered that some services of  server are not accessible to client programs on other machines. So, the dummy’s guide to fixing it by changing config and restarting the server program is not an option as we can’t risk to disconnect existing customers. So, hot patching is the only option until we can restart the program at next scheduled down time.

I could have fixed this in couple of ways either by adding few lines to iptables configuration or by writing simple socket program in python. The task is to forward data coming in on local IP port 8080 to loop back IP (127.0.0.1) port 8080 and send replies back to source address. Forwarding one socket data to other socket is pretty trivial using Python’s socket library and Twisted made it even more trivial, so I went with the following solution using Twisted.

That’s it. Now, all I needed to do is to run this program by the following command

This simple program is made possible by the heavy lifting done by twisted library. Interested folks can look under hood at twisted’s portforward.py module.

Sending emails asynchronously using Twisted – Part 2

In Part 1 of article, we saw how to send blocking emails using ‘smtplib’ module & non-blocking emails using Twisted framework. In this part, we will see how to send asynchronous emails to multiple recipients using Twisted

  • Sending multiple emails

    Refer following script.This script sends emails to given recipients asynchronously. Here we have used twisted.internet.defer.DeferredList API. This API is very useful in some scenarios. Suppose you have to finish multiple task asynchronously and then you have to finish one final task. For examples, your program is connected to 4 different clients & and before shutting it down, you have to make sure that all connections are closed properly. In such cases, DeferredList API is used. Create deferrands of each task & make their list. Pass this list to ‘DeferredList‘ API which will return you another deferrand. This final deferrand will be fired when all deferrands in list will be fired.


     
  • Sending multiple emails using coiterator

    Though above script runs fine, there is one problem. Here, recipients number is very small. But suppose you have to send emails to millions recipients then will this code work ?. Refer function ‘send_multiple_emails’.


    Here we have used ‘for’ loop which is blocking. So until this ‘for’ loop is iterated, program will not move to next line of code. For 3 recipients iteration will not take much time however for millions of recipients, it will not work.
    So lets modify our code to work like generators.

    Here, we have used twisted.internet.task.coiterate API. This API iterates over iterator by dividing reactor runtime between all iterators. Thus we can send millions of emails asynchronously.

Sending emails asynchronously using Twisted – Part 1

  • Using ‘smtplib‘ module

It is very easy to send emails using ‘smtplib‘ module of python. Check following recipe.

But ‘smtplib’ module sends emails synchronously. So code execution is blocked until email is sent. To overcome this, lets try to send email asynchornously.

  • Using Twisted

For this tutorial we are going to use Twisted framework. Twisted is event-driven networking engine. It uses reactor-pattern. Twisted uses deferred objects to address waiting IOs. Deferred is more like subset of promises. Check following recipe to send asynchronously MIME message using Twisted.

 

 

Building hello world using Python Tkinter

Continuing our series of desktop GUIs, in this post we are going to go over the topic of building a small hello world program using Python Tkinter.

What is Tkinter ?

Tkinter is python wrapper/binding to Tk library. Tk was developed as GUI library for Tcl language by John Ousterhout. For many other high level language authors coming to programming scene in 1990s this seemed like easy tool to capitalize on in bringing GUI library to their language. That’s why you can find many different language bindings to Tk. Tk brought easy GUI building to programming masses.

Often times many people write off Tk or Tkinter as ugly old GUI library. Their claims are not completely unfounded. But, Tk in recent times made strides to bring modern look by implementing Tile, themeing engine. Tile is also called as ttk. With ttk in place look and feel issues of Tk are addressed.

ttk xp blue theme demo
Tile xp blue theme demo

But, Tkinter still lags behind in terms available collection of default widgets compared to other libraries like PySide or wxPython . However, Tkinter is still a good candidate if you quickly want to dish out a GUI without having programming styles imposed on you. (You will see these style restrictions in our next posts about wxPython and PySide)

Hello world

Now, we will go over building a simple hello world program using Tkinter. In this program we will create a simple window with “Hello world” text in it. And a button that say “Kill Me”. Clicking that button will cause the python program to exit.

tkinter hello world

That import * line saves your some typing.

This line instantiates Tk and creates main window of the program. Next we create a Label (a widget which displays text ) and call it’s pack method. pack is one three available layout methods provided by Tkinter (rest of the two being grid and place). The widget will not show up until you call pack.  Next, we create a Button widget and pack it, similar to that of Label. The interesting point to note while creating button widget is, we provide command argument to it with a reference of sys.exit function. A newbie mistake that is often made is, passing function with () next to it. That will cause the target function to be invoked immediately. You are only supposed to pass the reference of function and Tkinter will call it when button is pressed. So, when this “Kill Me” button is pressed sys.exit will be called and program exits. One, last important line to notice is root.mainloop() . This line sets the event monitoring loop in motion and Tkinter take control of the program from here.

 

Common constructs of GUI programs

In this post we will go over some of the basic and common constructs that you come across while building GUI programs almost in any language or library. The following are some of the common items that I can think of

  1. Loop
  2. Widgets
  3. Events
  4. Layout

Loop

Loop or Mainloop as it is called in some GUI libraries is one of the most common element of all GUI programs. In a GUI program this is usually called at the at end of constructing the all the items (widgets) in program. When you call the mainloop, it takes over the control and keeps running until you terminate the GUI.

The main purpose of loop is to poll for user input (mouse and keyboard actions) and fire events which your program can handle in asynchronous way. Following diagram explains loop in simple terms.

GUI main loop
GUI main loop

As you can see from the above diagram the GUI loop keeps polling for user input. If there is any event via mouse or keyboard or other input devices (joysticks, game pads ) when input is available loop gives control to callback functions that are bound to input events.  Loop keeps running until the main window (also called parent window) exits. Once the main window is destroyed the loop exits and gives control back to your code. In most cases end of the mainloop would be end of many GUI programs.

Widgets

GUIs are collection of widgets. Example of widgets are button, scroll bar, check box etc. Widget is an element of user interaction. Their main purpose is to display or collect information from user. A label displays text to user. A text input widget collects text from user. One way to decide the richness of GUI tool kits is to look at how many widgets they offer. The more widgets a library offers you the better it is for you. So, that you don’t have to write extra code to implement custom widgets.  Widgets usually are sub classes of a Widget parent class. So, most of the widgets inherit properties of widget class in respective GUI libraries. In some GUI libraries main window also has properties of a widget. gui window

Events

When ever user presses a button or enters text using keyboard those actions generate events in GUI libraries. GUI libraries provide a way to hook into these events so that your program can respond to user actions. The event hooking/binding style differs from library to library. But, the aim is same across all libs, that is to provide programmer a way to respond to user input.

Layout

Layout facilities provided by GUI libs helps to make programmer’s life easier when placing widgets in main window. Layouts are simple rules you use to convey to GUI library on how you would like your widgets to be laid out on screen. Instead of manually placing widgets on screen by specifying distance units and placement pixel positions, once you construct your widgets you hand them over to certain layout scheme to let GUI library handle the placement. The benefits are, layouts will take care of expansions, size proportions and other nitty-gritty detail of placing your widgets in orderly way.

The layout features provided by libraries vary greatly. You often have to pick the one that suits your needs. These layouts can be mixed in order to achieve the placement and look you want.

 

Desktop GUI libraries in Python

If you want to build desktop GUIs, Python is right candidate to pick. It offers you plethora of GUI libraries compared to any other language. So, there are high number of chances that, some library in there fits your bill which is readily available, in whole of lot of GUI libs python has to offer. Following is a list of some of the popular GUI libs

  1. Tkinter 
  2. wxPython
  3. PyQt / PySide
  4. PyGTK

(Complete list of available libraries can be found on python wiki page.)

All of the above mentioned ones are cross-platform capable libraries. So, that you don’t have to rewrite any part of your GUI and gladly run it across all the main contenders Windows, Linux and Mac . There are other platform specific ones like win32gui , WinForms using IronPython. But, it helps to stick with cross-platform capable libs to avoid extra work when building your app for other platforms.

Over the next few posts I’ll present you with examples of simple hello world programs using above mentioned list of libraries.

Usually the best way to build your GUI application is to write compute intensive part (if there is any ) of your business logic in low level languages like C/C++ and then write GUI code using the above mentioned Python libs. This saves you lot of time and energy. As, compared to low level languages, in Python you get the benefit of automatic memory management and you only have to write less number of lines of code.

Desktop GUIs are on decline with the advent of web. More and more programs are being coded as services using web technologies ( HTML, CSS, JavaScript ). It has gotten so far, now UI/GUI by default means web UI.  Each passing day web technologies are filling up  the gap in terms of number of features (Dynamic content, Media capabilities ) . But, desktop GUIs still have their place in certain areas and there are numerous programs already written using desktop GUI technologies that needs to be managed.

How to implement PayPal payment gateway

The PayPal REST APIs are supported in two environments. Use the Sandbox environment for testing purposes, then move to the live environment for production processing.

The following endpoints address are two environments:

A complete REST operation is formed by combining an HTTP method with the full URI to the resource you’re addressing. For example, here is the operation to create a new payment:

The PayPal REST sdk can be obtained through pip

OAuth Request / Response

Client Id and Secret Id can be obtained from the application created in the paypal account.

For the following each API call, you’ll need to set request headers, including the access token.

Creating a WebProfile:

name:

The name of the web experience profile which should be unique among the profiles for a given merchant.

presentation:

It contains the parameters for style and presentation.

input_fields:

Parameters for input fields customization:

  1. allow_note : It enables the buyer to enter a note to the merchant on the paypal page during checkout.

  2. no_shipping : Determines whether or not PayPal displays shipping address fields on the experience pages.

    • 0 – PayPal displays the shipping address on the PayPal pages.
    • 1 – PayPal does not display shipping address fields whatsoever.
    • 2 – If you do not pass the shipping address, PayPal obtains it from the buyer’s account profile.
  3. address_override : Determines if the PayPal pages should display the shipping address supplied in this call, rather than the shipping address on file with PayPal for this buyer.

    • 0 – PayPal pages should display the address on file
    • 1 – PayPal pages should display the addresses supplied in this call instead of the address from the buyer’s PayPal account.
flow_config:

Parameters for flow configuration

  1. landing_page_type : Type of PayPal page to be displayed when a user lands on the PayPal site for checkout.

    • Billing – The Non-PayPal account landing page is used
    • Login – The paypal account login page is used.

Creating a Payment:


 

intent:

Payment intent. Allowed values are:

  • “sale” – For immediate payment
  • “authorize” – To authorize a payment for capture later
  • “order” – To create an order
experience_profile_id:

Id that will be obtained from the response of web profile request

payer:

Source of the funds for this payment represented by a PayPal account or a credit card.

  • payment_method : Payment method used. Must be either credit_card or paypal.
  • funding_instruments : A list of funding instruments for the current payment
  • payer_info : Information related to the payer
  • status : Status of the payer’s PayPal account. VERIFIED or UNVERIFIED
transactions:

Transactional details including the amount and item details.

redirect_urls:

Set of redirect URLs you provide only for PayPal-based payments.

  • return_url : The payer is redirected to this URL after approving the payment.
  • cancel_url : The payer is redirected to this URL after canceling the payment.

Execute an approved PayPal payment:

Use this call to complete a PayPal payment that has been approved by the payer.

The payment_id and payer_id are passed in the return_url. Once the payment is executed,it returns an array of payment object.

In the response state of the payment is obtained as : created approved failed, canceled, expired or pending.

The transaction details in the response contains state of the sale which is obtained as: pending, completed, refunded or partially_refunded

If the payment state is approved and the sale state is completed, the payment is successfully executed.

How to measure the duration of a function call or code block in python

The simple way to measure the duration of function call in python using context management.

Some times we may encounter the situation where we need to know total time taken by the function call. Here is the code which is pretty much handy and simple to measure the duration of function call  or  time taken by the function(call) in python

Here is how you apply the above code to get the time taken by the function call


 

Output would look like as follows,


 

How to implement social Login for Django app

In this article we will get to know about how to login to your django app by using social logins like Facebook and Google.

Start a simple Django project


 

Running ./manage.py syncdb and then ./manage.py runserver and navigating to localhost:8000 will show the familiar “It worked!” Django page. Let’s put some custom application code in place, so that we can tell whether the current user is authenticated or anonymous.

Show current user’s authentication status

Now, the very small customizations we’ll add are:

Relevant portion of settings.py:


 

Template: thirdauth/base.html:

Template: thirdauth/home.html:

File views.py:


 

File urls.py:


 

Install Python Social Auth

Second, let’s make some modifications to our settings.py to include python-social-auth in our project:


 

Lets update the urls module to include the new group of URLs:


 

And finally, let’s update the database models:

./manage.py syncdb

Add links for logging in and logging out.

Since we’ll be logging in and out multiple times, let’s include django.contrib.auth URLs into our URLs configuration:


 

Let’s modify our Home page template like this:

For the login and logout links in this template to work correctly, we need to modify a few things. First, let’s take care of logout, it’s easier. Just add ‘request’ to the context object that we pass into template-rendering code.

Updated views.py:


 

For login to work, let’s first add a LOGIN_REDIRECT_URL parameter to settings (to prevent the default /account/profile from raising a 404):

LOGIN_REDIRECT_URL = ‘/’

Get Client IDs for the social sites.

For All the social networks we are using in this demo, the process of obtaining an OAuth2 client ID (also known as application ID) is pretty similar. All of them will require that your application has a “real” URL – that is, not http://127.0.0.1 or http://localhost. You can add an entry in your /etc/hosts file that maps 127.0.0.1 to something like “test1.com”, and the URL of your application becomes http://test1.com:8000 – that is good enough for testing. You can change it in the social app settings when it goes into production.

Facebook:

Google: