How to shuffle lines in the file in linux

We can shuffle lines in the file in linux using following commands

  • shuf
  • sed and sort
  • awk
  • python

As an example we will take a file shuffle_mylines.txt  having numbers till 10 each digit in a new line.

Create a file using following command

$ seq 10  > shuffle_mylines.txt

Command shuf

This command is light wight and straight forward. You just need to call this command with file name as an argument.

Shuffle lines using sed

You may have already know about command sed(Stream Editor). It is one of the command widely used for text processing in unix/linux. We can’t shuffle line using single sed command, but we will do by combining other commands. Let’s take a look at following command,

How does it work?

Breakdown of above command,

Commands we have used in the above example are,

  • cat
  • while loop
  • $RANDOM   environment variable
  • soft
  • tail
  • sed

 

Now, lets come to see how this command work. First command cat will read the file content and will pipe it to shell while loop

Where, while loop will read the piped input into variable x and will iterate over all lines to generate  output  <random_number>:<line> as you can see $RANDOM:$x. Where $RANDOM is the environment variable, each time you query this variable you will get random number. Which is useful for to shuffle lines.

Then, we will sort output of above while loop using sort command

Out put of this command will always be randomly shuffled lines. It’s because $RANDOM.

Output here would look like,

To remove preceded random values we will use sed.

That’s it. On every execution of this command you will get shuffled lines. You can redirect output to new file if you want to store using (>) or (>>).

 

Shuffle lines using awk

The awk is the programming language which is specially designed for text processing. We will use it to shuffle lines.

 

Another example using awk. It’s is similar to sed and sort example.

Shuffle lines in file using python

Python is popular scripting language widely used today from big projects to small scripts. We will see, how you can shuffle lines using python.

Python Example 1

In this example, we are passing file name as command line argument. Reading it and shuffling the lines of file and printing them on terminal.

The output can be redirected to  a file using redirect operator (> or >>)

Conclusion:

If you are looking for a quick shuffle command shuf is best choice or you can have a fun of using other ways to shuffle lines in the file.

 

 

How to see stashed changes using git stash

The command git stash is used to stash the changes in a dirty working directory away.

You can list all stashed change using the command  git stash list,

Every time you stash your working directory, git will save the state of working directory into somethine which mantins history of stash tree. Every time you stash your changes it will be save as a kind of commit in stash tree. All stashed changes are stacked in the order with serial or reference. stash@{0} is the first or top most or recent stash. Every time if you want to refer a particular stash you have to use the it’s reference id something like stash@{3}, stash@{5}… etc

Think of each stash as a separate  commit. These commits are stored and stacked differently and not overlapped with conventional git commit history 

A stash is represented as a commit whose tree records the state of the working directory, and its first parent is the commit at HEAD when the stash was created.

The following command can be used to extract diff of stashed change againest any other stash or commit or branch or HEAD.

  • git stash show
  • git show
  • git diff
  • git difftool

Let’s see, how we can use each of the above mentioned commands.

Command git stash show

The simple command git stash show  gives very brief summary of changes of file, but will not show the diff of changes against current HEAD. Something like,

Something like below

Some times this is not so useful. You may want to see the difference against  current HEAD or any specific commit or current directory.

If you use git stash show along with option -p, It will show all changes.

 

Check diff against selected stash.

Command git show

The command git-show  is used to see  various types of objects.

The command git-show is not only used to visualize  stash changes, but also used to see one or more objects like blobs, trees, tags and commits. For more information check git-show

Synopsis: 


To see top most stash difference against  HEAD:


To get diff of of selected stash against HEAD:


See selected complete whole file as if stash is applied  from selected stash:

Where,

stash@{0} is the reference of stash. I could be any one of stash@{0}, stash@{1}, stash@{2}… etc.

<file_name>  is the name of the file relative to project/git repository

Command git diff 

The command git-diff  is also one of common command which is used to show changes between commits, commit and working tree, etc.

By default, git diff will show the diff of selected stash against(modified files) current state of repository unless other stash  reference or commit is specified.


To get difference between top most stash stash@{0} and master branch:


Only display the names of file not diff of changes:


See the diff between selected stashes for a selected file:

 

Command git difftool

The command git-difftool can also be used to find diff between selected stash and selected commit or branch or stash

See the difference between latest two stashes:

 

 

Summary: 

Commands which are  useful to extract the diff from selected stash git stash show, git show,  git diff, git difftool . 

See difference using command git stash show,

See the changes in the stash using command git show,

See the difference between latest stash and selected commit using command git diff,

 

References:

[1] https://git-scm.com/docs/git-show
[2] https://git-scm.com/docs/git-stash

 

How to delete files older than specified number of days using find command

The Linux command utility find will allow us to perform arbitrary commands on files which are filtered by the command.  Using this opportunity we can delete the files which are older than specified days by passing either command or action to find command.

Find command syntax would look like,

How to delete files older than specified days

To delete files which are older than specified number of days. We have to filter those files using criteria with action which will delete those files.

We can filter the files which are older than specified number of days using test -mtime.  Filtered files can b deleted using either of the following actions  -exec or -delete.

The command we need would look like,

find <path-to-files> -mtime +n -exec rm {} \;

or

find <path-to-files> -mtime +n -delete

Where,
n specifies number of days old that file should be to get not filtered(to get included in output)

Let’s consider an example to  delete files which are older than 7 days.

Example: Delete files older than 7 days

or, as mentioned we can also use -delete action.

As you can in the above command. We also mention other filters(tests) like -name along with -mtime to control what files should be deleted.

Break Down Of Command

First argument:  this could be either absolute path or relative path or wildcard specifies that where to search for files

Second argument: this is the criteria to filter files based on name, path, pattern and how many number of days older etc. More tests can be added to reduce final outcome of files.  Here we used -mtime +7 to filter all files which are older than 7 days.

Third argument: this argument will be the action. Which specifies what action should be preformed on found files. By default it is -print, means it will just print the result. As per our requirement we are using action -delete or -exec to delete files

Action -exec this is generic action, which can be used to perform any shell command on each file which is being located. Here use are using rm {} \;  Where {} represents the current file, it will expand to the name/path of found file.
Note: There is space between {} and \. If you omit this space it will throw and error.

🙂

How to build and install FreeSWITCH 1.6 on Debian 8 Jessie

FreeSWITCH is an opensource telephony soft switch created in 2006. As per official wiki  page,

It is a scalable open source cross-platform telephony platform designed to route and interconnect popular communication protocols using audio, video, text or any other form of media.

Sounds good. right ?

We are using Debian for this tutorial as it is very stable & mature linux distribution and FreeSWITCH core developers’ choice-of-distribution .
You can read more about FreeSWITCH on there wiki page.

Now lets cut a crap & start an action, assuming you  already have working Debian 8 OS.

Build & Install FreeSWITCH

There are different ways to install FreeSWITCH. In this tutorial, we will see how to install it from source.

  1. First update your Debian box & install curl & git.

  2. Add FreeSWITCH GPG key to APT sources keyring.

  3. Add FreeSWITCH repository to APT sources.

  4. Once again update your system.

  5. Now lets first install FreeSWITCH dependencies.

  6. Though above step takes care of most of dependencies, few still remains required to compile mod_fsv. So install them as,

  7. Grab source code of FreeSWITCH as follows,

  8. Now lets compile FreeSWITCH source for version 1.6

  9. Now lets compile sounds

  10. Lets create simlinks to required binaries to access them from anywhere.

     

Set Owner & Permissions

Starting FreeSWITCH service on boot automatically

To start FreeSWITCH after each boot automatically we need to set up init script. Init script is script used by init system to manipulate services. Debian 8 is now migrated to systemd init system, we will add systemd unit file.

Copy following content to ‘/lib/systemd/system/freeswitch.service’

Now execute following commands in your shell

Start FreeSWITCH

Now we are all set. Lets start hacking FreeSWITCH.

 

Notes

  1. If something goes wrong & you try compilation again by ‘make clean’, sometimes you get errors regarding ‘spandsp’. To resolve them try to clean using
    ‘git clean -fdx’. For more info check this ticket – https://freeswitch.org/jira/browse/FS-6405

 

How to write port-forwarding program using Twisted

Recently I was faced with an issue where a long running process is listening on loop back IP (127.0.0.1) on port 8080 on one of our servers and client programs on other machines are trying to access it on server’s local IP 10.91.20.66.  We ended up at this situation when we have updated server configuration and restarted the server program and forgot to change IP binding info in config file from loop back to local IP. Server got busy with it’s work, with lots of customer’s connections already, by the time we have discovered that some services of  server are not accessible to client programs on other machines. So, the dummy’s guide to fixing it by changing config and restarting the server program is not an option as we can’t risk to disconnect existing customers. So, hot patching is the only option until we can restart the program at next scheduled down time.

I could have fixed this in couple of ways either by adding few lines to iptables configuration or by writing simple socket program in python. The task is to forward data coming in on local IP port 8080 to loop back IP (127.0.0.1) port 8080 and send replies back to source address. Forwarding one socket data to other socket is pretty trivial using Python’s socket library and Twisted made it even more trivial, so I went with the following solution using Twisted.

That’s it. Now, all I needed to do is to run this program by the following command

This simple program is made possible by the heavy lifting done by twisted library. Interested folks can look under hood at twisted’s portforward.py module.

How to Fix G729a CODEC NEGOTIATION ERROR in FreeSWITCH

Are you facing issue of failing calls that are having G729a codec with 488 response? This article tells you how to fix that issue. When using G729 codec in FreeSWITCH if it receives following SDP in INVITE packet, that call is going to fail with 488 Incompatible Destination printing the error message mod_sofia.c:2226 CODEC NEGOTIATION ERROR.  SDP:

FreeSWITCH is not the offending party here, but the device that is sending G729 as G729a. The correct encoding name assigned to this codec by IANA is G729 not G729a.

To fix interop quirks like these FreeSWITCH has implemented some options that can be enabled. They are prefixed with NDLB means No Device Left Behind. The particular option that helps us with the current issue is

If you set this option to true in sofia profile as shown below, FreeSWITCH will be more forgiving to devices that are using non standard IANA codec names in SDP.

Note:- The situation described in this article has commercial G729 module loaded in FreeSWITCH

Installing SoftEther VPN server on Ubuntu

This tutorial is to show you how to install a SoftEther VPN on Ubuntu 12.04 & Ubuntu 14.04.
After installing a fresh copy of Ubuntu or your machine, run following commands to update the packages.

 

After updating add SoftEther PPA:

 

Update package list again & install softether package.

After installing run following commands to check for any errors.

Select option 1.  And press Enter for the rest. Then run command check.

 

2015-12-25 19_51_20-Clipboard

 

If you get output same as above image, then you have successfully installed SoftEther VPN server on your machine.

 

Introduction to Erlang part 1

Erlang is a functional programming language. If you have ever worked with imperative languages, statements such as i++ may be normal to you; in functional programming they are not allowed. In fact, changing the value of any variable is strictly forbidden.

We will start with installing Erlang.

Installation:

In Erlang, you can test most of your stuff in an emulator; it will run your scripts
and  it will also let you edit stuff live. To start,  open a terminal and then type $ erl. If you’ve set up everything fine, you should see text like this:

Shell

Shell commands:

If you type in i then c, Erlang should stop the currently running code and bring you back to a responsive shell. J will give you a list of processes running (a star after a number indicates this is the job you are currently running), which you can then interrupt with i followed by the number. If you use k , you will kill the shell as it is instead of just interrupting it. Press s to start a new one.

Now we will go through some basics

In the Erlang shell, expressions have to be terminated with a period followed by whitespace(line break, a space etc.), otherwise they won’t be executed. You can separate expressions with commas, but only the result of the last one will be shown (the others are still executed). This is certainly unusual syntax for most people and it comes from the days Erlang was implemented directly in Prolog, a logic programming language.

Here is example,

 

Erlang doesn’t care if you enter floating point numbers or integers: both types are supported when dealing with arithmetic.Integers and floating values are pretty much the only types of data Erlang’s mathematical operators will handle transparently for you.

Note that we can use several operators in a single expression, and mathematical operations obey the normal precedence rules.

If you want to express integers in other bases than base 10, just enter the number as Base#Value (given Base is in the range 2..36):


 

Invariable Variables

Doing arithmetic is alright, but you won’t go far without being able to store results somewhere. For that, we’ll use variables. If you have read the intro to this book, you’ll know that variables can’t be variable in functional programming. The basic behavior of variables can be demonstrated with these 7 expressions (note that variables begin with an uppercase letter):

The first thing these commands tell us is that you can assign a value to a variable exactly once; then you can ‘pretend’ to assign a value to a variable if it’s the same value it already has. If it’s different, Erlang will complain. It’s a correct observation, but the explanation is a bit more complex and depends on the = operator. The = operator (not the variables) has the role of comparing values and complaining if they’re different. If they’re the same, it returns the value:

IMPORTANT POINTS TO NOTE:

Data Types:

1) Terms: A piece of data of any data type is called a term.

2) Number: There are two types of numeric literals, integers and floats. Besides the conventional notation, there are two Erlang-specific notations:

$char ASCII value or unicode code-point of the character char.

base#value Integer with the base base, that must be an integer in the range 2..36. In Erlang 5.2/OTP R9B and earlier versions, the allowed range is 2..16.

Examples:

3) Atom

Examples:

4) Bit Strings and Binaries

A bit string is used to store an area of untyped memory. Bit strings are expressed using the bit syntax. Bit strings that consist of a number of bits that are evenly divisible by eight, are called binaries.

Examples:

5) Reference

A reference is a term that is unique in an Erlang runtime system, created by calling make_ref/0.

6) Fun

A fun is a functional object. Funs make it possible to create an anonymous function and pass the function itself — not its name — as argument to other functions.

Example:

7) Port Identifier

A port identifier identifies an Erlang port. open_port/2, which is used to create ports, returns a value of this data type.

8) Pid

A process identifier, pid, identifies a process. The following BIFs, which are used to create processes, return values of this data type:

spawn/1,2,3,4 spawn_link/1,2,3,4 spawn_opt/4

Example:

9) Tuple

A tuple is a compound data type with a fixed number of terms: Each term Term in the tuple is called an element. The number of elements is said to be the size of the tuple. There exists a number of BIFs to manipulate tuples.

Examples:

10) Map

A map is a compound data type with a variable number of key-value associations: ex: {Key1=>Value1,…,KeyN=>ValueN}

Each key-value association in the map is called an association pair. The key and value parts of the pair are called elements. The number of association pairs is said to be the size of the map.

Examples:

11) List

A list is a compound data type with a variable number of terms. [Term1,…,TermN] Each term Term in the list is called an element. The number of elements is said to be the length of the list.

Formally, a list is either the empty list [] or consists of a head (first element) and a tail (remainder of the list). The tail is also a list. The latter can be expressed as [H|T]. The notation [Term1,…,TermN] above is equivalent with the list [Term1|[…|[TermN|[]]]].

A list where the tail is a list is sometimes called a proper list. It is allowed to have a list where the tail is not a list, for example, [a|b]. However, this type of list is of little practical use.

Examples:

13) Record A record is a data structure for storing a fixed number of elements. It has named fields and is similar to a struct in C. However, a record is not a true data type. Instead, record expressions are translated to tuple expressions during compilation. Therefore, record expressions are not understood by the shell unless special actions are taken.

14) Boolean

There is no Boolean data type in Erlang. Instead the atoms true and false are used to denote Boolean values.

Examples:

15) Type converstions


 

 

How to integrate Celery into Django project

What is Celery?

Celery is a distributed task queue that allows us to execute jobs in background. This article explains how to set up Celery with Django to perform a background task.

Advantages:

  • Large or small, Celery makes scheduling such periodic tasks easy.
  • You never want end users to have to wait unnecessarily for pages to load or actions to complete. If a long process is part of your application’s workflow, you can use Celery to execute that process in the background, as resources become available, so that your application can continue to respond to client requests.

Celery uses brokers to pass messages between a Django Project and the Celery workers. We will use Redis as the message broker.

Installation

Before diving into Celery, follow the below setup points

Create a new virtualenv ‘venv’ using following command:

To activate the environment use command:

Install django and create a django project ‘myproject’. Make sure to activate a virtualenv, create a requirements.txt file and run the migrations. Then fire up the server and navigate to http://localhost:8000/ in your browser. You should see the familiar “Congratulations on your first Django-powered page” text. When done, kill the sever.

Let’s install Celery:

Now we will integrate Celery into our Django project with the following steps:

Step 1:

Inside the myproject directory i.e beside your settings.py create a new file called celery.py and add the following code in that:

Let’s break down what happens in the first module, first we import absolute imports from the future, so that our celery.py module will not clash with the library:

Then we set the default DJANGO_SETTINGS_MODULE for the celery command-line program:

Specifying the settings here means the celery command line program will know where your Django project is. This statement must always appear before the app instance is created, which is what we do next:

This is your instance of the library, you can have many instances but there’s probably no reason for that when using Django.

We also add the Django settings module as a configuration source for Celery. This means that you don’t have to use multiple configuration files, and instead configure Celery directly from the Django settings.

You can pass the object directly here, but using a string is better since then the worker doesn’t have to serialize the object when using Windows or execv:

Next, a common practice for reusable apps is to define all tasks in a separate tasks.py module, and Celery does have a way to autodiscover these modules:

Step 2:

To ensure that the Celery app is loaded when Django starts, add the following code into the __init__.py file that sits next to your settings.py file:

Project layout should look like:

Step 3:

Celery uses “brokers” to pass messages between a Django Project and the Celery workers. In this article, we will use Redis as the message broker.

First, install Redis from the official download page and then turn to your terminal, in a new terminal window, fire up the server:

You can test that Redis is working properly by typing this into your terminal:

Redis should reply with PONG – try it!

Once Redis is up, add the following code to your settings.py file:

You also need to add Redis as a dependency in the Django Project:

Test that the Celery worker is ready to receive tasks:

Kill the process with CTRL-C. Now, test that the Celery task scheduler is ready for action:

That’s it! You can now use Celery with Django. For more information on setting up Celery with Django, please check out the official Celery documentation.

Sending emails asynchronously using Twisted – Part 2

In Part 1 of article, we saw how to send blocking emails using ‘smtplib’ module & non-blocking emails using Twisted framework. In this part, we will see how to send asynchronous emails to multiple recipients using Twisted

  • Sending multiple emails

    Refer following script.This script sends emails to given recipients asynchronously. Here we have used twisted.internet.defer.DeferredList API. This API is very useful in some scenarios. Suppose you have to finish multiple task asynchronously and then you have to finish one final task. For examples, your program is connected to 4 different clients & and before shutting it down, you have to make sure that all connections are closed properly. In such cases, DeferredList API is used. Create deferrands of each task & make their list. Pass this list to ‘DeferredList‘ API which will return you another deferrand. This final deferrand will be fired when all deferrands in list will be fired.


     
  • Sending multiple emails using coiterator

    Though above script runs fine, there is one problem. Here, recipients number is very small. But suppose you have to send emails to millions recipients then will this code work ?. Refer function ‘send_multiple_emails’.


    Here we have used ‘for’ loop which is blocking. So until this ‘for’ loop is iterated, program will not move to next line of code. For 3 recipients iteration will not take much time however for millions of recipients, it will not work.
    So lets modify our code to work like generators.

    Here, we have used twisted.internet.task.coiterate API. This API iterates over iterator by dividing reactor runtime between all iterators. Thus we can send millions of emails asynchronously.