Data Analysis with Pandas & Python

What is Data Analysis?
Data analysis is a process of inspecting, cleansing, transforming, and modeling data with the goal of discovering useful information, informing conclusions, and supporting decision-making. In today’s business world, data analysis plays a role in making decisions more scientific and helping businesses operate more effectively
Python is a great language for doing data analysis, primarily because of the fantastic ecosystem of data-centric Python packages. Pandas is one of those packages providing fast, flexible, and expressive data structures designed to make working with “relational” or “labeled” data both easy and intuitive. It aims to be the fundamental high-level building block for doing practical, real-world data analysis in Python.
In this article, I have used Pandas to know more about doing data analysis.
Mainly pandas have two data structures, series, data frames, and Panel.

Installation
The easiest way to install pandas is to use pip:

or, Download it from here.

  • pandas Series

pandas series can be used for the one-dimensional labeled array.

Labels can be accessed using index attribute
print(a.index)

You can use array indexing or labels to access data in the series.
You can use array indexing or labels to access data in the series
print(a[1])
print(a[‘test4’])

You can also apply mathematical operations on pandas series.
b = a * 2
c = a ** 1.5
print(b)
print(c)

You can even create a series of heterogeneous data.
s = pd.Series([‘test1’, 1.2, 3, ‘test2’], index=[‘test3’, ‘test4’, 2, ‘4.3’])

print(s)

  • pandas DataFrame

pandas DataFrame is a two-dimensional array with heterogeneous data.i.e., data is aligned in a tabular fashion in rows and columns.
Structure
Let us assume that we are creating a data frame with the student’s data.

Name Age Gender Rating
Steve 32 Male 3.45
Lia 28 Female 4.6
Vin 45 Male 3.9
Katie 38 Female 2

You can think of it as an SQL table or a spreadsheet data representation.
The table represents the data of a sales team of an organization with their overall performance rating. The data is represented in rows and columns. Each column represents an attribute and each row represents a person.
The data types of the four columns are as follows −

Column Type
Name String
Age Integer
Gender String
Rating Float

Key Points
• Heterogeneous data
• Size Mutable
• Data Mutable

A pandas DataFrame can be created using the following constructor −
pandas.DataFrame( data, index, columns, dtype, copy)

•  data
data takes various forms like ndarray, series, map, lists, dict, constants and also another DataFrame.
•  index
For the row labels, the Index to be used for the resulting frame is Optional Default np.arrange(n) if no index is passed.
•  columns
For column labels, the optional default syntax is – np.arrange(n). This is only true if no index is passed.
•  dtype
The data type of each column.
•  copy
This command (or whatever it is) is used for copying of data if the default is False.

There are many methods to create DataFrames.
• Lists
• dict
• Series
• Numpy ndarrays
• Another DataFrame

Creating DataFrame from the dictionary of Series
The following method can be used to create DataFrames from a dictionary of pandas series.

print(df)

print(df.index)

print(df.columns)

Creating DataFrame from list of dictionaries
l = [{‘orange’: 32, ‘apple’: 42}, {‘banana’: 25, ‘carrot’: 44, ‘apple’: 34}]
df = pd.DataFrame(l, index=[‘test1’, ‘test2’])

print(df)

You might have noticed that we got a DataFrame with NaN values in it. This is because we didn’t the data for that particular row and column.

Creating DataFrame from Text/CSV files
Pandas tool comes in handy when you want to load data from a CSV or a text file. It has built-in functions to do this for use.

df = pd.read_csv(‘happiness.csv’)

Yes, we created a DataFrame from a CSV file. This dataset contains the outcome of the European quality of life survey. This dataset is available here. Now we have stored the DataFrame in df, we want to see what’s inside. First, we will see the size of the DataFrame.

print(df.shape)

It has 105 Rows and 4 Columns. Instead of printing out all the data, we will see the first 10 rows.
df.head(10)

There are many more methods to create a DataFrames. But now we will see the basic operation on DataFrames.

Operations on DataFrame
We’ll recall the DataFrame we made earlier.

print(df)

Now we want to create a new row column from current columns. Let’s see how it is done.
df[‘column3’] = (2 * df[‘column1’] + 3 * df[‘column2’])/5

We have created a new column column3 from column1 and  column2. We’ll create one more using boolean.
df[‘flag’] = df[‘column1’] > 99.5

We can also remove columns.
column3 = df.pop(‘column3’)

print(column3)

print(df)

Descriptive Statistics using pandas
It’s very easy to view descriptive statistics of a dataset using pandas. We are gonna use, Biomass data collected from this source. Let’s load the data first.

url = ‘https://raw.github.com/vincentarelbundock/Rdatasets/master/csv/DAAG/biomass.csv’
df = pd.read_csv(url)
df.head()

We are not interested in the unnamed column. So, let’s delete that first. Then we’ll see the statistics with one line of code.

It’s simple as that. We can see all the statistics. Count, mean, standard deviation and other statistics. Now we are gonna find some other metrics which are not available in the describe() summary.

Mean :
print(df.mean())

Min and Max
print(df.min())

print(df.max())

Pairwise Correlation
df.corr()

Data Cleaning
We need to clean our data. Our data might contain missing values, NaN values, outliers, etc. We may need to remove or replace that data. Otherwise, our data might make any sense.
We can find null values using the following method.

print(df.isnull().any())

We have to remove these null values. This can be done by the method shown below.

newdf = df.dropna()

print(newdf.shape)

print(newdf.shape)

Pandas .Panel()
A panel is a 3D container of data. The term Panel data is derived from econometrics and is partially responsible for the name pandas − pan(el)-da(ta)-s.
The names for the 3 axes are intended to give some semantic meaning to describing operations involving panel data. They are −
• items − axis 0, each item corresponds to a DataFrame contained inside.
• major_axis − axis 1, it is the index (rows) of each of the DataFrames.
• minor_axis − axis 2, it is the columns of each of the DataFrames.

A Panel can be created using the following constructor −
The parameters of the constructor are as follows −
• data – Data takes various forms like ndarray, series, map, lists, dict, constants and also another DataFrame
• items – axis=0
• major_axis – axis=1
• minor_axis – axis=2
• dtype – the Data type of each column
• copy – Copy data. Default, false

A Panel can be created using multiple ways like −
• From ndarrays
• From dict of DataFrames
• From 3D ndarray

print(p)

Note − Observe the dimensions of the empty panel and the above panel, all the objects are different.

From dict of DataFrame Objects

print(p)

Selecting the Data from Panel
Select the data from the panel using −
• Items
• Major_axis
• Minor_axis

Using Items

print p[‘Item1’]

We have two items, and we retrieved item1. The result is a DataFrame with 4 rows and 3 columns, which are the Major_axis and Minor_axis dimensions.

Using major_axis
Data can be accessed using the method panel.major_axis(index).

Using minor_axis
Data can be accessed using the method panel.minor_axis(index).

print(p.minor_xs(1))

 

Fusebill AJAX Transparent Redirect

To facilitate PCI compliant credit card collections Fusebill provides a AJAX Transparent Redirect endpoint which you can use to securely capture customer’s credit cards. If you are adding the first payment method on a customer, it will be set to the default payment method automatically.

This API action is authenticated with a separate Public API Key. If you do not have that key, please contact Fusebill Support. The Public Key can only be used to authenticate the Transparent Redirect action.

Ajax Transparent Redirect

Google reCAPTCHA required.

Fusebill leverages reCAPTCHA technology to ensure payment method data captured is provided by a human and to protect against bots and scripting.

We use Google reCAPTCHA V2 in order to accomplish this.
https://developers.google.com/recaptcha/intro
The basic workflow for how this is accomplished is as follows:

  • Using Fusebill’s public site key, the client is presented with a captcha widget.
  • The user then verifies that they are human, starting with a check box. The user may be presented with additional verification steps such as an image recognition task.
  • The captcha widget then verifies with Google that the user is human, and returns a response token.
  • That response token is then sent to Fusebill with the payment method data for our system to validate and verify.
Fusebill Environment
reCAPTCHA Public Site Key

Staging (stg-payments.subscriptionplatform.com)

6LcI_GwUAAAAAJZu0VvB68DdxNxb5ZcBIwAX7RVj

Sandbox and Production (payments.subscriptionplatform.com)

6LfVtGwUAAAAALHn9Ycaig9801f6lrPmouzuKF11

Create Credit Card Payment Method

Field Name
Details
Required
Type

CustomerID

This is the Fusebill customer ID of the customer you wish to add the card to

Yes

Number

PublicAPIKey

This is your public API key.
This is found in fusebill account under Settings > Integrations > Transparent Redirect.

Yes

String

CardNumber

This is the credit card number.

Yes

Number

FirstName

 

The first name of the cardholder.

Yes

String

LastName

The last name of the card holder.

Yes

String

ExpirationMonth

Expiration month on the credit card.

Yes

Number

ExpirationYear

Expiration on the credit card.

Yes

Number

CVV

The credit card verification number.

Yes

Number

recaptcha

Recaptcha token response.

Yes

String

riskToken

WePay Risk token

No+

String

clientIp

Client/Customer IP address

No+

String

email

Customer Email address

No+

String

address1

First line of payment method address.

No*

String

address2

Second line of payment method address.

No*

String

city

City of the payment method

No*

String

stateId

State ID of the Payment method.
These can be found by performing a GET to v1/countries

No*

Number

countryId

Country ID of the payment method.
These can be found by performing a GET to v1/countries

No*

Number

postalZip

PostalZip of the payment method

No*

String

paymentCollectOptions

Object that allows specifying an amount to collect when creating the card.

Only works through Json
{
"collectionAmount": 1.0
}

No

Object

+ Denotes a field required for Fusebill Payments API Risk Fields
* Denotes fields required for AVS and may be required by your account’s Gateway. These fields are also required if using Fusebill Payments accounts as AVS is mandatory.

Notes:- Address information can optionally be captured as well.

Sample Code


Sample Response

Fusebill Payments

When using Fusebill Payments as your gateway processing account, some additional processing and data is required.

These are the ClientIP and a Risk token.

Additional information is available here.

Fusebill Test Gateways

Available here.

Task Notification Bot for slack with Django

Slack is a great platform for team collaboration not just that it also has one of the best API interfaces to build Chatbots.

In this post, I will walk you through building a minimal Slack Bot with Django backend. The idea is to set up a Slack Bot that will notify event when greeted through a backend.

Before we start let us understand the Slack bots life cycle:

  • If you are new to Slack, It’s a messaging platform focused on team collaboration. Slack lets you create custom applications, including bots (sort of Messaging Platform as a Service). You will run the application back end to process business logic in your own server.
  • To start with, you need to be part of the Slack team and have admin privilege to create a new Slack App. If you are not part of a Slack team you may create one.
  • GIve the name of your company or team.

  • Enter Channel Name.
  • Click on See your channel in slack
  • We will create a Slack App for the Slack team then we will add a Bot User to the app.
  • We will create a Django based backend web application to post the messages into slack.
  • After setting up the Slack App and have the backend ready to notified events.

Create a Slack App

Start by creating a Slack app here, click Create App. Then proceed with app creation, give it a name and select the Slack team.

Then you will be taken to App configuration where you need do following to get our Bot up and running.

  1. Create a Bot User
  2. Install Slack App to your Team

Create a BOT User

On the left pane click on Bot User then choose a user name for the Bot and set “Always Show My Bot as Online” slider to on. Click on Add Bot User to create our shipment bot.

Install Slack App to Team

Now on the left pane click Install App and install the app to your Slack team.

Once installed you will get Bot User OAuth Access Token, note down this token we will need it later while configuring Django app. This token is the passphrase for our Bot to interact with the Slack Team.

Slack Client Credentials

Also, note down the App Credentials from Basic Information on the left pane. These credentials let us talk to Slack API, so every time we send a message to Slack we should send our Client ID(CLIENT_ID) & Client Secret(CLIENT_SECRET) to identify ourselves to Slack. Similarly, we can verify if an incoming message is sent by Slack checking if the Verification Token (VERIFICATION_TOKEN) in the message is the same as the one in App Credentials.

Now we should have four key values with us.

  1. Client ID — SLACK_CLIENT_ID/li>
  2. Client Secret — SLACK_CLIENT_SECRET
  3. Verification Token — SLACK_VERIFICATION_TOKEN
  4. Bot User Token — SLACK_BOT_USER_TOKEN

Environment Setup

Let us create a new virtual environment “venv” for our project with python version 3.6.x and activate the virtual environment.

You need to activate the virtual environment before installation of other dependencies.

Now let’s install required packages

Create a Django Application

Configure Django Settings

we need to add our own application shipment as a dependency. Add the line mentioned below in the file slack/settings.py

# slack/settings.py

Then add following configurations in slack_bot/settings.py with your authentication keys from Slack.

# slack/settings.py

Now start the Django development server

Once the server is started it will print something similar to this

Ignore the migration warnings and open the URL in your browser.

Create an API endpoint

Now that we have our app server up and running we need to create an endpoint for Slack to send event messages. We will create an API view with Django as follows:

Shipment/view.py

Configure Django Routes

If you are new to web applications, routing is the way to tell web server which functions to invoke when an URL is hit with a request. When the URL is hit with a request message the corresponding function will be invoked, passing the requester message as a parameter to the function.
Add following lines in shipment/urls.py to tie shipment API class to http://localhost:8000/shipment/

slackapi.py

Functions written in slackapi.py are used to post notification/messages to slack.

 

 

 

Network namespaces – part 2

How to run OpenVPN tunnel inside a network namespace

Linux network namespaces can be used to control which processes should be tunneled by OpenVPN.

First create an –up and –down script for OpenVPN. This script will create the VPN tunnel interface inside a network namespace called vpn, instead of the default namespace.

Then start OpenVPN and tell it to use our –up script instead of executing ifconfig and route.

Now you can start programs to be tunneled like this:

Or start a separate shell

 

How to Debug the Execution of a Program in Linux

strace is a useful diagnostic, instructional, and debugging tool. System administrators, diagnosticians and trouble-shooters will find it invaluable for solving problems with programs for which the source is not readily available since they do not need to be recompiled in order to trace them. Students, hackers and the overly-curious will find that a great deal can be learned about a system and its system calls by tracing even ordinary programs. And programmers will find that since system calls and signals are events that happen at the user/kernel interface, a close examination of this boundary is very useful for bug isolation, sanity checking and attempting to capture race conditions.

Trace the Execution

You can use strace command to trace the execution of any executable. The following example shows the output of strace for the Linux uname command.

Counting number of syscalls

Run the ls command counting the number of times each system call was made and print totals showing the number and time spent in each call (useful for basic profiling or bottleneck isolation):

Save the Trace Execution to a File Using Option -o

The following examples stores the strace output to output.txt file.

Print Timestamp for Each Trace Output Line Using Option -t

To print the timestamp for each strace output line, use the option -t as shown below.

Tracing only network related system calls

Trace just the network related system calls of ping command

 

Viewing files opened by a process/daemon using tracefile

tracefile: Output from cmd on stdout can mess up output from strace.

Notes

It is a pity that so much tracing clutter is produced by systems employing shared libraries.

It is instructive to think about system call inputs and outputs as data-flow across the user/kernel boundary. Because user-space and kernel-space are separate and address-protected, it is sometimes possible to make deductive inferences about process behavior using inputs and outputs as propositions.

In some cases, a system call will differ from the documented behavior or have a different name. For example, on System V-derived systems the true time(2) system call does not take an argument and the stat function is called xstat and takes an extra leading argument. These discrepancies are normal but idiosyncratic characteristics of the system call interface and are accounted for by C library wrapper functions.

On some platforms a process that has a system call trace applied to it with the -p option will receive a SIGSTOP . This signal may interrupt a system call that is not restartable. This may have an unpredictable effect on the process if the process takes no action to restart the system call.

 

Using WebSockets in Javascript

WebSockets is a next-generation bidirectional communication technology for web applications which operates over a single socket and is exposed via a JavaScript interface in HTML 5 compliant browsers.

Using websockets in javascript

Once you get a Web Socket connection with the web server, you can send data from browser to server by calling a send() method, and receive data from server to browser by an onmessage event handler.

Following is the API which creates a new WebSocket object.

Here first argument, url, specifies the URL to which to connect. The second attribute, protocol is optional, and if present, specifies a sub-protocol that the server must support for the connection to be successful.

A simple example

To open a websocket connection, we need to create new WebSocket using the special protocol ws in the url:

There’s also encrypted wss:// protocol. It’s like HTTPS for websockets.

Always prefer wss://

The wss:// protocol not only encrypted, but also more reliable.

That’s because ws:// data is not encrypted, visible for any intermediary. Old proxy servers do not know about WebSocket, they may see “strange” headers and abort the connection.

On the other hand, wss:// is WebSocket over TLS, (same as HTTPS is HTTP over TLS), the transport security layer encrypts the data at sender and decrypts at the receiver, so it passes encrypted through proxies. They can’t see what’s inside and let it through.

WebSocket Attributes

Following are the attribute of WebSocket object. Assuming we created Socket object as mentioned above:-

Sr.No. Attribute & Description
1 Socket.readyState

The readonly attribute readyState represents the state of the connection. It can have the following values:-

  • A value of 0 indicates that the connection has not yet been established.
  • 1 value indicates that the connection is established and communication is possible.
  • 2 value indicates that the connection is going through the closing handshake.
  • 3 value indicates that the connection has been closed or could not be opened.

2 Socket.bufferedAmount

The readonly attribute bufferedAmount represents the number of bytes of UTF-8 text that have been queued using send() method.

WebSocket Events

Following are the events associated with WebSocket object. Assuming we created Socket object as mentioned above:-

Event Event Handler Description
open Socket.onopen This event occurs when socket connection is established.
message Socket.onmessage This event occurs when client receives data from server.
error Socket.onerror This event occurs when there is any error in communication.
close Socket.onclose This event occurs when connection is closed.

WebSocket Methods

These are the methods associated with WebSocket object. Assuming we created Socket object as mentioned above:-

Sr.No. Method & Description
1 Socket.send()

The send(data) method transmits data using the connection.

2 Socket.close()

The close() method would be used to terminate any existing connection.

WebSocket Example

WebSocket is a standard bidirectional TCP socket between the client and the server. The socket starts out as a HTTP connection and then “Upgrades” to a TCP socket after a HTTP handshake. After the handshake, either side can send data.

Client Side HTML & JavaScript Code

At the time of writing this tutorial, there are only few web browsers supporting WebSocket() interface. You can try following example with latest version of Chrome, Mozilla, Opera and Safari.

 

How to update Route53 records after EC2 instance restart

Amazon Route 53 is a highly available and scalable cloud Domain Name System (DNS) web service. If you are not using Elastic IPs for your EC2 instances, chances are stopping and starting the server will result in different IPs after the instance comes back online. If you have A records pointing to those IPs in Route53 you will need a way to update them.After the script is executed, it will automatically gather the new Public IP and update the DNS record for it in Route53.

 

How to use rsync with ssh

Rsync is a fast and extraordinarily versatile file copying tool. It can copy locally, to/from another host over any remote shell, or to/from a remote rsync daemon. It offers a large number of options that control every aspect of its behavior and permit very flexible specification of the set of files to be copied. It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. Rsync is widely used for backups and mirroring and as an improved copy command for everyday use.

Rsync finds files that need to be transferred using a lqquick checkrq algorithm (by default) that looks for files that have changed in size or in last-modified time. Any changes in the other preserved attributes (as requested by options) are made on the destination file directly when the quick check indicates that the file’s data does not need to be updated.

While tar over ssh is ideal for making remote copies of parts of a filesystem, rsync is even better suited for keeping the filesystem in sync between two machines. Typically, tar is used for the initial copy, and rsync is used to pick up whatever has changed since the last copy. This is because tar tends to be faster than rsync when none of the destination files exist, but rsync is much faster than tar when there are only a few differences between the two filesystems.
To run an rsync over ssh, pass it the -e switch, like this:
Notice the trailing / on the file spec from the source side  On the source specification, a trailing / tells rsync to copy the contents of the directory, but not the directory itself. To include the directory as the top level of whatever is being copied, leave off the /:
By default, rsync will only copy files and directories, but not remove them from the destination copy when they are removed from the source. To keep the copies exact, include the — delete flag:
If you run a command like this in cron, leave off the v switch. This will keep the output quiet (unless rsync has a problem running, in which case you’ll receive an email with the error output).
Using ssh as your transport for rsync traffic has the advantage of encrypting the data over the network and also takes advantage of any trust relationships you already have established using ssh client keys. For keeping large, complex directory structures in sync between two machines (especially when there are only a few differences between them), rsync is a very handy (and fast) tool to have at your disposal.

How to Dial USSD code in iOS?

What is USSD?
USSD (Unstructured Supplementary Service Data) is a communications protocol used by GSM cellular telephones to communicate with the mobile network operator’s computers.

Generally, the USSD code is used by apps for start Call Forwarding, To get some system info, USSD based banking services, etc.

Previously, iOS didn’t allow to dial special chars like “*” and “#” programmatically from the iOS app, to prevent users from malicious redirecting phone calls or changing the behavior of a phone or account. So dialing codes programmatically was not possible till iOS 10.

With the launch of iOS 11 Apple enabled developers to programmatically dial USSD code from iOS app with some limitations.

 

How to dial USSD code:                                                       

Use below piece of Swift code to invoke dialing of USSD codes

When the above code gets executed, the app user will be shown a Dial prompt like below and will have to choose an action.

Note:  To dial USSD code with “#” you have to encode it like shown below else you will not get prompt for a number which contains “#”.

 

If you are interested to know about the status of this call you have made to dial USSD, check below

How to Track Ongoing Call Status:

We can track Dialled call status like call dialling, running, completed using callKit framework. CallKit is a new framework that aims to improve the VoIP experience by allowing apps to integrate tightly with the native Phone UI. CallKit has many functionalities that work with VoIP calling but here we are using it only to track dialed call status.

Implement Callkit in swift to check call status as follows:

Import required Library

Use Protocol and Delegate method :-

Thats it for now folks. Happy Coding!

Country Picker With Flag jQuery plugin

About single/multiple country picker jQuery plugin :-

This single/multiple country picker jQuery plugin allows you to easily display a list of countries with flag in your Bootstrap form.

Dependencies :-

Usage :-

Create your <select> with the .country_selector class and add option of required countries.

Improtant Notes:
For multiple country picker add multiple attribute in <select> tag.

Add CSS class under <head> tag.


Add JS function under <script> tag in bottom or add in your JS file.

Configuration :-

Refer to this documentation for more configuration.