Category Archives: begginers

Here you will find articles from beginners for beginners!

nginx

Howto reverse proxy in nginx

Proxying is typically used to distribute the load among several servers, seamlessly show content from different websites, or pass requests for processing to application servers over protocols other than HTTP.

When NGINX proxies a request, it sends the request to a specified proxied server, fetches the response, and sends it back to the client. It is possible to proxy requests to an HTTP server (another NGINX server or any other server) or a non-HTTP server (which can run an application developed with a specific framework, such as PHP or Python) using a specified protocol.

1. To pass a request to an HTTP proxied server, the proxy_pass directive is specified inside a location. For example:

location /some/path/ {
proxy_pass http://www.example.com/link/;
}

 2. This address can be specified as a domain name or an IP address. The address may also include a port:

location ~ \.php {
    proxy_pass http://127.0.0.1:8000;
}

3. To pass a request to a non-HTTP proxied server, the appropriate **_pass directive should be used:

  • fastcgi_pass passes a request to a FastCGI server
  • uwsgi_pass passes a request to a uwsgi server
  • scgi_pass passes a request to an SCGI server
  • memcached_pass passes a request to a memcached server

4. Passing Request Headers

location /some/path/ {
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_pass http://localhost:8000;
}

 

5. To disable buffering in a specific location, place the proxy_buffering directive in the location with the off parameter, as follows:

location /some/path/ {
    proxy_buffering off;
    proxy_pass http://localhost:8000;
}

 

 

WSL vs WSL 2 – performance

WSL 2 is a new version of the architecture that powers the Windows Subsystem for Linux to run ELF64 Linux binaries on Windows. Its primary goals are to increase file system performance, as well as adding full system call compatibility. This new architecture changes how these Linux binaries interact with Windows and your computer’s hardware, but still provides the same user experience as in WSL 1 (the current widely available version). Individual Linux distros can be run either as a WSL 1 distro, or as a WSL 2 distro, can be upgraded or downgraded at any time, and you can run WSL 1 and WSL 2 distros side by side. WSL 2 uses an entirely new architecture that uses a real Linux kernel.

It’s a major reworking of the original WSL concept, moving away from translating Linux system calls to Windows to shipping a complete Linux kernel that runs alongside Windows’ own kernel.

The reasons for doing this are many, but the main one is simple: It’s impossible for an emulator that ships twice a year to keep up with the changes in the Linux kernel, changes that Linux binaries depend on. If Windows is to support developers building Linux apps for the cloud, then it needs to be more than consistent, it needs to be compatible.

 

Linux kernel in WSL 2

The Linux kernel in WSL 2 is built in house from the latest stable branch, based on the source available at kernel.org. This kernel has been specially tuned for WSL 2. It has been optimized for size and performance to give an amazing Linux experience on Windows and will be serviced through Windows updates, which means you will get the latest security fixes and kernel improvements without needing to manage it yourself.

Increased file IO performance

File intensive operations like git clone, npm install, apt update, apt upgrade, and more will all be noticeably faster. The actual speed increase will depend on which app you’re running and how it is interacting with the file system. Initial versions of WSL 2 run up to 20x faster compared to WSL 1 when unpacking a zipped tarball, and around 2-5x faster when using git clone, npm install and cmake on various projects.

Sockets performance benchmarks

WSL

wsl

 

WSL 2

wsl2

The Ubuntu 18.04 LTS WSL instance was used for testing with its default packages. In addition to looking at the WSL1 vs. WSL2 performance of Ubuntu 18.04, Ubuntu 18.04.2 LTS itself was also tested bare metal on the same system for looking at the raw performance of Ubuntu on the Intel desktop being tested.

Full System Call Compatibility

Linux binaries use system calls to perform many functions such as accessing files, requesting memory, creating processes, and more. In WSL 1 we created a translation layer that interprets many of these system calls and allows them to work on the Windows NT kernel. However, it’s challenging to implement all of these system calls, resulting in some apps being unable to run in WSL 1. Now that WSL 2 includes its own Linux kernel it has full system call compatibility. This introduces a whole new set of apps that you can run inside of WSL. Some exciting examples are the Linux version of Docker, as well as FUSE!

Using WSL 2 means you can also get the most recent improvements to the Linux kernel much faster than in WSL 1, as we can simply update the WSL 2 kernel rather than needing to reimplement the changes ourselves.

WSL 2 will be a much more powerful platform for you to run your Linux apps on and will empower you to do more with a Linux environment on Windows.

 

How to fix different times in Dual boot mode ( Windows and Linux)

Your PC stores the time in a hardware clock on its motherboard. The clock keeps track of time, even when the computer is off. By default, Windows assumes the time is stored in local time, while Linux assumes the time is stored in UTC time and applies an offset. This leads to one of your operating systems showing the wrong time in a dual boot situation.

To fix this, you have two options: Disable RTC in Linux, or make Windows use UTC time. Don’t follow both steps of instructions or they still won’t be speaking the same language! We recommend you make Linux use local time, if possible.

1. Disable RTC on Linux

“`timedatectl set-local-rtc 1 –adjust-system-clock“`

                                        OR

2. Use UTC in windows

How to use ipset command on linux to block bulk IPs

ipset is a companion application for the iptables Linux firewall. It allows you to setup rules to quickly and easily block a set of IP addresses, among other things.

Installation

Debian based system

“`# apt install ipset“`

Redhat based system

“`# yum install ipset“`

Blocking a list of network

Start by creating a new “set” of network addresses. This creates a new “hash” set of “net” network addresses named “myset”.

# ipset create myset hash:net

or

# ipset -N myset nethash

Add any IP address that you’d like to block to the set.

# ipset add myset 14.144.0.0/12
# ipset add myset 27.8.0.0/13
# ipset add myset 58.16.0.0/15
# ipset add myset 1.1.1.0/24

Finally, configure iptables to block any address in that set. This command will add a rule to the top of the “INPUT” chain to “-m” match the set named “myset” from ipset (–match-set) when it’s a “src” packet and “DROP”, or block, it.

# iptables -I INPUT -m set --match-set myset src -j DROP

Blocking a list of IP addresses

Start by creating a new “set” of ip addresses. This creates a new “hash” set of “ip” addresses named “myset-ip”.

# ipset create myset-ip hash:ip

or

# ipset -N myset-ip iphash

Add any IP address that you’d like to block to the set.

# ipset add myset-ip 1.1.1.1
# ipset add myset-ip 2.2.2.2

Finally, configure iptables to block any address in that set.

# iptables -I INPUT -m set --match-set myset-ip src -j DROP

Making ipset persistent

The ipset you have created is stored in memory and will be gone after reboot. To make the ipset persistent you have to do the followings:

First save the ipset to /etc/ipset.conf:

# ipset save > /etc/ipset.conf

Then enable ipset.service, which works similarly to iptables.service for restoring iptables rules.

Other Commands

To view the sets:

# ipset list

or

# ipset -L

To delete a set named “myset”:

# ipset destroy myset

or

# ipset -X myset

To delete all sets:

# ipset destroy

How to Debug the Execution of a Program in Linux

strace is a useful diagnostic, instructional, and debugging tool. System administrators, diagnosticians and trouble-shooters will find it invaluable for solving problems with programs for which the source is not readily available since they do not need to be recompiled in order to trace them. Students, hackers and the overly-curious will find that a great deal can be learned about a system and its system calls by tracing even ordinary programs. And programmers will find that since system calls and signals are events that happen at the user/kernel interface, a close examination of this boundary is very useful for bug isolation, sanity checking and attempting to capture race conditions.

Trace the Execution

You can use strace command to trace the execution of any executable. The following example shows the output of strace for the Linux uname command.

Counting number of syscalls

Run the ls command counting the number of times each system call was made and print totals showing the number and time spent in each call (useful for basic profiling or bottleneck isolation):

Save the Trace Execution to a File Using Option -o

The following examples stores the strace output to output.txt file.

Print Timestamp for Each Trace Output Line Using Option -t

To print the timestamp for each strace output line, use the option -t as shown below.

Tracing only network related system calls

Trace just the network related system calls of ping command

 

Viewing files opened by a process/daemon using tracefile

tracefile: Output from cmd on stdout can mess up output from strace.

Notes

It is a pity that so much tracing clutter is produced by systems employing shared libraries.

It is instructive to think about system call inputs and outputs as data-flow across the user/kernel boundary. Because user-space and kernel-space are separate and address-protected, it is sometimes possible to make deductive inferences about process behavior using inputs and outputs as propositions.

In some cases, a system call will differ from the documented behavior or have a different name. For example, on System V-derived systems the true time(2) system call does not take an argument and the stat function is called xstat and takes an extra leading argument. These discrepancies are normal but idiosyncratic characteristics of the system call interface and are accounted for by C library wrapper functions.

On some platforms a process that has a system call trace applied to it with the -p option will receive a SIGSTOP . This signal may interrupt a system call that is not restartable. This may have an unpredictable effect on the process if the process takes no action to restart the system call.

 

Network namespaces – part 1

Linux namespaces are a relatively new kernel feature which is essential for implementation of containers. A namespace wraps a global system resource into an abstraction which will be bound only to processes within the namespace, providing resource isolation. In this article I discuss network namespace and show a practical example.

What is namespace?

A namespace is a way of scoping a particular set of identifiers. Using a namespace, you can use the same identifier multiple times in different namespaces. You can also restrict an identifier set visible to particular processes.

For example, Linux provides namespaces for networking and processes, among other things. If a process is running within a process namespace, it can only see and communicate with other processes in the same namespace. So, if a shell in a particular process namespace ran ps waux, it would only show the other processes in the same namespace.

Linux network namespaces

In a network namespace, the scoped ‘identifiers’ are network devices; so a given network device, such as eth0, exists in a particular namespace. Linux starts up with a default network namespace, so if your operating system does not do anything special, that is where all the network devices will be located. But it is also possible to create further non-default namespaces, and create new devices in those namespaces, or to move an existing device from one namespace to another.

Each network namespace also has its own routing table, and in fact this is the main reason for namespaces to exist. A routing table is keyed by destination IP address, so network namespaces are what you need if you want the same destination IP address to mean different things at different times – which is something that OpenStack Networking requires for its feature of providing overlapping IP addresses in different virtual networks.

Each network namespace also has its own set of iptables (for both IPv4 and IPv6). So, you can apply different security to flows with the same IP addressing in different namespaces, as well as different routing.

Any given Linux process runs in a particular network namespace. By default this is inherited from its parent process, but a process with the right capabilities can switch itself into a different namespace; in practice this is mostly done using the ip netns exec NETNS COMMAND… invocation, which starts COMMAND running in the namespace named NETNS. Suppose such a process sends out a message to IP address A.B.C.D, the effect of the namespace is that A.B.C.D will be looked up in that namespace’s routing table, and that will determine the network device that the message is transmitted through.

Lets play with ip namespaces

By convention a named network namespace is an object at /var/run/netns/NAME that can be opened. The file descriptor resulting from opening /var/run/netns/NAME refers to the specified network namespace.

create a namespace

power up loopback device

open up a namespace shell

now we can use this shell like user shell where it uses ns1 namespace only

 

In part-2  , I will explain how to connect to internet from ns1 namespace and adding custom routes.

How to update Route53 records after EC2 instance restart

Amazon Route 53 is a highly available and scalable cloud Domain Name System (DNS) web service. If you are not using Elastic IPs for your EC2 instances, chances are stopping and starting the server will result in different IPs after the instance comes back online. If you have A records pointing to those IPs in Route53 you will need a way to update them. After the script is executed, it will automatically gather the new Public IP and update the DNS record for it in Route53.

Edit 1:

As I am getting many comments on hosted zone parsing error… I wanted to add this

Before running script please check your host name is set to fully qualified domain name (FQDN) with this command
“`hostname -f“`

or

“`hostname -d“`

or

“`hostnamectl“`

If you don’t want to set host name system wide, feel free to modify the script manually

#!/bin/bash
# Author: Akhil Jalagam
# Description: update route53 after ec2 instance restart

HOSTNAME=$(hostname -f)
PUBLIC_IP=$(curl ident.me)
HOSTED_ZONE=$(hostname -d | awk -F"." '{print $(NF-2)".")
ZONE_ID=$(aws route53 --output json list-hosted-zones | jq --arg hosted_zone $HOSTED_ZONE. '.HostedZones[]  | select(.Name == $hosted_zone) | .Id' | awk -F"/" '{print $3}' | tr -d "\"")
RECORD_TYPE=$(aws route53 --output json list-resource-record-sets --hosted-zone-id $ZONE_ID --query "ResourceRecordSets[?Name == '$HOSTNAME.']" | jq ".[].Type" | tr -d "\"")
RECORD_SET="/tmp/updateroute53.json"

if [ -e "$RECORD_SET" ]
then
  rm -f $RECORD_SET
fi

echo "Updating resource record set"
echo "
{
    \"Comment\": \"Update record to reflect new public IP address\",
    \"Changes\": [
        {
            \"Action\": \"UPSERT\",
            \"ResourceRecordSet\": {
                \"Name\": \"$HOSTNAME.\",
                \"Type\": \"$RECORD_TYPE\",
                \"TTL\": 300,
                \"ResourceRecords\": [
                    {
                        \"Value\": \"$PUBLIC_IP\"
                    }
                ]
            }
        }
    ]
}" | tee -a /tmp/updateroute53.json

CHANGE_ID=$(aws route53 --output json change-resource-record-sets --hosted-zone-id $ZONE_ID --change-batch file:///$RECORD_SET | jq ".ChangeInfo.Id" | awk -F"/" '{print $3}' | tr -d "\"")
CHANGE_STATUS=$(aws route53 --output json get-change --id $CHANGE_ID | jq ".ChangeInfo.Status" | tr -d "\"")
declare -i COUNT=0

while [ "$CHANGE_STATUS" == "PENDING" ]
do
  COUNT=COUNT+1
  if [ "$COUNT" -ge 6 ]
  then
    echo "Update timed out, exiting..."
    exit 1
  fi
  sleep 10
  CHANGE_STATUS=$(aws route53 --output json get-change --id $CHANGE_ID | jq ".ChangeInfo.Status" | tr -d "\"")
done

echo "Record updated!"

*note: use “`set -ex“` option to debug the script

Setup Xamarin Environment on Mac & Visual Studio

Below I have explained how to setup Xamarin environment on mac operating system step by step.

1. Download Visual studio : 

      Download Visual Studio with below link

     https://visualstudio.microsoft.com/downloads/

      

At Microsoft website, you will have three options of  Visual Studio edition to choose from. Choose one according to your need. To download Visual Studio just click on download button and an installer .dmg file will be downloaded.

2. Install Visual Studio:

   Click on downloaded dmg file and below screen will be presented

    

Select from the different Platforms you need to develop apps for on Xamarin and press the Install button. Once Visual Studio installation is complete, we need to setup environment for both Android and Apple.

3. Setup Android SDK:

    To setup Android SDK open Visual Studio and go to :-

    Tools -> SDK Manager ->Android -> Locations

    

Set path for SDK ,NDK and JDK to your local machine locations. Once correct  path is given a green tick will appear on right.This completes our Android SDK   setup.

4. Apple Setup (for both iOS and Mac apps development):

You need latest Xcode to setup Apple environment. If you have Xcode preinstalled on your machine then it automatically configures and we don’t  have to do anything. If you are installing Xcode after installation of Visual  Studio then follow below steps to setup.

a. Download latest Xcode from apple store and Install it on your machine.

b. Go to Tool -> SDK Manager -> Apple

 

Give path to your Xcode.app . You will see green check mark once the correct path is given. This completes Apple environment setup.

That is all.  Now you can start your Android and iOS development on Xamarin. Happy Coding!

How to create Gridview using Recylerview Android

First let’s understand what Gridview and Recylerview are, in Android.

Gridview

A view that shows items in two-dimensional scrolling grid is known as Gridview. GridView layout in one of the most useful layouts in android to create a scrolling grid (rows & columns).

Recylerview

Recylerview is introduced in Android 5.0 Lollipop. The Recylerview widget is a more advanced and flexible version of Listview. It is a container used to display a large number of data sets that can be scrolled very efficiently by maintaining a limited number of views.

Now let’s start implementing Gridview

First, we need to add below dependency in build.gradle file at app level module.

dependencies {
      implementation 'com.android.support:recyclerview-v7:28.0.0'
}

After that, we need to add Recylerview widget in your main XML file.

<android.support.v7.widget.RecyclerView
         android:id="@+id/recyclerView"
         android:layout_width="match_parent"
         android:layout_height="wrap_content" />

Now we need to create item_logo.xml for Gridview row item.

<?xml version="1.0" encoding="utf-8"?>
<LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
   xmlns:app="http://schemas.android.com/apk/res-auto"
   android:id="@+id/root"
   android:layout_width="match_parent"
   android:layout_height="wrap_content"
   android:gravity="center"
   android:orientation="vertical">

    <ImageView
         android:id="@+id/ivLogo"
         android:layout_width="match_parent"
         android:layout_height="match_parent"
         android:scaleType="centerInside"
    />

</LinearLayout>

We need to create Adapter Object. An adapter in Android carries the data from a source (e.g. List<> ) and delivers it to a layout (.xml file).  The Adapter provides access to the data items.

public class LogoGridAdapter extends RecyclerView.Adapter<LogoGridAdapter.ViewHolder> {

    private List<Logo> logoList;
    private Context mContext;


    public LogoGridAdapter(Context mContext, List<Logo> logoList) {

        this.mContext = mContext;
        this.logoList = logoList;

    }

    @Override
    public ViewHolder onCreateViewHolder(ViewGroup parent, int viewType) {
        return new ViewHolder(LayoutInflater.from(parent.getContext()).inflate(R.layout.item_logo, null));
    }

    @Override
    public void onBindViewHolder(ViewHolder holder, int position) {

        Logo logo = getLogo(position);

        if(logo.getLogoUrl() != null) {
            ImageUtils.displayImage(logo.getLogoUrl(), holder.ivLogo, ContextCompat.getDrawable(mContext, R.drawable.image_placeholder));
        }

    }


    private Logo getLogo(int position) {
        return logoList.get(position);
    }

    public List<Logo> getLogoList() {
        return logoList;
    }

    @Override
    public int getItemCount() {
        return logoList.size();
    }

    public class ViewHolder extends RecyclerView.ViewHolder {
       
        LinearLayout root;
        ImageView ivLogo;

        public ViewHolder(View itemView) {
            super(itemView);
            // get logo view 
            ivLogo = (ImageView)itemView.findViewById(R.id.ivLogo);
            root =  (LinearLayout)itemView.findViewById(R.id.root);
        }
    }
}

To display images we can use Glide dependency.

Glide.with(imageView.getContext()).load(imageUrl)
                .apply(new RequestOptions().placeholder(placeHolder).dontAnimate().diskCacheStrategy(DiskCacheStrategy.ALL))
                .into(imageView);

Now we need to set data into Adapter.

LogoGridAdapter mLogoGridAdapter = new LogoGridAdapter(this, logoList);
recyclerView.setLayoutManager(new GridLayoutManager(getActivity(), 3));
recyclerView.setAdapter(mLogoGridAdapter);

GridLayoutManager is a Recylerview Layout Manager implementation to lay out items in a grid.

In the above code “3” is a number of columns in per row.

Output

 

 

 

 

 

 

 

 

 

 

That’s it, Happy Coding 🙂

Reference:-  https://developer.android.com/guide/topics/ui/layout/recyclerview

How Implement Multiservice in Twisted.

Multiservice module is service collection provided by twisted, which is useful for creating a new service and combines with two or more existing services.

The major tools that manages Twisted application is a command line utility called twistd. twistd is a cross-platform, and is the recommended tool for running twisted applications.

The core component of the Twisted Application infrastructure is the

twisted.application.service.Application

object. which represents your application. Application acts as a container of any “Services” that your application provides. This will be done through Services.

Services manages application that can be started and stopped. In Application object can contain many services, or can even hierarchies of Services using “Multiservice” or your own custom IServiceCollection implementations.

Multiservice Implementaion:

To use multiserivce, which implements IService. For this, import internet and service module.

from twisted.application import internet, service

 

Example :

from twisted.application import internet,service
from serviceone import getServiceOne
from servicesecond import getServiceSecond
from twisted.web.server import Site

agentService = service.MultiService()
agentService.addService(getServiceOne())
agentService.addService(getServiceSecond())
application = service.Application("Receive Request")
agentService.setServiceParent(application)

To run, Save above code in a file as serviceexample.tac . Here, “tac ” file is regular python file. Twisted application infrastructure, protocol implementations live in a module, services, using those protocols are registered in a Twisted Application Configuration(TAC) file and the reactor and configuration are managed by an external utility.

Here, I use multiservice functionality from service. agentservice create object of multiservice. Then add services using add service method. In service, you can add web servers, FTP servers and SSH clients. After this, set application name and pass application to serviceparent method.

now, add service on port 8082 as :

from twisted.web.server import Site, NOT_DONE_YET
from twisted.web.resource import Resource
import logging
from twisted.application import internet

log = logging.getLogger("State Home")

class StateHome(Resource):
    isLeaf = True


    def reqComplete(self, req):
        req.write("Successfully Called.")
        req.finish()

    def render_GET(self, request):
        log.info("Run Render Get Method.")
        self.reqComplete(request)
        return NOT_DONE_YET

def getServiceOne():
    root = StateHome()
    factory = Site(root)
    return internet.TCPServer(8082, factory)

add another service same as above on port 8083 as:

from twisted.web.server import Site, NOT_DONE_YET
from twisted.web.resource import Resource
import logging
from twisted.application import internet

log = logging.getLogger("State Home")

class StateHome(Resource):
    isLeaf = True


    def reqComplete(self, req):
        req.write("Successfully Called second service.")
        req.finish()

    def render_GET(self, request):
        log.info("Run Render Get Method.")
        self.reqComplete(request)
        return NOT_DONE_YET

def getServiceSecond():
    root = StateHome()
    factory = Site(root)
    return internet.TCPServer(8083, factory)

To run serviceexample.tac file using twistd program, use command twistd -y serviceexample.tac -n. After this, open browser and enter url localhost:8082 and localhost:8083. You can see result on web page and both TCP servers are active.