How to run twisted script as daemon without twistd command

Warning:- this article assumes you are familiar with twisted
we have scenarios like run different services on different ports
to create multiservice architecture.
So, here we cannot run the services on from the command line
we have to run them as daemon service, like using cron jobs in linux.

– Create a simple twisted with protocol and factory and save to test.py or any name you like.

#!/usr/bin/env python2.7

from twisted.internet import protocol, reactor

class echo(protocol.Protocol):
        def dataReceived(self, data):
                self.transport.write(data.upper())

class echoFactory(protocol.Factory):
        def buildProtocol(sef,addr):
                print 'called'
                return echo()

reactor.listenTCP(8000,echoFactory())
reactor.run()

this is a normal echo server script which can run using

./test.py # assuming the script is saved in test.py

But our script cannot run this is as a daemon service
daemon service is some task runs in the background, and it logs to some specific location, not on stdout. When we run this script like specified above, it blocks the current terminal ( if you are running from a terminal ) which is not something daemon script do.
To make this script run as a daemon, we need to use twistd inside the script
means it’ll run and execute, create a process in the background.

from twisted.scripts import twistd
import sys
sys.argv.append('-y dummy')
sys.argv.append('--pidfile={0} --logfile=/dev/null'.format('/tmp/echo.pid'))

application = service.Application('echo_daemon')
tcp_service = internet.TCPServer(interface='127.0.0.1',port=8000, factory=factory)
tcp_service.setServiceParent(application)

class ApplicationRunner(twistd._SomeApplicationRunner):
    def createOrGetApplication(self):
        return application

    def run(self):
        self.preApplication()
        self.application = self.createOrGetApplication()
        self.postApplication()


twistd._SomeApplicationRunner = ApplicationRunner
twistd.run()

#~ python2.7 test.py

How should we cross-check if the service is running or not
Use the following command on Linux system (in shell/terminal for simple word).

netstat -ntulp

You get a list of all ports (UDP, TCP) open on the system.

How to edit file with sed

This tutorial is about how to replace the pattern with sed command in the same file you are reading from.

What is sed?
Sed stands for stream editor, as per the Linux man page.
But sed is more than it. We can write more powerful bash scripts with just a single line using sed. It can be used for fetching specific lines, a bunch of lines, replace the character with patterns and so much. Basically, just manipulate the stream of characters with whatever logic you projectile

Let’s get started,
We understand more with the task in hand. So, our task for this tutorial is to replace some characters with our characters.
Suppose we have a file like content below,

  304  PULSEaudio -k
  306  PULSEaudio --cleanup-shm 
  310  PULSEaudio --check 
  311  PULSEaudio --start 
  312  PULSEaudio --kill 
  322  PULSEaudio -k
  323  PULSEaudio --check 
  324  killall PULSEaudio
  325  PULSEaudio --check 
  331  ls * | grep -e PULSE
  332  cd PULSE/
  340  PULSEaudio -k
  344  PULSEaudio -D
  345  PULSEaudio -d
  346  service PULSEaudio status
  348  ps -eo "user args" | grep PULSE
  350  ps -eo "user args" | grep PULSE
  351  PULSEaudio -k
  352  killall PULSEaudio 

And we are supposed to replace the characters PULSE to pulse.
Either, we can open this file in VIM and type command like

 %s/PULSE/pulse/gc

If you are familiar with vim. You’ll know what I am talking about
But if you are using this output for some reason in your bash script, you need to do this with a single command.

Here comes our savior sed.

sed s/pattern/replace_char/ <file_name>

This command does our task, but the problem is it gives output on stdout.

Common error:- we generally try to redirect that output to the file we are editing
If we are editing the file named replace.txt then the command will be
sed s/PULSE/pulse/ replace.txt 2> replace.txt

But sed creates a problem here. It doesn’t work that way it’s not sed’s problem, Its problem with the order of file descriptors it set.

This is a common error, we want to modify a file using something that reads from a file and writes the result to stdout. To do this, we redirect stdout to the file we want to modify. The problem here is that, as we have seen, the redirections are setup before the command is actually executed.
So BEFORE sed starts, standard output has already been redirected, with the additional side effect that, because we used >, “file” gets truncated. When sed starts to read the file, it contains nothing
( if you don’t know what it is read this link https://wiki.bash-hackers.org/howto/redirection_tutorial ).

Sed added one feature which internally sets this redirection of a file descriptor for us. Use -i option to overcome this problem.

The final command will be

sed s/PULSE/pulse/ replace.txt -i

How to make hello world program in wxPython

In this article we will look at creating a simple hello world program using wxPython. This program will create and display simple window with a big button on it. Up on clicking the button program will exit. Use the following code to create hello world program. You must already have wxPython library installed

Hello world program

import wx

class MainWindow(wx.Frame):
    def __init__(self,parent):
        wx.Frame.__init__(self,parent,title="Hello World")
        self.killButton = wx.Button(self,label="Kill Me")
        self.killButton.Bind(wx.EVT_BUTTON,self.kill)
        self.Show()

    def kill(self,event):
        self.Close()
        print("Bye Bye cruel world")
        
app = wx.App(False)
frame = MainWindow(None)
app.MainLoop()

Explanation

We will go line by line here and try to explain what’s going on in this program. Most of the lines of self explanatory. If you are just getting started in programming the following explanation will be helpful.

  1. Import wxpython library
  2. Inherit from wx.Frame class. This is useful style for most of the programs that you will build. You can create one base frame or window and put rest of GUI widgets on top of it like Text controls, buttons,images, tables etc.
  3. Instantiate the inherited frame the desired title. parent argument is usually None for main windows.
  4. Create a button with label “Kill Me”. The first argument is parent. In this case we use “self” which is the main window we have just created.
  5. Bind the button click event (EVT_BUTTON) of the killButton to kill method. Whenever, EVT_BUTTON event is fired aka the killButton is clicked, kill method will be called.
  6. This line will cause the window to get displayed on screen. It’s customary to call this method after being done with construction of GUI i.e. create main window, place widgets, bind event like we did here.
  7. Create wxPython application by call wx.App. Every wxPython program must have this application.
  8. Start the main loop. Which will hand over control to wxPython library. This post explains why main loop has to be called.

Output

This program will launch the following window. The button takes all the available space on the window since there are no other widgets. You need a few more lines of code to make the button look like what users are used to – small and horizontal. You can exit the program by clicking the button.

How to install wxpython

In this post we will go over the topic of easy way to install wxpython. The following command will install wxpython easily in python3.

pip install wxpython

Python 2

Older versions of wxpython can be installed by downloading binaries (for windows) from sourceforge . The binaries are available for both Windows and MacOS.

Linux

The pip install command should work on linux also. However, if you are stuck on old linux vesions. You can try installing wxpython using your distro’s package manager like apt-get or yum. You can search for exact version and package name available on your platform using the following commands for debian variants.

apt-cache search wxpython

┌──(kali㉿kali)-[~]
└─$ apt-cache search wxpython
gnumed-client - medical practice management - Client
psychopy - environment for creating psychology stimuli in Python
pyscanfcs - scientific tool for perpendicular line scanning FCS
python3-genx - differential evolution algorithm for fitting
python3-opengl - Python bindings to OpenGL (Python 3)
python3-pyface - traits-capable windowing framework
python3-squaremap - wxPython control to display hierarchic data as nested squares
python3-wxgtk-media4.0 - Python 3 interface to the wxWidgets Cross-platform C++ GUI toolkit (wx.media)
python3-wxgtk-webview4.0 - Python 3 interface to the wxWidgets Cross-platform C++ GUI toolkit (wx.html2)
python3-wxgtk4.0 - Python 3 interface to the wxWidgets Cross-platform C++ GUI toolkit
soundgrain - Graphical interface to control granular sound synthesis modules
wxglade - GUI designer written in Python with wxPython
wxpython-tools - Tools from the wxPython distribution

Successful installation

Use the following command to check if the installation is successful or not.

PS C:\Users\godson> python
Python 3.8.6 (tags/v3.8.6:db45529, Sep 23 2020, 15:52:53) [MSC v.1927 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import wx
>>> wx.version()
'4.1.1 msw (phoenix) wxWidgets 3.1.5'
>>>

As shown above it should print current version of wxpython installed if the installation is successful.

A more extensive tutorial is available here

mysql

How To Import and Export Databases in MySQL

MySQL is an open-source relational database management system. Its name is a combination of “My”, the name of co-founder Michael Widenius’s daughter, and “SQL”, the abbreviation for Structured Query Language.

A relational database organizes data into one or more data tables in which data types may be related to each other; these relations help structure the data. SQL is a language programmers use to create, modify and extract data from the relational database, as well as control user access to the database. In addition to relational databases and SQL, an RDBMS like MySQL works with an operating system to implement a relational database in a computer’s storage system, manages users, allows for network access and facilitates testing database integrity and creation of backups.

What is mysql?

mysql is a simple SQL shell (with GNU readline capabilities). It supports interactive and
non-interactive use. When used interactively, query results are presented in an ASCII-table format.
When used non-interactively (for example, as a filter), the result is presented in tab-separated
format. The output format can be changed using command options.

What is mysqldump?

The mysqldump client is a backup program originally written by Igor Romanenko. It can be used to dump
a database or a collection of databases for backup or transfer to another SQL server (not necessarily
a MariaDB server). The dump typically contains SQL statements to create the table, populate it, or
both. However, mysqldump can also be used to generate files in CSV, other delimited text, or XML
format.

Export a MySQL Database

Use mysqldump to export your database:

mysqldump -u username -p database_name > database_name-dump.sql

You can compress the data on the run using pipe and gzip.

mysqldump -u username -p database_name | gzip > database_name-dump.sql.gz

*Using GZIP will save a lot of space on disk for huge databases.

Import a MySQL Database

Use mysql to import your database:

Create the database first.

mysql > CREATE DATABASE database_name;

Import the database now.

mysql -u username -p database_name < database_name-dump.sql

If the file is compressed with gzip. use zcat to extract on the run.

zcat database_name-dump.sql.gz | mysql -u username -p database_name

Handy scripts for admins who do backups daily

bkpmysqlgz() {
    set -x
    DEST=${1:-/opt/backups}
    shift
    sudo mkdir -p $DEST
    DATE=$(DATE)
    if command -v pv > /dev/null 2>&1; then
        sudo mysqldump $@ | pv | gzip > $DEST/"${@: -1}"-$DATE.sql.gz
    else
        sudo mysqldump $@ | gzip > $DEST/"${@: -1}"-$DATE.sql.gz
    fi
    ls -lh $DEST/"${@: -1}"-$DATE.sql.gz
    set +x
}

using script.

$ bkpmysqlgz /opt/backups -u root -p secret dbname

How to find out CPU make and model in Linux

Use the following simple command to find out make and model of CPU/Processor – cat /proc/cpuinfo

┌──(kali㉿kali)-[~]
└─$ cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 94
model name : Intel(R) Core(TM) i7-6700HQ CPU @ 2.60GHz
stepping : 3
cpu MHz : 2591.996
cache size : 6144 KB
physical id : 0
siblings : 1
core id : 0
cpu cores : 1
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 22
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq monitor ssse3 cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase avx2 invpcid rdseed clflushopt md_clear flush_l1d
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit srbds
bogomips : 5183.99
clflush size : 64
cache_alignment : 64
address sizes : 39 bits physical, 48 bits virtual
power management:

There are other commands like

lscpu

dmidecode

But, cat /proc/cpuinfo works reliably on all machines.

How to use Free Clipboard Manager in Windows 10

I have been using clipboard managers for a long time. I came across a variety of clipboard managers both on Linux and Windows. Long ago when I was using KDE 3 Klipper was my favorite clipboard manager. After moving to windows I didn’t find any thing that matched my taste. A few free ones were sluggish and others came with restriction on usage unless you buy them. Number of times I have resisted the idea of writing my own clipboard history manager after failing to find some features I wanted in existing ones out there.

Windows 10 packs a neat little clipboard history manager for some one whose needs are simple. It won’t automatically process links or other text you put into clipboard. It’s pretty basic. But it will save your day with out having to install 3rd party clipboard managers

How to enable Windows Clipboard History

Goto Settings search for Clipboard . Turn on the toggle button under Clipboard History section. Look at the following screen shot

Once you have enabled it. You will be able to summon the clipboard history pop up by pressing Win + V key combination

Ideas are worthless

When building new products or services it’s better to share them with potential customers as early as possible. This will help close the feedback loop. Which is precious when a business is in early phase and haven’t burnt too much cash on the new product.

Start ups operating in stealth mode, shrouded in secrecy, developing the product in private, spend years in making it perfect. Then unveiling it to the public. Though that’s very tempting to entrepreneur. This path ruins careers and empties bank accounts.

I have been on the both sides of this game. There is a service which we have built and shared with potential customers very early on. It became successful. And there are a few products which were developed in stealth mode for a long time and they ultimately went no where.

Stealth mode hinders businesses from early learning opportunities and puts them at a huge disadvantage.

MVP (Minimum Viable Product) /Prototype – is early representation of what your offering will look like. It can rough around the edges. But, try to present it to the potential customers early on in the product’s life cycle. The feedback a business gets from early users is priceless and will help shape the whole thing into a better product that people actually want to use. Coming out of stealth mode and finding out that no one really wants to use the product is a very tough situation to spring back from. At this point one has to battle with lots of psychological effects like Sunk Cost Fallacy

Ideas are cheap or like they say dime a dozen. What really counts is the ability to transform that Idea into usable product/service. The journey from idea to taking the first payment from the customer is fraught with peril.

Unless you are working in an industry where there is heavy competition and competitors with deep pockets that can copy your idea over night, it’s a bad idea to develop the product in the dark.

How to whitelist Google IP address ranges in firewall using iptables

As an administrator, when you need to obtain a range of IP addresses for Google APIs and services’ default domains, you can refer to the following sources of information.

The default domains’ IP address ranges for Google APIs and services fit within the list of ranges between these 2 sources. (Subtract the usable ranges from the complete list.)

Once you get the IP address ranges, use the “`xargs“` command to update iptables.

google-ips-whitelist.sh

echo "8.8.4.0/24
8.8.8.0/24
8.34.208.0/20
8.35.192.0/20
23.236.48.0/20
23.251.128.0/19
34.64.0.0/10
34.128.0.0/10
35.184.0.0/13
35.192.0.0/14
35.196.0.0/15
35.198.0.0/16
35.199.0.0/17
35.199.128.0/18
35.200.0.0/13
35.208.0.0/12
35.224.0.0/12
35.240.0.0/13
64.15.112.0/20
64.233.160.0/19
66.102.0.0/20
66.249.64.0/19
70.32.128.0/19
72.14.192.0/18
74.114.24.0/21
74.125.0.0/16
104.154.0.0/15
104.196.0.0/14
104.237.160.0/19
107.167.160.0/19
107.178.192.0/18
108.59.80.0/20
108.170.192.0/18
108.177.0.0/17
130.211.0.0/16
136.112.0.0/12
142.250.0.0/15
146.148.0.0/17
162.216.148.0/22
162.222.176.0/21
172.110.32.0/21
172.217.0.0/16
172.253.0.0/16
173.194.0.0/16
173.255.112.0/20
192.158.28.0/22
192.178.0.0/15
193.186.4.0/24
199.36.154.0/23
199.36.156.0/24
199.192.112.0/22
199.223.232.0/21
207.223.160.0/20
208.65.152.0/22
208.68.108.0/22
208.81.188.0/22
208.117.224.0/19
209.85.128.0/17
216.58.192.0/19
216.73.80.0/20
216.239.32.0/19" | xargs -I% iptables -I INPUT -p tcp -s % -j ACCEPT

how to setup apache proxy for django application

Apache HTTP Server can be configured in both a forward and reverse proxy (also known as gateway) mode.

forward proxy

An ordinary forward proxy is an intermediate server that sits between the client and the origin server. In order to get content from the origin server, the client sends a request to the proxy naming the origin server as the target. The proxy then requests the content from the origin server and returns it to the client. The client must be specially configured to use the forward proxy to access other sites.

A typical usage of a forward proxy is to provide Internet access to internal clients that are otherwise restricted by a firewall. The forward proxy can also use caching (as provided by mod_cache) to reduce network usage.

The forward proxy is activated using the ProxyRequests directive. Because forward proxies allow clients to access arbitrary sites through your server and to hide their true origin, it is essential that you secure your server so that only authorized clients can access the proxy before activating a forward proxy.

reverse proxy

reverse proxy (or gateway), by contrast, appears to the client just like an ordinary web server. No special configuration on the client is necessary. The client makes ordinary requests for content in the namespace of the reverse proxy. The reverse proxy then decides where to send those requests and returns the content as if it were itself the origin.

A typical usage of a reverse proxy is to provide Internet users access to a server that is behind a firewall. Reverse proxies can also be used to balance load among several back-end servers or to provide caching for a slower back-end server. In addition, reverse proxies can be used simply to bring several servers into the same URL space.

A reverse proxy is activated using the ProxyPass directive or the [P] flag to the RewriteRule directive. It is not necessary to turn ProxyRequests on in order to configure a reverse proxy.

django application

I am running my gunicorn application on port 8090 using following command.

“`/opt/venv/bin/python3.6 /opt/venv/bin/gunicorn –config /etc/controlpanel/gunicorn/controlpanel.py –pid /var/run/controlpanel.pid controlpanel.wsgi:application“`

static files path is “`/opt/controlpanel/ui-ux/static/“`

apache config (/etc/apache2/sites-enabled/cp.conf)

  • enable mod_proxy module in apache
<VirtualHost *:80>

ServerName devcontrol.lintel.com
ErrorLog /var/log/httpd/cp_error.log
CustomLog /var/log/httpd/cp_access.log combined

ProxyPreserveHost On
ProxyPass /static !
ProxyPass / http://127.0.0.1:8090/
ProxyPassReverse / http://127.0.0.1:8090/
ProxyTimeout 300

Alias /static/ /opt/controlpanel/ui-ux/static/
<Directory "/opt/controlpanel/ui-ux/static/">
Options Indexes FollowSymLinks
AllowOverride None
Require all granted
</Directory>

</VirtualHost>

after deploying on Apache you can use lets encrypt to install SSL certificates.