Category Archives: networking

How to run twisted script as daemon without twistd command

Warning:- this article assumes you are familiar with twisted
we have scenarios like run different services on different ports
to create multiservice architecture.
So, here we cannot run the services on from the command line
we have to run them as daemon service, like using cron jobs in linux.

– Create a simple twisted with protocol and factory and save to test.py or any name you like.

#!/usr/bin/env python2.7

from twisted.internet import protocol, reactor

class echo(protocol.Protocol):
        def dataReceived(self, data):
                self.transport.write(data.upper())

class echoFactory(protocol.Factory):
        def buildProtocol(sef,addr):
                print 'called'
                return echo()

reactor.listenTCP(8000,echoFactory())
reactor.run()

this is a normal echo server script which can run using

./test.py # assuming the script is saved in test.py

But our script cannot run this is as a daemon service
daemon service is some task runs in the background, and it logs to some specific location, not on stdout. When we run this script like specified above, it blocks the current terminal ( if you are running from a terminal ) which is not something daemon script do.
To make this script run as a daemon, we need to use twistd inside the script
means it’ll run and execute, create a process in the background.

from twisted.scripts import twistd
import sys
sys.argv.append('-y dummy')
sys.argv.append('--pidfile={0} --logfile=/dev/null'.format('/tmp/echo.pid'))

application = service.Application('echo_daemon')
tcp_service = internet.TCPServer(interface='127.0.0.1',port=8000, factory=factory)
tcp_service.setServiceParent(application)

class ApplicationRunner(twistd._SomeApplicationRunner):
    def createOrGetApplication(self):
        return application

    def run(self):
        self.preApplication()
        self.application = self.createOrGetApplication()
        self.postApplication()


twistd._SomeApplicationRunner = ApplicationRunner
twistd.run()

#~ python2.7 test.py

How should we cross-check if the service is running or not
Use the following command on Linux system (in shell/terminal for simple word).

netstat -ntulp

You get a list of all ports (UDP, TCP) open on the system.

How to whitelist Google IP address ranges in firewall using iptables

As an administrator, when you need to obtain a range of IP addresses for Google APIs and services’ default domains, you can refer to the following sources of information.

The default domains’ IP address ranges for Google APIs and services fit within the list of ranges between these 2 sources. (Subtract the usable ranges from the complete list.)

Once you get the IP address ranges, use the “`xargs“` command to update iptables.

google-ips-whitelist.sh

echo "8.8.4.0/24
8.8.8.0/24
8.34.208.0/20
8.35.192.0/20
23.236.48.0/20
23.251.128.0/19
34.64.0.0/10
34.128.0.0/10
35.184.0.0/13
35.192.0.0/14
35.196.0.0/15
35.198.0.0/16
35.199.0.0/17
35.199.128.0/18
35.200.0.0/13
35.208.0.0/12
35.224.0.0/12
35.240.0.0/13
64.15.112.0/20
64.233.160.0/19
66.102.0.0/20
66.249.64.0/19
70.32.128.0/19
72.14.192.0/18
74.114.24.0/21
74.125.0.0/16
104.154.0.0/15
104.196.0.0/14
104.237.160.0/19
107.167.160.0/19
107.178.192.0/18
108.59.80.0/20
108.170.192.0/18
108.177.0.0/17
130.211.0.0/16
136.112.0.0/12
142.250.0.0/15
146.148.0.0/17
162.216.148.0/22
162.222.176.0/21
172.110.32.0/21
172.217.0.0/16
172.253.0.0/16
173.194.0.0/16
173.255.112.0/20
192.158.28.0/22
192.178.0.0/15
193.186.4.0/24
199.36.154.0/23
199.36.156.0/24
199.192.112.0/22
199.223.232.0/21
207.223.160.0/20
208.65.152.0/22
208.68.108.0/22
208.81.188.0/22
208.117.224.0/19
209.85.128.0/17
216.58.192.0/19
216.73.80.0/20
216.239.32.0/19" | xargs -I% iptables -I INPUT -p tcp -s % -j ACCEPT

Speed up SSH with multiplexing

SSH multiplexing is the ability to carry multiple SSH sessions over a single TCP connection.

OpenSSH can reuse an existing TCP connection for multiple concurrent SSH sessions. This results into reduction of the overhead of creating new TCP connections.

Advantage of using SSH multiplexing is that it speeds up certain operations that rely on or occur over SSH. For example, let’s say that you’re using SSH to regularly execute a command on a remote host. Without multiplexing, every time that command is executed your SSH client must establish a new TCP connection and a new SSH session with the remote host. With multiplexing, you can configure SSH to establish a single TCP connection that is kept alive for a specific period of time, and SSH sessions are established over that connection.

You can see the difference below

without multiplexing, we see the normal connection time:

“`$ time ssh lintel-blog“`

real    0m0.658s
user    0m0.016s
sys     0m0.008s

Then we do the same thing again, but with a multiplexed connection to see a faster result:

“`$ time ssh lintel-blog“`

real    0m0.029s
user    0m0.004s
sys     0m0.004s

Configure Multiplexing

OpenSSH client supports multiplexing its outgoing connections, since version 3.9, using the ControlMaster, ControlPath and ControlPersist configuration directives which get defined in ssh_config. The client configuration file usually defaults to the location ~/.ssh/config.

ControlMaster determines whether ssh will listen for control connections and what to do about them. ControlPath sets the location for the control socket used by the multiplexed sessions. These can be either globally or locally in ssh_config or else specified at run time. Control sockets are removed automatically when the master connection has ended. ControlPersist can be used in conjunction with ControlMaster. If ControlPersist is set to ‘yes’, then it will leave the master connection open in the background to accept new connections until either killed explicitly or closed with -O or ends at a pre-defined timeout. If ControlPersist is set to a time, then it will leave the master connection open for the designated time or until the last multiplexed session is closed, whichever is longer.

Here is a sample excerpt from ssh_config applicable for starting a multiplexed session to server1.example.org via the shortcut server1.

Host server1
  HostName server1.example.org
  ControlPath ~/.ssh/controlmasters/%r@%h:%p
  ControlMaster auto
  ControlPersist 10m

 

Parallel command execution – Linux Cluster

The pdsh parallel shell tool allows you and lets you run a shell command across multiple nodes in a cluster.

This is a high performance, parallel pdsh shell remote shell utility for admins. Chaos Pdsh is a multithreaded remote shell client which executes commands on multiple remote hosts in parallel.  A parallel shell permits your clusters Linux Ubuntu RedHat to run the same similar command on many designated hosts or nodes within the hadoop cluster. In this case you do not have to really log in to each node individually.

High-performance and parallel remote shell utility with dshgroup module allows dsh on pdsh (or otherwise known as Dancer’s shell sudo) files from /etc/dsh/group directory. Now download Parallel Distributed Shell free of charge.

What is pdsh?

pdsh is a variant of the rsh(1) command. Unlike rsh(1), which runs commands on a single remote host, pdsh can run multiple remote commands in parallel. pdsh uses a “sliding window” (or fanout) of threads to conserve resources on the initiating host while allowing some connections to time out.

When pdsh receives SIGINT (ctrl-C), it lists the status of current threads. A second SIGINT within one second terminates the program. Pending threads may be canceled by issuing ctrl-Z within one second of ctrl-C. Pending threads are those that have not yet been initiated, or are still in the process of connecting to the remote host.

If a remote command is not specified on the command line, pdsh runs interactively, prompting for commands and executing them when terminated with a carriage return. In interactive mode, target nodes that time out on the first command are not contacted for subsequent commands, and commands prefixed with an exclamation point will be executed on the local system.

The core functionality of pdsh may be supplemented by dynamically loadable modules. The modules may provide a new connection protocol (replacing the standard rcmd(3) protocol used by rsh(1)), filtering options (e.g. removing hosts that are “down” from the target list), and/or host selection options (e.g., -a selects all hosts from a configuration file.). By default, pdsh must have at least one “rcmd” module loaded. See the RCMD MODULES section for more information.

Installing pdsh

Debian based:

“`apt install pdsh“`

RHEL based:

“`yum install pdsh“`

Running

The following command installs telegraf on all 4 nodes in cluster02

[akhil@PHOENIX:~]$ pdsh -w root@cluster02-node0[1-4] yum install -y telegraf

Running multiple commands

[akhil@PHOENIX:~]$ pdsh -w root@cluster02-node0[1-4] "date;sleep 5;date"

Pipe redirection

[akhil@PHOENIX:~]$ pdsh -w root@cluster02-node0[1-4] "chkconfig|grep collectl"

 

Example

 

When using ssh for remote execution, expect the stderr of ssh to be folded in with that of the remote command. When invoked by pdsh, it is not possible for ssh to prompt for passwords if RSA/DSA keys are configured properly, etc.. For ssh implementations that suppport a connect timeout option, pdsh attempts to use that option to enforce the timeout (e.g. -oConnectTimeout=T for OpenSSH), otherwise connect timeouts are not supported when using ssh. Finally, there is no reliable way for pdsh to ensure that remote commands are actually terminated when using a command timeout. Thus if -u is used with ssh commands may be left running on remote hosts even after timeout has killed local ssh processes.

Output from multiple processes per node may be interspersed when using qshell or mqshell rcmd modules.

The number of nodes that pdsh can simultaneously execute remote jobs on is limited by the maximum number of threads that can be created concurrently, as well as the availability of reserved ports in the rsh and qshell rcmd modules. On systems that implement Posix threads, the limit is typically defined by the constant PTHREADS_THREADS_MAX.

Manhole service in Twisted Application.

What is Manhole?

Manhole is an in-process service, that will accept UNIX domain socket connections and present the stack traces for all threads and an interactive prompt.

Using it we can access and modify objects or definition in the running application, like change or add the method in any class, change the definition of any method of class or module.

This allows us to make modifications in running an application without restarting the application, it makes work easy like debugging the application, you are able to check the values of the object while the program is running.

How to configure it?

from twisted.internet import reactor
from twisted.conch import manhole, manhole_ssh
from twisted.conch.ssh.keys import Key
from twisted.cred import portal, checkers

DATA = {"Service": "Manhole"}


def get_manhole_factory(namespace, **passwords):

    def get_manhole(arg):
        return manhole.ColoredManhole(namespace)
            
    realm = manhole_ssh.TerminalRealm()
    realm.chainedProtocolFactory.protocolFactory = get_manhole
    p = portal.Portal(realm)
    p.registerChecker(checkers.InMemoryUsernamePasswordDatabaseDontUse(**passwords))
    f = manhole_ssh.ConchFactory(p)
    f.publicKeys = {"ssh-rsa": Key.fromFile("keys/manhole.pub")}
    f.privateKeys = {"ssh-rsa": Key.fromFile("keys/manhole")}
    return f


reactor.listenTCP(2222, get_manhole_factory(globals(), admin='admin'))
reactor.run()

Once you run above snippet, the service will start on TCP port 2222.

You need to use SSH command to get login into the service.

See below how it looks like.

[lalit : ~]₹ ssh admin@localhost -p 2222
admin@localhost's password:
>>> dir() 
['DATA', '__builtins__', '__doc__', '__file__', '__name__', '__package__', 'checkers', 'get_manhole_factory', 'manhole', 'manhole_ssh', 'portal', 'reactor'] 
>>> DATA 
{'Service': 'Manhole'}
>>> DATA['Service'] = "Edited" 
>>> DATA 
{'Service': 'Edited'}
[lalit : ~]₹ ssh admin@localhost -p 2222
admin@localhost's password: 
>>> dir() 
['DATA', '__builtins__', '__doc__', '__file__', '__name__', '__package__', 'checkers', 'get_manhole_factory', 'manhole', 'manhole_ssh', 'portal', 'reactor'] 
>>> DATA 
{'Service': 'Edited'} 

Here In the first login, we change the value in DATA dictionary in running application, as we can see we get the new value in the second login.

Simple port scanner in python

a port scanner is an application designed to probe a server or host for open ports. Such an application may be used by administrators to verify the security policies of their networks and by attackers to identify network services running on a host and exploit vulnerabilities.

port-scanner.py

#!/usr/bin/env python2
from socket import * 

if __name__ == '__main__':
    target = raw_input('Enter host to scan: ')
    targetIP = gethostbyname(target)
    print 'Starting scan on host ', targetIP

    #scan reserved ports
    for i in range(20, 1025):
        s = socket(AF_INET, SOCK_STREAM)

        result = s.connect_ex((targetIP, i))

        if(result == 0) :
            print 'Port %d: OPEN' % (i,)
        s.close()

Example

Howto use ssh as VPN tunnel

SSH is typically used to log into a remote machine and execute commands, but it also supports tunneling, forwarding TCP ports and X11 connections.

What is SSH Tunneling?

A tunneling protocol may, for example, allow a foreign protocol to run over a network that does not support that particular protocol, such as running IPv6 over IPv4.

SSH tunneling is a method of transporting arbitrary networking data over an encrypted SSH connection. It can be used to add encryption to legacy applications. … It also provides a way to secure the data traffic of any given application using port forwarding, basically tunneling any TCP/IP port over SSH.

sshuttle

sshuttle is not exactly a VPN, and not exactly port forwarding. It’s kind of both, and kind of neither.

It’s like a VPN, since it can forward every port on an entire network, not just ports you specify. Conveniently, it lets you use the “real” IP addresses of each host rather than faking port numbers on localhost.

On the other hand, the way it works is more like ssh port forwarding than a VPN. Normally, a VPN forwards your data one packet at a time, and doesn’t care about individual connections; ie. it’s “stateless” with respect to the traffic. sshuttle is the opposite of stateless; it tracks every single connection.

Installation

“` sudo pip install sshuttle“`

Example

“`$ sshuttle –dns -v -r <remote-host> 0/0“`

* This will forward all connections including DNS requests…

Usage

usage: sshuttle [-l [ip:]port] [-r [user@]sshserver[:port]] <subnets...>

positional arguments:
 IP/MASK[:PORT[-PORT]]...
 capture and forward traffic to these subnets
 (whitespace separated)

optional arguments:
 -h, --help show this help message and exit
 -l [IP:]PORT, --listen [IP:]PORT
 transproxy to this ip address and port number
 -H, --auto-hosts continuously scan for remote hostnames and update
 local /etc/hosts as they are found
 -N, --auto-nets automatically determine subnets to route
 --dns capture local DNS requests and forward to the remote
 DNS server
 --ns-hosts IP[,IP] capture and forward DNS requests made to the following
 servers
 --to-ns IP[:PORT] the DNS server to forward requests to; defaults to
 servers in /etc/resolv.conf on remote side if not
 given.
 --method TYPE auto, nat, nft, tproxy, pf, ipfw
 --python PATH path to python interpreter on the remote server
 -r [USERNAME@]ADDR[:PORT], --remote [USERNAME@]ADDR[:PORT]
 ssh hostname (and optional username) of remote
 sshuttle server
 -x IP/MASK[:PORT[-PORT]], --exclude IP/MASK[:PORT[-PORT]]
 exclude this subnet (can be used more than once)
 -X PATH, --exclude-from PATH
 exclude the subnets in a file (whitespace separated)
 -v, --verbose increase debug message verbosity
 -V, --version print the sshuttle version number and exit
 -e CMD, --ssh-cmd CMD
 the command to use to connect to the remote [ssh]
 --seed-hosts HOSTNAME[,HOSTNAME]
 comma-separated list of hostnames for initial scan
 (may be used with or without --auto-hosts)
 --no-latency-control sacrifice latency to improve bandwidth benchmarks
 --wrap NUM restart counting channel numbers after this number
 (for testing)
 --disable-ipv6 disable IPv6 support
 -D, --daemon run in the background as a daemon
 -s PATH, --subnets PATH
 file where the subnets are stored, instead of on the
 command line
 --syslog send log messages to syslog (default if you use
 --daemon)
 --pidfile PATH pidfile name (only if using --daemon) [./sshuttle.pid]
 --user USER apply all the rules only to this linux user
 --firewall (internal use only)
 --hostwatch (internal use only)
 --no-sudo-pythonpath do not set PYTHONPATH when invoking sudo

Still Confused With SMTP, IMAP, POP Ports?

Configuring SMTP, IMAP and POP ports can be confusing. Users and sometimes even system administrators aren’t sure when to use port 25, 587, or 465.

This article will clarify all ports related to the mail server.

SMTP 25
SMTP-SSL/TLS 587,465
IMAP 143
IMAP-SSL/TLS 993
POP3 110
POP3-SSL/TLS 995

IMAP uses port 143, but SSL/TLS encrypted IMAP uses port 993.

POP uses port 110, but SSL/TLS encrypted POP uses port 995.

SMTP uses port 25, but SSL/TLS encrypted SMTP uses port 465.

587 vs. 465
These port assignments are specified by the Internet Assigned Numbers Authority (IANA):

Port 587: [SMTP] Message submission (SMTP-MSA), a service that accepts submission of email from email clients (MUAs). Described in RFC 6409.
Port 465: URL Rendezvous Directory for SSM (entirely unrelated to email)
Historically, port 465 was initially planned for the SMTPS encryption and authentication “wrapper” over SMTP, but it was quickly deprecated (within months, and over 15 years ago) in favor of STARTTLS over SMTP (RFC 3207). Despite that fact, there are probably many servers that support the deprecated protocol wrapper, primarily to support older clients that implemented SMTPS. Unless you need to support such older clients, SMTPS and its use on port 465 should remain nothing more than a historical footnote.

Howto reverse proxy in nginx

Proxying is typically used to distribute the load among several servers, seamlessly show content from different websites, or pass requests for processing to application servers over protocols other than HTTP.

When NGINX proxies a request, it sends the request to a specified proxied server, fetches the response, and sends it back to the client. It is possible to proxy requests to an HTTP server (another NGINX server or any other server) or a non-HTTP server (which can run an application developed with a specific framework, such as PHP or Python) using a specified protocol.

1. To pass a request to an HTTP proxied server, the proxy_pass directive is specified inside a location. For example:

location /some/path/ {
proxy_pass http://www.example.com/link/;
}

 2. This address can be specified as a domain name or an IP address. The address may also include a port:

location ~ \.php {
    proxy_pass http://127.0.0.1:8000;
}

3. To pass a request to a non-HTTP proxied server, the appropriate **_pass directive should be used:

  • fastcgi_pass passes a request to a FastCGI server
  • uwsgi_pass passes a request to a uwsgi server
  • scgi_pass passes a request to an SCGI server
  • memcached_pass passes a request to a memcached server

4. Passing Request Headers

location /some/path/ {
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_pass http://localhost:8000;
}

 

5. To disable buffering in a specific location, place the proxy_buffering directive in the location with the off parameter, as follows:

location /some/path/ {
    proxy_buffering off;
    proxy_pass http://localhost:8000;
}

 

 

Openvas installation in CentOS 7

What is Openvas?

OpenVAS (Open Vulnerability Assessment System, originally known as GNessUs) is a software framework of several services and tools offering vulnerability scanning and vulnerability management.

All OpenVAS products are free software, and most components are licensed under the GNU General Public License (GPL). Plugins for OpenVAS are written in the Nessus Attack Scripting Language, NASL.

The primary reason to use this scan type is to perform comprehensive security testing of an IP address. It will initially perform a port scan of an IP address to find open services. Once listening services are discovered they are then tested for known vulnerabilities and misconfiguration using a large database (more than 53000 NVT checks). The results are then compiled into a report with detailed information regarding each vulnerability and notable issues discovered.

Once you receive the results of the tests, you will need to check each finding for relevance and possibly false positives. Any confirmed vulnerabilities should be re-mediated to ensure your systems are not at risk.

Vulnerability scans performed from externally hosted servers give you the same perspective as an attacker. This has the advantage of understanding exactly what is exposed on external-facing services.

Step 1: Disable SELinux

“`

sed -i ‘s/=enforcing/=disabled/’ /etc/selinux/config

“`

and reboot the machine.

Step 2:  Install dependencies

“`

yum -y install wget rsync curl net-tools

“`

Step 3: Install OpenVAS repository

install the official repository so that OpenVAS works appropriately in the analysis of vulnerabilities.

“`

wget -q -O – http://www.atomicorp.com/installers/atomic |sh

“`

Step 4: Install OpenVAS

“`

yum -y install openvas

“`

Step 5: Run OpenVAS

Once OpenVAS is installed, we continue to start it by executing the following command:

“`

openvas-setup

“`

Once downloaded it will be necessary to configure the GSAD IP address, Greenbone Security Assistant, which is a web interface to manage system scans.

Step 6: Configure OpenVAS Connectivity

We go to our browser and enter the IP address of the CentOS 7 server where we have installed OpenVAS, and we will see that the following message is displayed:

Openvas dashboard

 

Automatic NVT Updates With Cron

“`

35 1 * * * /usr/sbin/greenbone-nvt-sync > /dev/null
5 0 * * * /usr/sbin/greenbone-scapdata-sync > /dev/null
5 1 * * * /usr/sbin/greenbone-certdata-sync > /dev/null

“`