Develop Reference

How to feed information to a Python daemon?

I have a Python daemon running on a Linux system.
I would like to feed information such as "Bob", "Alice", etc. and have the daemon print "Hello Bob." and "Hello Alice" to a file.
This has to be asynchronous. The Python daemon has to wait for information and print it whenever it receives something.
What would be the best way to achieve this?
I was thinking about a named pipe or the Queue library but there could be better solutions.

Here is how you can do it with a fifo:
# receiver.py
import os
import sys
import atexit
# Set up the FIFO
thefifo = 'comms.fifo'
os.mkfifo(thefifo)
# Make sure to clean up after ourselves
def cleanup():
os.remove(thefifo)
atexit.register(cleanup)
# Go into reading loop
while True:
with open(thefifo, 'r') as fifo:
for line in fifo:
print "Hello", line.strip()
You can use it like this from a shell session
$ python receiver.py &
$ echo "Alice" >> comms.fifo
Hello Alice
$ echo "Bob" >> comms.fifo
Hello Bob

There are several options
1) If the daemon should accept messages from other systems, make the daemon an RPC server - Use xmlrpc/jsonrpc.
2) If it is all local, you can use either TCP sockets or Named PIPEs.
3) If there will be a huge set of clients connecting concurrently, you can use select.epoll.

python has a built-in rpc library (using xml for data encoding). the documentation is well written; there is a complete example there:
https://docs.python.org/2.7/library/xmlrpclib.html
(python 2.7) or
https://docs.python.org/3.3/library/xmlrpc.server.html#module-xmlrpc.server
(python 3.3)
that may be worth considering.

Everyone mentioned FIFO-s (that's named pipes in Linux terminology) and XML-RPC, but if you learning these things right now, you have to check TCP/UDP/Unix sockets as well, since they are platform independent (at least, TCP/UDP sockets are). You can check this tutorial for a working example or the Python documentation if you want to go deper in this direction. It's also useful since most of the modern communication platforms (XML-RPC, SOAP, REST) uses these basic things.

There are a few mechanisms you could use, but everything boils down to using IPC (inter-process communication).
Now, the actual mechanism you will use depends on the details of what you can achieve, a good solution though would be to use something like zmq.
Check the following example on pub/sub on zmq
http://learning-0mq-with-pyzmq.readthedocs.org/en/latest/pyzmq/patterns/pubsub.html
also this
http://learning-0mq-with-pyzmq.readthedocs.org/en/latest/pyzmq/multisocket/zmqpoller.html
for the non-blocking way.

I'm not good in python so I would like to share
**Universal Inter process communcation **
nc a.k.a netcat is a server client model program which allow to send data such as text,files over network.
Advantages of nc
Very easy to use
IPC even between different programming langauges
Inbuilt on most linux OS
Example
On deamon
nc -l 1234 > output.txt
From other program or shell/terminal/script
echo HELLO | nc 127.0.0.1 1234
nc can be python by using the system command calling function ( may be os.system ) and read the stdout.

Why not use signals?
I am not a python programmer but presumably you can register a signal handler within your daemon and then signal it from the terminal. Just use SIGUSR or SIGHUP or similar.
This is the usual method you use to rotate logfiles or similar.

One solution could be to use the asynchat library which simplify calls between a server and a client.
Here is an example you could use (adapted from this site)
In deamon.py, a ChatServer object is created. Each time a connection is done, a ChatHandler object is created, inherited from asynchat.async_chat. This object collects data and fills it in self.buffer.
When a special string call the terminator is encountered, data is supposed to be complete and method found_terminator is called. It is in this method that you write your own code.
In sender.py, you create a ChatClient object, inherited from asynchat.async_chat, setup the connection in the constructor, define the terminator (in case the server answers !) and call the push method to send your data. You must append your terminator string to your data for the server to know when it can stop reading data.
daemon.py :
import asynchat
import asyncore
import socket
# Terminator string can be changed here
TERMINATOR = '\n'
class ChatHandler(asynchat.async_chat):
def __init__(self, sock):
asynchat.async_chat.__init__(self, sock=sock)
self.set_terminator(TERMINATOR)
self.buffer = []
def collect_incoming_data(self, data):
self.buffer.append(data)
def found_terminator(self):
msg = ''.join(self.buffer)
# Change here what the daemon is supposed to do when a message is retrieved
print 'Hello', msg
self.buffer = []
class ChatServer(asyncore.dispatcher):
def __init__(self, host, port):
asyncore.dispatcher.__init__(self)
self.create_socket(socket.AF_INET, socket.SOCK_STREAM)
self.bind((host, port))
self.listen(5)
def handle_accept(self):
pair = self.accept()
if pair is not None:
sock, addr = pair
print 'Incoming connection from %s' % repr(addr)
handler = ChatHandler(sock)
server = ChatServer('localhost', 5050)
print 'Serving on localhost:5050'
asyncore.loop()
sender.py :
import asynchat
import asyncore
import socket
import threading
# Terminator string can be changed here
TERMINATOR = '\n'
class ChatClient(asynchat.async_chat):
def __init__(self, host, port):
asynchat.async_chat.__init__(self)
self.create_socket(socket.AF_INET, socket.SOCK_STREAM)
self.connect((host, port))
self.set_terminator(TERMINATOR)
self.buffer = []
def collect_incoming_data(self, data):
pass
def found_terminator(self):
pass
client = ChatClient('localhost', 5050)
# Data sent from here
client.push("Bob" + TERMINATOR)
client.push("Alice" + TERMINATOR)

Reading from pipe in python is imposiible

Hello I have the following code in python 2.6:
command = "tcpflow -c -i any port 5559"
port_sniffer = subprocess.Popen(command, stdout=subprocess.PIPE, bufsize=1, shell=True)
while True:
line = port_sniffer.stdout.readline()
#do some stuff with line
The purpose of this code is to sniff the traffic between two processes (A and B) that communicate on port 5559.
Now let me describe the different scenarios I am having:
1) Code above is not running:
A and B are communicating and i can see it clearly using logs and the linux command netstat -napl | grep 5559 shows that the processes are communicating on the desired port.
2) Code above is not running and I am sniffing by running tcpflow -c -i any port 5559 directly from shell:
I can see the communication on console clearly :-).
3) Code above is running: Proccesses can't communicate. netstat -napl | grep 5559 prints nothing and logs give out errors!!!
4) Code above is running in debug mode: I can't seem to be able to step after the line line = port_sniffer.stdout.readline()
I tried using an iterator instead of a while loop (not that it should matter but still I am pointing it out). I also tried different values for bufsize (none, 1, and 8).
Please help!!

So after a quick read through the docs I found these two sentences:
On Unix, if args is a string, the string is interpreted as the name or
path of the program to execute
and
The shell argument (which defaults to False) specifies whether to use
the shell as the program to execute. If shell is True, it is
recommended to pass args as a string rather than as a sequence.
Based on this, I would recommend recreating your command as a list:
command = ["tcpflow -c", "-i any port 5559"] #I don't know linux, so double check this line!!
The general idea is this (also from the docs):
If args is a sequence, the first item specifies the command string,
and any additional items will be treated as additional arguments to
the shell itself. That is to say, Popen does the equivalent of:
Popen(['/bin/sh', '-c', args[0], args[1], ...])
Additionally, it seems that to read from your process, you should use communicate(). So
while True:
line = port_sniffer.stdout.readline()
would become
while True:
line = port_sniffer.communicate()[0]
But keep in mind this note from the docs:
Note The data read is buffered in memory, so do not use this method if the data size is large or unlimited.

If I had to guess, I think the problem that you're having is that you aren't running your program as root. TCPFlow needs to be run as a privelaged user if you want to be able to sniff other people's traffic (otherwise that'd be a serious security vulnerability). I wrote the following programs and they worked just fine for your scenario
server.py
#!/usr/bin/python
import socket
s = socket.socket()
host = socket.gethostname()
port = 12345
s.bind((host,port))
s.listen(5)
while True:
c, addr = s.accept()
print 'Connection from', addr
c.send('Test string 1234')
c.recv(1024)
while x != 'q':
print "Received " + x
c.send('Blah')
x = c.recv(1024)
print "Closing connection"
c.close()
client.py
#!/usr/bin/python
import socket, sys
from time import sleep
from datetime import datetime
s = socket.socket()
host = socket.gethostname()
port = 12345
s.connect((host,port))
c = sys.stdin.read(1) # Type a char to send to initate the sending loop
while True:
s.send(str(datetime.now()))
s.sleep(3)
msg = s.recv(1024)
flow.py
#!/usr/bin/python
import subprocess
command = 'tcpflow -c -i any port 12345'
sniffer = subprocess.Popen(command, stdout=subprocess.PIPE, shell=True)
while True:
print sniffer.stdout.readline()

Check network connection from an IP address with Python

How can I check if there is still connection from a specific ip address using python.

ping the ip address
import os
#192.168.1.10 is the ip address
ret = os.system("ping -o -c 3 -W 3000 192.168.1.10")
if ret != 0:
print "pc still alive"
well in any case you really want to check for availability of incoming connection on the PC you are trying to connect you need to make a program that will receive the connection which is already out of the question.

As far as I understood the OP is looking for active connection FROM certain ip, meaning he wants to check locally if there is active connection exists. It looks like something along lines of netstat to me. There are several options:
You can use psutils as demonstrated in this post. You will want to cycle the active processes and query active connections.
You could use netstat.py - a clone of netstat by Jay Loden, Giampaolo Rodola' to do the job for you.
Added:
You can do something like that:
import psutil
def remote_ips():
'''
Returns the list of IPs for current active connections
'''
remote_ips = []
for process in psutil.process_iter():
try:
connections = process.get_connections(kind='inet')
except psutil.AccessDenied or psutil.NoSuchProcess:
pass
else:
for connection in connections:
if connection.remote_address and connection.remote_address[0] not in remote_ips:
remote_ips.append(connection.remote_address[0])
return remote_ips
def remote_ip_present(ip):
return ip in remote_ips()
This is how it works:
>>>remote_ips()
['192.168.1.50', '192.168.1.15', '192.168.1.52', '198.252.206.16', '198.252.206.17']
>>>remote_ip_present('192.168.1.52')
True
>>>remote_ip_present('10.1.1.1')
False

You can use socket library:
import socket
try:
socket.gethostbyaddr(your_ip_adrress)
except socket.herror:
print u"Unknown host"

if you are on Windows:
import os
ret = os.system("ping -n 3 1.1.1.1")
if ret != 0:
print("Ip address responding")
The -n argument is for how many times to ping it
if you are on LInux:
import os
ret = os.system("ping -c 3 1.1.1.1")
if ret != 0:
print("Ip address responding")

import os
address = "my_ip_address"
os.system('ping ' + address)

Continuous check for VPN Connectivity - Python

Is there any efficient way to check and report in a log file or on the console may be... when ever the VPN is disconnected?
import time
print time.asctime( time.localtime(time.time()) )
Can print the time but I do not know what is the code to recursively find whether the VPN is active or not. Pinging it in a while(1) would be a stupid way to check if the connection is active or not. Any way to achieve this?

This solution is system dependent, I do know that it works on Linux because I've done something similar, but not sure about Windows though. I don't know if you want a solution not involving ping, but I think this is a good solution.
import logging, os, time
PING_HOST='10.10.10.10' # some host on the other side of the VPN
while True:
retcode = os.system('ping -c 1 %s' % PING_HOST)
if retcode:
# perform action for lost connection
logging.warn('Lost visibility with %s' % PING_HOST)
time.sleep(10) # sleep 10 seconds
This works because ping returns a return code of 0 for success. All other return codes signify an error.

In case the IP of vpn changes, you can check if a tunnel has been established at all.
import psutil
import logging, os, time
import subprocess
import sys
procname = "yourprocess_name"
while True:
cmdout = subprocess.Popen(["ifconfig | grep tun"],stdout = subprocess.PIPE, shell=True).communicate()[0]
print "cmdout: "+str(cmdout)
time.sleep(2)
#-----
if "tun" in cmdout:
print "seems to be ok"
if not "tun" in cmdout:
# perform action for lost connection
print "killing "+str(procname)
for proc in psutil.process_iter():
# check whether the process name matches
print "Listing procname: "+str(proc.name())
if proc.name() == procname:
proc.kill()
sys.exit()

This method uses the NAME of the HOST "Connection-specific DNS Suffix" associated with your IP (Mostly the corporation's VPN):
import os
import platform
def check_ping():
hostname = "xyz.com" #hostname will be..Name under: "Connection-specific DNS Suffix" when you type "ipconfig" in cmd..
response = os.system("ping " + ("-n 1 " if platform.system().lower()=="windows" else "-c 1 ") + hostname)
# and then check the response...
if response == 0:
pingstatus = "Network Active: Connected"
else:
pingstatus = "Network Error: Not Connected"
return pingstatus
response = check_ping()
print(response)

Check if remote host is up in Python

How would I check if the remote host is up without having a port number? Is there any other way I could check other then using regular ping.
There is a possibility that the remote host might drop ping packets

This worked fine for me:
HOST_UP = True if os.system("ping -c 1 " + SOMEHOST) is 0 else False

A protocol-level PING is best, i.e., connecting to the server and interacting with it in a way that doesn't do real work. That's because it is the only real way to be sure that the service is up. An ICMP ECHO (a.k.a. ping) would only tell you that the other end's network interface is up, and even then might be blocked; FWIW, I have seen machines where all user processes were bricked but which could still be pinged. In these days of application servers, even getting a network connection might not be enough; what if the hosted app is down or otherwise non-functional? As I said, talking sweet-nothings to the actual service that you are interested in is the best, surest approach.

HOST_UP = True if os.system("ping -c 5 " + SOMEHOST.strip(";")) is 0 else False
to remove nasty script execution just add .strip(";")
-c 5
to increase the number of ping requests, if all pass than True
PS. Works only on Linux, on Windows always returns True

The best you can do is:
Try and connect on a known port (eg port 80 or 443 for HTTP or HTTPS); or
Ping the site. See Ping a site in Python?
Many sites block ICMP (the portocol used to ping sites) so you must know beforehand if the host in question has it enabled or not.
Connecting to a port tells you mixed information. It really depends on what you want to know. A port might be open but the site is effectively hung so you may get a false positive. A more stringent approach might involve using a HTTP library to execute a Web request against a site and see if you get back a response.
It really all depends on what you need to know.

Many firewalls are configured to drop ping packets without responding. In addition, some network adapters will respond to ICMP ping requests without input from the operating system network stack, which means the operating system might be down, but the host still responds to pings (usually you'll notice if you reboot the server, say, it'll start responding to pings some time before the OS actually comes up and other services start up).
The only way to be certain that a host is up is to actually try to connect to it via some well-known port (e.g. web server port 80).
Why do you need to know if the host is "up", maybe there's a better way to do it.

What about trying something that requires a RPC like a 'tasklist' command in conjunction with a ping?

I would use a port scanner. Original question states that you don't want to use a port. Then you need to specify which Protocol (Yes, this needs a port) you want to monitor: HTTP, VNC, SSH, etc. In case you want to monitor via ICMP you can use subprocess and control ping parameters, number of pings, timeout, size, etc.
import subprocess
try:
res = subprocess.Popen(['ping -t2 -c 4 110.10.0.254 &> /dev/null; echo $?'],shell=True,stdout=subprocess.PIPE,stderr=subprocess.PIPE)
out, err = res.communicate()
out = out.rstrip()
err = err.rstrip()
print 'general.connectivity() Out: ' + out
print 'general.connectivity() Err: ' + err
if(out == "0"):
print 'general.connectivity() Successful'
return True
print 'general.connectivity() Failed'
return False
except Exception,e:
print 'general.connectivity() Exception'
return False
In case you want a port scanner
import socket
from functools import partial
from multiprocessing import Pool
from multiprocessing.pool import ThreadPool
from errno import ECONNREFUSED
NUM_CORES = 4
def portscan(target,port):
try:
# Create Socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
socketTimeout = 5
s.settimeout(socketTimeout)
s.connect((target,port))
print('port_scanner.is_port_opened() ' + str(port) + " is opened")
return port
except socket.error as err:
if err.errno == ECONNREFUSED:
return False
# Wrapper function that calls portscanner
def scan_ports(server=None,port=None,portStart=None,portEnd=None,**kwargs):
p = Pool(NUM_CORES)
ping_host = partial(portscan, server)
if portStart and portStart:
return filter(bool, p.map(ping_host, range(portStart, portStart)))
else:
return filter(bool, p.map(ping_host, range(port, port+1)))
# Check if port is opened
def is_port_opened(server=None,port=None, **kwargs):
print('port_scanner.is_port_opened() Checking port...')
try:
# Add More proccesses in case we look in a range
pool = ThreadPool(processes=1)
try:
ports = list(scan_ports(server=server,port=int(port)))
print("port_scanner.is_port_opened() Port scanner done.")
if len(ports)!=0:
print('port_scanner.is_port_opened() ' + str(len(ports)) + " port(s) available.")
return True
else:
print('port_scanner.is_port_opened() port not opened: (' + port +')')
return False
except Exception, e:
raise
except Exception,e:
print e
raise