I'm making a simple multi-threaded port scanner. It scans all ports on host and returns open ports. The trouble is interrupting the scan. It take a lot of time for a scan to complete and sometimes I wish to kill program with C-c while in the middle of scan. Trouble is the scan won't stop. Main thread is locked on queue.join() and oblivious to KeyboardInterrupt, until all data from queue is processed thus deblocking main thread and exiting program gracefully. All my threads are daemonized so when main thread dies they should die with him.
I tried using signal lib, no success. Overriding threading.Thread class and adding method for graceful termination didn't work... Main thread just won't receive KeyboardInterrupt while executing queue.join()
import threading, sys, Queue, socket
queue = Queue.Queue()
def scan(host):
while True:
port = queue.get()
if port > 999 and port % 1000 == 0:
print port
try:
#sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
#sock.settimeout(2) #you need timeout or else it will try to connect forever!
#sock.connect((host, port))
#----OR----
sock = socket.create_connection((host, port), timeout = 2)
sock.send('aaa')
data = sock.recv(100)
print "Port {} open, message: {}".format(port, data)
sock.shutdown()
sock.close()
queue.task_done()
except:
queue.task_done()
def main(host):
#populate queue
for i in range(1, 65536):
queue.put(i)
#spawn worker threads
for port in range(100):
t = threading.Thread(target = scan, args = (host,))
t.daemon = True
t.start()
if __name__ == '__main__':
host = ""
#does input exist?
try:
host = sys.argv[1]
except:
print "No argument was recivied!"
exit(1)
#is input sane?
try:
host = socket.gethostbyname(host)
except:
print "Adress does not exist"
exit(2)
#execute main program and wait for scan to complete
main(host)
print "Post main() call!"
try:
queue.join()
except KeyboardInterrupt:
print "C-C"
exit(3)
EDIT:
I have found a solution by using time module.
#execute main program and wait for scan to complete
main(host)
#a little trick. queue.join() makes main thread immune to keyboardinterrupt. So use queue.empty() with time.sleep()
#queue.empty() is "unreliable" so it may return True a bit earlier then intented.
#when queue is true, queue.join() is executed, to confirm that all data was processed.
#not a true solution, you can't interrupt main thread near the end of scan (when queue.empty() returns True)
try:
while True:
if queue.empty() == False:
time.sleep(1)
else:
break
except KeyboardInterrupt:
print "Alas poor port scanner..."
exit(1)
queue.join()
You made your threads daemons already, but you need to keep your main thread alive while daemon threads are there, there's how to do that: Cannot kill Python script with Ctrl-C
When you create the threads add them to a list of running threads and when dealing with ctrl-C send a kill signal to each thread on the list. That way you are actively cleaning up rather than relying on it being done for you.
Related
I have a program which runs on 2 threads. The main thread is for its own work and the other thread keeps calling recv() on a UDP socket.
Basically, the code structure looks like this:
done = False
def run_sock():
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.bind(('localhost', 12345))
while not done: # receive data until work done
data = sock.recv(1500)
print(data)
sock.close()
thread = threading.Thread(target=run_sock, daemon=True)
thread.start()
# Main thread
while not done:
... # Do work here
if some_condition: # Stop running, thread should as well
done = True
thread.join()
I want to close the socket when the main thread changes done to True, but when that happens, the socket is still in its current blocking recv call and it has to receive another message before it finally stops.
Is there a way to gracefully close the socket (without having to handle errors)? I've tried sock.shutdown(socket.SHUT_RDWR), sock.setblocking(False) and but they all raise errors.
So sock.recv(1500) will block until it receives something. If it receives nothing, then it waits.
But if you set a timeout then periodically that wait will throw an exception and you can do other stuff (like look at the done flag) before trying to read again.
sock.settimeout(1.0)
sock.bind(...)
while not done:
try:
data = sock.recv(1500)
except timeout:
continue
sock.close()
Of course, if the remote end closes the connection that is different. Then you need to look at data to see if it is empty.
while not done:
try:
data = sock.recv(1500)
if not data:
break
except timeout:
continue
I am using socket in this code to connect with other machine.I want to terminate thread when i get message from other machine but how to terminate Thread in Python ?
I refer Many SO Questions and I found that there is no method in python to Close thread.Can anyone tell me the alternate way to close the thread ?
code:
from threading import Thread
import time
import socket
def background(arg):
global thread
thread = Thread(target=arg)
thread.start()
def display():
for i in range(0,20):
print(i)
time.sleep(5)
background(display)
s = socket.socket()
s.bind((ip,6500))
s.listen(5)
print("listening")
val,addr = s.accept()
cmd = val.recv(1024)
if cmd == "Terminate Process":
print("Connected")
thread.close()
print("Process Closed")
Error:
AttributeError: 'Thread' object has no attribute 'close'
Short answer:
thread.join()
The rule of thumb is: don't kill threads (note that in some environments this may not even be possible, e.g. standard C++11 threads). Let the thread fetch the information and terminate itself. Controlling threads from other threads leads to hard to maintain and debug code.
E.g.
SHOULD_TERMINATE = False
def display():
for i in range(0,20):
print(i)
time.sleep(5)
if SHOULD_TERMINATE:
return
thread = Thread(target=display)
thread.start()
// some other code
if cmd == "Terminate Process":
SHOULD_TERMINATE = True
thread.join()
This is of course heavily simplified. Your code can be further refined with event objects (instead of .sleep) or thread pools.
I am developing a server (daemon).
The server has one "worker thread". The worker thread runs a queue of commands. When the queue is empty, the worker thread is paused (but does not exit, because it should preserve certain state in memory). To have exactly one copy of the state in memory, I need to run all time exactly one (not several and not zero) worker thread.
Requests are added to the end of this queue when a client connects to a Unix socket and sends a command.
After the command is issued, it is added to the queue of commands of the worker thread. After it is added to the queue, the server replies something like "OK". There should be not a long pause between server receiving a command and it "OK" reply. However, running commands in the queue may take some time.
The main "work" of the worker thread is split into small (taking relatively little time) chunks. Between chunks, the worker thread inspects ("eats" and empties) the queue and continues to work based on the data extracted from the queue.
How to implement this server/daemon in Python?
This is a sample code with internet sockets, easily replaced with unix domain sockets. It takes whatever you write to the socket, passes it as a "command" to worker, responds OK as soon as it has queued the command. The single worker simulates a lengthy task with sleep(30). You can queue as many tasks as you want, receive OK immediately and every 30 seconds, your worker prints a command from the queue.
import Queue, threading, socket
from time import sleep
class worker(threading.Thread):
def __init__(self,q):
super(worker,self).__init__()
self.qu = q
def run(self):
while True:
new_task=self.qu.get(True)
print new_task
i=0
while i < 10:
print "working ..."
sleep(1)
i += 1
try:
another_task=self.qu.get(False)
print another_task
except Queue.Empty:
pass
task_queue = Queue.Queue()
w = worker(task_queue)
w.daemon = True
w.start()
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.bind(('localhost', 4200))
sock.listen(1)
try:
while True:
conn, addr = sock.accept()
data = conn.recv(32)
task_queue.put(data)
conn.sendall("OK")
conn.close()
except:
sock.close()
Yes, yes I know I could just use nmap but I want to try this out myself.
I'm trying to write a threaded script to find open ports on a target IP address. This is what I have right now:
import socket, Queue
from threading import Thread
print "Target to scan: "
targetIP = raw_input("> ")
print "Number of threads: "
threads = int(raw_input("> "))
q = Queue.Queue()
# Fill queue with port numbers
for port in range(1, 1025):
q.put(port)
def scan(targetIP, port):
try:
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.settimeout(4)
result = s.connect_ex((targetIP, port))
if result == 0:
print 'Port {0} is open'.format(port)
s.close
q.task_done()
while q.full:
for i in range(threads):
port = q.get()
t = Thread(target=scan, args =(targetIP, port))
t.daemon = True
t.start()
However I have a few issues:
1) When I run this as is, it will iterate through the port queue but then just hang, never breaking from the while loop even though the queue empties.
2) If I add a print line to scan to see whats happening, basically add a "Scanning port X" line in the beginning and a print result line at the end, stdout gets flooded with the "Scanning port" line for all ports in the queue, and THEN the result lines get printed. Meaning, it looks like currently the script is not waiting for result to get a value, and just continue iterating on as if it had.
What am I doing wrong here?
Your actual question has already been answered by a few people, so here's an alternative solution with multiprocessing.Pool instead of threading:
import socket
from multiprocessing import Pool
def scan(arg):
target_ip, port = arg
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.settimeout(2)
try:
sock.connect((target_ip, port))
sock.close()
return port, True
except (socket.timeout, socket.error):
return port, False
if __name__ == '__main__':
target_ip = raw_input('Target IP: ')
num_procs = int(raw_input('Number of processes: '))
ports = range(1, 1025)
pool = Pool(processes=num_procs)
for port, status in pool.imap_unordered(scan, [(target_ip, port) for port in ports]):
print port, 'is', 'open' if status else 'closed'
You have several problems here, the first is:
while q.full:
Presumably you meant to call the function:
while q.full():
But you have an infinite queue (you created it with no maxsize), so it's never full; so if you make that change, it won't call scan() at all.
Assuming you fix this in some other way (e.g., using q.empty()), what happens if range(threads) does not evenly divide the items in the queue? For instance, suppose you use 3 threads and put port numbers 1, 2, 3, and 4 into q. You'll call q.get() three times (getting 1, 2, and 3) in the first trip through the outer while, and then call it three times again in the second trip—but it only has one more value in it, 4, so the call to q.get() after that will wait for someone to execute a q.put(), and you will get stuck.
You need to rewrite the logic, in other words.
Edit: same problem with s.close vs s.close(). Others addressed the whole pool-of-threads aspect. #Blender's version, using multiprocessing, is a lot simpler since multiprocessing takes care of that for you.
There are a few issues with your code. First, the while loop continues until q.full, which is a function, is falsy. But actually there's no need to loop in your main thread.
I would add sentinel values to the end of the queue, one per worker thread. When the worker thread gets a sentinel, it quits its processing loop. This way you don't have to daemonize the Threads.
So you code should be like:
put ports into queue
put sentinels into queue
start the desired number of threads, have them take ports from the queue and process them, put the results in another queue
wait for the threads to terminate, calling t.join() on the workers
use the results
Well you have to know that by just iterating through the numbers in the range of the number of threads and executing the thread, you don't keep the number of desired threads. It just loops 4 times, creates 4 threads, loops again and enters the another same loop and creates another 4 without being sure that those 4 have finish their task, so you get that flood of messages when you put prints in scan function.
You would have to wait for the childs to finish at the end of the while body section.
I think:
threading.wait()
does the thing.
Try this:
import socket
import threading
from queue import Queue
print_lock = threading.Lock()
target = 'pythonprogramming.net'
def portscan(port):
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
try:
con = s.connect((target,port))
with print_lock:
print('port',port,'is open!')
con.close()
except:
pass
def threader():
while True:
worker = q.get()
portscan(worker)
q.task_done()
q = Queue()
for x in range(30):
t = threading.Thread(target=threader)
t.daemon = True
t.start()
for worker in range(1,10000):
q.put(worker)
q.join()
I am trying to implement a simpley portscanner with Python. It works by creating a number of worker threads which scan ports that are provided in a queue. They save the results in another queue. When all ports are scanned the threads and the application should terminate. And here lies the problem: For small numbers of ports everything works fine, but if I try to scan 200 or more ports, the application will get caught in a deadlock. I have no idea, why.
class ConnectScan(threading.Thread):
def __init__(self, to_scan, scanned):
threading.Thread.__init__(self)
self.to_scan = to_scan
self.scanned = scanned
def run(self):
while True:
try:
host, port = self.to_scan.get()
except Queue.Empty:
break
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
try:
s.connect((host, port))
s.close()
self.scanned.put((host, port, 'open'))
except socket.error:
self.scanned.put((host, port, 'closed'))
self.to_scan.task_done()
class ConnectScanner(object):
def scan(self, host, port_from, port_to):
to_scan = Queue.Queue()
scanned = Queue.Queue()
for port in range(port_from, port_to + 1):
to_scan.put((host, port))
for i in range(20):
ConnectScan(to_scan, scanned).start()
to_scan.join()
Does anybody see what might be wrong? Also I would appreciate some tipps how to debug such threading issues in Python.
I don't see anything obviously wrong with your code, but as it stands the break will never be hit - self.to_scan.get() will wait forever rather than raising Queue.Empty. Given that you're loading up the queue with ports to scan before starting the threads, you can change that to self.to_scan.get(False) to have the worker threads exit correctly when all the ports have been claimed.
Combined with the fact that you have non-daemon threads (threads that will keep the process alive after the main thread finishes), that could be the cause of the hang. Try printing something after the to_scan.join() to see whether it's stopped there, or at process exit.
As Ray says, if an exception other than socket.error is raised between self.to_scan.get() and self.to_scan.task_done(), then the join call will hang. It could help to change that code to use a try/finally to be sure:
def run(self):
while True:
try:
host, port = self.to_scan.get(False)
except Queue.Empty:
break
try:
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
try:
s.connect((host, port))
s.close()
self.scanned.put((host, port, 'open'))
except socket.error:
self.scanned.put((host, port, 'closed'))
finally:
self.to_scan.task_done()
In general, debugging multithreaded processes is tricky. I try to avoid anything blocking indefinitely - it's better to have something crash noisily because a timeout was too short than to have it just stop forever waiting for an item that will never appear. So I'd specify timeouts for your self.to_scan.get, socket.connect and to_scan.join calls.
Use logging to work out the order events are occurring - printing can get interleaved from different threads, but loggers are thread-safe.
Also, something like this recipe can be handy for dumping the current stack trace for each thread.
I haven't used any debuggers with support for debugging multiple threads in Python, but there are some listed here.
It is likely that not all items on the to_scan queue are consumed and that you're not calling the task_done method enough times to unblock ConnectScanner.
Could it be that an exception is thrown during the runtime of ConnectScan.run that you're not catching and your threads prematurely terminate?