Yes, yes I know I could just use nmap but I want to try this out myself.
I'm trying to write a threaded script to find open ports on a target IP address. This is what I have right now:
import socket, Queue
from threading import Thread
print "Target to scan: "
targetIP = raw_input("> ")
print "Number of threads: "
threads = int(raw_input("> "))
q = Queue.Queue()
# Fill queue with port numbers
for port in range(1, 1025):
q.put(port)
def scan(targetIP, port):
try:
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.settimeout(4)
result = s.connect_ex((targetIP, port))
if result == 0:
print 'Port {0} is open'.format(port)
s.close
q.task_done()
while q.full:
for i in range(threads):
port = q.get()
t = Thread(target=scan, args =(targetIP, port))
t.daemon = True
t.start()
However I have a few issues:
1) When I run this as is, it will iterate through the port queue but then just hang, never breaking from the while loop even though the queue empties.
2) If I add a print line to scan to see whats happening, basically add a "Scanning port X" line in the beginning and a print result line at the end, stdout gets flooded with the "Scanning port" line for all ports in the queue, and THEN the result lines get printed. Meaning, it looks like currently the script is not waiting for result to get a value, and just continue iterating on as if it had.
What am I doing wrong here?
Your actual question has already been answered by a few people, so here's an alternative solution with multiprocessing.Pool instead of threading:
import socket
from multiprocessing import Pool
def scan(arg):
target_ip, port = arg
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.settimeout(2)
try:
sock.connect((target_ip, port))
sock.close()
return port, True
except (socket.timeout, socket.error):
return port, False
if __name__ == '__main__':
target_ip = raw_input('Target IP: ')
num_procs = int(raw_input('Number of processes: '))
ports = range(1, 1025)
pool = Pool(processes=num_procs)
for port, status in pool.imap_unordered(scan, [(target_ip, port) for port in ports]):
print port, 'is', 'open' if status else 'closed'
You have several problems here, the first is:
while q.full:
Presumably you meant to call the function:
while q.full():
But you have an infinite queue (you created it with no maxsize), so it's never full; so if you make that change, it won't call scan() at all.
Assuming you fix this in some other way (e.g., using q.empty()), what happens if range(threads) does not evenly divide the items in the queue? For instance, suppose you use 3 threads and put port numbers 1, 2, 3, and 4 into q. You'll call q.get() three times (getting 1, 2, and 3) in the first trip through the outer while, and then call it three times again in the second trip—but it only has one more value in it, 4, so the call to q.get() after that will wait for someone to execute a q.put(), and you will get stuck.
You need to rewrite the logic, in other words.
Edit: same problem with s.close vs s.close(). Others addressed the whole pool-of-threads aspect. #Blender's version, using multiprocessing, is a lot simpler since multiprocessing takes care of that for you.
There are a few issues with your code. First, the while loop continues until q.full, which is a function, is falsy. But actually there's no need to loop in your main thread.
I would add sentinel values to the end of the queue, one per worker thread. When the worker thread gets a sentinel, it quits its processing loop. This way you don't have to daemonize the Threads.
So you code should be like:
put ports into queue
put sentinels into queue
start the desired number of threads, have them take ports from the queue and process them, put the results in another queue
wait for the threads to terminate, calling t.join() on the workers
use the results
Well you have to know that by just iterating through the numbers in the range of the number of threads and executing the thread, you don't keep the number of desired threads. It just loops 4 times, creates 4 threads, loops again and enters the another same loop and creates another 4 without being sure that those 4 have finish their task, so you get that flood of messages when you put prints in scan function.
You would have to wait for the childs to finish at the end of the while body section.
I think:
threading.wait()
does the thing.
Try this:
import socket
import threading
from queue import Queue
print_lock = threading.Lock()
target = 'pythonprogramming.net'
def portscan(port):
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
try:
con = s.connect((target,port))
with print_lock:
print('port',port,'is open!')
con.close()
except:
pass
def threader():
while True:
worker = q.get()
portscan(worker)
q.task_done()
q = Queue()
for x in range(30):
t = threading.Thread(target=threader)
t.daemon = True
t.start()
for worker in range(1,10000):
q.put(worker)
q.join()
Related
I am developing a server (daemon).
The server has one "worker thread". The worker thread runs a queue of commands. When the queue is empty, the worker thread is paused (but does not exit, because it should preserve certain state in memory). To have exactly one copy of the state in memory, I need to run all time exactly one (not several and not zero) worker thread.
Requests are added to the end of this queue when a client connects to a Unix socket and sends a command.
After the command is issued, it is added to the queue of commands of the worker thread. After it is added to the queue, the server replies something like "OK". There should be not a long pause between server receiving a command and it "OK" reply. However, running commands in the queue may take some time.
The main "work" of the worker thread is split into small (taking relatively little time) chunks. Between chunks, the worker thread inspects ("eats" and empties) the queue and continues to work based on the data extracted from the queue.
How to implement this server/daemon in Python?
This is a sample code with internet sockets, easily replaced with unix domain sockets. It takes whatever you write to the socket, passes it as a "command" to worker, responds OK as soon as it has queued the command. The single worker simulates a lengthy task with sleep(30). You can queue as many tasks as you want, receive OK immediately and every 30 seconds, your worker prints a command from the queue.
import Queue, threading, socket
from time import sleep
class worker(threading.Thread):
def __init__(self,q):
super(worker,self).__init__()
self.qu = q
def run(self):
while True:
new_task=self.qu.get(True)
print new_task
i=0
while i < 10:
print "working ..."
sleep(1)
i += 1
try:
another_task=self.qu.get(False)
print another_task
except Queue.Empty:
pass
task_queue = Queue.Queue()
w = worker(task_queue)
w.daemon = True
w.start()
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.bind(('localhost', 4200))
sock.listen(1)
try:
while True:
conn, addr = sock.accept()
data = conn.recv(32)
task_queue.put(data)
conn.sendall("OK")
conn.close()
except:
sock.close()
I need a little help on my port scanner in Python. I have to learn how to add the threading module to it, but have no clue and do not really understand the few tutorials and help I've found. Here is a little test script:
from socket import *
remote = raw_input ("Website: ")
remote_ip = gethostbyname(remote)
print "scaning at:",remote_ip
for i in range (20,100):
s = socket(AF_INET,SOCK_STREAM)
result = s.connect_ex((remote,i))
if result == 0):
print "port %d: open"%(i)
s.close
Something like the below. You force each function into a separate thread and kick it off. You also keep a reference to each thread so you can block your main thread until they have all executed. You also need a lock because "print" in Python isn't thread safe and you'll end up getting the output from each thread mangled up otherwise (you can use the logger instead which is thread safe and neater). This will also generate a lot of threads as I haven't even bothered to show you pooling. This is just a rough and ready example to get you going on your journey to discover multi threading in Python.
print_lock = Lock()
def socket_test(address, port):
s = socket(AF_INET,SOCK_STREAM)
result = s.connect_ex((address,port))
if result == 0:
with print_lock:
print "port %s: open" % port
s.close()
def main():
remote = raw_input("Website: ")
remote_ip = gethostbyname(remote)
print "scaning at:",remote_ip
threads = []
for i in range(20, 100):
new_thread = threading.Thread(socket_test(remote_ip, i))
new_thread.start()
threads.append(new_thread)
[this_thread.join() for this_thread in threads]
I am trying to run two tcp clients from the same code using multithreading. The issue is that the second thread never runs. And main() never reaches the last 'Its here!' string print. I have the following code:
def main():
t = Thread(None,connect(),None,)
t2 = Thread(None,connect2(),None,)
t.start()
t2.start()
print "it's here!"
def connect_mktData():
# create Internet TCP socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
# connect to server
s.connect(('IP', PORT))
while(1):
print 'data1'
k = 'enter a letter1:'
s.send(k) # send k to server
v = s.recv(1024) # receive v from server (up to 1024 bytes)
print v
time.sleep(1)
s.close() # close socket
def connect_mktData2():
# create Internet TCP socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
# connect to server
s.connect(('IP', PORT))
while(1):
print 'data2'
# get letter
k = raw_input('enter a letter2:')
s.send(k) # send k to server
v = s.recv(1024) # receive v from server (up to 1024 bytes)
print v
time.sleep(1)
s.close() # close socket
main()
I get the following output:
data1
enter a letter1:
data1
enter a letter1:
data1
enter a letter1:
data1
enter a letter1:
data1
Even though both functions are mostly identical, ultimately I will have two different connections doing two different things 'simultaneously' and alternating between each other. Shouldn't both threads run independently? thanks for the help!
It looks like your issue is that this:
t = Thread(None,connect(),None,)
t2 = Thread(None,connect2(),None,)
Should be this:
t = Thread(None,connect,None,)
t2 = Thread(None,connect2,None,)
You want to pass the function objects connect and connect2 to the Thread object. When you use connect() instead of connect, you end up calling connect in the main thread, and then pass its return value to the Thread object, which isn't what you want.
Also, it is much more readable to create the Thread objects like this:
t = Thread(target=connect)
t2 = Thread(target=connect2)
Use the target kwarg, so you don't have to include the None for the group.
Also note that while this will make both functions run in concurrently, they will only truly being running at the same time while they're doing blocking I/O operations (meaning inside send, recv, or raw_input). Because of Python's Global Interpeter Lock (GIL), only one thread can be doing CPU-bound operations at a time. So your threads will end up doing a mixture of true concurrency (during I/O) and cooperative multitasking (during CPU-bound operations).
I am trying to write a program using which I wish to alternate between two threads, thread1 and thread2. The tricky part is that the thread should begin execution first must be thread1.
This is the code I have so far:
Class Client:
#member variables
def sendFile(self,cv1,lock1):
sent=0;
while (i<self.size):
message = self.data[i:1024+i]
cv1.acquire()
BadNet.transmit(self.clientSocket,message,self.serverIP,self.serverPort)
cv1.notify()
cv1.release()
i = i+1024
sent+=1
lock1.wait()
print "File sent successfully !"
self.clientSocket.close()
def receiveAck(self,cv1,lock2):
i=0
while (1):
lock1.clear()
cv1.acquire()
cv1.wait()
print "\nentered ack !\n"
self.ack, serverAddress = self.clientSocket.recvfrom(self.buf)
cv1.release()
lock1.set()
if __name__ == "__main__":
lock1 = Event()
cv1 = Condition()
cv2= Condition()
client = Client();
client.readFile();
thread1 = Thread(target = client.sendFile, args=[cv1,lock1])
thread2 = Thread(target = client.receiveAck, args=[cv1,lock1])
thread1.start()
thread2.start()
thread1.join()
thread2.join()
The problem I am currently facing is that initially the program does alternate between two threads (confirmed by the output on the console. But after an arbitrary number of iterations (usually between 20 and 80) the program just hangs and no further iterations are performed.
There are at least two problems with your synchronization.
First, you're using cv1 wrong. Your receive thread has to loop around its cv, checking the condition and calling wait each time. Otherwise, you're just using a cv as a broken event + lock combination. You don't have such a loop. More importantly, you don't even have a condition to wait for.
Second, you're using lock1 wrong. Your receive thread sets the event and then immediately clears it. But there's no guarantee that the send thread has gotten to the wait yet. (The race from the previous problem makes this more of a problem, but it's still a problem even if you fix that.) On a multi-core machine, it will usually get there in time, but "usually" is even worse than never in threaded programming. So, eventually the send thread will get to the wait after the receive thread has already done the clear, and therefore it will wait forever. The receive thread, meanwhile, will be waiting to be notified by the send thread, which will never happen. So you're deadlocked.
For future reference, adding print statements before and after every blocking operation, especially sync operations, would make this a lot to debug: you would see the receive thread's last message was "receive waiting on cv1", while the send thread's last message was "send waiting on lock1", and it would be obvious where the deadlock was.
Anyway, I'm not sure what it would even mean to "fix" a cv with no condition, or an event that you're trying to use as a cv, so instead I'll show how to write something sensible with two cvs. In this case, we might as well just use a flag that we flip back and forth as the condition for both cvs.
While I'm at it, I'll fix a couple other problems that made your code not even testable (e.g., i is never initialized), and include the debugging information, and what I had to fill in to make this a complete example, but otherwise I'll try to leave your structure and irrelevant problems (like Client being an old-style class) intact.
class Client:
def __init__(self):
self.clientSocket = socket(AF_INET, SOCK_DGRAM)
self.serverIP = '127.0.0.1'
self.serverPort = 11111
self.buf = 4
self.waitack = False
def readFile(self):
self.data = ', '.join(map(str, range(100000)))
self.size = len(self.data)
#member variables
def sendFile(self,cv1,lock1):
i = 0
sent=0
while (i<self.size):
message = self.data[i:1024+i]
print "s cv1 acquire"
with cv1:
print "s sendto"
self.clientSocket.sendto(message, (self.serverIP, self.serverPort))
self.waitack = True
print "s cv1 notify"
cv1.notify()
i = i+1024
sent+=1
print "s cv2 acquire"
with cv2:
print "s cv2 wait"
while self.waitack:
cv2.wait()
print "File sent successfully !"
self.clientSocket.close()
def receiveAck(self,cv1,lock2):
i=0
while (1):
print "r cv1 acquire"
with cv1:
while not self.waitack:
print "r cv1 wait"
cv1.wait()
print "r recvfrom"
self.ack, serverAddress = self.clientSocket.recvfrom(self.buf)
i += 1
print self.ack, i
print "r cv2 acquire"
with cv2:
self.waitack = False
print "r cv2 notify"
cv2.notify()
And here's a test server for it:
from itertools import *
from socket import *
s = socket(AF_INET, SOCK_DGRAM)
s.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1)
s.bind(('127.0.0.1', 11111))
for i in count():
data, addr = s.recvfrom(1024)
print(i)
s.sendto('ack\n', addr)
Start the server, start the client, the server will count up to 672, the client will count up to 673 (since your code counts 1-based) with 673 balanced pairs of messages and a "File sent successfully !" at the end. (Of course the client will then hang forever because receiveAck has no way to finish, and the server because I wrote it as an infinite loop.)
I'm making a simple multi-threaded port scanner. It scans all ports on host and returns open ports. The trouble is interrupting the scan. It take a lot of time for a scan to complete and sometimes I wish to kill program with C-c while in the middle of scan. Trouble is the scan won't stop. Main thread is locked on queue.join() and oblivious to KeyboardInterrupt, until all data from queue is processed thus deblocking main thread and exiting program gracefully. All my threads are daemonized so when main thread dies they should die with him.
I tried using signal lib, no success. Overriding threading.Thread class and adding method for graceful termination didn't work... Main thread just won't receive KeyboardInterrupt while executing queue.join()
import threading, sys, Queue, socket
queue = Queue.Queue()
def scan(host):
while True:
port = queue.get()
if port > 999 and port % 1000 == 0:
print port
try:
#sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
#sock.settimeout(2) #you need timeout or else it will try to connect forever!
#sock.connect((host, port))
#----OR----
sock = socket.create_connection((host, port), timeout = 2)
sock.send('aaa')
data = sock.recv(100)
print "Port {} open, message: {}".format(port, data)
sock.shutdown()
sock.close()
queue.task_done()
except:
queue.task_done()
def main(host):
#populate queue
for i in range(1, 65536):
queue.put(i)
#spawn worker threads
for port in range(100):
t = threading.Thread(target = scan, args = (host,))
t.daemon = True
t.start()
if __name__ == '__main__':
host = ""
#does input exist?
try:
host = sys.argv[1]
except:
print "No argument was recivied!"
exit(1)
#is input sane?
try:
host = socket.gethostbyname(host)
except:
print "Adress does not exist"
exit(2)
#execute main program and wait for scan to complete
main(host)
print "Post main() call!"
try:
queue.join()
except KeyboardInterrupt:
print "C-C"
exit(3)
EDIT:
I have found a solution by using time module.
#execute main program and wait for scan to complete
main(host)
#a little trick. queue.join() makes main thread immune to keyboardinterrupt. So use queue.empty() with time.sleep()
#queue.empty() is "unreliable" so it may return True a bit earlier then intented.
#when queue is true, queue.join() is executed, to confirm that all data was processed.
#not a true solution, you can't interrupt main thread near the end of scan (when queue.empty() returns True)
try:
while True:
if queue.empty() == False:
time.sleep(1)
else:
break
except KeyboardInterrupt:
print "Alas poor port scanner..."
exit(1)
queue.join()
You made your threads daemons already, but you need to keep your main thread alive while daemon threads are there, there's how to do that: Cannot kill Python script with Ctrl-C
When you create the threads add them to a list of running threads and when dealing with ctrl-C send a kill signal to each thread on the list. That way you are actively cleaning up rather than relying on it being done for you.