I am trying to run two tcp clients from the same code using multithreading. The issue is that the second thread never runs. And main() never reaches the last 'Its here!' string print. I have the following code:
def main():
t = Thread(None,connect(),None,)
t2 = Thread(None,connect2(),None,)
t.start()
t2.start()
print "it's here!"
def connect_mktData():
# create Internet TCP socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
# connect to server
s.connect(('IP', PORT))
while(1):
print 'data1'
k = 'enter a letter1:'
s.send(k) # send k to server
v = s.recv(1024) # receive v from server (up to 1024 bytes)
print v
time.sleep(1)
s.close() # close socket
def connect_mktData2():
# create Internet TCP socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
# connect to server
s.connect(('IP', PORT))
while(1):
print 'data2'
# get letter
k = raw_input('enter a letter2:')
s.send(k) # send k to server
v = s.recv(1024) # receive v from server (up to 1024 bytes)
print v
time.sleep(1)
s.close() # close socket
main()
I get the following output:
data1
enter a letter1:
data1
enter a letter1:
data1
enter a letter1:
data1
enter a letter1:
data1
Even though both functions are mostly identical, ultimately I will have two different connections doing two different things 'simultaneously' and alternating between each other. Shouldn't both threads run independently? thanks for the help!
It looks like your issue is that this:
t = Thread(None,connect(),None,)
t2 = Thread(None,connect2(),None,)
Should be this:
t = Thread(None,connect,None,)
t2 = Thread(None,connect2,None,)
You want to pass the function objects connect and connect2 to the Thread object. When you use connect() instead of connect, you end up calling connect in the main thread, and then pass its return value to the Thread object, which isn't what you want.
Also, it is much more readable to create the Thread objects like this:
t = Thread(target=connect)
t2 = Thread(target=connect2)
Use the target kwarg, so you don't have to include the None for the group.
Also note that while this will make both functions run in concurrently, they will only truly being running at the same time while they're doing blocking I/O operations (meaning inside send, recv, or raw_input). Because of Python's Global Interpeter Lock (GIL), only one thread can be doing CPU-bound operations at a time. So your threads will end up doing a mixture of true concurrency (during I/O) and cooperative multitasking (during CPU-bound operations).
Related
Hi i have an exercise to build with sockets select and msvcrt, server and clients of mltiplie chat(the server and the clients need to be built non-blocking) that every client will send message and the server will send the message to all the clients except the one who sent it, the server:
import socket
import select
IP = "192.168.1.154"
port = 123
default_buffer_size = 1024
open_client_sockets = []
messages_to_send = []
def send_waiting_messages(wlist):
for message in messages_to_send:
(client_sock, data) = message
if client_sock in wlist:
for sock in open_client_sockets:
if sock is not client_sock:
sock.send(data)
messages_to_send.remove(message)
def main():
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.bind((IP, port))
sock.listen(5)
print("The server is on and waiting for client...")
while True:
rlist, wlist, xlist = select.select([sock] + open_client_sockets, open_client_sockets, [])
for current_socket in rlist:
if current_socket is sock:
(new_socket, addr) = sock.accept()
open_client_sockets.append(new_socket)
else:
data = current_socket.recv(default_buffer_size)
if data == "":
open_client_sockets.remove(current_socket)
print("Connection with client closed")
else:
messages_to_send.append((current_socket, 'Hello ' + data))
send_waiting_messages(wlist)
if __name__ == '__main__':
main()
Building the server wasnt hard because it was guided(if it was not guided i would never got this code working) by the book but i have problem building the client and the main reason is that i dont understand how select.select works, couldn't find answer that will simplify enough this module for me.
this is what i did with the client:
import socket
import select
import msvcrt
IP = "192.168.1.154"
port = 123
sockets = []
def write():
pass
def main():
sock = socket.socket()
sock.connect((IP, port))
while True:
rlist, wlist, xlist = select.select(sockets, sockets, [])
for current_socket in rlist:
if current_socket is sock:
data = current_socket.recv(1024)
print(data)
else:
sockets.append(current_socket)
write()
if __name__ == '__main__':
main()
This probably shows you that I have low understanding of the module select and the exercise actually. I saw some threads that has similar question but I understand nothing from them so I realy need good explantion.
In conclusion I realy am lost...
select takes as parameters a list of sockets to wait for readablity, a list of sockets to wait for writability, and a list of sockets to wait for errors. It returns lists of ready to read, ready to write, and error sockets. From help:
>>> help(select.select)
Help on built-in function select in module select:
select(...)
select(rlist, wlist, xlist[, timeout]) -> (rlist, wlist, xlist)
Wait until one or more file descriptors are ready for some kind of I/O.
The first three arguments are sequences of file descriptors to be waited for:
rlist -- wait until ready for reading
wlist -- wait until ready for writing
xlist -- wait for an ``exceptional condition''
If only one kind of condition is required, pass [] for the other lists.
A file descriptor is either a socket or file object, or a small integer
gotten from a fileno() method call on one of those.
The optional 4th argument specifies a timeout in seconds; it may be
a floating point number to specify fractions of seconds. If it is absent
or None, the call will never time out.
The return value is a tuple of three lists corresponding to the first three
arguments; each contains the subset of the corresponding file descriptors
that are ready.
*** IMPORTANT NOTICE ***
On Windows, only sockets are supported; on Unix, all file
descriptors can be used.
So to fix your client, you need to add the socket you opened (sock) to the sockets list. Your write function can then be called if your socket is ready to be written.
In write, use msvcrt.kbhit() to test for characters typed. You can't just use input because it will block. Then read the character if one has been typed. Collect up the characters until you hit enter, then build a message and write it to the socket. Something like:
message = []
def write(sock):
if msvcrt.kbhit():
c = msvcrt.getche()
if c == '\r':
data = ''.join(message)
print 'sending:',data
sock.sendall(data)
message.clear()
else:
message.append(c)
In Python how do I fill a buffer with lines of data (strings) and consume it with a second process? There are ample of examples here adding and reading lines from a string, but I need to remove the consumed line from the string for the string to work as a buffer.
Example: read sporadic data from a serial port and send it via TCP/IP to a server. Line-by-line within one loop and no buffering = no problem, but in case the destination is unreachable the data should be stored in the buffer and then sent once connection is available.
#!/usr/bin/python
import serial
import socket
from multiprocessing import Process
ip = "someURL"
port = 12345
ser = serial.Serial("/dev/ttyUSB0", 57600, timeout=0)
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
def serial_reader():
while True:
for line in ser.read():
try:
response = ser.readlines(None)
response = str(response)
message = response[7:]
except:
print datetime.datetime.now(), " No data from serial connection."
##
def data_sender():
s.connect((ip, port))
while True:
for line in queue():
try:
s.send(message)
except:
try:
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((ip, port))
continue
except:
s.close()
##
if __name__ == '__main__':
Process(target=serial_reader).start()
Process(target=data_sender).start()
I think the best way to achieve what you want is to use a queue:
from multiprocessing import Queue
specifically use queue.put() to put a string on the queue, queue.get() to retrieve it, and queue.task_done() to indicate that the task is complete.
https://docs.python.org/2/library/queue.html#Queue.Queue
if you need a bigger gun take a look at RabbitMQ and python libraries that implement the AMPQ protocol such as rabbitpy. This is the defacto standard for inter process/inter service communication and has a lot of usefyl stuff already baked in, such as persisting messages in case the processes shut down, load balancing tasks across multiple processes, etc.
I am trying to write a program using which I wish to alternate between two threads, thread1 and thread2. The tricky part is that the thread should begin execution first must be thread1.
This is the code I have so far:
Class Client:
#member variables
def sendFile(self,cv1,lock1):
sent=0;
while (i<self.size):
message = self.data[i:1024+i]
cv1.acquire()
BadNet.transmit(self.clientSocket,message,self.serverIP,self.serverPort)
cv1.notify()
cv1.release()
i = i+1024
sent+=1
lock1.wait()
print "File sent successfully !"
self.clientSocket.close()
def receiveAck(self,cv1,lock2):
i=0
while (1):
lock1.clear()
cv1.acquire()
cv1.wait()
print "\nentered ack !\n"
self.ack, serverAddress = self.clientSocket.recvfrom(self.buf)
cv1.release()
lock1.set()
if __name__ == "__main__":
lock1 = Event()
cv1 = Condition()
cv2= Condition()
client = Client();
client.readFile();
thread1 = Thread(target = client.sendFile, args=[cv1,lock1])
thread2 = Thread(target = client.receiveAck, args=[cv1,lock1])
thread1.start()
thread2.start()
thread1.join()
thread2.join()
The problem I am currently facing is that initially the program does alternate between two threads (confirmed by the output on the console. But after an arbitrary number of iterations (usually between 20 and 80) the program just hangs and no further iterations are performed.
There are at least two problems with your synchronization.
First, you're using cv1 wrong. Your receive thread has to loop around its cv, checking the condition and calling wait each time. Otherwise, you're just using a cv as a broken event + lock combination. You don't have such a loop. More importantly, you don't even have a condition to wait for.
Second, you're using lock1 wrong. Your receive thread sets the event and then immediately clears it. But there's no guarantee that the send thread has gotten to the wait yet. (The race from the previous problem makes this more of a problem, but it's still a problem even if you fix that.) On a multi-core machine, it will usually get there in time, but "usually" is even worse than never in threaded programming. So, eventually the send thread will get to the wait after the receive thread has already done the clear, and therefore it will wait forever. The receive thread, meanwhile, will be waiting to be notified by the send thread, which will never happen. So you're deadlocked.
For future reference, adding print statements before and after every blocking operation, especially sync operations, would make this a lot to debug: you would see the receive thread's last message was "receive waiting on cv1", while the send thread's last message was "send waiting on lock1", and it would be obvious where the deadlock was.
Anyway, I'm not sure what it would even mean to "fix" a cv with no condition, or an event that you're trying to use as a cv, so instead I'll show how to write something sensible with two cvs. In this case, we might as well just use a flag that we flip back and forth as the condition for both cvs.
While I'm at it, I'll fix a couple other problems that made your code not even testable (e.g., i is never initialized), and include the debugging information, and what I had to fill in to make this a complete example, but otherwise I'll try to leave your structure and irrelevant problems (like Client being an old-style class) intact.
class Client:
def __init__(self):
self.clientSocket = socket(AF_INET, SOCK_DGRAM)
self.serverIP = '127.0.0.1'
self.serverPort = 11111
self.buf = 4
self.waitack = False
def readFile(self):
self.data = ', '.join(map(str, range(100000)))
self.size = len(self.data)
#member variables
def sendFile(self,cv1,lock1):
i = 0
sent=0
while (i<self.size):
message = self.data[i:1024+i]
print "s cv1 acquire"
with cv1:
print "s sendto"
self.clientSocket.sendto(message, (self.serverIP, self.serverPort))
self.waitack = True
print "s cv1 notify"
cv1.notify()
i = i+1024
sent+=1
print "s cv2 acquire"
with cv2:
print "s cv2 wait"
while self.waitack:
cv2.wait()
print "File sent successfully !"
self.clientSocket.close()
def receiveAck(self,cv1,lock2):
i=0
while (1):
print "r cv1 acquire"
with cv1:
while not self.waitack:
print "r cv1 wait"
cv1.wait()
print "r recvfrom"
self.ack, serverAddress = self.clientSocket.recvfrom(self.buf)
i += 1
print self.ack, i
print "r cv2 acquire"
with cv2:
self.waitack = False
print "r cv2 notify"
cv2.notify()
And here's a test server for it:
from itertools import *
from socket import *
s = socket(AF_INET, SOCK_DGRAM)
s.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1)
s.bind(('127.0.0.1', 11111))
for i in count():
data, addr = s.recvfrom(1024)
print(i)
s.sendto('ack\n', addr)
Start the server, start the client, the server will count up to 672, the client will count up to 673 (since your code counts 1-based) with 673 balanced pairs of messages and a "File sent successfully !" at the end. (Of course the client will then hang forever because receiveAck has no way to finish, and the server because I wrote it as an infinite loop.)
Yes, yes I know I could just use nmap but I want to try this out myself.
I'm trying to write a threaded script to find open ports on a target IP address. This is what I have right now:
import socket, Queue
from threading import Thread
print "Target to scan: "
targetIP = raw_input("> ")
print "Number of threads: "
threads = int(raw_input("> "))
q = Queue.Queue()
# Fill queue with port numbers
for port in range(1, 1025):
q.put(port)
def scan(targetIP, port):
try:
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.settimeout(4)
result = s.connect_ex((targetIP, port))
if result == 0:
print 'Port {0} is open'.format(port)
s.close
q.task_done()
while q.full:
for i in range(threads):
port = q.get()
t = Thread(target=scan, args =(targetIP, port))
t.daemon = True
t.start()
However I have a few issues:
1) When I run this as is, it will iterate through the port queue but then just hang, never breaking from the while loop even though the queue empties.
2) If I add a print line to scan to see whats happening, basically add a "Scanning port X" line in the beginning and a print result line at the end, stdout gets flooded with the "Scanning port" line for all ports in the queue, and THEN the result lines get printed. Meaning, it looks like currently the script is not waiting for result to get a value, and just continue iterating on as if it had.
What am I doing wrong here?
Your actual question has already been answered by a few people, so here's an alternative solution with multiprocessing.Pool instead of threading:
import socket
from multiprocessing import Pool
def scan(arg):
target_ip, port = arg
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.settimeout(2)
try:
sock.connect((target_ip, port))
sock.close()
return port, True
except (socket.timeout, socket.error):
return port, False
if __name__ == '__main__':
target_ip = raw_input('Target IP: ')
num_procs = int(raw_input('Number of processes: '))
ports = range(1, 1025)
pool = Pool(processes=num_procs)
for port, status in pool.imap_unordered(scan, [(target_ip, port) for port in ports]):
print port, 'is', 'open' if status else 'closed'
You have several problems here, the first is:
while q.full:
Presumably you meant to call the function:
while q.full():
But you have an infinite queue (you created it with no maxsize), so it's never full; so if you make that change, it won't call scan() at all.
Assuming you fix this in some other way (e.g., using q.empty()), what happens if range(threads) does not evenly divide the items in the queue? For instance, suppose you use 3 threads and put port numbers 1, 2, 3, and 4 into q. You'll call q.get() three times (getting 1, 2, and 3) in the first trip through the outer while, and then call it three times again in the second trip—but it only has one more value in it, 4, so the call to q.get() after that will wait for someone to execute a q.put(), and you will get stuck.
You need to rewrite the logic, in other words.
Edit: same problem with s.close vs s.close(). Others addressed the whole pool-of-threads aspect. #Blender's version, using multiprocessing, is a lot simpler since multiprocessing takes care of that for you.
There are a few issues with your code. First, the while loop continues until q.full, which is a function, is falsy. But actually there's no need to loop in your main thread.
I would add sentinel values to the end of the queue, one per worker thread. When the worker thread gets a sentinel, it quits its processing loop. This way you don't have to daemonize the Threads.
So you code should be like:
put ports into queue
put sentinels into queue
start the desired number of threads, have them take ports from the queue and process them, put the results in another queue
wait for the threads to terminate, calling t.join() on the workers
use the results
Well you have to know that by just iterating through the numbers in the range of the number of threads and executing the thread, you don't keep the number of desired threads. It just loops 4 times, creates 4 threads, loops again and enters the another same loop and creates another 4 without being sure that those 4 have finish their task, so you get that flood of messages when you put prints in scan function.
You would have to wait for the childs to finish at the end of the while body section.
I think:
threading.wait()
does the thing.
Try this:
import socket
import threading
from queue import Queue
print_lock = threading.Lock()
target = 'pythonprogramming.net'
def portscan(port):
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
try:
con = s.connect((target,port))
with print_lock:
print('port',port,'is open!')
con.close()
except:
pass
def threader():
while True:
worker = q.get()
portscan(worker)
q.task_done()
q = Queue()
for x in range(30):
t = threading.Thread(target=threader)
t.daemon = True
t.start()
for worker in range(1,10000):
q.put(worker)
q.join()
I made a better chat client following help from people:
They told me that if I didn't want to be blocked on .recv when waiting for messages, I would need to use threads, classes, functions, and queues to do so.
So I followed some help a specific person gave me where I created a thread from a class and then defined a function that was supposed to read incoming messages and print them.
I also created a function that allows you to enter stuff to be sent off.
Thing is, when I run the program. Nothing happens.
Can somebody help point out what is wrong? (I've asked questions and researched for 3 days, without getting anywhere, so I did try)
from socket import *
import threading
import json
import select
print("Client Version 3")
HOST = input("Connect to: ")
PORT = int(input("On port: "))
# Create Socket
s = socket(AF_INET,SOCK_STREAM)
s.connect((HOST,PORT))
print("Connected to: ",HOST,)
#-------------------Need 2 threads for handling incoming and outgoing messages--
# 1: Create out_buffer:
Buffer = []
rlist,wlist,xlist = select.select([s],Buffer,[])
class Incoming(threading.Thread):
# made a function a thread
def Incoming_messages():
while True:
for i in rlist:
data = i.recv(1024)
if data:
print(data.decode())
# Now for outgoing data.
def Outgoing():
while True:
user_input=("Your message: ")
if user_input is True:
Buffer += [user_input.encode()]
for i in wlist:
s.sendall(Buffer)
Buffer = []
Thanks for taking a look, thanks also to Tony The Lion for suggesting this
Take a look at this revised version of your code: (in python3.3)
from socket import *
import threading
import json
import select
print("client")
HOST = input("connect to: ")
PORT = int(input("on port: "))
# create the socket
s = socket(AF_INET, SOCK_STREAM)
s.connect((HOST, PORT))
print("connected to:", HOST)
#------------------- need 2 threads for handling incoming and outgoing messages--
# 1: create out_buffer:
out_buffer = []
# for incoming data
def incoming():
rlist,wlist,xlist = select.select([s], out_buffer, [])
while 1:
for i in rlist:
data = i.recv(1024)
if data:
print("\nreceived:", data.decode())
# now for outgoing data
def outgoing():
global out_buffer
while 1:
user_input=input("your message: ")+"\n"
if user_input:
out_buffer += [user_input.encode()]
# for i in wlist:
s.send(out_buffer[0])
out_buffer = []
thread_in = threading.Thread(target=incoming, args=())
thread_out = threading.Thread(target=outgoing, args=())
thread_in.start() # this causes the thread to run
thread_out.start()
thread_in.join() # this waits until the thread has completed
thread_out.join()
in your program you had various problems, namely you need to call the threads; to just define them isn't enough.
you also had forgot the function input() in the line: user_input=input("your message: ")+"\n".
the "select()" function was blocking until you had something to read, so the program didn't arrive to the next sections of the code, so it's better to move it to the reading thread.
the send function in python doesn't accept a list; in python 3.3 it accepts a group of bytes, as returned by the encoded() function, so that part of the code had to be adapted.