Develop Reference

Printing on console while typing inputs in Python

I'm trying to write a chat application in Python as a school project.
It should be receiving messages from a server and at the same time, it also should be able to send messages to that server. In order to do that, I created two threads: one waits for the incoming messages and the other one is takes inputs from me to send over. The problem is that it can't print the messages which are coming from the server because the other thread is always asking for input. Is there any way to make the message-receiving thread print the incoming messages while the input function asks for input?
Here is the troublesome part of the code:
def sendmsg(conn):
while True:
msg=input("Your message: ")
conn.send(bytes(msg,"utf-8"))
def getmsg(conn):
while True:
data=conn.recv(1024)
print(data.decode("utf-8"))
def server():
soket=socket.socket(socket.AF_INET, socket.SOCK_STREAM)
soket.bind((HOST,PORT))
soket.listen()
print("Listining")
conn, addr = soket.accept()
print("Connection established!")
send = threading.Thread(target = sendmsg(conn))
get = threading.Thread(target = getmsg(conn))
get.start()
send.start()

The basic problem is with the following two lines:
send = threading.Thread(target = sendmsg(conn))
get = threading.Thread(target = getmsg(conn))
There is a big difference in Python between the function object sendmsg and the result of calling that object sendmsg(conn).
The Thread object send never gets created, much less started, because the parameter you are trying to pass in is the result of the call sendmsg(conn), but the function never returns. You have effectively entered an infinite loop at that point, always asking for user input in the main thread.
Instead, you should be passing in the sendmsg function object, and using the args parameter to Thread to let it know that you want to pass in an extra parameter when it does get called. The same applies to getmsg:
send = threading.Thread(target=sendmsg, args=(conn,))
get = threading.Thread(target=getmsg, args=(conn,))
Be careful to include the comma in args=(conn,), or the argument won't be interpreted as a tuple. You can use a list instead if you prefer: args=[conn].

How to use python socket.settimeout() properly

As far as I know, when you call socket.settimeout(value) and you set a float value greater than 0.0, that socket will raise a scocket.timeout when a call to, for example, socket.recv has to wait longer than the value specified.
But imagine I have to receive a big amount of data, and I have to call recv() several times, then how does settimeout affect that?
Given the following code:
to_receive = # an integer representing the bytes we want to receive
socket = # a connected socket
socket.settimeout(20)
received = 0
received_data = b""
while received < to_receive:
tmp = socket.recv(4096)
if len(tmp) == 0:
raise Exception()
received += len(tmp)
received_data += tmp
socket.settimeout(None)
The third line of the code sets the timeout of the socket to 20 seconds. Does that timeout reset every iteration? Will timeout be raised only if one of those iteration takes more than 20 seconds?
A) How can I recode it so that it raises an exception if it is taking more than 20 seconds to receive all the expected data?
B) If I don't set the timeout to None after we read all data, could anything bad happen? (the connection is keep-alive and more data could be requested in the future).

The timeout applies independently to each call to socket read/write operation. So next call it will be 20 seconds again.
A) To have a timeout shared by several consequential calls, you'll have to track it manually. Something along these lines:
deadline = time.time() + 20.0
while not data_received:
if time.time() >= deadline:
raise Exception() # ...
socket.settimeout(deadline - time.time())
socket.read() # ...
B) Any code that's using a socket with a timeout and isn't ready to handle socket.timeout exception will likely fail. It is more reliable to remember the socket's timeout value before you start your operation, and restore it when you are done:
def my_socket_function(socket, ...):
# some initialization and stuff
old_timeout = socket.gettimeout() # Save
# do your stuff with socket
socket.settimeout(old_timeout) # Restore
# etc
This way, your function will not affect the functioning of the code that's calling it, no matter what either of them do with the socket's timeout.

The timeout applies to each call to recv().
A) simply use your existing timeout and call recv(to_receive) - I.e. Try to receive all the data in one recv call - in fact I don't see why you shouldn't use this as the default way it works
B) No nothing bad could happen, but any other code which uses that socket needs to be aware of handling timeout.
On your existing code, shouldn't the recv() call be recv(max(4096,to_receive-received)) - that way you won't unintentionally consume any data which follows after the to_receive bytes.

See my server script, you will get the idea to use it properly.
import socket
import sys
fragments = []
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind(("192.168.1.4",9001))
s.listen(5)
while True:
c,a = s.accept()
c.settimeout(10.0)
print "Someone came in Server from %s and port %s" %(a[0],a[1])
c.send("Welcome to system")
while True:
chunk = c.recv(2048)
if not chunk.strip():
break
else:
fragments.append(chunk)
continue
combiner = "".join(fragments)
print combiner
shutdown = str(raw_input("Wanna Quit(Y/y) or (N/n): "))
if shutdown == 'Y' or shutdown == 'y':
c.close()
sys.exit()
else:
continue
This script is just to give you an idea about the socket.settimeout().

https://docs.python.org/3/library/socket.html#socket.socket.settimeout
"Changed in version 3.5: The socket timeout is no more reset each time data is sent successfully. The socket timeout is now the maximum total duration to send all data."

Python select.select and thread that put socket in output list

I have a problem with a Python client socket.
I need to connect to a server and I want to:
Send a chr if the socket doesn't receive a chr in the last 10
seconds
Send back every chr that the socket receive
Have a thread where (asynchronously) we can decide to send a chr
whenever we want
I try to create the script using select.select method.
Here is my code:
class ThreadCollector(threading.Thread):
def __init__(self, s, outputs):
threading.Thread.__init__(self)
self.__s = s
self.__outputs = outputs
def run(self):
global message_queues
time.sleep(12)
message_queues.put("Y")
self.__outputs.append(self.__s)
time.sleep(30)
print "Start main process"
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
address = "XXX.XXX.XXX.XXX"
port = 999999
s.connect((address, port))
inputs = [s]
outputs = []
message_queues = Queue.Queue()
ThreadCollector(s, outputs).start()
timeout = 10
while 1 == 1:
readable, writable, exceptional = select.select(inputs, outputs, inputs, timeout)
if not (readable or writable or exceptional):
message_queues.put("T")
outputs.append(s)
else:
for socketactivated in readable:
if socketactivated is s:
input = s.recv(1)
message_queues.put(input)
outputs.append(s)
for socketactivated in writable:
if socketactivated is s:
message = message_queues.get_nowait()
s.send(messaggio)
outputs.remove(s)
print "End main process"
The script works well.
If the timeout is reached, the "T" chr is immediately sent to the destination.
But I have a problem.
When the thread push chr "Y" to "message_queues" and append the socket to the output list, the script doesn't send the "Y" immediately. It sends "Y" only after timeout expires and the main script append "T" and socket to lists.
Only after this operation "select.select" wakes up knowing that there is something to send. And main script correctly sees that the output list contains two socket, and message_queues two chr ("Y" and "T").
If the thread sleeps, and the timeout is reached, the process is fine and the "T" is immediately sent after the main script fills "message_queues" and "outputs". In this case, I can see that the script return to select.select, which immediately understand that the output list has something to send.
Why can't my thread tells "select.select" that the socket has something to send? What can I write to wake up "select.select" from the thread?
Where can I put that information?

Multithreading - Alternating between two threads using Conditions and Events in Python

I am trying to write a program using which I wish to alternate between two threads, thread1 and thread2. The tricky part is that the thread should begin execution first must be thread1.
This is the code I have so far:
Class Client:
#member variables
def sendFile(self,cv1,lock1):
sent=0;
while (i<self.size):
message = self.data[i:1024+i]
cv1.acquire()
BadNet.transmit(self.clientSocket,message,self.serverIP,self.serverPort)
cv1.notify()
cv1.release()
i = i+1024
sent+=1
lock1.wait()
print "File sent successfully !"
self.clientSocket.close()
def receiveAck(self,cv1,lock2):
i=0
while (1):
lock1.clear()
cv1.acquire()
cv1.wait()
print "\nentered ack !\n"
self.ack, serverAddress = self.clientSocket.recvfrom(self.buf)
cv1.release()
lock1.set()
if __name__ == "__main__":
lock1 = Event()
cv1 = Condition()
cv2= Condition()
client = Client();
client.readFile();
thread1 = Thread(target = client.sendFile, args=[cv1,lock1])
thread2 = Thread(target = client.receiveAck, args=[cv1,lock1])
thread1.start()
thread2.start()
thread1.join()
thread2.join()
The problem I am currently facing is that initially the program does alternate between two threads (confirmed by the output on the console. But after an arbitrary number of iterations (usually between 20 and 80) the program just hangs and no further iterations are performed.

There are at least two problems with your synchronization.
First, you're using cv1 wrong. Your receive thread has to loop around its cv, checking the condition and calling wait each time. Otherwise, you're just using a cv as a broken event + lock combination. You don't have such a loop. More importantly, you don't even have a condition to wait for.
Second, you're using lock1 wrong. Your receive thread sets the event and then immediately clears it. But there's no guarantee that the send thread has gotten to the wait yet. (The race from the previous problem makes this more of a problem, but it's still a problem even if you fix that.) On a multi-core machine, it will usually get there in time, but "usually" is even worse than never in threaded programming. So, eventually the send thread will get to the wait after the receive thread has already done the clear, and therefore it will wait forever. The receive thread, meanwhile, will be waiting to be notified by the send thread, which will never happen. So you're deadlocked.
For future reference, adding print statements before and after every blocking operation, especially sync operations, would make this a lot to debug: you would see the receive thread's last message was "receive waiting on cv1", while the send thread's last message was "send waiting on lock1", and it would be obvious where the deadlock was.
Anyway, I'm not sure what it would even mean to "fix" a cv with no condition, or an event that you're trying to use as a cv, so instead I'll show how to write something sensible with two cvs. In this case, we might as well just use a flag that we flip back and forth as the condition for both cvs.
While I'm at it, I'll fix a couple other problems that made your code not even testable (e.g., i is never initialized), and include the debugging information, and what I had to fill in to make this a complete example, but otherwise I'll try to leave your structure and irrelevant problems (like Client being an old-style class) intact.
class Client:
def __init__(self):
self.clientSocket = socket(AF_INET, SOCK_DGRAM)
self.serverIP = '127.0.0.1'
self.serverPort = 11111
self.buf = 4
self.waitack = False
def readFile(self):
self.data = ', '.join(map(str, range(100000)))
self.size = len(self.data)
#member variables
def sendFile(self,cv1,lock1):
i = 0
sent=0
while (i<self.size):
message = self.data[i:1024+i]
print "s cv1 acquire"
with cv1:
print "s sendto"
self.clientSocket.sendto(message, (self.serverIP, self.serverPort))
self.waitack = True
print "s cv1 notify"
cv1.notify()
i = i+1024
sent+=1
print "s cv2 acquire"
with cv2:
print "s cv2 wait"
while self.waitack:
cv2.wait()
print "File sent successfully !"
self.clientSocket.close()
def receiveAck(self,cv1,lock2):
i=0
while (1):
print "r cv1 acquire"
with cv1:
while not self.waitack:
print "r cv1 wait"
cv1.wait()
print "r recvfrom"
self.ack, serverAddress = self.clientSocket.recvfrom(self.buf)
i += 1
print self.ack, i
print "r cv2 acquire"
with cv2:
self.waitack = False
print "r cv2 notify"
cv2.notify()
And here's a test server for it:
from itertools import *
from socket import *
s = socket(AF_INET, SOCK_DGRAM)
s.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1)
s.bind(('127.0.0.1', 11111))
for i in count():
data, addr = s.recvfrom(1024)
print(i)
s.sendto('ack\n', addr)
Start the server, start the client, the server will count up to 672, the client will count up to 673 (since your code counts 1-based) with 673 balanced pairs of messages and a "File sent successfully !" at the end. (Of course the client will then hang forever because receiveAck has no way to finish, and the server because I wrote it as an infinite loop.)

Does Queue.get block main?

I know that Queue.get() method in python is a blocking function. I need to know if I implemented this function inside the main, waiting for an object set by a thread, does this means that all the main will be blocked.
For instance, if the main contains functions for transmitter and receiver, will the two work together or not?

Yes -- if you call some_queue.get() within either the thread or the main function, the program will block there until some object as passed through the queue.
However, it is possible to use queues so that they don't block, or so that they have a timeout of some kind:
import Queue
while True:
try:
data = some_queue.get(False)
# If `False`, the program is not blocked. `Queue.Empty` is thrown if
# the queue is empty
except Queue.Empty:
data = None
try:
data2 = some_queue.get(True, 3)
# Waits for 3 seconds, otherwise throws `Queue.Empty`
except Queue.Empty:
data = None
You can do the same for some_queue.put -- either do some_queue.put(item, False) for non-blocking queues, or some_queue.put(item, True, 3) for timeouts. If your queue has a size limit, it will throw a Queue.Full exception if there is no more room left to append a new item.

Yes it will block main/thread. if you want to get all messages without blocking try this
def get_messages(q):
messages = []
while q.qsize():
messages.append(q.get())
# or process message here
return messages
If messages are like stream above code might get caught in loop.
to avoid that use "for loop" and get all the messages sent so far
def get_messages(q):
messages = []
for _ in range(q.qsize()):
messages.append(q.get())
# or process message here
return messages