Develop Reference

Why Thrift raises AssertionError when requested by multiple threads?

i'm building a kind of simulator that uses thrift protocol.
But when im executing multiple threads of my virtual equipments sending messages, the program breaks after a short time by receiving them, think the buffer is overloaded or something like that, or not, so i'm here asking for some help if its possible.
Here's the main pieces of my code
A class for threading:
class ThreadManager (threading.Thread):
def __init__(self, name, obj, client, layout):
threading.Thread.__init__(self)
self.name = name
self.obj = obj
self.client = client
self.layout = layout
def run(self):
print ("Starting " + self.name)
while True:
sleep(2)
self.obj.auto_gen_msg(self.client, layout=self.layout)
The method for generating messages:
def auto_gen_msg(self, client, layout='', min_delay=15, max_delay=30):
if not layout:
msg = self.gen_message(self.draw_random_model())
else:
msg = self.gen_message(layout)
wait = randint(min_delay, max_delay)
sleep(wait)
print(self.eqp_type, " delivered a message ...")
getattr(client, msg[0])(*msg[1])
The main:
def start(layout, equipment, number):
try:
host = 'localhost'
transport = TSocket.TSocket(host, port=9090)
transport = TTransport.TBufferedTransport(transport)
protocol = TCompactProtocol.TCompactProtocol(transport)
client = SuiteService.Client(protocol)
transport.open()
equips = [Equipment(equipment) for i in range(number)]
threads = [ThreadManager(i.eqp_type, i, client, layout) for i in equips]
for i in range(len(threads)):
threads[i].start()
sleep(2)
while True:
pass
transport.close()
except Thrift.TException as tx:
print ("%s " % (tx.message))
The error haunting me:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/Users/lem4fia/Documents/sg/loki/loki-thrift/loki_thrift/loki_thrift/lib/thread_manager.py", line 39, in run
self.obj.auto_gen_msg(self.client, layout=self.layout)
File "/Users/lem4fia/Documents/sg/loki/loki-thrift/loki_thrift/loki_thrift/lib/virtual.py", line 281, in auto_gen_msg
getattr(client, msg[0])(*msg[1])
File "/Users/lem4fia/Documents/sg/loki/thrift-server/thrift_server/suite/SuiteService.py", line 4895, in v1
self.send_v1(ir, ts, ch, o1, o2, o3, o4, o5, o6, o7)
File "/Users/lem4fia/Documents/sg/loki/thrift-server/thrift_server/suite/SuiteService.py", line 4899, in send_v1
self._oprot.writeMessageBegin('v1', TMessageType.CALL, self._seqid)
File "/Users/lem4fia/Documents/sg/loki/lokiv/lib/python3.6/site-packages/thrift-0.11.0-py3.6-macosx-10.6-intel.egg/thrift/protocol/TCompactProtocol.py", line 156, in writeMessageBegin
assert self.state == CLEAR
AssertionError
Curiously, it doesnt bug if instancing 2 virtual equipments in thread, but 10 virtual equipments (sometimes less than this) is sufficient to raise this error.
Can someone please gimme a light? :)

The problem there is that it seems that you have to use one diferent Transport object for each thread. This is probably related to Thrift's implementation!
Reference here : http://grokbase.com/t/thrift/user/134s16ks4m/single-connection-and-multiple-threads

As a general rule 1), Thrift is not intended to be used across multiple threads.
This is, at least to my knowledge, true for all currently supported languages.
One instance per thread will do.
1) aside from server-end things like TThreadedServer or TThreadPoolServer

Why is my Python multiprocess crashing when I try to run infinite loop along with gui

I haven’t coded since college and I’m new to Python so I’m coming at this as a relative novice. My ultimate goal will be to have a gui program that will react based on commands people enter into twitch chat. I have a barebones gui created with tkinter and a bot that will monitor twitch chat. When I run these two components separately in their own .py files they work fine. When they are together in same program it locks up because I have two infinite loops running. I tried threading them but this also didn’t work. I tried using using multiprocess instead of threading, the code for which I’ve included below. I've looked at the other multiprocess and threading questions but I can't seem to figure this out.
from tkinter import *
# settings contains values for HOST, PORT, NICK, PASS, CHANNEL
from settings import *
import socket
import multiprocessing
class Gui:
def __init__(self, master):
self.master=master
master.title("AftA_Bot beta")
self.doodle = Canvas(master, width=720, height=576, bg="white")
self.doodle.pack()
self.out = Button(master, text="Quit", command=master.quit)
self.out.pack()
class createbot:
def __init__(self, server, port, username, token, channel):
self.s = socket.socket()
self.s.connect((server, port))
self.channel = channel
self.s.send(bytes("PASS " + token + "\r\n", "UTF-8"))
self.s.send(bytes("NICK " + username + "\r\n", "UTF-8"))
self.s.send(bytes("JOIN #" + channel + " \r\n", "UTF-8"))
print("Connected to host")
# Clear initial connection text
while True:
line = str(self.s.recv(1024))
if "End of /NAMES list" in line:
break
print("Initial login text cleared.")
def read_chat(self):
# Infinite Loop to monitor chat
while True:
for line in str(self.s.recv(1024)).split('\\r\\n'):
parts = line.split(':')
if len(parts) < 3:
continue
message = parts[2][:len(parts[2])]
usernamesplit = parts[1].split("!")
username = usernamesplit[0]
print(username + ": " + message)
def send_message(self, msg):
self.s.send(bytes("PRIVMSG #" + self.channel + " :" + msg + "\r\n", "UTF-8"))
bot = createbot(HOST, PORT, NICK, PASS, CHANNEL)
botthread = multiprocessing.Process(target=bot.read_chat)
root = Tk()
mainapp = Gui(root)
guithread = multiprocessing.Process(target=root.mainloop)
try:
botthread.start()
except:
print("Could not start bot thread")
try:
guithread.start()
except:
print("Could not start gui thread")
The code above results in the following output/errors:
Connected to host
Initial login text cleared.
Connected to host
Could not start gui thread
Initial login text cleared.
Could not start bot thread
Could not start gui thread
Connected to host
Initial login text cleared.
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Users\Thom\AppData\Local\Programs\Python\Python36-32\Lib\multiprocessing\spawn.py", line 105, in spawn_main
exitcode = _main(fd)
File "C:\Users\Thom\AppData\Local\Programs\Python\Python36-32\Lib\multiprocessing\spawn.py", line 115, in _main
self = reduction.pickle.load(from_parent)
EOFError: Ran out of input
Could not start bot thread
Could not start gui thread

Using one socket in UDP chat using threading

I am working on UDP chat which should be listening and being able to send message any time using only one socket. Example, I will have the chat program done, I will open it first time, then second time and I must be able to communicate over UDP from both programs, simply each program has only one opened socket.
My two threads are for listening, which is deamon thread, because I want it to listen to new messages nonstop, and my other is sending the messages, which is just like a normal thread.
First of all, my problem is that it looks like my threads are blocking each other, because if I run the program, I only get output from the first thread I start.
Second problem is that I am not sure if my sending function or the entire class is written properly, or if there is something missing or incorrect.
Thanks in advance. Btw, I am new into python and I am using python 3, just to make it clear.
import socket
import threading
import logging
import time
from sys import byteorder
class Sending():
def __init__(self, name, tHost, tPort):
self.name = name
self.host = tHost
self.port = tPort
def set_name(self, name):
self.name = name
def send(self, name, tHost, tPort, msgType, dgramSize):
logging.debug('Starting send run')
message = input('Enter message: ')
data = bytearray()
data.extend( (name.encode('utf-8'), message.encode('utf-8'), msgType.to_bytes(1, byteorder = 'little')) )
#data.extend(message.encode(encoding='utf_8'))
self.sock.sendto(bytearray(data), (tHost, tPort))
def run(self):
th2 = threading.Thread(name = 'send', target=self.send('username', 'localhost', 8001, 1, 1400))
th2.start()
class Receiving():
def __init__(self, host, port):
self.host = host
self.port = port
def create_socket(self, host, port):
logging.debug('Starting socket')
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.bind((host, port))
#print ('socket ready')
time.sleep(5)
while True:
data, addr = sock.recvfrom(1500)
print('Prijata:' + data + addr)
def run(self):
th1 = threading.Thread(name = 'rec', target=self.create_socket('localhost', 8000))
th1.setDaemon(True)
th1.start()
if __name__ == '__main__':
#print ('running')
rec = Receiving('localhost', 8000)
send = Sending('username', 'localhost', 8001)
send.run()
rec.run()

Congrats on your introduction to Python! It looks like you're using Python 3, and in future questions it's helpful if you are explicit about which version you're using because there are minor but program-breaking incompatibilities in some code (including this code!).
I found a few errors in your program:
The most major issue - as Trevor Barnwell says, you're not calling threading.Thread quite correctly. The target= argument needs to be a callable object (i.e. function), but in this case it should just be a reference to the function. If you add brackets to the function, self.create_socket(host, port) as you have above, it actually runs the function immediately. As Trevor explained, your Sending.send() method was called early, but additionally there was a similar bug in Receiving. Because Receiving.create_socket() creates an infinite loop, it never returns program execution. While the console output looks correct to the user, the actual program execution has never made it to running the listener in a separate thread.
bytearray.extend() takes an iterable of ints, what you're passing right now is a tuple of byte objects.
In Sending.send() you call self.sock, but you never assign self.sock a value, so it fails.
Sending.run() only runs Sending.send() one time. After completing input for the user, it immediately exits, because the program has finished.
If you're looking for an in-depth, project based introduction to Python appropriate for an experienced programmer (including an exercise very similar to this question on basic sockets, and another on threading), I highly recommend you check out Wesley Chun's "Core Python Applications Programming". The most recent edition (3rd) has a lot of Python 2 code, but it's easily portable to Python 3 with some minor work on the reader's part.
I tried to modify your code as little as possible to get it working, here it is:
import socket
import threading
import logging
import time
class Sending():
def __init__(self, name, tHost, tPort, target):
self.name = name
self.host = tHost
self.port = tPort
self.target_port = target
self.sock = self.create_socket()
def create_socket(self):
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.bind((self.host, self.port))
return sock
def set_name(self, name):
self.name = name
def send_loop(self):
while True:
logging.debug('Starting send run')
message = input('Enter message: ')
data = bytearray()
data.extend(message.encode('utf-8'))
self.sock.sendto(bytearray(data), (self.host, self.target_port))
def run(self):
th2 = threading.Thread(name='send', target=self.send_loop)
th2.start()
class Receiving():
def __init__(self, host, port):
self.host = host
self.port = port
def create_socket(self):
logging.debug('Starting socket')
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.bind((self.host, self.port))
print ('socket ready')
time.sleep(5)
while True:
data, addr = sock.recvfrom(1500)
print('\nPrijata:' + data.decode('utf-8') + str(addr))
def run(self):
th1 = threading.Thread(name='rec', target=self.create_socket)
print("Made it here")
th1.daemon = True
th1.start()
return
if __name__ == '__main__':
print('running')
rec = Receiving('localhost', 8000)
send = Sending('username', 'localhost', 8001, 8000)
rec.run()
send.run()

The threads are not blocking each other. send is called before a thread is even created.
th2 = threading.Thread(name = 'send', target=self.send('username', 'localhost', 8001, 1, 1400))
This line makes a call to send at:
self.send('username', 'localhost', 8001, 1, 1400)
I think you meant to do this:
th2 = threading.Thread(
target=self.send
args=('username', 'localhost', 8001, 1, 1400))
That way a thread will start that calls send on the next line.
Two other things:
You will want to loop in your functions because the thread terminates once the function does.
I think you mean raw_input instead of input

Make parent script continue immediately after calling another script

I'm using Python 2.7.
class Client():
def __init__(self, host, server_port):
"""
This method is run when creating a new Client object
"""
self.host = 'localhost'
self.server_Port = 1337
# Set up the socket connection to the server
self.connection = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.receiver = None
self.myParser = MessageParser()
# TODO: Finish init process with necessary code
self.run()
def run(self):
self.connection.connect((self.host, self.server_Port))
self.receiver = MessageReceiver(self, self.connection) #On this line, a MessageReceiver object is instantiated.
self.take_input()
class MessageReceiver(Thread):
def __init__(self, client, connection):
super(MessageReceiver, self).__init__()
self.myClient = client
self.connection = connection
self.daemon = True
self.run()
def run(self):
self.myClient.receive_message(self.connection.recv(1024)) #This line blocks further progress in the code.
When the run-method in the Client object instantiates a MessageReceiver object, I want the next line of code in Client to be executed immediately, without waiting for an exit code from MessageReceiver. Is there a way to do this?

self.run()
Call start() instead. run() executes the run method in the current thread. start() spins up another thread and calls it there.
self.start()

cPickle.UnpicklingError: pickle data was truncated

i use a Remote Procedure call to communicate between two prozesses. I send objects from the on hand to the other. the object is a object by a django model. The object has different variables, integers and strings.
If i change only integer variables everything works fine. if i change a string variable in the first time it works, too but if i change a string second time my code crashes and i get the following error message
Traceback (most recent call last):
File "/home/manch011/disserver/src/disserver/gui/backends/receiver.py", line 69, in run
name, args, kwargs = cPickle.load(connFile)
cPickle.UnpicklingError: pickle data was truncated
This is my code,
on the server-side:
_exportedMethods = {
'changes': signal_when_changes,
}
class ServerThread(QtCore.QThread):
def __init__(self):
super(ServerThread,self).__init__()
st = self
#threading.Thread.__init__(self)
def run(self):
HOST = '' # local host
PORT = 50000
SERVER_ADDRESS = HOST, PORT
# set up server socket
s = socket.socket()
s.bind(SERVER_ADDRESS)
s.listen(5)
while True:
conn, addr = s.accept()
connFile = conn.makefile()
name, args, kwargs = cPickle.load(connFile)
res = _exportedMethods[name](*args,**kwargs)
cPickle.dump(res,connFile) ; connFile.flush()
conn.close()
And this is the client-side:
class RemoteFunction(object):
def __init__(self,serverAddress,name):
self.serverAddress = serverAddress
self.name = name
def __call__(self,*args,**kwargs):
s = socket.socket()
s.connect(self.serverAddress)
f = s.makefile()
cPickle.dump((self.name,args,kwargs), f)
f.flush()
res = cPickle.load(f)
s.close()
return res
def machine_changed_signal(machine):
HOST = ''
PORT = 50000
SERVER_ADDRESS = HOST, PORT
advise = RemoteFunction(SERVER_ADDRESS,'changes')
advise(machine)
I am not familiar with cPickle and hence cannot figure this one out, can someone explain it to me?
Thanks in advance Chis

I solved my own problem. But first the error message I describe in my question is not meaningful.
I am the problem tackled new and have used the Pyro4 Framework. So i got a new error message which was equivalent to the old but clearly. U cant pickle class objects.
Because I need in my case only the attribute values ​​I pass this in a simple dictionary.
At first download Pyro4 and install it
A simple example similar to the example on the Pyro homepage:
# saved as helloworld.py
import Pyro4
import threading
import os
class HelloWorld(object):
def get_hello_world(self, name):
return "HelloWorld,{0}.".format(name)
#The NameServer had to run in a own thread because he has his own eventloop
class NameServer(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
def run(self):
os.system("python -m Pyro4.naming")
ns = NameServer()
ns.start()
hello_world=HelloWorld()
daemon=Pyro4.Daemon() # make a Pyro daemon
ns=Pyro4.locateNS() # find the name server
uri=daemon.register(hello_world) # register the greeting object as a Pyro object
ns.register("example.helloworld", uri) # register the object with a name in the name server
print "Ready."
daemon.requestLoop() # start the event loop of the server to wait for calls
run this programm and execute the next after
# saved as client.py
import Pyro4
name=raw_input("What is your name? ").strip()
helloworld=Pyro4.Proxy("PYRONAME:example.helloworld") # use name server object lookup uri shortcut
print helloworld.get_hello_world(name)
Important u cant transfer class instances. So "name" could not be a class instance.