Develop Reference

Python multiprocess queue can get() some objects but not others

In the code below when I call .get() in get_query_result I actually get an ImportError as described in the function comment. The actual result in do_query is a large dictionary. It might be in the 10s of MB range. I've verified that this large object can be pickled, and that when adding it to the output queue there are no errors. Simpler objects work just fine in place of result.
Of course I've distilled this code down from it's actual form, but it seems like perhaps there's some issue with the size of result and perhaps it's just not deserializeable when trying to read it off the queue? I've even tried adding a sleep before calling .get() in case the internal mechanisms of SimpleQueue need time to get all the bytes into the queue but it didn't make a difference. I've also tried using multiprocess.Queue and pipes with the same result each time. Also the module name in the error message is slightly different each time making me think this has something to do with the object in the queue only being incorrectly serialized or something.
Your help is greatly appreciated.
import multiprocessing
from multiprocessing import queues
import pickle
def do_query(recv_queue, send_queue):
while True:
item = recv_queue.get()
#result = long_running_function(item)
try:
test = pickle.loads(pickle.dumps(result))
#this works and prints this large object out just fine
print(test)
except pickle.PicklingError:
print('obj was not pickleable')
try:
send_queue.put([result])
except Exception as e:
#This also never happens
print('Exception when calling .put')
class QueueWrapper:
def __init__(self)
self._recv_queue = queues.SimpleQueue()
self._send_queue = queues.SimpleQueue()
self._proc = None
def send_query(self, query):
self._send_queue.put(query)
def get_query_result(self):
'''
When calling .get() I get "ImportError: No module named tmp_1YYBC\n"
If I replace 'result' in do_query with something like ['test'] it works fine.
'''
return self._recv_queue.get()
def init(self):
self._proc = multiprocess.Process(target=do_query, args=(self._send_queue, self._recv_queue))
self._proc.start()
if __name__=='__main__':
queue_wrapper = QueueWrapper()
queue_wrapper.init()
queue_wrapper.send_query('test')
#import error raised when calling below
queue_wrapper.get_query_result()
Edit1: If I change the code to pickle result myself and then send that pickled result in the queue I'm able to successfully call .get on the other end of the queue. However when I go to unpickle that item I get the same error as before. To recap that means I can pickle and unpickle the object I want to send between processes in the process running do_query just fine, if I pickle it myself I can send it between processes just fine, but when I go to unpickle manually I get an error. Almost seems like I'm able to read off the queue before it's done being written to or something? That shouldn't be the case if I'm understanding .get and .put correctly.
Edit2: After some more digging I see that the type(result) is returning <class tmp_1YYBC._sensor_msgs__Image> which is not correct and should be just sensor_msgs.msg._Image.Image but it's interesting to note that the weird prefix appears in my error message. If I try to construct a new Image, copy all the data from result into that image and send that newly created object in the queue I get the exact same error message... It seems like pickle or the other process in general has trouble knowing how to construct this image object?

Occasional error with shelve module (pickle.UnpicklingError: pickle data was truncated and EOFError: Ran out of input)

I'm writing a bot that listens to streams and notifies it's subscribers about posts. I'm working with module shelve which is giving me what to me seems as random errors (they usually don't occur, but sometimes when starting the bot they do and the bot is no longer being able to start until I remove my database files). This means that some posts that were already sent to subscribers are going to be resent because of loss of data.
Two errors are being raised:
EOFError: Ran out of input
and
pickle.UnpicklingError: pickle data was truncated
I was (I believe) able to diagnose the cause of EOFError which happens when bot is interrupted (e.g. KeyboardInterrupt) during IO operations. This error is less of an issue as it won't occur often in real world use but when it does I'm still forced to delete the whole database.
The "pickle data was truncated" error however stays a mystery to me as I can't seem to figure out when it happens or what exactly goes wrong. It also happens more often. The line that raises the error is: if message.id in self.db['processed_messages']:. This is the first line apart from the constructor that does anything with the database.
I'm hoping for some help in resolving these two errors. I included code that relates to the errors in case anyone sees what could be causing them from it.
Code that is relative to my problem:
import shelve
class Bot():
def __init__(self):
# get data from file or create it if it does not exist
self.db = shelve.open('database')
# init keys if they do not exist (example: first time running the program with empty database):
if 'processed_messages' not in self.db:
self.db['processed_messages'] = []
if 'processed_submissions' not in self.db:
self.db['processed_submissions'] = []
if 'subscribers' not in self.db:
self.db['subscribers'] = []
def handler(self, message):
if message.id in self.db['processed_messages']:
return
self.store_data('processed_messages', message.id)
def store_data(self, key, obj):
""" stores data locally with shelve module """
temp = self.db[key]
temp.append(obj)
self.db[key] = temp
On completely different note: if someone knows a better (more elegant) way of handling case of empty database I would also love to hear some input as current approach in constructor is rather wonky

Tweet Feels: Always returns the same Sentiment Score, regardless tags

I am trying to use this library to generate sentiment score for cryptocurrencies:
https://github.com/uclatommy/tweetfeels/blob/master/README.md
When I use the code from the example trump, it returns a sentiment score of -0.00082536637608123106.
I have changed the tags to the following:
btc_feels = TweetFeels(login, tracking=['bitcoin'])
btc_feels.start(20)
btc_feels.sentiment.value
and it still gives me the same value.
I did notice something strange when I installed the library.
from the instructions:
If for some reason pip did not install the vader lexicon:
python3 -m nltk.downloader vader_lexicon
When I ran this, I got:
/anaconda/lib/python3.6/runpy.py:125: RuntimeWarning:
'nltk.downloader' found in sys.modules after import of package 'nltk',
but prior to execution of 'nltk.downloader'; this may result in
unpredictable behaviour warn(RuntimeWarning(msg))
Could this be why it appears not to be working?

By default, tweetfeels creates a database in your current directory. The next time you start the program, it will continue using the same database, and pick up where it left off. I don't know what tweetfeels does to handle you changing the keyword on it, but this behaviour of tweetfeels could be a problem. The solution would be to use a different database for different keywords, and then pass in the location of your database to the TweetFeels constructor.
I don't know that much about Tweetfeels, it just sounded interesting, so I've downloaded the project, and I have a working script that will perform the sentiment analysis on any keyword I give it. I can add a copy of the script here, if you're still having problems getting TweetFeels to work.
Edit: here the script I am using
I am currently having the following problems with the script.
1) I was getting some error that was different from the one you'd got, but I was able to fix the issue by replacing the tweetfeels library from pip with the latest code in their Github repository.
2) If a sentiment value does not get reported, sometimes tweetfeels fails to come to a complete stop, without forcefully sending a ctrl+c keyboard interrupt.
import os, sys, time
from threading import Thread
from pathlib import Path
from tweetfeels import TweetFeels
consumer_key = 'em...'
consumer_secret = 'aF...'
access_token = '25...'
access_token_secret = 'd3...'
login = [consumer_key, consumer_secret, access_token, access_token_secret]
try:
kw = sys.argv[1]
except IndexError:
kw = "iota"
try:
secs = int(sys.argv[2])
except IndexError:
secs = 15
for arg in sys.argv:
if (arg == "-h" or arg == "--help"):
print("Gets sentiment from twitter.\n"
"Pass in a search term, and how frequently you would like the sentiment recalculated (defaults to 15 seconds).\n"
"The keyword can be a comma seperated list of keywords to look at.")
sys.exit(0)
db = Path(f"~/tweetfeels/{kw}.sqlite").expanduser()
if db.exists():
print("existing db detected. Continueing from where the last sentiment stream left off")
else:
#ensure the parent folder exists, the db will be created inside of this folder
Path(f"~/tweetfeels").expanduser().mkdir(exist_ok=True)
feels = TweetFeels(login, tracking=kw.split(","), db=str(db))
go_on = True
def print_feels(feels, seconds):
while go_on:
if feels.sentiment:
print(f"{feels.sentiment.volume} tweets analyzed from {feels.sentiment.start} to {feels.sentiment.end}")
print(f'[{time.ctime()}] Sentiment Score: {feels.sentiment.value}')
print(flush=True)
else:
print(f"The datastream has not reported a sentiment value.")
print(f"It takes a little bit for the first tweets to be analyzed (max of {feels._stream.retry_time_cap + seconds} seconds).")
print("If this problem persists, there may not be anyone tweeting about the keyword(s) you used")
print(flush=True)
time.sleep(seconds)
t = Thread(target=print_feels, kwargs={"feels":feels,"seconds":secs}, daemon=True)
print(f'Twitter posts containing the keyword(s) "{kw}" will be streamed, and a new sentiment value will be recalculated every {secs} seconds')
feels.start()
time.sleep(5)
t.start()
try:
input("Push enter at any time to stop the feed...\n\n")
except (Exception, KeyboardInterrupt) as e:
feels.stop()
raise e
feels.stop()
go_on = False
print(f"Stopping feed. It may take up to {feels._stream.retry_time_cap} for the feed to shut down.\n")
#we're waiting on the feels thread to stop

No, the same sentiment value that you see printed is not related to the warning you've got when downloading the dataset.
The problem with the same sentiment score is coming from these lines:
for s in sentiments:
pass
return s
I suspect that this unbound variable s remembers the previous value of the sentiment score.
But, the problem itself is that you are printing out the score right after you execute the start() function which starts a multi-threaded program to constantly update data from twitter - you should not expect the sentiment score to arrive right after you started the update.
Note that the examples in the README are shown from the Python terminal where they wait after the execution of start() function until the Timer completed. Disconnecting now... message appears.

Python 3 (Bot) script stops working

I'm trying to connect to a TeamSpeak server using the QueryServer to make a bot. I've taken advice from this thread, however I still need help.
This is The TeamSpeak API that I'm using.
Before the edits, this was the summary of what actually happened in my script (1 connection):
It connects.
It checks for channel ID (and it's own client ID)
It joins the channel and starts reading everything
If someone says an specific command, it executes the command and then it disconnects.
How can I make it so it doesn't disconnect? How can I make the script stay in a "waiting" state so it can keep reading after the command is executed?
I am using Python 3.4.1.
I tried learning Threading but either I'm dumb or it doesn't work the way I thought it would. There's another "bug", once waiting for events, if I don't trigger anything with a command, it disconnects after 60 seconds.
#Librerias
import ts3
import threading
import datetime
from random import choice, sample
# Data needed #
USER = "thisisafakename"
PASS = "something"
HOST = "111.111.111.111"
PORT = 10011
SID = 1
class BotPrincipal:
def __init__(self, manejador=False):
self.ts3conn = ts3.query.TS3Connection(HOST, PORT)
self.ts3conn.login(client_login_name=USER, client_login_password=PASS)
self.ts3conn.use(sid=SID)
channelToJoin = Bot.GettingChannelID("TestingBot")
try: #Login with a client that is ok
self.ts3conn.clientupdate(client_nickname="The Reader Bot")
self.MyData = self.GettingMyData()
self.MoveUserToChannel(ChannelToJoin, Bot.MyData["client_id"])
self.suscribirEvento("textchannel", ChannelToJoin)
self.ts3conn.on_event = self.manejadorDeEventos
self.ts3conn.recv_in_thread()
except ts3.query.TS3QueryError: #Name already exists, 2nd client connect with this info
self.ts3conn.clientupdate(client_nickname="The Writer Bot")
self.MyData = self.GettingMyData()
self.MoveUserToChannel(ChannelToJoin, Bot.MyData["client_id"])
def __del__(self):
self.ts3conn.close()
def GettingMyData(self):
respuesta = self.ts3conn.whoami()
return respuesta.parsed[0]
def GettingChannelID(self, nombre):
respuesta = self.ts3conn.channelfind(pattern=ts3.escape.TS3Escape.unescape(nombre))
return respuesta.parsed[0]["cid"]
def MoveUserToChannel(self, idCanal, idUsuario, passCanal=None):
self.ts3conn.clientmove(cid=idCanal, clid=idUsuario, cpw=passCanal)
def suscribirEvento(self, tipoEvento, idCanal):
self.ts3conn.servernotifyregister(event=tipoEvento, id_=idCanal)
def SendTextToChannel(self, idCanal, mensajito="Error"):
self.ts3conn.sendtextmessage(targetmode=2, target=idCanal, msg=mensajito) #This works
print("test") #PROBLEM HERE This doesn't work. Why? the line above did work
def manejadorDeEventos(sender, event):
message = event.parsed[0]['msg']
if "test" in message: #This works
Bot.SendTextToChannel(ChannelToJoin, "This is a test") #This works
if __name__ == "__main__":
Bot = BotPrincipal()
threadprincipal = threading.Thread(target=Bot.__init__)
threadprincipal.start()
Prior to using 2 bots, I tested to launch the SendTextToChannel when it connects and it works perfectly, allowing me to do anything that I want after it sends the text to the channel. The bug that made entire python code stop only happens if it's triggered by the manejadorDeEventos
Edit 1 - Experimenting with threading.
I messed it up big time with threading, getting to the result where 2 clients connect at same time. Somehow i think 1 of them is reading the events and the other one is answering. The script doesn't close itself anymore and that's a win, but having a clone connection doesn't looks good.
Edit 2 - Updated code and actual state of the problem.
I managed to make the double connection works more or less "fine", but it disconnects if nothing happens in the room for 60 seconds. Tried using Threading.timer but I'm unable to make it works. The entire question code has been updated for it.
I would like an answer that helps me to do both reading from the channel and answering to it without the need of connect a second bot for it (like it's actually doing...) And I would give extra points if the answer also helps me to understand an easy way to make a query to the server each 50 seconds so it doesn't disconnects.

From looking at the source, recv_in_thread doesn't create a thread that loops around receiving messages until quit time, it creates a thread that receives a single message and then exits:
def recv_in_thread(self):
"""
Calls :meth:`recv` in a thread. This is useful,
if you used ``servernotifyregister`` and you expect to receive events.
"""
thread = threading.Thread(target=self.recv, args=(True,))
thread.start()
return None
That implies that you have to repeatedly call recv_in_thread, not just call it once.
I'm not sure exactly where to do so from reading the docs, but presumably it's at the end of whatever callback gets triggered by a received event; I think that's your manejadorDeEventos method? (Or maybe it's something related to the servernotifyregister method? I'm not sure what servernotifyregister is for and what on_event is for…)
That manejadorDeEventos brings up two side points:
You've declared manejadorDeEventos wrong. Every method has to take self as its first parameter. When you pass a bound method, like self.manejadorDeEventos, that bound self object is going to be passed as the first argument, before any arguments that the caller passes. (There are exceptions to this for classmethods and staticmethods, but those don't apply here.) Also, within that method, you should almost certainly be accessing self, not a global variable Bot that happens to be the same object as self.
If manejadorDeEventos is actually the callback for recv_in_thread, you've got a race condition here: if the first message comes in before your main threads finishes the on_event assignment, the recv_on_thread won't be able to call your event handler. (This is exactly the kind of bug that often shows up one time in a million, making it a huge pain to debug when you discover it months after deploying or publishing your code.) So, reverse those two lines.
One last thing: a brief glimpse at this library's code is a bit worrisome. It doesn't look like it's written by someone who really knows what they're doing. The method I copied above only has 3 lines of code, but it includes a useless return None and a leaked Thread that can never be joined, not to mention that the whole design of making you call this method (and spawn a new thread) after each event received is weird, and even more so given that it's not really explained. If this is the standard client library for a service you have to use, then you really don't have much choice in the matter, but if it's not, I'd consider looking for a different library.

Why is my file getting closed if I don't do anything with it for a while?

Original situation:
The application I'm working on at the moment will receive notification from another application when a particular file has had data added and is ready to be read. At the moment I have something like this:
class Foo(object):
def __init__(self):
self.myFile = open("data.txt", "r")
self.myFile.seek(0, 2) #seeks to the end of the file
self.mainWindow = JFrame("Foo",
defaultCloseOperation = JFrame.EXIT_ON_CLOSE,
size = (640, 480))
self.btn = JButton("Check the file", actionPerformed=self.CheckFile)
self.mainWindow.add(self.btn)
self.mainWindow.visible = True
def CheckFile(self, event):
while True:
line = self.myFile.readline()
if not line:
break
print line
foo = Foo()
Eventually, CheckFile() will be triggered when a certain message is received on a socket. At the moment, I'm triggering it from a JButton.
Despite the fact that the file is not touched anywhere else in the program, and I'm not using with on the file, I keep on getting ValueError: I/O operation on closed file when I try to readline() it.
Initial Solution:
In trying to figure out when exactly the file was being closed, I changed my application code to:
foo = Foo()
while True:
if foo.myFile.closed == True:
print "File is closed!"
But then the problem went away! Or if I change it to:
foo = Foo()
foo.CheckFile()
then the initial CheckFile(), happening straight away, works. But then when I click the button ~5 seconds later, the exception is raised again!
After changing the infinite loop to just pass, and discovering that everything was still working, my conclusion was that initially, with nothing left to do after instantiating a Foo, the application code was ending, foo was going out of scope, and thus foo.myFile was going out of scope and the file was being closed. Despite this, swing was keeping the window open, which was then causing errors when I tried to operate on an unopened file.
Why I'm still confused:
The odd part is, if foo had gone out of scope, why then, was swing still able to hook into foo.CheckFile() at all? When I click on the JButton, shouldn't the error be that the object or method no longer exists, rather than the method being called successfully and giving an error on the file operation?
My next idea was that maybe, when the JButton attempted to call foo.CheckFile() and found that foo no longer existed, it created a new Foo, somehow skipped its __init__ and went straight to its CheckFile(). However, this doesn't seem to be the case either. If I modify Foo.__init__ to take a parameter, store that in self.myNum, and print it out in CheckFile(), the value I pass in when I instantiate the initial object is always there. This would seem to suggest that foo isn't going out of scope at all, which puts me right back where I started!!!
EDIT: Tidied question up with relevant info from comments, and deleted a lot of said comments.

* Initial, Partial Answer (Added to Question) *
I think I just figured this out. After foo = Foo(), with no code left to keep the module busy, it would appear that the object ceases to exist, despite the fact that the application is still running, with a Swing window doing stuff.
If I do this:
foo = Foo()
while True:
pass
Then everything works as I would expect.
I'm still confused though, as to how foo.CheckFile() was being called at all. If the problem was that foo.myFile was going out of scope and being closed, then how come foo.CheckFile() was able to be called by the JButton?
Maybe someone else can provide a better answer.

I think the problem arises from memory being partitioned into two types in Java, heap and non-heap.Your class instance foo gets stored in heap memory while its method CheckFile is loaded into the method area of non-heap memory. After your script finishes, there are no more references to foo so it gets marked for garbage collection, while the Swing interface is still referring to CheckFile, so it gets marked as in-use. I am assuming that foo.myFile is not considered static so it also is stored in heap memory. As for the Swing interface, it's presumably still being tracked as in-use as long as the window is open and being updated by the window manager.
Edit: your solution of using a while True loop is a correct one, in my opinion. Use it to monitor for events and when the window closes or the last line is read, exit the loop and let the program finish.
Edit 2: alternative solution - try having foo inherit from JFrame to make Swing keep a persistent pointer to it in its main loop as long as the window is open.