In the code below when I call .get() in get_query_result I actually get an ImportError as described in the function comment. The actual result in do_query is a large dictionary. It might be in the 10s of MB range. I've verified that this large object can be pickled, and that when adding it to the output queue there are no errors. Simpler objects work just fine in place of result.
Of course I've distilled this code down from it's actual form, but it seems like perhaps there's some issue with the size of result and perhaps it's just not deserializeable when trying to read it off the queue? I've even tried adding a sleep before calling .get() in case the internal mechanisms of SimpleQueue need time to get all the bytes into the queue but it didn't make a difference. I've also tried using multiprocess.Queue and pipes with the same result each time. Also the module name in the error message is slightly different each time making me think this has something to do with the object in the queue only being incorrectly serialized or something.
Your help is greatly appreciated.
import multiprocessing
from multiprocessing import queues
import pickle
def do_query(recv_queue, send_queue):
while True:
item = recv_queue.get()
#result = long_running_function(item)
try:
test = pickle.loads(pickle.dumps(result))
#this works and prints this large object out just fine
print(test)
except pickle.PicklingError:
print('obj was not pickleable')
try:
send_queue.put([result])
except Exception as e:
#This also never happens
print('Exception when calling .put')
class QueueWrapper:
def __init__(self)
self._recv_queue = queues.SimpleQueue()
self._send_queue = queues.SimpleQueue()
self._proc = None
def send_query(self, query):
self._send_queue.put(query)
def get_query_result(self):
'''
When calling .get() I get "ImportError: No module named tmp_1YYBC\n"
If I replace 'result' in do_query with something like ['test'] it works fine.
'''
return self._recv_queue.get()
def init(self):
self._proc = multiprocess.Process(target=do_query, args=(self._send_queue, self._recv_queue))
self._proc.start()
if __name__=='__main__':
queue_wrapper = QueueWrapper()
queue_wrapper.init()
queue_wrapper.send_query('test')
#import error raised when calling below
queue_wrapper.get_query_result()
Edit1: If I change the code to pickle result myself and then send that pickled result in the queue I'm able to successfully call .get on the other end of the queue. However when I go to unpickle that item I get the same error as before. To recap that means I can pickle and unpickle the object I want to send between processes in the process running do_query just fine, if I pickle it myself I can send it between processes just fine, but when I go to unpickle manually I get an error. Almost seems like I'm able to read off the queue before it's done being written to or something? That shouldn't be the case if I'm understanding .get and .put correctly.
Edit2: After some more digging I see that the type(result) is returning <class tmp_1YYBC._sensor_msgs__Image> which is not correct and should be just sensor_msgs.msg._Image.Image but it's interesting to note that the weird prefix appears in my error message. If I try to construct a new Image, copy all the data from result into that image and send that newly created object in the queue I get the exact same error message... It seems like pickle or the other process in general has trouble knowing how to construct this image object?
Related
Hello :) I´m a complete beginner when it comes to COM objects, any help is appreciated!
I´m working on a Python program supposed to read incoming MS-Word documents in a client/server fashion, i.e. the client sends a request (one or multiple MS-Word documents) and the server reads specific content from those requests using pythoncom and win32com.
Because I want to minimize waiting time for the client (client needs a status message from server, I do not want to open an MS-Word instance for every request. Hence, I intend to have a pool of running MS-Word instances from which the server can pick and choose. This, in turn, means I have to reuse those instances from the pool in different threads and this is what causes trouble right now. After I read Using win32com with multithreading, my dummy code for the server looks like this:
import pythoncom, win32com.client, threading, psutil, os, queue, time, datetime
appPool = {'WINWORD.EXE': queue.Queue()}
def initAppPool():
global appPool
wordApp = win32com.client.DispatchEx('Word.Application')
appPool["WINWORD.EXE"].put(wordApp) # For testing purpose I only use one MS-Word instance currently
def run_in_thread(appid, path):
#open doc, read do some stuff, close it and reattach MS-Word instance to pool
pythoncom.CoInitialize()
wordApp = win32com.client.Dispatch(pythoncom.CoGetInterfaceAndReleaseStream(appid, pythoncom.IID_IDispatch))
doc = wordApp.Documents.Open(path)
time.sleep(3) # read out some content ...
doc.Close()
appPool["WINWORD.EXE"].put(wordApp)
if __name__ == '__main__':
initAppPool()
pathOfFile2BeRead1 = r'C:\Temp\file4.docx'
pathOfFile2BeRead2 = r'C:\Temp\file5.doc'
#treat first request
wordApp = appPool["WINWORD.EXE"].get(True, 10)
pythoncom.CoInitialize()
wordApp_id = pythoncom.CoMarshalInterThreadInterfaceInStream(pythoncom.IID_IDispatch, wordApp)
readDocjob1 = threading.Thread(target=run_in_thread,args=(wordApp_id,pathOfFile2BeRead1), daemon=True)
readDocjob1.start()
#wait here until readDocjob1 is done
wait = True
while wait:
try:
wordApp = appPool["WINWORD.EXE"].get(True, 1)
wait = False
except queue.Empty:
print(f"[{datetime.datetime.now()}] error: appPool empty")
except BaseException as err:
print(f"[{datetime.datetime.now()}] error: {err}")
So far everything works as expected, but when I start a second request similar to the first one:
(x) wordApp_id = pythoncom.CoMarshalInterThreadInterfaceInStream(pythoncom.IID_IDispatch, wordApp)
readDocjob2 = threading.Thread(target=run_in_thread,args=(wordApp_id,pathOfFile2BeRead2), daemon=True)
readDocjob2.start()
I receive the following error message: "The application called an interface that was marshaled for a different thread" for the (x) marked line.
I thought that is why I have to use pythoncom.CoGetInterfaceAndReleaseStream to jump between threads with the same COM object? And besides that why does it work the first time but not the second time?
I searched for different solutions on StackOverflow which use CoMarshalInterface instead of CoMarshalInterThreadInterfaceInStream, but they all gave me the same error. I´m really confused right now.
EDIT:
After fixing the error as mentioned in the comments, I ran into a mysterious behavior.
When the second job is executed:
wordApp_id = pythoncom.CoMarshalInterThreadInterfaceInStream(pythoncom.IID_IDispatch, wordApp)
readDocjob2 = threading.Thread(target=run_in_thread,args=(wordApp_id,pathOfFile2BeRead2), daemon=True)
readDocjob2.start()
The function run_in_thread terminates immediately without executing any line, respectively it seems that the pythoncom.CoInitialize() is not working properly.
The script finishes without any error messages though.
def run_in_thread(instance,appid, path):
#open doc, read do some stuff, close it and reattach MS-Word instance to pool
pythoncom.CoInitialize()
wordApp = win32com.client.Dispatch(pythoncom.CoGetInterfaceAndReleaseStream(appid, pythoncom.IID_IDispatch))
doc = wordApp.Documents.Open(path)
time.sleep(3) # read out some content ...
doc.Close()
instance.flag = True
What happens is you put back in the "activePool" a COM reference that you got from CoGetInterfaceAndReleaseStream. But this reference was created specially for this new thread and then you call CoMarshalInterThreadInterfaceInStream on this new reference.
That's what is wrong.
You must always use the original COM reference you got from the thread that created it, to be able to call CoMarshalInterThreadInterfaceInStream repeatedly.
So, to solve the problem, you must change how your apppool works, use some kind of a "in use" flag but don't touch the original COM reference.
I am just starting to learn Python and in order to do so, I was working on implementing a simple chat-bot. That worked fine until I wanted to implement some text-to-speech functionality which speaks the lines while they are appearing on the screen.
To achieve that, I had to dive into multi-threading and that's where I'm stuck:
import concurrent.futures
import pyttsx3
from time import sleep
import sys
# Settings
engine = pyttsx3.init()
voices = engine.getProperty('voices')
engine.setProperty('voice', voices[0].id)
typing_delay=0.035
def textToSpeech(text):
engine.say(text)
engine.runAndWait()
def typing(sentence):
for char in sentence:
sleep(typing_delay)
sys.stdout.write(char)
sys.stdout.flush()
# def parallel(string):
# tasks = [lambda: textToSpeech(string), lambda: typing("\n> "+string+"\n\n")]
# with ThreadPoolExecutor(max_workers=2) as executor:
# futures = [executor.submit(task) for task in tasks]
# for future in futures:
# try:
# future.result()
# except Exception as e:
# print(e)
def parallel(text):
with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor:
future_tasks = {executor.submit(textToSpeech, text), executor.submit(typing, "\n> "+text+"\n\n")}
for future in concurrent.futures.as_completed(future_tasks):
try:
data = future.result()
except Exception as e:
print(e)
# Test Sentence
parallel("Greetings Professor Falken")
The two functions on top are supposed to run in parallel. I've tried two different implementations for my parallel() function (one is commented out), both of which yield the same result however. For the first line of text that the chat-bot puts out, I do actually get both text & speech, but then I get the error:
'NoneType' object has no attribute 'earlierDate_'
After that, I only get text, no more speech and the error: run loop already started
I assume that somewhere in concurrent.futures is the attribute 'earlierDate_' and that I'm not handling it correctly, so that the text-to-speech thread never stops. But I have no idea how to fix it.
I hope someone here has an idea that might help. I've cut down my code to something that is as small as possible but still can be run and tested.
Addendum: I had issues with importing pyttsx3 on Python 3.8, so I downgraded to Python 3.7 where it seems to work.
UPDATE:
So it occurred to me, that while I was focusing on the multithreading, the issue might have been with my text-to-speech implementation all along.
The obvious bit was me initialising my speech engine globally. So I moved my settings into the textToSpeech function:
def textToSpeech(text):
engine = pyttsx3.init()
voices = engine.getProperty('voices')
engine.setProperty('voice', voices[0].id)
engine.say(text)
engine.runAndWait()
The run loop already started Error now doesn't appear right away and I get text & speech throughout the first couple of chatbot interactions.
I still get the 'NoneType' object has no attribute 'earlierDate_' error, now after every chat-bot output though and eventually the run loop already started Error kicks in again and I lose the sound. Still, one step closer I guess.
UPDATE2:
After another day of digging, I think I'm another step closer.
This seems to be a Mac-specific issue related to multi-threading. I've found multiple issues across different areas where people ran into this problem.
I've located the issue within PyObjCTools/AppHelper.py
There we have the following function:
def runConsoleEventLoop(
argv=None, installInterrupt=False, mode=NSDefaultRunLoopMode, maxTimeout=3.0
):
if argv is None:
argv = sys.argv
if installInterrupt:
installMachInterrupt()
runLoop = NSRunLoop.currentRunLoop()
stopper = PyObjCAppHelperRunLoopStopper.alloc().init()
PyObjCAppHelperRunLoopStopper.addRunLoopStopper_toRunLoop_(stopper, runLoop)
try:
while stopper.shouldRun():
nextfire = runLoop.limitDateForMode_(mode)
if not stopper.shouldRun():
break
soon = NSDate.dateWithTimeIntervalSinceNow_(maxTimeout)
nextfire = nextfire.earlierDate_(soon)
if not runLoop.runMode_beforeDate_(mode, nextfire):
stopper.stop()
finally:
PyObjCAppHelperRunLoopStopper.removeRunLoopStopperFromRunLoop_(runLoop)
This line close to the button is the culprit: nextfire = nextfire.earlierDate_(soon)
The object nextfire seems to be a date. In Objective-C, NSDate objects do indeed have an earlierDate() method, so it should work. But something's wrong with the initialization. When I print(nextfire), I get None. No surprise then that a NoneType object doesn't have the attribute 'earlierDate_'.
So, I have kinda solved both of my issues, however, I solved them both in an unsatisfying way. It's a bit hacky. But it's the best I could do, short of dissecting pyttsx3.
1) First, for my issue with the run loop already started error:
I have moved my engine initialisation back onto a global level outside the textToSpeech function (like in my initial code snippet).
Then, every time before I call my textToSpeech function, I put in the following code:
try:
engine.endLoop()
except Exception as e:
pass
This way, when there is a loop already running, it gets stopped before the new call, preventing the error from happening. If there's no loop running, nothing happens.
2) My main issue with the 'NoneType' object has no attribute 'earlierDate_' error runs a bit deeper. I have looked at the various sources, but I don't completely follow as to what is going on there.
As I wrote in my second update, the error originates in PyObjCTools/AppHelper.py. nextfire is initialized with the limitDateForMode method from NSRunLoop which according to the Apple documentation returns nil if there are no input sources for this mode.
The next step was to look at pyttsx3/nsss.py where this method from PyObjCTools/AppHelper.py gets instantiated. But I didn't really grasp how that instantiation works and how to fix the fact that NSRunLoop apparently gets instantiated without an input source or if that should be fixed at all.
So I went for a dirty hack and changed PyObjCTools/AppHelper.py by switching nextfire and soon in the earlierDate_ call:
try:
while stopper.shouldRun():
nextfire = runLoop.limitDateForMode_(mode)
if not stopper.shouldRun():
break
soon = NSDate.dateWithTimeIntervalSinceNow_(maxTimeout)
nextfire = soon.earlierDate_(nextfire)
if not runLoop.runMode_beforeDate_(mode, nextfire):
stopper.stop()
finally:
PyObjCAppHelperRunLoopStopper.removeRunLoopStopperFromRunLoop_(runLoop)
The soon variable is always correctly instantiated as an NSDate object, so the method earlierDate is available. And apparently it also works with nil as an argument.
Of course it would be nicer if there was a proper fix within one of the two involved libraries. My guess is that pyttsx3/nsss.py needs some work, but I don't know enough about NSRunLoops to come to an informed opinion.
I'm writing a bot that listens to streams and notifies it's subscribers about posts. I'm working with module shelve which is giving me what to me seems as random errors (they usually don't occur, but sometimes when starting the bot they do and the bot is no longer being able to start until I remove my database files). This means that some posts that were already sent to subscribers are going to be resent because of loss of data.
Two errors are being raised:
EOFError: Ran out of input
and
pickle.UnpicklingError: pickle data was truncated
I was (I believe) able to diagnose the cause of EOFError which happens when bot is interrupted (e.g. KeyboardInterrupt) during IO operations. This error is less of an issue as it won't occur often in real world use but when it does I'm still forced to delete the whole database.
The "pickle data was truncated" error however stays a mystery to me as I can't seem to figure out when it happens or what exactly goes wrong. It also happens more often. The line that raises the error is: if message.id in self.db['processed_messages']:. This is the first line apart from the constructor that does anything with the database.
I'm hoping for some help in resolving these two errors. I included code that relates to the errors in case anyone sees what could be causing them from it.
Code that is relative to my problem:
import shelve
class Bot():
def __init__(self):
# get data from file or create it if it does not exist
self.db = shelve.open('database')
# init keys if they do not exist (example: first time running the program with empty database):
if 'processed_messages' not in self.db:
self.db['processed_messages'] = []
if 'processed_submissions' not in self.db:
self.db['processed_submissions'] = []
if 'subscribers' not in self.db:
self.db['subscribers'] = []
def handler(self, message):
if message.id in self.db['processed_messages']:
return
self.store_data('processed_messages', message.id)
def store_data(self, key, obj):
""" stores data locally with shelve module """
temp = self.db[key]
temp.append(obj)
self.db[key] = temp
On completely different note: if someone knows a better (more elegant) way of handling case of empty database I would also love to hear some input as current approach in constructor is rather wonky
I have a pull queue being serviced by a backend and when the queue is empty I need to trigger another script.
At the moment I am using a very crude detection in the method that leases the tasks from the queue, so that if the task list returned is empty we presume that there are no more to lease and trigger the next step. However, while this works most of the time, occasionally a lease request seems to return an empty list even though there are tasks available.
Anyway, the better way to do it I think is to use the fetch_statistics method of the Queue. That way the script can monitor whats going on in the pull queue and know that there are no more items left in the queue. Now this is obviously available via the REST api for queues, but it seems rather backward to use this when I am using these internally.
So I am making the Queue.fetch_statistics() call, but it throws an error. I've tried putting the stated error into Google, but it returns nothing. Same here on stackoverflow.
It always throws:
AttributeError: type object 'QueueStatistics' has no attribute '_QueueStatistics__TranslateError'
My code is:
q = taskqueue.Queue('reporting-pull')
try:
logging.debug(q.fetch_statistics())
except Exception, e:
logging.exception(e)
Can anyone shed any light on this? I am I doing something really stupid here?
Just incase it is useful to anyone else, here is an example function to get you started getting queue info from your app. Its only an example, and could do with better error handling, but it should get you up and running. Previously we have used the Taskqueue client but I thought that was a bit overkill as we can lease and delete in the code anyway, so I used app identity, and it worked a treat.
from google.appengine.api import taskqueue
from google.appengine.api import app_identity
from google.appengine.api import urlfetch
try:
import json
except ImportError:
import simplejson as json
import logging
def get_queue_info(queue_name, stats=False):
'''
Uses the Queue REST API to fetch queue info
Args:
queue_name: string - the name of the queue
stats: boolean - get the stats info too
RETURNS:
DICT: from the JSON response or False on fail
'''
scope = 'https://www.googleapis.com/auth/taskqueue'
authorization_token, _ = app_identity.get_access_token(scope)
app_id = app_identity.get_application_id()
#note the s~ denoting HRD its not mentioned in the docs as far as
#I can see, but it wont work without it
uri = 'https://www.googleapis.com/taskqueue/v1beta1/projects/s~%s/taskqueues/%s?getStats=%s' % (app_id, queue_name, stats)
#make the call to the API
response = urlfetch.fetch(uri, method="GET", headers = {"Authorization": "OAuth " + authorization_token})
if response.status_code == 200:
result = json.loads(response.content)
else:
logging.error('could not get queue')
logging.error(response.status_code)
logging.error(response.content)
return False
return result
Don't forget to update your queue.yaml with the acl for your app identity
-name: queue_name
mode: pull
acl:
- user_email: myappid#appspot.gserviceaccount.com
I hope someone finds this useful.
In the meantime I have posted a Feature request so we can do this with the Queue object, please go and star it if you want it too. http://goo.gl/W8Pk1
The Task Queue Statistics API is now documented and publicly available. The error no longer occurs.
The immediate reason for the specific error you're getting seems to be a bug in the code; Queue.fetch_statistics() calls QueueStatistics.fetch() calls QueueStatistics._FetchMultipleQueues() which apparently encounters an apiproxy_errors.ApplicationError and then tries to call cls.__TranslateError() but there is no such method on the QueueStatistics class.
I don't know the deeper reason for the ApplicationError, but it may mean that the feature is not yet supported by the production runtime.
I need to pickle object [wxpython frame object] and send it as a prameter to this function apply_async on multiproccessing pool module
could someone provide me an example how can I do it
I tried the following and get an error message :
myfile = file(r"C:\binary.dat", "w")
pickle.dump(self, myfile)
myfile.close()
self.my_pool.apply_async(fun,[i,myfile])
def fun(i,self_object):
window = pickle.load(self_oject)
wx.CallAfter(window.LogData, msg)
could someone tell me what could be the problem
If the error give some indicator below the last error message i get:
File "C:\Python26\lib\copy_reg.py", line 70, in _reduce_ex
raise TypeError, "can't pickle %s objects" % base.name
TypeError: can't pickle PySwigObject objects
You can not serialize a widget for use in another process. I guess you want to change the GUI content from another process that is started by the multiprocessing module. In that case, you should define a callback function in the parent process that gets called when the result of the sub-process is ready. Therefore you can use the "callback" parameter of apply_async.
Something like:
def fun(i):
# do something in this sub-process and then return a log message
return "finished doing something"
def cb(resultFromFun):
wx.CallAfter(window.LogData, resultFromFun)
my_pool.apply_async(fun, [i], callback = cb)
I don't believe that wxPython objects can be pickled. They are just wrappers around C objects, which contain lots of pointers and other stateful stuff. The pickle module doesn't know enough about them to be able to restore their state afterwards.