Develop Reference

TIS/TSM non-main thread error; pygame script triggered by hotkey (rumps, pygame, keyboard)

I'm writing a python app, whose main purpose is to run a minigame (using the 'pygame' library), whenever I use a hotkey (which currently uses the 'keyboard' library). I want this hotkey to be recognized universally, so I'm packaging the whole thing as a status bar app (using the 'rumps' library).
So far, I can start the whole thing, select Play from the status bar dropdown, and it works! Great.
HOWEVER, if I attempt to use the bound hotkey, to run the same function that Play triggers, I get:
python[58226:599749] pid(58226)/euid(0) is calling TIS/TSM in non-main thread
environment, ERROR : This is NOT allowed. Please call TIS/TSM in main thread!!!
This shows up four times, and is followed up
python[58226:599749] WARNING: nextEventMatchingMask should only be
called from the Main Thread! This will throw an exception in the future.
To check that it wasn't the hotkey itself, I did a test: If I connect the hotkey to a simpler function, like setting an alert, it works fine. It still complains, and gives me the error:
python[60308:620099] -[NSAlert runModal] may only be invoked from the main thread. Behavior on other threads is undefined. (
0 AppKit 0x00007fff2b7f563f -[NSAlert runModal] + 178
1 _objc.cpython-36m-darwin.so 0x000000010c1358c7 ffi_call_unix64 + 79
2 ??? 0x000070000f3b2e50 0x0 + 123145557847632
)
but it does RUN. However if it connects to the Play function, it breaks (and produces the first error, above).
To summarize:
1) Dropdown => run minigame: Success!
2) Hotkey => run minigame: Incomprehensible errors!
I've googled this error, but have only seen explanations that are way over my head; is there a way that I could get around this error, by using different software, or a different approach -- but without having to leave python, or performing some deep and evil hack to an underlying system?
Could I get the hotkey/game to USE the Main Thread environment, somehow? How?
Could I use a vehicle other than a status bar app to listen for the hotkey? (Update: I tried pynput, and got the same non-main thread error.)
Could I do something inside of pygame to not make it as offensive? (At the moment, it's literally just animating a rolling sine wave.)
Could I get the rumps callback function to work? At the moment, it's just not doing ANYTHING, no matter where or how it's called.
(And unfortunately, this on a Mac, because pyhk3 is for windows only, and
wx.Window (which can have hotkeys) is also for windows only)
FURTHER failure: Can't get it to run with Keyboard Maestro either -- I've never used it before, so may well be using it wrong, but it also seems like a VERY simple command, that just flat-out has no response whatsoever.
Aaand ... the Automator script runs, but -- not with the hotkey! (EDIT: The hotkey was apparently taken. But a different one worked! See below.)

This problem is a MacOS 10.13 Sierra error. It is being reported with Processing.org 3d applications, as well as a Program called Synergy.

I'm accepting my own answer because it is the only one so far that solves the problem I was most interested in, i.e. getting the thing to run with a global shortcut, BUT, since I would ideally like to package and distribute this, I would still be very glad to see answers that solved the problem within python, and would gladly award the bounty to any answer that did that.
Finally, finally, made it work with a Service in Automator, and System Preferences > Keyboard > Shortcuts. Along the way, I had to
Explicitly call the python binary I wanted to use, because the bash shell that Automator uses in executing a text script is apparently different from the one in Terminal
Realize that the System Preferences shortcut wasn't working because of the key-combination I first chose, NOT that it was being blocked, because there were no errors or messages when I called it.
Probably all sorts of other things. Good god.

Python hanging in a loop

Thanks to a lot of help from some people, I've got a threadsafe PyQt gui, where sys.stdout prints to a QTextEdit and works fine. Except when a large loop is run in the slave thread.
In a 300,000 iteration loop, I just calculate sqrt, power, and logs, and print the results, but the application just stops and hangs (on my own 64 bit Windows 7 machine, it's after 79%, on an older Mac running Lion it's after ~60%).
Running the loop directly in the python terminal results in the program finishing normally.
I'm not sure I know where to start debugging - is it likely just to be a memory issue, or is there some subtle problem with the threading?

As implied in the comments, modifying the QTextEdit to a QPlainTextEdit fixed the issue; QTextEdit is not designed for handling very large paragraphs , which is effectively what I was creating. I didn't find it necessary to specify the maximumBlockCount

Python crashes in rare cases when running code - how to debug?

I have a problem that I seriously spent months on now!
Essentially I am running code that requires to read from and save to HD5 files. I am using h5py for this.
It's very hard to debug because the problem (whatever it is) only occurs in like 5% of the cases (each run takes several hours) and when it gets there it crashes python completely so debugging with python itself is impossible. Using simple logs it's also impossible to pinpoint to the exact crashing situation - it appears to be very random, crashing at different points within the code, or with a lag.
I tried using OllyDbg to figure out whats happening and can safely conclude that it consistently crashes at the following location: http://i.imgur.com/c4X5W.png
It seems to be shortly after calling the python native PyObject_ClearWeakRefs, with an access violation error message. The weird thing is that the file is successfully written to. What would cause the access violation error? Or is that python internal (e.g. the stack?) and not file (i.e. my code) related?
Has anyone an idea whats happening here? If not, is there a smarter way of finding out what exactly is happening? maybe some hidden python logs or something I don't know about?
Thank you

PyObject_ClearWeakRefs is in the python interpreter itself. But if it only happens in a small number of runs, it could be hardware related. Things you could try:
Run your program on a different machine. if it doesn't crash there, it is probably a hardware issue.
Reinstall python, in case the installed version has somehow become corrupted.
Run a memory test program.

Thanks for all the answers. I ran two versions this time, one with a new python install and my same program, another one on my original computer/install, but replacing all HDF5 read/write procedures with numpy read/write procedures.
The program continued to crash on my second computer at odd times, but on my primary computer I had zero crashes with the changed code. I think it is thus safe to conclude that the problems were HDF5 or more specifically h5py related. It appears that more people encountered issues with h5py in that respect. Given that any error in my application translates to potentially large financial losses I decided to dump HDF5 completely in favor of other stable solutions.

Use a try catch statement. This can be put into the program in order to stop the program from crashing when erroneous data is entered

Is there a simple way to launch a background task from a Python CGI script without waiting around for it to terminate?

In Windows, that is.
I think the answer to this question is that I need to create a Windows service. This seems ludicrously heavyweight for what I am trying to do.
I'm just trying to slap together a little prototype here for my manager, I'm not going to be responsible for productizing it... in fact, it may never even BE productized; it might just be something that a few researchers play around with.
I have a CGI script that receives a file for upload, stores it to a temporary location, then launches a background process to do some serious number-crunching on the file. Then some Javascript stuff sits around calling other CGI scripts to check on the status and update the page as needed.
All of this works, except the damn web server won't close the connection as long as the subrocess is running. I've done some searching, and it appears the answer on Unix is to make it a daemon, but I'm stuck on Windows right now and I guess the answer there is to make it a Windows service?!? This seems incredibly heavyweight to just, you know, launch a damn process and then close the server connection.
That's really the only way?
Edit: Okay, found a nifty little hack over here (the choice (3) that the guy gives):
How to completely background a process in Perl CGI under IIS
I was able to modify this to make it even simpler, and although this is a klugey solution, it is perfect for the quick-and-dirty little prototype I am trying to make.
So I initially had my main script doing this:
subprocess.Popen("python.exe","myscript.py","arg1","arg2")
Which doesn't work, as I've described. Instead, I now have my main script emit this little bit of Javascript which runs after the document is fully loaded:
$("#somecrap").load("launchBackgroundProcess.py", {arg1:"foo",arg2:"bar"});
And then launchBackgroundProcess.py does the subprocess.Popen.
This solution would never scale, since it still leaves the browser connection open during the entire time the background task is running. But since this little thinger I am whipping up might someday have two simultaneous users at most (even then I doubt it) resources are not a concern. This allows the user to see the main page and get the Javascript updates even though there is still an http connection hanging open for no good reason.
Thanks for the answers! If I'm ever asked to productize this, I'll take at the resources Profane recommends.

If you haven't much experience with windows programming and don't wish to peruse the MSDN docs-- I don't blame you-- you may want to try to pick up a copy of Mark Hammond's cannonical guide to all things python and windows. It somehow never goes out-of-date on many of these sorts of recurring questions. Instead of launching the process with the every-platform solution, you'd probably be better off using the win32process module. Chapter 17 of the Hammond book covers this extensively, but you could probably get all you need by downloading the pywin ide (I think it comes bundled in the windows extensions which you can download from pypi), and looking through the help docs it has on python's windows' api. Here's an example of using the api, from a project I was working on recently. It may in fact do some of what you want with a little adaptation. You'd probably want to focus on CreationFlags. In particular, win32process.DETACHED_PROCESS is "often used to execute console programs in the background." Many other flags are available and conveniently wrapped however.
if subprocess.mswindows:
su=subprocess.STARTUPINFO()
su.dwFlags |= subprocess._subprocess.STARTF_USESHOWWINDOW
process = subprocess.Popen(['program', 'flag', 'flag2'], bufsize=-1,
stdout=subprocess.PIPE, startupinfo=su)

Simplest, but not most efficient way would be to just run another python executable
from subprocess import Popen
Popen("python somescript.py")

You can just use a system call using the "start" windows command. This way your python script will not wait for the completion of the started program.

CGI scripts are run with standard output redirected, either directly to the TCP socket or to a pipe. Typically, the connection won't close until the handle, and all copies of it, are closed. By default, the subprocess will inherit a copy of the handle.
There are two ways to prevent the connection from waiting on the subprocess. One is to prevent the subprocess from inheriting the handle, the other is for the subprocess to close its copy of the handle when it starts.
If the subprocess is in Perl, I think you could close the handle very simply:
close(STDOUT);
If you want to prevent the subprocess from inheriting the handle, you could use the SetHandleInformation function (if you have access to the Win32 API) or set bInheritHandles to FALSE in the call to CreateProcess. Alternatively, close the handle before launching the subprocess.

Numeric GUI bottleneck

I've made a GUI to set up and start a numerical integrator using PyQT4, Wing, QT, and Python 2.6.6, on my Mac. The thing is, when I run the integrator form the GUI, it takes very many times longer than when I crudely run the integrator from the command line.
As an example, a 1000 year integration took 98 seconds on the command line and ~570 seconds from the GUI.
In the GUI, the integration runs from a thread and then returns. It uses a a queue to communicate back to the GUI.
Does anyone have any ideas as to where the bottleneck is? I suspect that others may be experiencing something like this just on a smaller scale.
t = threading.Thread(target=self.threadsafe_start_thread, args=(self.queue, self.selected))
t.start()

In general it is not a good idea to use python threads within a pyqt application. Instead use a QThread.
Both python and QThreads call the same underlying mechanisms, but they don't play well together. I don't know if this will solve your problem or not, but it might be part of the issue.

Is your thread code mostly Python code? If yes, then you might be a victim of the Global Interpreter Lock.