I currently have a working python application, gui with wxpython. I send this application a folder which then gets processed by a command line application via Popen. Each time I run this application it take about 40 mins+ to process before it finishes. While a single job processes I would like to queue up another job, I don't want to submit multiple jobs at the same time, I want to submit one job, while it's processing I want to submit another job, so when the first one finishes it would then just process the next, and so on, but I am unsure of how to go about this and would appreciate some suggestions.
Presumably you have either a notification that the task has finished being passed back to the GUI or the GUI is checking the state of the task periodically. In either case you can allow the user to just add to a list of directories to be processed and when your popen task has finished take the first one off of the list and start a new popen task, (remembering to remove the started one off of the list.
Use subprocess.call() instead of Popen, or use Popen.wait().
Related
I've written a script that uses PARAMIKO library to log on to a server and executes a command. This command actually invokes the server to execute another python script (resulting in a child process I believe). I believe the server returns back signal indicating that the command was executed successfully, however it doesn't seem to wait for the new child process to complete - only that the original parent process has been completed. Is there anyway of waiting to reference any/all child processes that were generated as a result of this command and waiting that they are all completed before returning control to the initiating client?
Many thanks.
Without the code this will be difficult. I think you should create a rest service . So you would POST to http://0.0.0.0/runCode and this would kick off a process in a different thread. That would end that call. The thread is still running ...when done do a post to http:// 0.0.0.0/afterProcessIsDone this will be the response from the thread that was kicked off. Then in that route you can do whatever you want with thay response there. If you need help with REST check out Flask. It's pretty easy and straight to the point for small projects.
I am a bit confused with multiprocessing. I have a video processing script which can be run from the command line or launched from a PySide application using a subprocess call. The script seems to run fine from the command line and basically initializes a pool of workers which each process a separate video file.
When I run the program however the OS tells me my program is not responding. I would like to make use of all the cores on my system for multiprocessing but I would also like to prevent this annoyance. What should I do I get around this? Do I start the initial script in a thread or something?
As you are speaking of PySide, I assume you program is a GUI one. In a GUI program all processing must occurs in a worker thread if you want to keep the UI responsive. So yes, the initial script must be start in a thread distinct from main thread (main one is reserved for UI)
What I would like to do, is some kind of communication between two applications:
first application would run and call the second application (first application would be closed) then if the second application has finished its job I would like to sent some kind of signal to the first application and launch it again (second application would be closed this time).
The only one idea I got is to write to a file, when the second application has finished its job and check in the first application if the file exists... is there any other way to do that?
It's a little unclear what you're trying to do. Are you simply trying to chain applications together, i.e. when the first application finishes it calls the second application, and when the second finishes it calls the first, etc.? If this is true, then you simply have to have one application spawn the other and exit immediately instead of waiting. Take a look at subprocess.Popen, which lets you do this (subprocess.call always waits).
However, it sounds like maybe you want the first application to continue running, but to reset itself when the second one finishes. In this case the second application is in fact the child of the first. In the first, you can check if the second has finished after calling Popen by using poll; then, when the second has finished, have the first app respawn itself and then exit, just as described above using Popen. If you want it to work asynchronously, you can create a separate thread in the first app that calls wait, and then exits; but you have to be careful with synchronization in this case. I can't say more because I don't know what the first app is doing while the second runs.
If you want the second application to keep running but the send a signal back to the first application, which is also running, create a pipe between the two, so that when the child writes to its stdout, it's actually writing to a pipe, which the parent can read. If your parent (the first application) is able to block waiting for the second application to finish doing what it's doing, then you just do this:
p = subprocess.Popen('myprogram.exe', stdout = subprocess.PIPE)
str = p.stdout.readline()
Then, the call to readline will block until the second program prints something out (e.g. a line just containing "done"), at which point str in the first program will contain the line printed out by the second program.
If you need for the first program to do something else at the same time, and periodically poll the pipe in a non-blocking fashion, it gets trickier. Take a look at this answer for a solution that works on both Unix and Windows:
Non-blocking read on a subprocess.PIPE in python
I created a GUI with PyQt which implements the buttons "Start" and "Stop".
When I click on "Start" a huge python Script is started. The function of "Stop" has to end this python script, but when I start the script it runs and I can't stop it. I even can't activate anything else on the GUI and I get no reaction from it. So i have to wait the long time until the python script ends.
How can I implement the methods so that I can interrupt the script with the "Stop" button even when I want?
Since you do everything in the QButton.clicked signal, your GUI locks up until you exit that function.
My solution i used in a small project was to seperate it into a GUI and worker process.
Use multiprocessing.Process to do your processing and have it send the results over a multiprocessing.Pipe.
The worker also has a second Pipe to recieve commands (my project just uses one command - exit)
In the GUI, you create 2 Pipes: one for results, one for commands.
Initialize the worker with both pipes and start the process.
The next step would be to have a QTimer poll the pipe for results and display them.
By doing so, your UI stays responsive while the calculations happen in the background.
So I have a simple python cgi script. The web front end is used to add stuff to a database, and I have update() function that does some cleanup.
I want to run the update() function every time something is added to site, but it needs to be in the background. That is, the webpage should finish loading without waiting for the update() function to finish.
Now I use:
-add stuff to db
Thread(target=update).start()
-redirect to index page
The problem seems to be that python does not want to finish the request (redirect) until the update() thread is done.
Any ideas?
That is, the webpage should finish loading without waiting for the update() function to finish
CGI has to wait for the process -- as a whole -- to finish. Threads aren't helpful.
You have three choices.
subprocess. Spawn a separate "no wait" subprocess to do the update. Provide all the information as command-line parameters.
multiprocessing. Have your CGI connect place a work request in a Queue. You'd start a separate listener which handles the update requests from a Queue.
celery. Download Celery and use it to manage the separate worker process that does the background processing.
you could add a database trigger to update db in response to an event e.g., if a specific column has changed
start a subprocess e.g., subprocess.Popen([sys.executable, '-c', "from m import update; update()"]). It might not work depending on your cgi environment
or just touch update file to be picked up by an inotify script to run necessary updates in a separate process
switch to a different execution environment, e.g., some multithreaded wsgi-server
as a heave-weight option you could use celery if it is easy to deploy in your environment