I have a huge problem. I am working in a Web Python Project, where, after click in a button, a specific controller is called and then another function, present in a python module, is called as well, as shown in my code below. However , I need a second button that stops the process of the stream function controller.
import analyser
def stream():
analyser.get_texts()
response.flash = "Analysis Done."
return ""
I've been searching a lot how to stop a process by an external event (something similar to interruption), but the solutions that I've got, all of them, were about how to stop python script using sys.exit() ou programatically by a return statement, for example. None of these solutions actually work for me.
I want that the user be able to stop that function whenever he wants, once that my function analyser.get_texts() remains processing all the time.
So, my question is how can I stop the execution of stream function, through a button click on my view? Thanks.
If I understand you correctly, then your analyser doesn't provide its own way to terminate an ongoing calculation. You will therefore need to wrap it into something that allows you to terminate the analyser without its "consent".
The right approach for that depends on how bad terminating the analyser in that way is: does it leave resources in a bad state?
Depending on that, you have multiple options:
Run your analysis in a separate process. These can be cleanly killed from the outside. Note that it's usually not a good idea to forcefully stop a thread, so use processes instead.
Use some kind of asynchronous task management that lets you create and stop tasks (e.g. Celery).
Related
How to make cmds.duplicate execute immediately when called in maya? Instead of waiting for the entire script to run and then executing it in batches. For example, for this script below, all execution results will appear immediately after the entire script is executed
import time
for i in range(1, 6):
pm.select("pSphere{}".format(i))
time.sleep(0.5)
cmds.duplicate()
I have tried to use python multithreading, like this
import threading
import time
def test():
for i in range(50):
cmds.duplicate('pSphere1')
time.sleep(0.1)
thread = threading.Thread(target=test)
thread.start()
#thread.join()
Sometimes it can success, but sometimes it will crash maya. If the main thread join, it will not achieve the effect. When I want to do a large number of cmds.duplicate, it will resulting in a very high memory consumption, and the program runs more and more slowly. In addition, all duplicate results appear together after the entire python script runs, so I suspect that when I call cmds When duplicating, Maya did not finish executing and outputting the command, but temporarily put the results in a container with variable capacity. With the increase of my calls, the process of dynamic expansion of the container causes the program to become slower and slower, and the memory consumption also increase dramatically. Because I saw that other plug-ins can see the command execution results in real time, so I thought that this should be a proper way to do this just thath I haven't found yet
Your assumptions are not correct. Maya does not need to display anything to complete a tool. If you want to see the results inbetween you can try to use:
pm.refresh()
but this will not change the behaviour in general. I suppose your memory problems have a different source. You could check if it helps to turn off history or the undo queue temporarily.
And of course Ennakard is right with the answer, that most maya commands are not thread save unless mentioned in the docs. Every node creation and modificatons have to be done in the main thread.
The simple answer is you don't, maya command in general and most interaction with maya are not thread safe
threading is usually used for data manipulation before it get used to manipulate anything in maya, but once you start creating node or setting attribute, or any maya modification, no threading.
So I have this library that I use and within one of my functions I call a function from that library, which happens to take a really long time. Now, at the same time I have another thread running where I check for different conditions, what I want is that if a condition is met, I want to cancel the execution of the library function.
Right now I'm checking the conditions at the start of the function, but if the conditions happen to change while the library function is running, I don't need its results, and want to return from it.
Basically this is what I have now.
def my_function():
if condition_checker.condition_met():
return
library.long_running_function()
Is there a way to run the condition check every second or so and return from my_function when the condition is met?
I've thought about decorators, coroutines, I'm using 2.7 but if this can only be done in 3.x I'd consider switching, it's just that I can't figure out how.
You cannot terminate a thread. Either the library supports cancellation by design, where it internally would have to check for a condition every once in a while to abort if requested, or you have to wait for it to finish.
What you can do is call the library in a subprocess rather than a thread, since processes can be terminated through signals. Python's multiprocessing module provides a threading-like API for spawning forks and handling IPC, including synchronization.
Or spawn a separate subprocess via subprocess.Popen if forking is too heavy on your resources (e.g. memory footprint through copying of the parent process).
I can't think of any other way, unfortunately.
Generally, I think you want to run your long_running_function in a separate thread, and have it occasionally report its information to the main thread.
This post gives a similar example within a wxpython program.
Presuming you are doing this outside of wxpython, you should be able to replace the wx.CallAfter and wx.Publisher with threading.Thread and PubSub.
It would look something like this:
import threading
import time
def myfunction():
# subscribe to the long_running_function
while True:
# subscribe to the long_running_function and get the published data
if condition_met:
# publish a stop command
break
time.sleep(1)
def long_running_function():
for loop in loops:
# subscribe to main thread and check for stop command, if so, break
# do an iteration
# publish some data
threading.Thread(group=None, target=long_running_function, args=()) # launches your long_running_function but doesn't block flow
myfunction()
I haven't used pubsub a ton so I can't quickly whip up the code but it should get you there.
As an alternative, do you know the stop criteria before you launch the long_running_function? If so, you can just pass it as an argument and check whether it is met internally.
I'm using Python http.client.HTTPResponse.read() to read data from a stream. That is, the server keeps the connection open forever and sends data periodically as it becomes available. There is no expected length of response. In particular, I'm getting Tweets through the Twitter Streaming API.
To accomplish this, I repeatedly call http.client.HTTPResponse.read(1) to get the response, one byte at a time. The problem is that the program will hang on that line if there is no data to read, which there isn't for large periods of time (when no Tweets are coming in).
I'm looking for a method that will get a single byte of the HTTP response, if available, but that will fail instantly if there is no data to read.
I've read that you can set a timeout when the connection is created, but setting a timeout on the connection defeats the whole purpose of leaving it open for a long time waiting for data to come in. I don't want to set a timeout, I want to read data if there is data to be read, or fail if there is not, without waiting at all.
I'd like to do this with what I have now (using http.client), but if it's absolutely necessary that I use a different library to do this, then so be it. I'm trying to write this entirely myself, so suggesting that I use someone else's already-written Twitter API for Python is not what I'm looking for.
This code gets the response, it runs in a separate thread from the main one:
while True:
try:
readByte = dc.request.read(1)
except:
readByte = []
if len(byte) != 0:
dc.responseLock.acquire()
dc.response = dc.response + chr(byte[0])
dc.responseLock.release()
Note that the request is stored in dc.request and the response in dc.response, these are created elsewhere. dc.responseLock is a Lock that prevents dc.response from being accessed by multiple threads at once.
With this running on a separate thread, the main thread can then get dc.response, which contains the entire response received so far. New data is added to dc.response as it comes in without blocking the main thread.
This works perfectly when it's running, but I run into a problem when I want it to stop. I changed my while statement to while not dc.twitterAbort, so that when I want to abort this thread I just set dc.twitterAbort to True, and the thread will stop.
But it doesn't. This thread remains for a very long time afterward, stuck on the dc.request.read(1) part. There must be some sort of timeout, because it does eventually get back to the while statement and stop the thread, but it takes around 10 seconds for that to happen.
How can I get my thread to stop immediately when I want it to, if it's stuck on the call to read()?
Again, this method is working to get Tweets, the problem is only in getting it to stop. If I'm going about this entirely the wrong way, feel free to point me in the right direction. I'm new to Python, so I may be overlooking some easier way of going about this.
Your idea is not new, there are OS mechanisms(*) for making sure that an application is only calling I/O-related system calls when they are guaranteed to be not blocking . These mechanisms are usually used by async I/O frameworks, such as tornado or gevent. Use one of those, and you will find it very easy to run code "while" your application is waiting for an I/O event, such as waiting for incoming data on a socket.
If you use gevent's monkey-patching method, you can proceed using http.client, as requested. You just need to get used to the cooperative scheduling paradigm introduced by gevent/greenlets, in which your execution flow "jumps" between sub-routines.
Of course you can also perform blocking I/O in another thread (like you did), so that it does not affect the responsiveness of your main thread. Regarding your "How can I get my thread to stop immediately" problem:
Forcing a thread that's blocking in a system call to stop is usually not a clean or even valid process (also see Is there any way to kill a Thread in Python?). Either -- if your application has finished its jobs -- you take down the entire process, which also affects all contained threads, or you just leave the thread be and give it as much time to terminate as required (these 10 seconds you were referring to are not a problem -- are they?)
If you do not want to have such long-blocking system calls anywhere in your application (be it in the main thread or not), then use above-mentioned techniques to prevent blocking system calls.
(*) see e.g. O_NONBLOCK option in http://man7.org/linux/man-pages/man2/open.2.html
I had a program that ran recursively, and while 95% of the time it wasn't an issue sometimes I would hit a recursion limit if I was doing something that took too long. In my efforts to convert to and iterative code, I decided to try something along the lines of this:
while True:
do something
#check if task is done
if done:
print 'ALL DONE'
break
else:
time.sleep(600)
continue
I've tested my code and it works fine, but I was wondering if there is anything inherently wrong with this method? Will it eat up RAM or crash the box if it was left to run for too long?
Thanks in advance!
EDIT:
The "do something" I refer to is checking a log file for certain keywords periodically, as data is constantly being written to the log file. Once these lines are written, which happens at varying length of times, I have the script perform certain tasks, such as copying specific lines to a separate files.
My original program had two functions, one called itself periodically until it found keywords, which would then call the 'dosomething' function. The do something function upon completion would then call original function, and this would happen until the task was finished or I hit the recursion limit
There is nothing inherently wrong in this pattern. I have used the daemon function in init.d to start a very similar python script. As long as "do something" doesn't leak, it should be able to run forever.
I think that either way
time.sleep()
will not stop the recursion limit
Because sleep only pauses the execution , and doesn't free any kind of memory
check https://docs.python.org/2/library/time.html the Time.sleep() description
It suspends the operation , but it will not do any memory optimization
The pattern you describe is easy to implement, but usually not the best way to do things. If the task completes just after you check, you still have to wait 5 minutes to resume processing. However, sometimes there is little choice but to do this; for example, if the only way to detect the task is complete is to check for the existence of a file, you may have to do it this way. In such cases the time interval choice needs to balance the CPU consumed by the "spin" with wait time.
Another pattern that is also fairly easy is to simply block while waiting on the task to complete. Whether this is easy or not depends on the particular API you are using. But this technique does not scale because all processing must wait for a single activity to complete. Imagine not being able to open a new browser tab while a page is loading.
Best practice today generally uses one of several models for asynchronous processing. Much like writing event handlers for mouse clicks, etc. in a website or GUI, you write a callback function that handles the result of processing, and pass that callback to the task. No CPU is wasted and the response is handled immediately without waiting. Many frameworks support this model today. Tulip uses the actor model.
Specifically regarding the recursion limit, I don't think your sleep loop is responsible for hitting the stack frame limit. Maybe it was something happening within the task itself.
I am writing a VIM plugin in Python. I would like to be able to run a function that would wait for events in the background and update a buffer when needed, without freezing the whole window. Is that possible?
I tried running a separate thread using the threading module, which didn't help. The changes in the buffer are reflected only when the function returns (and the blocking thread terminates).
Clarification: I have a function that may take dozens of seconds to return the updates. I need to update one of the buffers with the returned data (and call the function again).
How do I call it "asynchronously" and not have the window frozen? If I can do it by spanning a separate process, how do I set up the IPC?
Another update: What if I create a temp file, have a separate process write to it, and watch for FileChangedShell as suggested here to automatically refresh it?
(If there is no "it's not going to work" answer and no better ideas in my mind, I will try this in a while and update with results).
This may be abstract without knowing the details. But would polling for the event data to be present works for your plugin? Way back i was writing some vim plugin for ensime ( scala compeletion daemon ) and had a similar problem. For me opening a new process and waiting for data to be present in socket was working pretty well. This was obviously due to the fact that the server itself was asynchronous and used to write result on the socket in the order they were recieved. Hope this helps.