I'm sure this is somewhat common, so I am curious what are the accepted/efficient ways of doing this in Python.
Put simply, I am just busy-waiting for a variable to be updated. At the same time I need a timeout scheme, but I feel there must be a better way of doing this.
Currently I do something like this:
wait_start = time.time()
while state != NEW_STATE:
if time.time() - wait_start > timeout:
print "Timed out!"
# Do something
# Continuing on...
I obviously just can't sleep, because I need to know when the state has changed.
So what is an efficient method of implementing a timeout for a state (variable) change?
Condition variables and events are often used for this type of thing. Both require cooperation from the side that's changing the variable.
Related
I have a process that does many things in python (scrapes data, reads csvs, preprocesses data, loads a model and scores data, pushes data to a database, et. al.). I want to time specific parts of the process separately for monitoring reasons. Before I knew it, my script looked something like this:
import time
import pandas
print('Doing foo...')
foo_start = time.time
foo = pd.read_csv('data.csv')
foo_end = time.time()
foo_delta = round(foo_end - foo_start)
print(f'Complete! ({foo_delta} seconds)')
print('Doing bar...')
bar_start = time.time
bar = 1+1 # (or some other operation)
bar_end = time.time()
bar_delta = round(bar_end - bar_start)
print(f'Complete! ({bar_delta} seconds)')
...
and so on. In the spirit of DRY (Don't Repeat Yourself), I figured I could reduce number of lines by making this a function
timedProcess(operation, msg):
print(msg)
operation_start = time.time()
var = operation
operation_end = time.time()
operation_delta = round(operation_end - operation_star)
print(f'Complete! ({operation_delta} sec)')
return operation_delta, var
And this works EDIT: This does not work. The operation is performed before the function is called. The operation runs, but the time is always 0.
load_time, data = timedProcess(
operation = pd.read_csv('data.csv'),
msg = 'Reading data...'
)
And the time of the process is returned along with the pandas dataframe of the data.
My question regards documentation. How do I document this function?
"""
This function times any given operation.
Parameters:
operation (*what goes here?*): Operation to be timed
msg (str): Message describing operation
Returns:
operation_delta (int): Time of operation in seconds
var (*what goes here?*): Outcome of operation
"""
Since operation can really be any type, I am not sure how to go about documenting this. The codebase won't really ever change and I will be the only one to ever run it since this is just a personal project of mine, but I really want to get into the good habit of well documented code.
So my first question is to how to navigate the documentation, and my second question would be the next level of that - how to go about this using type hinting should I so choose eventually.
My third question is, is this even good practice? Should I ignore DRY in this scenario, and go back to what I have in my original example? Thanks!
NOTE: I know this question may be opinion based, but I am looking for the most agreed upon or accepted solution to this type of problem so I can go forward doing it the most pythonic way.
EDIT(S): Grammar
I need to detect when the minutes of the clock/time change and do something,
This is mi code so far, for the clock but still can figuruate out in python how to detect the value has change and do action after. Any help will be apreciated i come from a c++ backgorund my implementations seems so far not working.
while True:
now = datetime.now()
print(now.strftime("%M), end = " ", flush = true)
time.sleep(1)
currentMin = now.srtftime("%M")
that worked for me:
from datetime import datetime
import time
past_min = None
while True:
#current min
now_min = int(datetime.now().strftime("%M"))
#first iteration
if not past_min:
past_min = now_min
if now_min != past_min:
#call your function here
print("Min change detected")
past_min = now_min
#print the seconds
print(datetime.now().strftime("%S"))
time.sleep(1.5)
I think you can create a class (in the below example Minute) with a property currenMin to store the current minute value. By using #<property>.setter function, when the property <property> is changed, it will trigger the setter function
from datetime import datetime
import time
class Minute(object):
def __init__(self):
self._currenMin = ''
#property
def currentMin(self):
return self._currenMin
#currentMin.setter
def currentMin(self, value):
if value != self._currenMin:
# ACTION CODE BELOW
print('Minute changed')
self._currenMin = value
minute = Minute()
while True:
now = datetime.now()
print(now.strftime("%M"), end=" ", flush = True)
time.sleep(1)
minute.currentMin = now.strftime("%M")
Well, for the general case with simple variables, you can't simply do it. There are two simple options to do something similar:
if you control EVERYTHING that writes it, make them trigger that action
write code that regularly checks it and triggers the action when it changes
use language tools like a custom setter (see #user696969's answer)
The first case needs you to control everything that could modify that value. At that point, you might not even need a variable, and just pass the new value (and you can reverse this by having a variable that is always updated). This is a very common pattern, called Event-driven programming, and heavily used for example in UIs, websites (client-side, see a list of DOM events for example) and game frameworks (see pygame's documentation on events)
The second-case of writing a loop or checking it regularly can also work, however, there are some downsides to it as well. You probably don't want to write an infinite loop waiting for it to change, especially not in a way that also blocks the changing of that variable, and thus dead-locking the entire program as it's preventing something it's waiting for. If you just check it regularly between other, it might be hard to ensure it will be checked regardless of what else is the program doing. You might use multiple threads for it, but that brings it's own set of problems. You also have to store and update the previous value, so you can compare it. This might be slow or memory-consuming if the variable holds too much data.
You can also use language tools with custom setters. This is clean, but can not be used for any variable, just for class attributes, so you still need some control over the rest of the program.
Generally I'd use the event-driven approach or the setter one, depending on the wider context. However, for such simple cases, the checking is also fine. The simplest solution might event be to remove the need for this entirely.
Is there an easy way to execute time delay (like time.sleep(3)) between every statement of Python code without having to explicitly write between every statement?
Like in the below Python Script which performs certain action on SAP GUI window. Sometimes, the script continues to the next statement before the previous statement is complete. So, I had to add a time delay between every statement so that it executes correctly. It is working with time delay, but I end up adding time.sleep(3) between every line. Just wondering if there is a better way?
import win32com.client
import time
sapgui = win32com.client.GetObject("SAPGUI").GetScriptingEngine
session = sapgui.FindById("ses[0]")
def add_record(employee_num, start_date, comp_code):
try:
time.sleep(3)
session.findById("wnd[0]/tbar[0]/okcd").text = "/npa40"
time.sleep(3)
session.findById("wnd[0]").sendVKey(0)
time.sleep(3)
session.findById("wnd[0]/usr/ctxtRP50G-PERNR").text = employee_num
time.sleep(3)
session.findById("wnd[0]").sendVKey(0)
time.sleep(3)
session.findById("wnd[0]/usr/ctxtRP50G-EINDA").text = start_date
time.sleep(3)
session.findById("wnd[0]/usr/tblSAPMP50ATC_MENU_EVENT/ctxtRP50G-WERKS[1,0]").text = comp_code
time.sleep(3)
session.findById("wnd[0]/usr/tblSAPMP50ATC_MENU_EVENT/ctxtRP50G-PERSG[2,0]").text = "1"
time.sleep(3)
session.findById("wnd[0]/usr/tblSAPMP50ATC_MENU_EVENT/ctxtRP50G-PERSK[3,0]").text = "U1"
time.sleep(3)
session.findById("wnd[0]/usr/tblSAPMP50ATC_MENU_EVENT").getAbsoluteRow(0).selected = True
time.sleep(3)
return "Pass"
except:
return "failed"
The right way to do what you asked for is almost certainly to use the debugger, pdb.
The right way to do what you want is probably something completely different: find some signal that tells you that the step is done, and wait for that signal. With problems like this, almost any time you pick will be way, way too long 99% of the time, but still too short 1% of the time. That signal may be joining a thread, or waiting on a (threading or multiprocessing) Condition, or getting from a queue, or awaiting a coroutine or future, or setting the sync flag on an AppleEvent, or… It really depends on what you're doing.
But if you really want to do this, you can use settrace:
def sleeper(frame, event, arg):
if event == 'line':
time.sleep(2)
return sleeper
sys.settrace(sleeper)
One small problem is that the notion of line used by the interpreter may well not be what you want. Briefly, a 'line' trace event is triggered whenever the ceval loop jumps to a different lnotab entry (see lnotab_notes.txt in the source to understand what that means—and you'll probably need at least a passing understanding of how bytecode is interpreted, at least from reading over the dis docs, to understand that). So, for example, a multiline expression is a single line; the line of a with statement may appear twice, etc.1
And there's probably an even bigger problem.
Sometimes, the script continues to next step before the previous step is fully complete.
I don't know what those steps are, but if you put the whole thread to sleep for 2 seconds, there's a good chance the step you're waiting for won't make any progress, because the thread is asleep. (For example, you're not looping through any async or GUI event loops, because you're doing nothing at all.) If so, then after 2 seconds, it'll still be just as incomplete as it was before, and you'll have wasted 2 seconds for nothing.
1. If your notion of "line" is closer to what's described in the reference docs on lexing and parsing Python, you could create an import hook that walks the AST and adds an expression statement with a Call to time.sleep(2) after each list element in each body with a module, definition, or compound statement (and then compiles and execs the result as usual).
Anything you want to happen in a program has to be explicitly stated - this is the nature of programming. This is like asking if you can print hello world without calling print("hello world").
I think the best advice to give you here is: don't think in terms of "lines", but think in term of functions.
use debugging mode and watch each and every line executing line by line.
I am wondering if it is possible to get a script to grab the "remaining time" in the "timeout counter" or whatever is used internally by pexpect:
index, match_obj, text = session.expect(["New Software Release is ready -- Version"], timeout=int(DOWNLOAD_TIMEOUT))
if match_obj:
logger.info('Download to primary card complete.')
logger.debug('Download took %d seconds', SOMETHINGGOESHERE)
else:
logger.critical('Download to primary card took too long.')
quit(session)
For the life of me, I can't seem to find any attempt to do this anywhere here or via Google :)
I assume I can do something more complicated where I keep track of my own timer, but I was hoping for a simpler solution, given that pexpect is already keeping track of a time value, in seconds.
pexpect source code shows that the remaining time is kept in a local variable therefore it is not accessible outside of the function after it returns.
It is not complicated to keep track of the remaining time yourself:
end_time = time.time() + timeout
index = child.expect([..], timeout=timeout)
remaining_time = end_time - time.time()
If you need to do it multiple times, you could subclass pexpect.spawn class and override expect() method to return the remaining time together with the index.
I have got stuck with a problem.
It goes like this,
A function returns a single result normally. What I want is it to return continuous streams of result for a certain time frame(optional).
Is it feasible for a function to repeatedly return results for a single function call?
While browsing through the net I did come across gevent and threading. Will it work if so any heads up how to solve it?
I just need to call the function carry out the work and return results immediately after every task is completed.
Why you need this is not specified in the question, so it is hard to know what you need, but I will give you a general idea, and code too.
You could return in that way: return var1, var2, var3 (but that's not what you need I think)
You have multiple options: either blocking or non-blocking. Blocking means your code will no longer execute while you are calling the function. Non-blocking means that it will run in parallel. You should also know that you will definitely need to modify the code calling that function.
That's if you want it in a thread (non-blocking):
def your_function(callback):
# This is a function defined inside of it, just for convenience, it can be any function.
def what_it_is_doing(callback):
import time
total = 0
while True:
time.sleep(1)
total += 1
# Here it is a callback function, but if you are using a
# GUI application (not only) for example (wx, Qt, GTK, ...) they usually have
# events/signals, you should be using this system.
callback(time_spent=total)
import thread
thread.start_new_thread(what_it_is_doing, tuple(callback))
# The way you would use it:
def what_I_want_to_do_with_each_bit_of_result(time_spent):
print "Time is:", time_spent
your_function(what_I_want_to_do_with_each_bit_of_result)
# Continue your code normally
The other option (blocking) involves a special kind of functions generators which are technically treated as iterators. So you define it as a function and acts as an iterator. That's an example, using the same dummy function than the other one:
def my_generator():
import time
total = 0
while True:
time.sleep(1)
total += 1
yield total
# And here's how you use it:
# You need it to be in a loop !!
for time_spent in my_generator():
print "Time spent is:", time_spent
# Or, you could use it that way, and call .next() manually:
my_gen = my_generator()
# When you need something from it:
time_spent = my_gen.next()
Note that in the second example, the code would make no sense because it is not really called at 1 second intervals, because there's the other code running each time it yields something or .next is called, and that may take time. But I hope you got the point.
Again, it depends on what you are doing, if the app you are using has an "event" framework or similar you would need to use that, if you need it blocking/non-blocking, if time is important, how your calling code should manipulate the result...
Your gevent and threading are on the right track, because a function does what it is programmed to do, either accepting 1 var at a time or taking a set and returning either a set or a var. The function has to be called to return either result, and the continuous stream of processing is probably taking place already or else you are asking about a loop over a kernel pointer or something similar, which you are not, so ...
So, your calling code which encapsulates your function is important, the function, any function, eg, even a true/false boolean function only executes until it is done with its vars, so there muse be a calling function which listens indefinitely in your case. If it doesn't exist you should write one ;)
Calling code which encapsulates is certainly very important.
Folks aren't going to have enough info to help much, except in the super generic sense that we can tell you that you are or should be within in some framework's event loop, or other code's loop of some form already- and that is what you want to be listening to/ preparing data for.
I like "functional programming's," "map function," for this sort of thing. I think. I can't comment at my rep level or I would restrict my speculation to that. :)
To get a better answer from another person post some example code and reveal your API if possible.