I've just learnt about the concept of callbacks, I decided to try implementing my own. My efforts were fruitful, I did indeed manage to simulate the functionality of a callback. Unfortunately, I noticed that my implementation resulted in the stack growing by 2 function calls every cycle which, I assume, would eventually result in a stack overflow if the code ran for a sufficient amount of time.
I was wondering, how could I implement this code to prevent the stack growing on each cycle? Or is this an inevitable product of this implementation, in which case how would this issue be circumvented?
import time
import inspect
def doSomething(x):
return x + 0.00000001
def continue_processing(runningTotal,termination_condition,callback,callback_args,timeout=5):
startTime = time.time()
while (time.time() - startTime < timeout and not(termination_condition(runningTotal))):
runningTotal = doSomething(runningTotal)
print(f"Returning control to calling function, running total is {runningTotal}")
return callback(runningTotal,*callback_args)
def process(runningTotal,n,beginTime):
if(runningTotal < n):
print(f"Continue processing, running total is {runningTotal}\nTime elapsed {time.time() - beginTime}\nCurrent stack size: {len(inspect.stack())}")
continue_processing(runningTotal,lambda x: x>n,process,(n,beginTime))
if __name__ == '__main__':
beginTime = time.time()
try:
process(0,1,beginTime)
except KeyboardInterrupt:
print("Program interrupted!")
exit(0)
print(f"Completed in {time.time() - beginTime}"
The problem is the callback is recursive, it calls itself (indirectly) — which is why the stack overflows. Below is how to avoid that. Note I also changed your code to conform to the PEP 8 - Style Guide for Python Code guidelines to make it more readable. I strongly suggest you read and follow it, especially if you're just learning the language.
import time
import inspect
def doSomething(x):
return x + 0.00000001
def continue_processing(runningTotal, termination_condition, timeout=5):
startTime = time.time()
while (time.time() - startTime < timeout
and not(termination_condition(runningTotal))):
runningTotal = doSomething(runningTotal)
print(f"Returning control to calling function, running total is "
f"{runningTotal}")
# Don't call back the callback
#return callback(runningTotal, *callback_args)
def process(runningTotal, n, beginTime):
while runningTotal < n:
print(f"Continue processing, running total is {runningTotal}\n"
f"Time elapsed {time.time() - beginTime}\n"
f"Current stack size: {len(inspect.stack())}")
continue_processing(runningTotal, lambda x: x>n)
if __name__ == '__main__':
beginTime = time.time()
try:
process(0, 1, beginTime)
except KeyboardInterrupt:
print("Program interrupted!")
exit(0)
print(f"Completed in {time.time() - beginTime}")
Related
I am trying to figure out how to do the following:
If "some logic" occurs for more than 15 seconds:
"execute some code"
Im not quite sure how to ask python to do this. I am working with stock prices and my index column is dtype='datetime64[ns].
I tried searching other threads for hints on how to do this but i wasn't even quite sure what to search for.
Sorry in advance if its a very elementary question.
try takinng the current time and comparing it every tick
One way to accomplish this is to use threads (multiprocessing) as they would be 2 tasks "some logic" and the "scheduler".
This can be addressed easily with the following logic:
import threading
import time
lock = threading.Lock()
cond = threading.Condition(threading.Lock())
def waitLock(timeout):
with cond:
current_time = start_time = time.time()
while current_time < start_time + timeout:
if lock.acquire(False):
return True
else:
cond.wait(timeout - current_time + start_time)
current_time = time.time()
return False
Source: How to implement a Lock with a timeout in Python 2.7
import time
def check():
if not some_logic(): return False
start_time = time.time()
while time.time() - start_time <= 15:
if not some_logic(): return False
return True
if check():
... # do stuff
This runs some_logic as many times as possible to ensure that it returns True throughout the whole 15-second period.
I would like to get a basic idea working in Python:
Let's say I have a blocking call to a function that reads the current speed of a car (I use python-can for this, and wait for the corresponding message to appear on the bus).
On the other hand, I have an infinite loop that prints as precisely as possible the speed of the car.
What I would like is the value of the speed to be updated directly, so that the infinite loop can print it as soon as it changes.
Right now I had three ideas:
Call my get_speed function in the loop, and block. This works, but only if this is the only value I am willing to print (spoiler alert: it's not). If I want to print precisely the RPM too, I have to wait for the speed to be found, possibly missing multiple RPM value before.
Create another "thread" (subprocess/multiprocessing/thread, whatever you call it) that updates a global variable. I know this works, but meh, I feel like I could have something smarter
From what I saw in javascript, there is a way to ask a function for a result, and keep going until this result is found (this rings some CPU bells right now ahah). I kinda want something like this, which could make me do the following
speed = get_speed()
rpm = get_rpm()
while True:
print("RPM: {0} - Speed: {1}".format(rpm, speed))
This would (in my fairy world) actually display something like this:
RPM: None - Speed: None # Function haven't returned yet, waiting
RPM: None - Speed: None # Same here
RPM: None - Speed: None # Same here
RPM: 300 - Speed: None # get_rpm has returned
RPM: 300 - Speed: None # Nothing happened
RPM: 303 - Speed: 0 # get_rpm and get_speed have returned
RPM: 303 - Speed: 0
RPM: 312 - Speed: 0 # etc.
Right now what I have is something like this, which doesn't work (at all)
#!/usr/bin/env python3
import datetime
import asyncio
import random
from time import sleep
async def _can_get_speed():
# My long function
print ("In can_get_speed")
sleep(4)
print ("Out can_get_speed")
r = random.randint(0, 10)
print(r)
return r
async def can_get_speed():
return await asyncio.gather(_can_get_speed())
if __name__ == "__main__":
loop = asyncio.get_event_loop()
speed = 0
speed = loop.call_soon(can_get_speed, loop)
loop.run_forever()
while True:
print("Speed: {0}".format(speed))
My two questions are:
Is this the most correct way to do it (--> asyncio, I mean), and
if not what is ?
If it is, can somebody help me understanding
asyncio better :)
Thanks in advance all!
I was able to get the expected result like this:
#!/usr/bin/env python3
import datetime
import asyncio
import random
from time import sleep
from multiprocessing import Process, Queue
def can_get_speed(q):
while True:
print("IN get_speed")
r = random.randint(1, 4)
sleep(r)
print("Return {0}".format(r))
q.put(r)
def can_get_rpm(q):
while True:
print("IN get_rpm")
r = random.randint(1, 4)
sleep(r)
print("Return {0}".format(r))
q.put(r)
if __name__ == "__main__":
q = Queue()
q2 = Queue()
p = Process(target=can_get_speed, args=(q,))
p2 = Process(target=can_get_rpm, args=(q2,))
p.start()
p2.start()
speed = None
rpm = None
while True:
if not q.empty():
speed = q.get()
if not q2.empty():
rpm = q2.get()
print("Speed: {0} - RPM: {1} - {2}".format(speed, rpm, datetime.datetime.now()))
Is there a smarter way ?
I writing an app based on the asyncio framework. This app interacts with an API that has a rate limit(maximum 2 calls per sec). So I moved methods which interact with an API to the celery for using it as rate limiter. But it is looks like as an overhead.
There are any ways to create a new asyncio event loop(or something else) that guarantees execution of a coroutins not more then n per second?
The accepted answer is accurate. Note however that, usually, one would want to get as close to 2QPS as possible. This method doesn't offer any parallelisation, which could be a problem if make_io_call() takes longer than a second to execute. A better solution would be to pass a semaphore to make_io_call, that it can use to know whether it can start executing or not.
Here is such an implementation: RateLimitingSemaphore will only release its context once the rate limit drops below the requirement.
import asyncio
from collections import deque
from datetime import datetime
class RateLimitingSemaphore:
def __init__(self, qps_limit, loop=None):
self.loop = loop or asyncio.get_event_loop()
self.qps_limit = qps_limit
# The number of calls that are queued up, waiting for their turn.
self.queued_calls = 0
# The times of the last N executions, where N=qps_limit - this should allow us to calculate the QPS within the
# last ~ second. Note that this also allows us to schedule the first N executions immediately.
self.call_times = deque()
async def __aenter__(self):
self.queued_calls += 1
while True:
cur_rate = 0
if len(self.call_times) == self.qps_limit:
cur_rate = len(self.call_times) / (self.loop.time() - self.call_times[0])
if cur_rate < self.qps_limit:
break
interval = 1. / self.qps_limit
elapsed_time = self.loop.time() - self.call_times[-1]
await asyncio.sleep(self.queued_calls * interval - elapsed_time)
self.queued_calls -= 1
if len(self.call_times) == self.qps_limit:
self.call_times.popleft()
self.call_times.append(self.loop.time())
async def __aexit__(self, exc_type, exc, tb):
pass
async def test(qps):
executions = 0
async def io_operation(semaphore):
async with semaphore:
nonlocal executions
executions += 1
semaphore = RateLimitingSemaphore(qps)
start = datetime.now()
await asyncio.wait([io_operation(semaphore) for i in range(5*qps)])
dt = (datetime.now() - start).total_seconds()
print('Desired QPS:', qps, 'Achieved QPS:', executions / dt)
if __name__ == "__main__":
asyncio.get_event_loop().run_until_complete(test(100))
asyncio.get_event_loop().close()
Will print Desired QPS: 100 Achieved QPS: 99.82723898022084
I believe you are able to write a cycle like this:
while True:
t0 = loop.time()
await make_io_call()
dt = loop.time() - t0
if dt < 0.5:
await asyncio.sleep(0.5 - dt, loop=loop)
INTRO: It is well known that the accuracy of time.sleep is OS and computation load dependent. The accuracy in Windows is very poor.
Similarly to /questions/17499837 a method can implement a busy wait using the time.clock method as an alternative to time.sleep. Such an approach creates unnecessary load affecting other modules in the system. That is not desirable while doing simulations.
To reduce the amount of time spent in busy wait and not relying on the time.sleep, a class employs the method select.select and exploits the timeout attribute. See code below:
from sys import platform as _platform
import time, select, socket
class HighResolutionTimeStamp():
__init = time.clock()
__base = time.time()
def __init__(self):
self.__fd = socket.socket()
self.dtts = time.clock if _platform == 'win32' else time.time
def __del__(self):
self.__fd.close()
def get_high_resolution_dt(self):
return HighResolutionTimeStamp.__base + self.dtts() if _platform == 'win32' else time.time()
def busy_wait(self, wait_time):
currentTime = self.dtts()
while (self.dtts() <= currentTime + wait_time):
pass
def sleep(self, wait_time):
currentTime = self.dtts()
while (self.dtts() < (currentTime + wait_time - 0.001)):
select.select([self.__fd], [], [], 0.001)
while (self.dtts() < currentTime + wait_time):
select.select([self.__fd], [], [], 0.0)
if __name__ == '__main__':
st = 1.0/80.0
it = 10
ts = 1
time.sleep(ts)
hrdr = HighResolutionTimeStamp()
total = hrdr.get_high_resolution_dt()
for i in range(it):
hrdr.busy_wait(st)
print 'Ellapsed:', hrdr.get_high_resolution_dt() - total
time.sleep(ts)
total = hrdr.get_high_resolution_dt()
for i in range(it):
hrdr.sleep(st)
print 'Ellapsed:', hrdr.get_high_resolution_dt() - total
time.sleep(ts)
total = hrdr.get_high_resolution_dt()
for i in range(it):
time.sleep(st)
print 'Ellapsed:', hrdr.get_high_resolution_dt() - total
ENVIRONMENT: I'm using PortablePython2.7.6.1
PROBLEM: When the code is executed at the PyScripter or in the command line with PyScripter open in the background, the script above performs very accurate. Once the PyScripter is closed, the method sleep becomes inaccurate. I'm aware that the timeout for select.select should be inaccurate as time.sleep but in all cases, not as described above.
RESULTS:
Without PyScripter running in the background
C:\..\PortablePython2.7.6.1\App\python.exe highresolutiondt.py
Busy wait. Ellapsed: 0.125249385834
Sleep. Ellapsed: 0.15624165535
Time.sleep. Ellapsed: 0.156844139099
With PyScripter running in the background
C:\..\PortablePython2.7.6.1\App\python.exe highresolutiondt.py
Busy wait. Ellapsed: 0.125702142715
Sleep. Ellapsed: 0.125874519348
Time.sleep. Ellapsed: 0.120799064636
This uses time since unix epoch, which, I'm pretty sure is more accurate, I don't use windows though, so I didn't test this out.
from time import time
def pause(secs):
init_time = time()
while time() < init_time+secs: pass
print("See ya in 10 seconds")
pause(10)
print("Heeeeeelooooo there")
Hope it helped
I am running pool.map on big data array and i want to print report in console every minute.
Is it possible? As i understand, python is synchronous language, it can't do this like nodejs.
Perhaps it can be done by threading.. or how?
finished = 0
def make_job():
sleep(1)
global finished
finished += 1
# I want to call this function every minute
def display_status():
print 'finished: ' + finished
def main():
data = [...]
pool = ThreadPool(45)
results = pool.map(make_job, data)
pool.close()
pool.join()
You can use a permanent threaded timer, like those from this question: Python threading.timer - repeat function every 'n' seconds
from threading import Timer,Event
class perpetualTimer(object):
# give it a cycle time (t) and a callback (hFunction)
def __init__(self,t,hFunction):
self.t=t
self.stop = Event()
self.hFunction = hFunction
self.thread = Timer(self.t,self.handle_function)
def handle_function(self):
self.hFunction()
self.thread = Timer(self.t,self.handle_function)
if not self.stop.is_set():
self.thread.start()
def start(self):
self.stop.clear()
self.thread.start()
def cancel(self):
self.stop.set()
self.thread.cancel()
Basically this is just a wrapper for a Timer object that creates a new Timer object every time your desired function is called. Don't expect millisecond accuracy (or even close) from this, but for your purposes it should be ideal.
Using this your example would become:
finished = 0
def make_job():
sleep(1)
global finished
finished += 1
def display_status():
print 'finished: ' + finished
def main():
data = [...]
pool = ThreadPool(45)
# set up the monitor to make run the function every minute
monitor = PerpetualTimer(60,display_status)
monitor.start()
results = pool.map(make_job, data)
pool.close()
pool.join()
monitor.cancel()
EDIT:
A cleaner solution may be (thanks to comments below):
from threading import Event,Thread
class RepeatTimer(Thread):
def __init__(self, t, callback, event):
Thread.__init__(self)
self.stop = event
self.wait_time = t
self.callback = callback
self.daemon = True
def run(self):
while not self.stop.wait(self.wait_time):
self.callback()
Then in your code:
def main():
data = [...]
pool = ThreadPool(45)
stop_flag = Event()
RepeatTimer(60,display_status,stop_flag).start()
results = pool.map(make_job, data)
pool.close()
pool.join()
stop_flag.set()
One way to do this, is to use main thread as the monitoring one. Something like below should work:
def main():
data = [...]
results = []
step = 0
pool = ThreadPool(16)
pool.map_async(make_job, data, callback=results.extend)
pool.close()
while True:
if results:
break
step += 1
sleep(1)
if step % 60 == 0:
print "status update" + ...
I've used .map() instead of .map_async() as the former is synchronous one. Also you probably will need to replace results.extend with something more efficient. And finally, due to GIL, speed improvement may be much smaller than expected.
BTW, it is little bit funny that you wrote that Python is synchronous in a question that asks about ThreadPool ;).
Consider using the time module. The time.time() function returns the current UNIX time.
For example, calling time.time() right now returns 1410384038.967499. One second later, it will return 1410384039.967499.
The way I would do this would be to use a while loop in the place of results = pool(...), and on every iteration to run a check like this:
last_time = time.time()
while (...):
new_time = time.time()
if new_time > last_time+60:
print "status update" + ...
last_time = new_time
(your computation here)
So that will check if (at least) a minute has elapsed since your last status update. It should print a status update approximately every sixty seconds.
Sorry that this is an incomplete answer, but I hope this helps or gives you some useful ideas.