Develop Reference

Forecast the remaining time of a loop

Good Evening,
I am trying to estimate the remaining time to the end of a loop; I've used:
start = datetime.now()
progress = 0
for i in range(1000):
#do a few calculations
progress += 1
stop = datetime.now()
execution_time = stop-start
remaining = execution_time * ( 1000 - progress )
print("Progress:", progress, "%, estimated", remaining, "time remaining")
But it does not seem to work properly, since it goes up to minutes, even though the loop would take 20 seconds in total, and decrease quickly when reaching the end.
How can I try to forecast the remaining time of a loop efficiently and correctly?

Simply use tqdm package:
from tqdm import tqdm
for i in tqdm(range(10000)):
dosomthing()
It will print everything for you:
76%|█████████████ | 7568/10000 [00:33<00:10, 229.00it/s]

Rather than using datetime.datetime.now() for this sort of thing you can use time.perf_counter(), which is available in Python 3.3+. From the docs:
Return the value (in fractional seconds) of a performance counter,
i.e. a clock with the highest available resolution to measure a short
duration. It does include time elapsed during sleep and is
system-wide. The reference point of the returned value is undefined,
so that only the difference between the results of consecutive calls
is valid.
Also, you can print using a carriage return instead of a newline so that the progress reports are printed on a single line. Here's a brief demo derived from your code.
from time import sleep, perf_counter
fmt = " Progress: {:>3}% estimated {:>3}s remaining"
num = 1000
start = perf_counter()
for i in range(1, num + 1):
# Simulate doing a few calculations
sleep(0.01)
stop = perf_counter()
remaining = round((stop - start) * (num / i - 1))
print(fmt.format(100 * i // num, remaining), end='\r')
print()
Depending on your terminal (and Python version) you may also need to add the flush=True keyword arg to the print call in order to get the progress reports to print as they are issued.

I think that in this line:
remaining = execution_time * ( 1000 - progress )
you should divide execution_time/progress, because you want to know how long it takes to complete one percent of progress.
remaining = execution_time/progress * ( 1000 - progress )

Your calculation for time remaining is wrong. If it takes execution_time for progress steps. Then how much does it take for 1000 steps ?
Simple cross multiply gives you the total time. Subtract it from the time already elapsed and that will give you the time remaining.
remaining_time = execution_time * 1000 / progress - execution_time
percent_complete = (progress / 1000) * 100 #You can simplify this if you like
print("Progress:", percent_complete , "%, Estimated", remaining_time, "time remaining")
Also your variable execution_time_1 is never defined

Python - Print index of for loop only once every five minutes

I'm trying to get a for loop to print the value of 'i' every 5 minutes
import threading
def f(i):
print(i)
threading.Timer(600, f).start()
for i in range(1,1000000000000):
f(i=i)
However, this method results in the code printing the value of i instantly since it calls 'f' as soon as it finds 'i'.
I know this is not the first time someone will ask, nor the last, but I can't get it to work on a for loop nested within a function.
I'm fairly new to Python and I'd appreciate any help.

How about just keeping track of how long has passed in the loop?
from timeit import default_timer as timer
start = timer()
freq = 5 * 60 # Time in seconds
last_time = 0.0
for i in range(int(1e8)):
ctime = timer()
if ctime - last_time > freq:
print(i)
last_time = ctime
I imagine you can make this more efficient by only checking the time every N iterations rather than every time. You may also want to look into using progressbar2 for a ready-made solution.

I prefer using datetime, as I think it's more intuitive and looks a bit cleaner. Otherwise, using more or less the same approach as Paul:
from datetime import datetime, timedelta
print_interval = timedelta(minutes=5)
# Initialize the next print time. Using now() will print the first
# iteration and then every interval. To avoid printing the first
# time, just add print_interval here (i.e. uncomment).
next_print = datetime.now() # + print_interval
for i in range(int(1e8)):
now = datetime.now()
if now >= next_print:
next_print = now + print_interval
print(i)
Also note that before python 3.x xrange would be preferable over range.

Generate some "random" start times for scripts to run based on a period of time in python

I'm trying to generate some random seeded times to tell my script when to fire each of the scripts from within a main script.
I want to set a time frame of:
START_TIME = "02:00"
END_TIME = "03:00"
When it reaches the start time, it needs to look at how many scripts we have to run:
script1.do_proc()
script2.alter()
script3.noneex()
In this case there are 3 to run, so it needs to generate 3 randomized times to start those scripts with a minimum separation of 5 mins between each script but the times must be within the time set in START_TIME and END_TIME
But, it also needs to know that script1.main is ALWAYS the first script to fire, other scripts can be shuffled around (random)
So we could potentially have script1 running at 01:43 and then script3 running at 01:55 and then script2 might run at 02:59
We could also potentially have script1 running at 01:35 and then script3 running at 01:45 and then script2 might run at 01:45 which is also fine.
My script so far can be found below:
import random
import pytz
from time import sleep
from datetime import datetime
import script1
import script2
import script3
START_TIME = "01:21"
END_TIME = "03:00"
while 1:
try:
# Set current time & dates for GMT, London
CURRENT_GMTTIME = datetime.now(pytz.timezone('Europe/London')).strftime("%H%M")
CURRENT_GMTDAY = datetime.now(pytz.timezone('Europe/London')).strftime("%d%m%Y")
sleep(5)
# Grab old day for comparisons
try:
with open("DATECHECK.txt", 'rb') as DATECHECK:
OLD_DAY = DATECHECK.read()
except IOError:
with open("DATECHECK.txt", 'wb') as DATECHECK:
DATECHECK.write("0")
OLD_DAY = 0
# Check for new day, if it's a new day do more
if int(CURRENT_GMTDAY) != int(OLD_DAY):
print "New Day"
# Check that we are in the correct period of time to start running
if int(CURRENT_GMTTIME) <= int(START_TIME.replace(":", "")) and int(CURRENT_GMTTIME) >= int(END_TIME.replace(":", "")):
print "Correct time, starting"
# Unsure how to seed the start times for the scripts below
script1.do_proc()
script2.alter()
script3.noneex()
# Unsure how to seed the start times for above
# Save the current day to prevent it from running again today.
with open("DATECHECK.txt", 'wb') as DATECHECK:
DATECHECK.write(CURRENT_GMTDAY)
print "Completed"
else:
pass
else:
pass
except Exception:
print "Error..."
sleep(60)
EDIT 31/03/2016
Let's say I add the following
SCRIPTS = ["script1.test()", "script2.test()", "script3.test()"]
MAIN_SCRIPT = "script1.test()"
TIME_DIFFERENCE = datetime.strptime(END_TIME, "%H:%M") - datetime.strptime(START_TIME, "%H:%M")
TIME_DIFFERENCE = TIME_DIFFERENCE.seconds
We now have the the number of scripts to run
We have the list of the script to run.
We have the name of the main script, the one to run first.
We have the time in seconds to show how much time we have in total to run all the scripts within.
Surely there is a way we can just plug some sort of loop to make it do it all..
for i in range(len(SCRIPTS)), which is 3 times
Generate 3 seeds, making sure the minimum time is of 300 and all together the 3 seeds must not exceed TIME_DIFFERENCE
Create the start time based on RUN_TIME = START_TIME and then RUN_TIME = RUN_TIME + SEED[i]
First loop would check that that MAIN_SCRIPT exists within SCRIPTS, if it does then it would run that script first, delete itself from SCRIPTS and then on next loops, as it doesn't exist in SCRIPTS it would switch to randomly calling one of the other scripts.
Seeding the times
The following appears to work, there might be an easier way of doing this though.
CALCULATE_SEEDS = 0
NEW_SEED = 0
SEEDS_SUCESSS = False
SEEDS = []
while SEEDS_SUCESSS == False:
# Generate a new seed number
NEW_SEED = random.randrange(0, TIME_DIFFERENCE)
# Make sure the seed is above the minimum number
if NEW_SEED > 300:
SEEDS.append(NEW_SEED)
# Make sure we have the same amount of seeds as scripts before continuing.
if len(SEEDS) == len(SCRIPTS):
# Calculate all of the seeds together
for SEED in SEEDS:
CALCULATE_SEEDS += SEED
# Make sure the calculated seeds added together is smaller than the total time difference
if CALCULATE_SEEDS >= TIME_DIFFERENCE:
# Reset and try again if it's not below the number
SEEDS = []
else:
# Exit while loop if we have a correct amount of seeds with minimum times.
SEEDS_SUCESSS = True

Use datetime.timedelta to compute time differences. This code assumes all three processes run on the same day
from datetime import datetime, timedelta
from random import randint
YR, MO, DY = 2016, 3, 30
START_TIME = datetime( YR, MO, DY, 1, 21, 00 ) # "01:21"
END_TIME = datetime( YR, MO, DY, 3, 0, 0 ) # "3:00"
duration_all = (END_TIME - START_TIME).seconds
d1 = ( duration_all - 600 ) // 3
#
rnd1 = randint(0,d1)
rnd2 = rnd1 + 300 + randint(0,d1)
rnd3 = rnd2 + 300 + randint(0,d1)
#
time1 = START_TIME + timedelta(seconds=rnd1)
time2 = START_TIME + timedelta(seconds=rnd2)
time3 = START_TIME + timedelta(seconds=rnd3)
#
print (time1)
print (time2)
print (time3)
Values of rnd1, rnd2and rnd3 are at least 5 minutes (300 seconds) apart.
Values of rnd3 cannot be greater than the total time interval (3 * d1 + 600). So all three times occur inside the interval.
NB You did not specify how much time each script runs. That is why I did not use time.sleep. A possible option would be threading.Timer (see python documentation).

Assume you store all the method.func() in an array and, as u described, subsequent scripts must be at least 5 mins after script1. They can be executed randomly, so we can launch multiple processes and let them sleep for a period before they can automatically start. (Timing is in seconds)
from multiprocessing import Process
import os
import random
import time
#store all scripts you want to execute here
eval_scripts = ["script1.test()","script2.test()", "script3.test()"]
#run job on different processes. non-blocking
def run_job(eval_string,time_sleep):
#print out script + time to test
print eval_string + " " + str(time_sleep)
time.sleep(time_sleep) #wait to be executed
#time to start
eval(eval_string)
def do_my_jobs():
start_time = []
#assume the duration between start_time and end_time is 60 mins, leave some time for other jobs after the first job (5-10 mins). This is just to be careful in case random.randrange returns the largest number
#adjust this according to the duration between start_time and end_time since calculating (end_time - star_time) is trivial.
proc1_start_time = random.randrange(60*60 - 10*60)
start_time.append(proc1_start_time)
#randomize timing for other procs != first script
for i in range(len(eval_scripts)-1):
#randomize time from (proc1_start_time + 5 mins) to (end_time - star_time)
start_time.append(random.randint(proc1_start_time+5*60, 60*60))
for i in range(len(eval_scripts)):
p_t = Process(target = run_job, args = (eval_scripts[i],start_time[i],))
p_t.start()
p_t.join()
Now all you need to do is to call do_my_jobs() only ONCE at START_TIME every day.

Displaying algorithm run time in python

I am required to display the time it took to run two different algorithms using functions available in the time library. I'm assuming I have to use the timeit() function however I'm not familiar as to how to incorporate that into the code. So far this is what I have:
import time
def time2Algorithms(sound):
# normalize(sound)
largest = 0
for s in getSamples(sound):
largest = max(largest,getSampleValue(s) )
multiplier = 32767.0 / largest
for s in getSamples(sound):
louder = multiplier * getSampleValue(s)
setSampleValue(s,louder)
explore(sound)
# onlyMaximize(sound)
for sample in getSamples(sound):
value = getSampleValue(sample)
if value >= 0:
setSampleValue(sample,32767)
if value < 0:
setSampleValue(sample,-32768)
explore(sound)
My goal is to display the run times of both the normalize and maximize algorithms after they execute.
Thanks.

The time module (which you are required to use) does not include timeit (different module).
Just add a
start = time.time()
just before the part you want to time, and e.g
print(time.time() - start)
just after said part -- this will display the elapsed time in seconds. Ornament and format that as required, of course:-)

You can use timeit like this
import timeit
start_time = timeit.default_timer()
# Your algo goes here
elapsed = timeit.default_timer() - start_time
and also time module which is easy
import time
start_time = time.time()
# Your algo goes here
elapsed = time.time() - start_time

In Python, how can I put a thread to sleep until a specific time?

I know that I can cause a thread to sleep for a specific amount of time with:
time.sleep(NUM)
How can I make a thread sleep until 2AM? Do I have to do math to determine the number of seconds until 2AM? Or is there some library function?
( Yes, I know about cron and equivalent systems in Windows, but I want to sleep my thread in python proper and not rely on external stimulus or process signals.)

Here's a half-ass solution that doesn't account for clock jitter or adjustment of the clock. See comments for ways to get rid of that.
import time
import datetime
# if for some reason this script is still running
# after a year, we'll stop after 365 days
for i in xrange(0,365):
# sleep until 2AM
t = datetime.datetime.today()
future = datetime.datetime(t.year,t.month,t.day,2,0)
if t.hour >= 2:
future += datetime.timedelta(days=1)
time.sleep((future-t).total_seconds())
# do 2AM stuff

You can use the pause package, and specifically the pause.until function, for this:
import pause
from datetime import datetime
pause.until(datetime(2015, 8, 12, 2))

Slightly more generalized solution (based off of Ross Rogers') in case you'd like to add minutes as well.
def sleepUntil(self, hour, minute):
t = datetime.datetime.today()
future = datetime.datetime(t.year, t.month, t.day, hour, minute)
if t.timestamp() > future.timestamp():
future += datetime.timedelta(days=1)
time.sleep((future-t).total_seconds())

Another approach, using sleep, decreasing the timeout logarithmically.
def wait_until(end_datetime):
while True:
diff = (end_datetime - datetime.now()).total_seconds()
if diff < 0: return # In case end_datetime was in past to begin with
time.sleep(diff/2)
if diff <= 0.1: return
Building on the answer of #MZA and the comment of #Mads Y

One possible approach is to sleep for an hour. Every hour, check if the time is in the middle of the night. If so, proceed with your operation. If not, sleep for another hour and continue.
If the user were to change their clock in the middle of the day, this approach would reflect that change. While it requires slightly more resources, it should be negligible.

I tried the "pause" pacakage. It does not work for Python 3.x. From the pause package I extracted the code required to wait until a specific datetime and made the following def.
def wait_until(execute_it_now):
while True:
diff = (execute_it_now - datetime.now()).total_seconds()
if diff <= 0:
return
elif diff <= 0.1:
time.sleep(0.001)
elif diff <= 0.5:
time.sleep(0.01)
elif diff <= 1.5:
time.sleep(0.1)
else:
time.sleep(1)

adapt this:
from datetime import datetime, timedelta
from time import sleep
now = datetime.utcnow
to = (now() + timedelta(days = 1)).replace(hour=1, minute=0, second=0)
sleep((to-now()).seconds)

Slightly beside the point of the original question:
Even if you don't want to muck around with crontabs, if you can schedule python scripts to those hosts, you might be interested to schedule anacron tasks? anacron's major differentiator to cron is that it does not rely the computer to run continuously. Depending on system configuration you may need admin rights even for such user-scheduled tasks.
A similar, more modern tool is upstart provided by the Ubuntu folks: http://upstart.ubuntu.com/
This does not yet even have the required features. But scheduling jobs and replacing anacron is a planned feature. It has quite some traction due to its usage as Ubuntu default initd replacement. (I am not affiliated with the project)
Of course, with the already provided answer, you can code the same functionality into your python script and it might suit you better in your case.
Still, for others, anacron or similar existing systems might be a better solution. anacron is preinstalled on many current linux distributions (there are portability issues for windows users).
Wikipedia provides a pointer page: https://en.wikipedia.org/wiki/Anacron
If you do go for a python version I'd look at the asynchronous aspect, and ensure the script works even if the time is changed (daylight savings, etc) as others have commented already. Instead of waiting til a pre-calculated future, I'd always at maximum wait one hour, then re-check the time. The compute cycles invested should be negligible even on mobile, embedded systems.

Asynchronous version of Omrii's solution
import datetime
import asyncio
async def sleep_until(hour: int, minute: int, second: int):
"""Asynchronous wait until specific hour, minute and second
Args:
hour (int): Hour
minute (int): Minute
second (int): Second
"""
t = datetime.datetime.today()
future = datetime.datetime(t.year, t.month, t.day, hour, minute, second)
if t.timestamp() > future.timestamp():
future += datetime.timedelta(days=1)
await asyncio.sleep((future - t).total_seconds())

I know is way late for this, but I wanted to post an answer (inspired on the marked answer) considering systems that might have - incorrect - desired timezone + include how to do this threaded for people wondering how.
It looks big because I'm commenting every step to explain the logic.
import pytz #timezone lib
import datetime
import time
from threading import Thread
# using this as I am, check list of timezone strings at:
## https://en.wikipedia.org/wiki/List_of_tz_database_time_zones
TIMEZONE = pytz.timezone("America/Sao_Paulo")
# function to return desired seconds, even if it's the next day
## check the bkp_time variable (I use this for a bkp thread)
## to edit what time you want to execute your thread
def get_waiting_time_till_two(TIMEZONE):
# get current time and date as our timezone
## later we remove the timezone info just to be sure no errors
now = datetime.datetime.now(tz=TIMEZONE).replace(tzinfo=None)
curr_time = now.time()
curr_date = now.date()
# Make 23h30 string into datetime, adding the same date as current time above
bkp_time = datetime.datetime.strptime("02:00:00","%H:%M:%S").time()
bkp_datetime = datetime.datetime.combine(curr_date, bkp_time)
# extract the difference from both dates and a day in seconds
bkp_minus_curr_seconds = (bkp_datetime - now).total_seconds()
a_day_in_seconds = 60 * 60 * 24
# if the difference is a negative value, we will subtract (- with + = -)
# it from a day in seconds, otherwise it's just the difference
# this means that if the time is the next day, it will adjust accordingly
wait_time = a_day_in_seconds + bkp_minus_curr_seconds if bkp_minus_curr_seconds < 0 else bkp_minus_curr_seconds
return wait_time
# Here will be the function we will call at threading
def function_we_will_thread():
# this will make it infinite during the threading
while True:
seconds = get_waiting_time_till_two(TIMEZONE)
time.sleep(seconds)
# Do your routine
# Now this is the part where it will be threading
thread_auto_update = Thread(target=function_we_will_thread)
thread_auto_update.start()

It takes only one of the very basic libraries.
import time
sleep_until = 'Mon Dec 25 06:00:00 2020' # String format might be locale dependent.
print("Sleeping until {}...".format(sleep_until))
time.sleep(time.mktime(time.strptime(sleep_until)) - time.time())
time.strptime() parses the time from string -> struct_time tuple. The string can be in different format, if you give strptime() parse-format string as a second argument. E.g.
time.strptime("12/25/2020 02:00AM", "%m/%d/%Y %I:%M%p")
time.mktime() turns the struct_time -> epoch time in seconds.
time.time() gives current epoch time in seconds.
Substract the latter from the former and you get the wanted sleep time in seconds.
sleep() the amount.
If you just want to sleep until whatever happens to be the next 2AM, (might be today or tomorrow), you need an if-statement to check if the time has already passed today. And if it has, set the wake up for the next day instead.
import time
sleep_until = "02:00AM" # Sets the time to sleep until.
sleep_until = time.strftime("%m/%d/%Y " + sleep_until, time.localtime()) # Adds todays date to the string sleep_until.
now_epoch = time.time() #Current time in seconds from the epoch time.
alarm_epoch = time.mktime(time.strptime(sleep_until, "%m/%d/%Y %I:%M%p")) # Sleep_until time in seconds from the epoch time.
if now_epoch > alarm_epoch: #If we are already past the alarm time today.
alarm_epoch = alarm_epoch + 86400 # Adds a day worth of seconds to the alarm_epoch, hence setting it to next day instead.
time.sleep(alarm_epoch - now_epoch) # Sleeps until the next time the time is the set time, whether it's today or tomorrow.

What about this handy and simple solution?
from datetime import datetime
import time
pause_until = datetime.fromisoformat('2023-02-11T00:02:00') # or whatever timestamp you gonna need
time.sleep((pause_until - datetime.now()).total_seconds())

from datetime import datetime
import time, operator
time.sleep([i[0]*3600 + i[1]*60 for i in [[H, M]]][0] - [i[0]*3600 + i[1]*60 for i in [map(int, datetime.now().strftime("%H:%M").split(':'))]][0])

Instead of using the wait() function, you can use a while-loop checking if the specified date has been reached yet:
if datetime.datetime.utcnow() > next_friday_10am:
# run thread or whatever action
next_friday_10am = next_friday_10am()
time.sleep(30)
def next_friday_10am():
for i in range(7):
for j in range(24):
for k in range(60):
if (datetime.datetime.utcnow() + datetime.timedelta(days=i)).weekday() == 4:
if (datetime.datetime.utcnow() + datetime.timedelta(days=i, hours=j)).hour == 8:
if (datetime.datetime.utcnow() + datetime.timedelta(days=i, hours=j, minutes=k)).minute == 0:
return datetime.datetime.utcnow() + datetime.timedelta(days=i, hours=j, minutes=k)
Still has the time-checking thread check the condition every after 30 seconds so there is more computing required than in waiting, but it's a way to make it work.