What my code does is generate an X amount of random lines, each is the length of X
My program is almost finished but I couldn't get the time to work, more specifically, get it out of the function, I know it calculates the time correctly because if I put a print in the break section it actually prints the time, but when I want to use it outside the function it never is defined
I tried tons of stuff, like globalizing it in the function, outside function (it doesn't change from 0, which means the function is not changing the variable for some reason) , tried float, tried adding the line elapsed = MainCode() , but all never working, I spent like 6 hours with this single issue, 3 of those hours with other programmers from Python's official discord server for extra help, all were also confused as to why is it still not considered defined
here's my full code, any help is appreciated
import random
import string
import multiprocessing
import os
import sys
import time
if __name__ == "__main__":
lines_wanted = input("input how many lines: ") #user can input how many lines he wants
length = input("How long: ") #user can enter how long each line should be
def MainCode(lines_wanted, length):
global elapsed
elapsed = 0.0
starttime = time.time() # start the time
file = open("blablabla.txt", "a",) # open/create the file
i = 0
while 1 == 1:
file.write(''.join(random.choice(string.ascii_letters + string.digits + ".") for i in range(int(length))) + "\n") # write the randomly generated lowercase string
i += 1 * multiprocessing.cpu_count() # count how many lines the loop is on
if i >= int(lines_wanted): # If it reaches the lines it would stop writing
endtime = time.time() # stop the time
elapsed = endtime - starttime #calculate the time
return elapsed
break
processes = []
if __name__ == '__main__':
for _ in range(multiprocessing.cpu_count()):
p = multiprocessing.Process(target=MainCode, args=(lines_wanted, length))
p.start()
processes.append(p)
for process in processes:
process.join()
elapsee = MainCode()
if __name__ == "__main__": # Prints this after finshing the func above
print(f"Finished Writting {lines_wanted} lines, took {elapsee:.1f} to generate")
elapsed is only written by MainCode, which is only called in subprocesses started by multiprocessing.Process. Being in a different process, your main program has no opportunity to observe those variables; unlike threads, subprocesses do not share your namespace. As such, you want to communicate the value somehow. This is conveniently handled by multiprocessing.Pool:
#! /usr/bin/env python3
import multiprocessing
import os
import time
def worker(arg):
starttime = time.time()
time.sleep(3)
endtime = time.time()
return endtime - starttime
if __name__ == '__main__':
wallstart = time.time()
with multiprocessing.Pool() as pool:
times = pool.map(worker, [3]*os.cpu_count())
wallstop = time.time()
print(f'Wall time: {wallstop-wallstart}, worker time: {sum(times)}')
Example run result:
$ python3 multiprocess.py
Wall time: 4.356384515762329, worker time: 24.002115488052368
Of course, the typos others pointed out would keep you from finding the intended variable as well.
One way to know this wouldn't do what you intended is the fact that __name__ must differ between the subprocess and main program. That means it's in a different module, even if they were sharing a process, so they don't share a global namespace.
First, you're trying to print "elapsee" instead of "elapsed".
Second, you probably want to put all the time code outside MainCode completely, instead starting before the process start and the joins, if you want an accurate total elapsed time.
Related
I want to execute two commands 1st in background and second on top of it.
import time
loop =[ 1,100]
start_time_loop = time.time()
for in loop:
print i
end_time_loop = time.time()
multi()
start_time_func = time.time()
c= 5*2
end_time_func = time.time()
Loop should run in background while multiplication is done.
I want to prove:
start_time_loop < start_time_func
end_time_func << end_time_loop
Any pointers will be helpful.
You need to use multiprocessing to do what you want. It basically starts a new (cloned) copy of python in the background.
import time
from multiprocessing import Process
def thing_1():
"""We'll run this in a subprocess."""
for i in range(10):
print('thing_1: {}'.format(i))
# let's make it take a bit longer
time.sleep(1)
def thing_2():
"""We'll run this as a normal function in the current python process."""
time.sleep(1)
c = 5 * 2
print('thing_2: {}'.format(c))
# let's make this take a bit longer too
time.sleep(1)
if __name__ == '__main__':
# start the first thing running "in the background"
p = Process(target=thing_1)
p.start()
# keep a record of when we started it running
start_thing_1 = time.time()
# let's run the other thing
start_thing_2 = time.time()
thing_2()
end_thing_2 = time.time()
# this will wait for the first thing to finish
p.join()
end_thing_1 = time.time()
print('thing 1 took {}'.format(end_thing_1 - start_thing_1))
print('thing 2 took {}'.format(end_thing_2 - start_thing_2))
At the end you'll see:
thing 1 took 10.020239114761353
thing 2 took 2.003588914871216
So, while thing_1 is running in the background your local python can carry on doing other things.
You need to use special mechanisms to transfer any information between the two copies of python. And printing will always be a bit weird because you don't really know which copy of python is going to print next.
I've got this program:
import multiprocessing
import time
def timer(sleepTime):
time.sleep(sleepTime)
fooProcess.terminate()
fooProcess.join() #line said to "cleanup", not sure if it is required, refer to goo.gl/Qes6KX
def foo():
i=0
while 1
print i
time.sleep(1)
i
if i==4:
#pause timerProcess for X seconds
fooProcess = multiprocessing.Process(target=foo, name="Foo", args=())
timer()
fooProcess.start()
And as you can see in the comment, under certain conditions (in this example i has to be 4) the timer has to stop for a certain X time, while foo() keeps working.
Now, how do I implement this?
N.B.: this code is just an example, the point is that I want to pause a process under certain conditions for a certain amount of time.
I am think you're going about this wrong for game design. Games always (no exceptions come to mind) use a primary event loop controlled in software.
Each time through the loop you check the time and fire off all the necessary events based on how much time has elapsed. At the end of the loop you sleep only as long as necessary before you got the next timer or event or refresh or ai check or other state change.
This gives you the best performance regarding lag, consistency, predictability, and other timing features that matter in games.
roughly:
get the current timestamp at the time start time (time.time(), I presume)
sleep with Event.wait(timeout=...)
wake up on an Event or timeout.
if on Event: get timestamp, subtract initial on, subtract result from timer; wait until foo() stops; repeat Event.wait(timeout=[result from 4.])
if on timeout: exit.
Here is an example, how I understand, what your Programm should do:
import threading, time, datetime
ACTIVE = True
def main():
while ACTIVE:
print "im working"
time.sleep(.3)
def run(thread, timeout):
global ACTIVE
thread.start()
time.sleep(timeout)
ACTIVE = False
thread.join()
proc = threading.Thread(target = main)
print datetime.datetime.now()
run(proc, 2) # run for 2 seconds
print datetime.datetime.now()
In main() it does a periodic task, here printing something. In the run() method you can say, how long main should do the task.
This code producess following output:
2014-05-25 17:10:54.390000
im working
im working
im working
im working
im working
im working
im working
2014-05-25 17:10:56.495000
please correct me, if I've understood you wrong.
I would use multiprocessing.Pipe for signaling, combined with select for timing:
#!/usr/bin/env python
import multiprocessing
import select
import time
def timer(sleeptime,pipe):
start = time.time()
while time.time() < start + sleeptime:
n = select.select([pipe],[],[],1) # sleep in 1s intervals
for conn in n[0]:
val = conn.recv()
print 'got',val
start += float(val)
def foo(pipe):
i = 0
while True:
print i
i += 1
time.sleep(1)
if i%7 == 0:
pipe.send(5)
if __name__ == '__main__':
mainpipe,foopipe = multiprocessing.Pipe()
fooProcess = multiprocessing.Process(target=foo,name="Foo",args=(foopipe,))
fooProcess.start()
timer(10,mainpipe)
fooProcess.terminate()
# since we terminated, mainpipe and foopipe are corrupt
del mainpipe, foopipe
# ...
print 'Done'
I'm assuming that you want some condition in the foo process to extend the timer. In the sample I have set up, every time foo hits a multiple of 7 it extends the timer by 5 seconds while the timer initially counts down 10 seconds. At the end of the timer we terminate the process - foo won't finish nicely at all, and the pipes will get corrupted, but you can be certain that it'll die. Otherwise you can send a signal back along mainpipe that foo can listen for and exit nicely while you join.
I would like to measure the time elapsed to evaluate a block of code in a Python program,
possibly separating between user cpu time, system cpu time and elapsed time.
I know the timeit module, but I have many self-written functions and it is not very easy
to pass them in the setup process.
I would rather have something that could be used like:
#up to here I have done something....
start_counting() #or whatever command used to mark that I want to measure
#the time elapsed in the next rows
# code I want to evaluate
user,system,elapsed = stop_counting() #or whatever command says:
#stop the timer and return the times
The user and system CPU times are not essential (though I would like to measure them),
but for the elapsed time I would like to be able to do something like this,
rather than using complicated commands or modules.
To get the elapsed time in seconds, you can use timeit.default_timer():
import timeit
start_time = timeit.default_timer()
# code you want to evaluate
elapsed = timeit.default_timer() - start_time
timeit.default_timer() is used instead of time.time() or time.clock() because it will choose the timing function that has the higher resolution for any platform.
I always use a decorator to do some extra work for a existing function, including to get the execution time. It is pythonic and simple.
import time
def time_usage(func):
def wrapper(*args, **kwargs):
beg_ts = time.time()
retval = func(*args, **kwargs)
end_ts = time.time()
print("elapsed time: %f" % (end_ts - beg_ts))
return retval
return wrapper
#time_usage
def test():
for i in xrange(0, 10000):
pass
if __name__ == "__main__":
test()
I found myself solving this problem again and again, so I finally created a library for it. Install with pip install timer_cm. Then:
from time import sleep
from timer_cm import Timer
with Timer('Long task') as timer:
with timer.child('First step'):
sleep(1)
for _ in range(5):
with timer.child('Baby steps'):
sleep(.5)
Output:
Long task: 3.520s
Baby steps: 2.518s (71%)
First step: 1.001s (28%)
You can achieve this through the Context Manager, for example:
from contextlib import contextmanager
import time
import logging
#contextmanager
def _log_time_usage(prefix=""):
'''log the time usage in a code block
prefix: the prefix text to show
'''
start = time.time()
try:
yield
finally:
end = time.time()
elapsed_seconds = float("%.2f" % (end - start))
logging.debug('%s: elapsed seconds: %s', prefix, elapsed_seconds)
use example:
with _log_time_usage("sleep 1: "):
time.sleep(1)
There is one more option which i loves a lot now for simplicity - ipython. In ipython you got a lot of useful stuff plus:
%time <expression> - to get straight cpu and wall time on expression
%timeit <expression> - to get cpu and wall time in a loop of expression
Python 3 - Simple solution using standard library
Option 1: Triple quote the code
import inspect
import timeit
code_block = inspect.cleandoc("""
base = 123456789
exponent = 100
return base ** exponent
""")
print(f'\Code block: {timeit.timeit(code_block, number=1, globals=globals())} elapsed seconds')
inspect.cleandoc handles the removal of extra tabs and whitespace so that blocks of code can be copied and pasted without getting indentation errors.
Option 2: Place code block in a function
import timeit
def my_function():
base = 123456789
exponent = 100
return base ** exponent
if __name__ == '__main__':
print(f'With lambda wrapper: {timeit.timeit(lambda: my_function(), number=1)} elapsed seconds')
Note that a function call will add additional execution time versus timing the function body directly.
I'm reading serial data with a while loop. However, I have no control over the sample rate.
The code itself seems to take 0.2s to run, so I know I won't be able to go any faster than that. But I would like to be able to control precisely how much slower I sample.
I feel like I could do it using 'sleep', but the problem is that there is potential that at different points the loop itself will take longer to read(depending on precisely what is being transmitted over serial data), so the code would have to make up the balance.
For example, let's say I want to sample every 1s, and the loop takes anywhere from 0.2s to 0.3s to run. My code needs to be smart enough to sleep for 0.8s (if the loop takes 0.2s) or 0.7s (if the loop takes 0.3s).
import serial
import csv
import time
#open serial stream
while True:
#read and print a line
sample_value=ser.readline()
sample_time=time.time()-zero
sample_line=str(sample_time)+','+str(sample_value)
outfile.write(sample_line)
print 'time: ',sample_time,', value: ',sample_value
Just measure the time running your code takes every iteration of the loop, and sleep accordingly:
import time
while True:
now = time.time() # get the time
do_something() # do your stuff
elapsed = time.time() - now # how long was it running?
time.sleep(1.-elapsed) # sleep accordingly so the full iteration takes 1 second
Of course not 100% perfect (maybe off one millisecond or another from time to time), but I guess it's good enough.
Another nice approach is using twisted's LoopingCall:
from twisted.internet import task
from twisted.internet import reactor
def do_something():
pass # do your work here
task.LoopingCall(do_something).start(1.0)
reactor.run()
An rather elegant method is you're working on UNIX : use the signal library
The code :
import signal
def _handle_timeout():
print "timeout hit" # Do nothing here
def second(count):
signal.signal(signal.SIGALRM, _handle_timeout)
signal.alarm(1)
try:
count += 1 # put your function here
signal.pause()
finally:
signal.alarm(0)
return count
if __name__ == '__main__':
count = 0
count = second(count)
count = second(count)
count = second(count)
count = second(count)
count = second(count)
print count
And the timing :
georgesl#cleese:~/Bureau$ time python timer.py
5
real 0m5.081s
user 0m0.068s
sys 0m0.004s
Two caveats though : it only works on *nix, and it is not multithread-safe.
At the beginning of the loop check if the appropriate amount of time has passed. If it has not, sleep.
# Set up initial conditions for sample_time outside the loop
sample_period = ???
next_min_time = 0
while True:
sample_time = time.time() - zero
if sample_time < next_min_time:
time.sleep(next_min_time - sample_time)
continue
# read and print a line
sample_value = ser.readline()
sample_line = str(sample_time)+','+str(sample_value)
outfile.write(sample_line)
print 'time: {}, value: {}'.format(sample_time, sample_value)
next_min_time = sample_time + sample_period
I received the following script from a fellow programmer:
from time import *
start = strptime(asctime())
end = strptime(asctime())
print 'you took %i minutes' % (end[4] - start[4])
As you might already know, the script measures the time between the third and fourth line. However, it seems to measure it in minutes. How can I go about to measure it in seconds, with at least one decimal place (ex. 7.4 seconds).
Also, I would like to add one extra piece of functionality. Say the script runs, and I am asked by the program to type any word. At the end of my typing, I have to press the enter key to exit the program and measure the time from the first keystroke to the moment I press enter. How can I go about measuring this?
First, I would avoid using import * as it's considered bad practice. You can use time.time() to get more precision:
>>> import time
>>> start = time.time()
>>> end = time.time()
>>> end - start
5.504057168960571
You could also use datetime.datetime.now().
#source: http://docs.python.org/library/timeit.html
def test():
"""Stupid test function"""
L = []
for i in range(100):
L.append(i)
if __name__ == '__main__':
from timeit import Timer
t = Timer("test()", "from __main__ import test")
print t.timeit()
If you are trying to optimize a python web service call, you can do the following.
import time
In the beginning of the function, right
start = time.time()
in the line put (540 is the line number),
l540 = time.time()
print("--------l541 ------------")
print(l540 - start)
in the next line put (608 is the line number),
l608 = time.time()
print("-------- 609 ------------")
print(l608 - l540)
You can add as many as you want and it will tell you where exactly the program is taking time.