I have the following code (Python 3.7 on windows 64bit):
from time import sleep
import time
from multiprocessing import Process
### function ###
def func(l):
for i in l:
sleep(1)
print (i)
t1 = time.time()
total = t1-t0
print ('time : ',total)
### main code ###
t0 = time.time()
l = list(range(1, 4))
if __name__ == '__main__':
p = Process(target=func, args=(l,))
p.start()
p.join()
sleep(10)
print ('done')
t1 = time.time()
total = t1-t0
print ('time : ',total)
The goal is to have a function run in parallel with the main block of code. When I run this I get the following result:
done
time : 10.000610828399658
1
time : 11.000777244567871
2
time : 12.001059532165527
3
time : 13.00185513496399
done
time : 23.11873483657837
However I was expecting the following:
1
time: ~1
2
time: ~2
3
time: ~3
done
time: ~10
So essentially I want the function to run in parallel with the main code. I am confused because without multiprocessing this code should run at most for 13 seconds but it is running for 23. The goal is to have it run in 10 seconds.
How can I fix this to have it work as intended?
I can't reproduce the problem where the first time printed is ~10, when I try it, I get times starting from ~1, as expected.
My final time from the parent process is ~13. This is because of p.join(), which waits for the child process to finish. If I remove that, the time printed in the parent is ~10.
Script:
from time import sleep
import time
from multiprocessing import Process
### function ###
def func(l):
for i in l:
sleep(1)
print (i)
t1 = time.time()
total = t1-t0
print ('time : ',total)
### main code ###
t0 = time.time()
l = list(range(1, 4))
if __name__ == '__main__':
p = Process(target=func, args=(l,))
p.start()
# p.join()
sleep(10)
print ('done')
t1 = time.time()
total = t1-t0
print ('time : ',total)
Output:
$ python testmultiproc.py
1
time : 1.0065689086914062
2
time : 2.0073459148406982
3
time : 3.0085067749023438
done
time : 10.008337020874023
Related
i want to check every 500 milliseconds if a process/application is running (Windows 10). The code should be very fast and resource efficient!
My Code is this but how to build the 500 milliseconds in. Is psutil the fastest and best way? Thank You.
import psutil
for p in psutil.process_iter(attrs=['pid', 'name']):
if "excel.exe" in (p.info['name']).lower():
print("Application is running", (p.info['name']).lower())
else:
print("Application is not Running")
How about doing it like this:
import psutil
import time
def running(pname):
pname = pname.lower()
for p in psutil.process_iter(attrs=['name']):
if pname in p.info['name'].lower():
print(f'{pname} is running')
return # early return
print(f'{pname} is not running')
while True:
running('excel.exe')
time.sleep(0.5)
First of all, psutil is a pretty good library. It has C Bindings so you won't be able to get much faster.
import psutil
import time
def print_app():
present = False
for p in psutil.process_iter(attrs=['pid', 'name']):
if "excel.exe" in (p.info['name']).lower():
present = True
print(f"Application is {'' if present else 'not'} present")
start_time = time.time()
print_app()
print("--- %s seconds ---" % (time.time() - start_time))
You can know how much time it takes. 0.06sec for me.
if you want to exec this every 0.5s you can simply put a time.sleep because 0.5 >> 0.06.
You can then write this kind of code:
import psutil
import time
def print_app():
present = False
for p in psutil.process_iter(attrs=['pid', 'name']):
if "excel.exe" in (p.info['name']).lower():
present = True
print(f"Application is {'' if present else 'not'} present")
while True:
print_app()
sleep(0.5)
PS: I changed your code to check if your app was running without printing it. This makes the code faster because print takes a bit of time.
I am currently executing tasks via a thread pool based on a for loop length, and it is ending its execution when it is not supposed to (before end of loop). Any ideas why? Here is the relavent code:
from classes.scraper import size
from multiprocessing import Pool
import threading
if __name__ == '__main__':
print("Do something")
size = size()
pool = Pool(processes=50)
with open('size.txt','r') as file:
asf = file.read()
for x in range(0,1000000):
if '{num:06d}'.format(num=x) in asf:
continue
else:
res = pool.apply_async(size.scrape, ('{num:06d}'.format(num=x),))
Here is the console output (I am printing out the values inside size.scrape().
...
...
...
013439
013440
013441
013442
013443
Process finished with exit code 0
I'm trying to launch two python scripts simultaneously from another script. It works with subprocess.Popen. But I also would like to recover the output of these two scripts launched simultaneously.
When I use subprocess.check_output, I manage to recover the outputs but the scripts are not launched at the same time.
I have made a simple example to illustrate the problem. The programm 2scripts.py calls the scripts aa.py and bb.py.
aa.py :
import time
delay = 0
t0 = time.time()
print "temps " + str(t0)
print("aa")
while delay < 5 :
delay = time.time() - t0
bb.py :
import time
delay = 0
t0 = time.time()
print "temps " + str(t0)
print("bb")
while delay < 5 :
delay = time.time() - t0
This is 2scripts.py and the output with subprocess.Popen :
2scripts.py :
import subprocess
x = subprocess.Popen((["python", "aa.py"]))
y = subprocess.Popen((["python", "bb.py"]))
temps 1460040113.05
aa
temps 1460040113.05
bb
And 2scripts.py and the output with subprocess.check_output()
import subprocess
x = subprocess.check_output((["python", "aa.py"]))
y = subprocess.check_output((["python", "bb.py"]))
print(x)
print(y)
temps 1460040186.3
aa
temps 1460040191.31
bb
You can use multiprocessing.pool to run both at the same time and only print the outputs when both have returned results. This way, they both start at the same time and you manage to get their outputs.
Code:
import subprocess
import multiprocessing.pool
import time
pool = multiprocessing.pool.ThreadPool(2)
x,y = pool.map(lambda x: x(), [
lambda: subprocess.check_output((["python", "aa.py"])),
lambda: subprocess.check_output((["python", "bb.py"]))
])
print(x)
print(y)
Output:
temps 1460050982.44
aa
temps 1460050982.44
bb
An enhanced example of #Ru Hasha with number of thread as first command line parameter and only one script to illustrate the different command called.
xscripts.py
import sys, subprocess, multiprocessing.pool
nb_threads = int(sys.argv[1])
pool = multiprocessing.pool.ThreadPool(nb_threads)
processes = []
for i in range(nb_threads):
processes.append(lambda i=i: subprocess.check_output((["python", "mic.py", "mic" + str(i)])))
outputs = pool.map(lambda x: x(), processes)
for o in outputs:
print o
mic.py
import sys, time
sys.stdout.write(sys.argv[1] + " start time\t" + str(time.time()) + '\n')
time.sleep(2)
sys.stdout.write(sys.argv[1] + " end time\t" + str(time.time()))
Output
$ python xscripts.py 4
mic0 start time 1460071350.1
mic0 end time 1460071352.1
mic1 start time 1460071350.1
mic1 end time 1460071352.1
mic2 start time 1460071350.1
mic2 end time 1460071352.1
mic3 start time 1460071350.1
mic3 end time 1460071352.1
When running a data streaming sub-process against a multiprocessing, after printing a single element in the dictionary program freezes,
#!/usr/bin/python
import subprocess,time, timeit
from multiprocessing import Process, Queue
import re, os, pprint, math
from collections import defaultdict
Dict = {}
count = defaultdict(int)
queueVar = Queue()
def __ReadRX__(RX_info):
lines = iter(RX_info.stdout.readline, "")
try:
start = time.clock()
for line in lines:
if re.match(r"^\d+.*$",line):
splitline = line.split()
del splitline[1:4]
identifier = splitline[1]
count[identifier] += 1
end = time.clock()
timing = round((end - start) * 10000, 100)
dlc = splitline[2]
hexbits = splitline[3:]
Dict[identifier] = [dlc, hexbits, count[identifier],int(timing)]
start = end
for identifier,hexbits in Dict.items():
queueVar.put(Dict)
except KeyboardInterrupt:
pass
procRX = subprocess.Popen('receivetest -f=/dev/pcan32'.split(), stdout=subprocess.PIPE)
if __name__ == '__main__':
munchCan = Process(target=__ReadRX__, args=(procRX,))
munchCan.start()
#munchCan.join()
printDict = queueVar.get()
for i in range(len(printDict)):
print printDict
I know if if I print from __ReadRX__ it prints a constant stream however when trying to print from outside of the function I only get a single entry in the dictionary.
add the following at the top:
from time import sleep
then after the print statement add:
sleep(1)
this will make the script wait for 1 second. You can adjust that number as necessary.
Trying to use line_profiler as an API. Following their docs and this tutorial (scroll down to Line Profiling), I get a minimalist test case for profiling some numpy ufuncs:
import numpy as np
import line_profiler
import time
shp = (1000,1000)
a = np.ones(shp)
o = np.zeros(shp)
def main():
t = time.time()
np.divide(a,1,o)
for i in xrange(200):
np.multiply(a,2,o)
np.add(a,1,o)
print 'duration', time.time()-t
profiler = line_profiler.LineProfiler()
profiler.add_function(main)
main()
profiler.print_stats()
I get this in stdout which indicates that main ran, but was not profiled:
duration 2.6779999733
Timer unit: 5.59936e-07 s
File: testprof.py
Function: main at line 9
Total time: 0 s
Line # Hits Time Per Hit % Time Line Contents
==============================================================
9 def main():
10 t = time.time()
11 np.divide(a,1,o)
12 for i in xrange(200):
13 np.multiply(a,2,o)
14 np.add(a,1,o)
15 print 'duration', time.time
()-t
I'm new to line_profiler. See my other q if curious why I don't use cProfile.
Try add
profiler.enable_by_count()
before
main()