Develop Reference

How to get every second's GPU usage in Python

I have a model which runs by tensorflow-gpu and my device is nvidia. And I want to list every second's GPU usage so that I can measure average/max GPU usage. I can do this mannually by open two terminals, one is to run model and another is to measure by nvidia-smi -l 1. Of course, this is not a good way. I also tried to use a Thread to do that, here it is.
import subprocess as sp
import os
from threading import Thread
class MyThread(Thread):
def __init__(self, func, args):
super(MyThread, self).__init__()
self.func = func
self.args = args
def run(self):
self.result = self.func(*self.args)
def get_result(self):
return self.result
def get_gpu_memory():
output_to_list = lambda x: x.decode('ascii').split('\n')[:-1]
ACCEPTABLE_AVAILABLE_MEMORY = 1024
COMMAND = "nvidia-smi -l 1 --query-gpu=memory.used --format=csv"
memory_use_info = output_to_list(sp.check_output(COMMAND.split()))[1:]
memory_use_values = [int(x.split()[0]) for i, x in enumerate(memory_use_info)]
return memory_use_values
def run():
pass
t1 = MyThread(run, args=())
t2 = MyThread(get_gpu_memory, args=())
t1.start()
t2.start()
t1.join()
t2.join()
res1 = t2.get_result()
However, this does not return every second's usage as well. Is there a good solution?

In the command nvidia-smi -l 1 --query-gpu=memory.used --format=csv
the -l stands for:
-l, --loop= Probe until Ctrl+C at specified second interval.
So the command:
COMMAND = 'nvidia-smi -l 1 --query-gpu=memory.used --format=csv'
sp.check_output(COMMAND.split())
will never terminate and return.
It works if you remove the event loop from the command(nvidia-smi) to python.
Here is the code:
import subprocess as sp
import os
from threading import Thread , Timer
import sched, time
def get_gpu_memory():
output_to_list = lambda x: x.decode('ascii').split('\n')[:-1]
ACCEPTABLE_AVAILABLE_MEMORY = 1024
COMMAND = "nvidia-smi --query-gpu=memory.used --format=csv"
try:
memory_use_info = output_to_list(sp.check_output(COMMAND.split(),stderr=sp.STDOUT))[1:]
except sp.CalledProcessError as e:
raise RuntimeError("command '{}' return with error (code {}): {}".format(e.cmd, e.returncode, e.output))
memory_use_values = [int(x.split()[0]) for i, x in enumerate(memory_use_info)]
# print(memory_use_values)
return memory_use_values
def print_gpu_memory_every_5secs():
"""
This function calls itself every 5 secs and print the gpu_memory.
"""
Timer(5.0, print_gpu_memory_every_5secs).start()
print(get_gpu_memory())
print_gpu_memory_every_5secs()
"""
Do stuff.
"""

Here is a more rudimentary way of getting this output, however just as effective - and I think easier to understand. I added a small 10-value cache to get a good recent average and upped the check time to every second. It outputs average of the last 10 seconds and the current each second, so operations that cause usage can be identified (what I think the original question was).
import subprocess as sp
import time
memory_total=8192 #found with this command: nvidia-smi --query-gpu=memory.total --format=csv
memory_used_command = "nvidia-smi --query-gpu=memory.used --format=csv"
isolate_memory_value = lambda x: "".join(y for y in x.decode('ascii') if y in "0123456789")
def main():
percentage_cache = []
while True:
memory_used = isolate_memory_value(sp.check_output(memory_used_command.split(), stderr=sp.STDOUT))
percentage = float(memory_used)/float(memory_total)*100
percentage_cache.append(percentage)
percentage_cache = percentage_cache[max(0, len(percentage_cache) - 10):]
print("curr: " + str(percentage) + " %", "\navg: " + str(sum(percentage_cache)/len(percentage_cache))[:4] + " %\n")
time.sleep(1)
main()

Python run function in parallel to main code

I have the following code (Python 3.7 on windows 64bit):
from time import sleep
import time
from multiprocessing import Process
### function ###
def func(l):
for i in l:
sleep(1)
print (i)
t1 = time.time()
total = t1-t0
print ('time : ',total)
### main code ###
t0 = time.time()
l = list(range(1, 4))
if __name__ == '__main__':
p = Process(target=func, args=(l,))
p.start()
p.join()
sleep(10)
print ('done')
t1 = time.time()
total = t1-t0
print ('time : ',total)
The goal is to have a function run in parallel with the main block of code. When I run this I get the following result:
done
time : 10.000610828399658
1
time : 11.000777244567871
2
time : 12.001059532165527
3
time : 13.00185513496399
done
time : 23.11873483657837
However I was expecting the following:
1
time: ~1
2
time: ~2
3
time: ~3
done
time: ~10
So essentially I want the function to run in parallel with the main code. I am confused because without multiprocessing this code should run at most for 13 seconds but it is running for 23. The goal is to have it run in 10 seconds.
How can I fix this to have it work as intended?

I can't reproduce the problem where the first time printed is ~10, when I try it, I get times starting from ~1, as expected.
My final time from the parent process is ~13. This is because of p.join(), which waits for the child process to finish. If I remove that, the time printed in the parent is ~10.
Script:
from time import sleep
import time
from multiprocessing import Process
### function ###
def func(l):
for i in l:
sleep(1)
print (i)
t1 = time.time()
total = t1-t0
print ('time : ',total)
### main code ###
t0 = time.time()
l = list(range(1, 4))
if __name__ == '__main__':
p = Process(target=func, args=(l,))
p.start()
# p.join()
sleep(10)
print ('done')
t1 = time.time()
total = t1-t0
print ('time : ',total)
Output:
$ python testmultiproc.py
1
time : 1.0065689086914062
2
time : 2.0073459148406982
3
time : 3.0085067749023438
done
time : 10.008337020874023

Number of vehicles each T seconds

I wrote the script below with python and i implemented it on sumo,in order to obtain the number of vehicles between two inductionLoop,every 60 seconds,in a lane.
But this one gives each second .
#!/usr/bin/env python
# -*-coding:Latin-1 -*
import os, sys
import optparse
import subprocess
import random
import threading
import time
SUMO_HOME = "/home/khadija/Téléchargements/sumo-0.25.0"
try:
sys.path.append(os.path.join(SUMO_HOME, "tools"))
from sumolib import checkBinary
except ImportError:
sys.exit("please declare environment variable 'SUMO_HOME' as the root directory of your sumo installation (it should contain folders 'bin', 'tools' and 'docs')")
import traci
routeFile="data2/cross.rou.xml"
PORT = 8873
#SIGN CONFIGURATIONS : NESW
NSgreen = "GrGr"
EWgreen = "rGrG"
PROGRAM = (NSgreen,EWgreen)
def nbr_veh_entr_indloop(i,o):
# i et j se sont les inductions loop input et output
threading.Timer(60.0, nbr_veh_entr_indloop).start()
x = traci.inductionloop.getLastStepMeanLength(i) - traci.inductionloop.getLastStepMeanLength(o)
return x
def run():
steps = open("data2/step.txt","w")
traci.init(int(PORT))
step = 0
while step < 7200 :
a = nbr_veh_entr_indloop("4i","40")
k=str(a)
print >> steps , "nombre des veh : " + k #concaténation
traci.simulationStep()
step +=1
steps.close()
traci.close()
sys.stdout.flush()
def get_options():
optParser = optparse.OptionParser()
optParser.add_option("--nogui", action="store_true",
default=False, help="run the commandline version of sumo")
options, args = optParser.parse_args()
return options
# this is the main entry point of this script
if __name__ == "__main__":
options = get_options()
# this script has been called from the command line. It will start sumo as a
# server, then connect and run
if options.nogui:
sumoBinary = checkBinary('sumo')
else:
sumoBinary = checkBinary('sumo-gui')
# this is the normal way of using traci. sumo is started as a
# subprocess and then the python script connects and runs
sumoProcess = subprocess.Popen([sumoBinary, "-c", "data2/cross.sumocfg", "--tripinfo-output","tripinfo.xml", "--remote-port", str(PORT)], stdout=sys.stdout, stderr=sys.stderr)
run()
sumoProcess.wait()
Thanks for help in advance.
Regards,

You probably want to have have the value every 60 simulation seconds not every 60 wallclock seconds, so a timer is pointless here. Simply ask for the value after 60 simulation steps (assuming you use sumo's default step length of one second). So you could write something like:
if step % 60 == 0:
print >> steps , "nombre des veh : " + k
This will print the value for the last step every 60 steps. If you want the value for the last minute you need to aggregate (sum up) yourself.

Data not printing correct

When running a data streaming sub-process against a multiprocessing, after printing a single element in the dictionary program freezes,
#!/usr/bin/python
import subprocess,time, timeit
from multiprocessing import Process, Queue
import re, os, pprint, math
from collections import defaultdict
Dict = {}
count = defaultdict(int)
queueVar = Queue()
def __ReadRX__(RX_info):
lines = iter(RX_info.stdout.readline, "")
try:
start = time.clock()
for line in lines:
if re.match(r"^\d+.*$",line):
splitline = line.split()
del splitline[1:4]
identifier = splitline[1]
count[identifier] += 1
end = time.clock()
timing = round((end - start) * 10000, 100)
dlc = splitline[2]
hexbits = splitline[3:]
Dict[identifier] = [dlc, hexbits, count[identifier],int(timing)]
start = end
for identifier,hexbits in Dict.items():
queueVar.put(Dict)
except KeyboardInterrupt:
pass
procRX = subprocess.Popen('receivetest -f=/dev/pcan32'.split(), stdout=subprocess.PIPE)
if __name__ == '__main__':
munchCan = Process(target=__ReadRX__, args=(procRX,))
munchCan.start()
#munchCan.join()
printDict = queueVar.get()
for i in range(len(printDict)):
print printDict
I know if if I print from __ReadRX__ it prints a constant stream however when trying to print from outside of the function I only get a single entry in the dictionary.

add the following at the top:
from time import sleep
then after the print statement add:
sleep(1)
this will make the script wait for 1 second. You can adjust that number as necessary.

How can I clear a line in console after using \r and printing some text?

For my current project, there are some pieces of code that are slow and which I can't make faster. To get some feedback how much was done / has to be done, I've created a progress snippet which you can see below.
When you look at the last line
sys.stdout.write("\r100%" + " "*80 + "\n")
I use " "*80 to override eventually remaining characters. Is there a better way to clear the line?
(If you find the error in the calculation of the remaining time, I'd also be happy. But that's the question.)
Progress snippet
#!/usr/bin/env python
import time
import sys
import datetime
def some_slow_function():
start_time = time.time()
totalwork = 100
for i in range(totalwork):
# The slow part
time.sleep(0.05)
if i > 0:
# Show how much work was done / how much work is remaining
percentage_done = float(i)/totalwork
current_running_time = time.time() - start_time
remaining_seconds = current_running_time / percentage_done
tmp = datetime.timedelta(seconds=remaining_seconds)
sys.stdout.write("\r%0.2f%% (%s remaining) " %
(percentage_done*100, str(tmp)))
sys.stdout.flush()
sys.stdout.write("\r100%" + " "*80 + "\n")
sys.stdout.flush()
if __name__ == '__main__':
some_slow_function()
Consoles
I use ZSH most of the time, sometimes bash (and I am always on a Linux system)

Try using the ANSI/vt100 "erase to end of line" escape sequence:
sys.stdout.write("\r100%\033[K\n")
Demonstration:
for i in range(4):
sys.stdout.write("\r" + ("."*i*10))
sys.stdout.flush()
if i == 3:
sys.stdout.write("\rDone\033[K\n")
time.sleep(1.5)
Reference: https://en.wikipedia.org/wiki/ANSI_escape_code#CSI_sequences

This is what I use
from msvcrt import putch, getch
def putvalue(value):
for c in str(value):
putch(c)
def overwrite(value):
""" Used to overwrite the current line in the command prompt,
useful when displaying percent or progress """
putvalue('\r'+str(value))
from time import sleep
for x in xrange(101):
overwrite("Testing Overwrite.........%s%% complete" % x)
sleep(.05)