Develop Reference

Python multiprocessing progress approach

I've been busy writing my first multiprocessing code and it works, yay.
However, now I would like some feedback of the progress and I'm not sure what the best approach would be.
What my code (see below) does in short:
A target directory is scanned for mp4 files
Each file is analysed by a separate process, the process saves a result (an image)
What I'm looking for could be:
Simple
Each time a process finishes a file it sends a 'finished' message
The main code keeps count of how many files have finished
Fancy
Core 0 processing file 20 of 317 ||||||____ 60% completed
Core 1 processing file 21 of 317 |||||||||_ 90% completed
...
Core 7 processing file 18 of 317 ||________ 20% completed
I read all kinds of info about queues, pools, tqdm and I'm not sure which way to go. Could anyone point to an approach that would work in this case?
Thanks in advance!
EDIT: Changed my code that starts the processes as suggested by gsb22
My code:
# file operations
import os
import glob
# Multiprocessing
from multiprocessing import Process
# Motion detection
import cv2
# >>> Enter directory to scan as target directory
targetDirectory = "E:\Projects\Programming\Python\OpenCV\\videofiles"
def get_videofiles(target_directory):
# Find all video files in directory and subdirectories and put them in a list
videofiles = glob.glob(target_directory + '/**/*.mp4', recursive=True)
# Return the list
return videofiles
def process_file(videofile):
'''
What happens inside this function:
- The video is processed and analysed using openCV
- The result (an image) is saved to the results folder
- Once this function receives the videofile it completes
without the need to return anything to the main program
'''
# The processing code is more complex than this code below, this is just a test
cap = cv2.VideoCapture(videofile)
for i in range(10):
succes, frame = cap.read()
# cv2.imwrite('{}/_Results/{}_result{}.jpg'.format(targetDirectory, os.path.basename(videofile), i), frame)
if succes:
try:
cv2.imwrite('{}/_Results/{}_result_{}.jpg'.format(targetDirectory, os.path.basename(videofile), i), frame)
except:
print('something went wrong')
if __name__ == "__main__":
# Create directory to save results if it doesn't exist
if not os.path.exists(targetDirectory + '/_Results'):
os.makedirs(targetDirectory + '/_Results')
# Get a list of all video files in the target directory
all_files = get_videofiles(targetDirectory)
print(f'{len(all_files)} video files found')
# Create list of jobs (processes)
jobs = []
# Create and start processes
for file in all_files:
proc = Process(target=process_file, args=(file,))
jobs.append(proc)
for job in jobs:
job.start()
for job in jobs:
job.join()
# TODO: Print some form of progress feedback
print('Finished :)')

I read all kinds of info about queues, pools, tqdm and I'm not sure which way to go. Could anyone point to an approach that would work in this case?
Here's a very simple way to get progress indication at minimal cost:
from multiprocessing.pool import Pool
from random import randint
from time import sleep
from tqdm import tqdm
def process(fn) -> bool:
sleep(randint(1, 3))
return randint(0, 100) < 70
files = [f"file-{i}.mp4" for i in range(20)]
success = []
failed = []
NPROC = 5
pool = Pool(NPROC)
for status, fn in tqdm(zip(pool.imap(process, files), files), total=len(files)):
if status:
success.append(fn)
else:
failed.append(fn)
print(f"{len(success)} succeeded and {len(failed)} failed")
Some comments:
tqdm is a 3rd-party library which implements progressbars extremely well. There are others. pip install tqdm.
we use a pool (there's almost never a reason to manage processes yourself for simple things like this) of NPROC processes. We let the pool handle iterating our process function over the input data.
we signal state by having the function return a boolean (in this example we choose randomly, weighting in favour of success). We don't return the filename, although we could, because it would have to be serialised and sent from the subprocess, and that's unnecessary overhead.
we use Pool.imap, which returns an iterator which keeps the same order as the iterable we pass in. So we can use zip to iterate files directly. Since we use an iterator with unknown size, tqdm needs to be told how long it is. (We could have used pool.map, but there's no need to commit the ram---although for one bool it probably makes no difference.)
I've deliberately written this as a kind of recipe. You can do a lot with multiprocessing just by using the high-level drop in paradigms, and Pool.[i]map is one of the most useful.
References
https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.Pool
https://tqdm.github.io/

ffmpeg - Poll folder for files, and stream as video with rtp

(I'm a newbie when it comes to ffmpeg).
I have an image source which saves files to a given folder in a rate of 30 fps. I want to wait for every (let's say) 30 frames chunk, encode it to h264 and stream it with RDP to some other app.
I thought about writing a python app which just waits for the images, and then executes an ffmpeg command. For that I wrote the following code:
main.py:
import os
import Helpers
import argparse
import IniParser
import subprocess
from functools import partial
from Queue import Queue
from threading import Semaphore, Thread
def Run(config):
os.chdir(config.Workdir)
iteration = 1
q = Queue()
Thread(target=RunProcesses, args=(q, config.AllowedParallelRuns)).start()
while True:
Helpers.FileCount(config.FramesPathPattern, config.ChunkSize * iteration)
command = config.FfmpegCommand.format(startNumber = (iteration-1)*config.ChunkSize, vFrames=config.ChunkSize)
runFunction = partial(subprocess.Popen, command)
q.put(runFunction)
iteration += 1
def RunProcesses(queue, semaphoreSize):
semaphore = Semaphore(semaphoreSize)
while True:
runFunction = queue.get()
Thread(target=HandleProcess, args=(runFunction, semaphore)).start()
def HandleProcess(runFunction, semaphore):
semaphore.acquire()
p = runFunction()
p.wait()
semaphore.release()
if __name__ == '__main__':
argparser = argparse.ArgumentParser()
argparser.add_argument("config", type=str, help="Path for the config file")
args = argparser.parse_args()
iniFilePath = args.config
config = IniParser.Parse(iniFilePath)
Run(config)
Helpers.py (not really relevant):
import os
import time
from glob import glob
def FileCount(pattern, count):
count = int(count)
lastCount = 0
while True:
currentCount = glob(pattern)
if lastCount != currentCount:
lastCount = currentCount
if len(currentCount) >= count and all([CheckIfClosed(f) for f in currentCount]):
break
time.sleep(0.05)
def CheckIfClosed(filePath):
try:
os.rename(filePath, filePath)
return True
except:
return False
I used the following config file:
Workdir = "C:\Developer\MyProjects\Streaming\OutputStream\PPM"
; Workdir is the directory of reference from which all paths are relative to.
; You may still use full paths if you wish.
FramesPathPattern = "F*.ppm"
; The path pattern (wildcards allowed) where the rendered images are stored to.
; We use this pattern to detect how many rendered images are available for streaming.
; When a chunk of frames is ready - we stream it (or store to disk).
ChunkSize = 30 ; Number of frames for bulk.
; ChunkSize sets the number of frames we need to wait for, in order to execute the ffmpeg command.
; If the folder already contains several chunks, it will first process the first chunk, then second, and so on...
AllowedParallelRuns = 1 ; Number of parallel allowed processes of ffmpeg.
; This sets how many parallel ffmpeg processes are allowed.
; If more than one chunk is available in the folder for processing, we will execute several ffmpeg processes in parallel.
; Only when on of the processes will finish, we will allow another process execution.
FfmpegCommand = "ffmpeg -re -r 30 -start_number {startNumber} -i F%08d.ppm -vframes {vFrames} -vf vflip -f rtp rtp://127.0.0.1:1234" ; Command to execute when a bulk is ready for streaming.
; Once a chunk is ready for processing, this is the command that will be executed (same as running it from the terminal).
; There is however a minor difference. Since every chunk starts with a different frame number, you can use the
; expression of "{startNumber}" which will automatically takes the value of the matching start frame number.
; You can also use "{vFrames}" as an expression for the ChunkSize which was set above in the "ChunkSize" entry.
Please note that if I set "AllowedParallelRuns = 2" then it allows multiple ffmpeg processes to run simultaneously.
I then tried to play it with ffplay and see if I'm doing it right.
The first chunk was streamed fine. The following chunks weren't so great. I got a lot of [sdp # 0000006de33c9180] RTP: dropping old packet received too late messages.
What should I do so I get the ffplay, to play it in the order of the incoming images? Is it right to run parallel ffmpeg processes? Is there a better solution to my problem?
Thank you!

As I stated in the comment, since you rerun ffmpeg each time, the pts values are reset, but the client perceives this as a single continuous ffmpeg stream and thus expects increasing PTS values.
As I said you could use a ffmpeg python wrapper to control the streaming yourself, but yeah that is quite an amount of code. But, there is actually a dirty workaround.
So, apparently there is a -itsoffset parameter with which you can offset the input timestamps (see FFmpeg documentation). Since you know and control the rate, you could pass an increasing value with this parameter, so that each next stream is offset with the proper duration. E.g. if you stream 30 frames each time, and you know the fps is 30, the 30 frames create a time interval of one second. So on each call to ffmepg you would increase the -itsoffset value by one second, thus that should be added to the output PTS values. But I can't guarantee this works.
Since the idea about -itsoffset did not work, you could also try feeding the jpeg images via stdin to ffmpeg - see this link.

Python Sounddevice.play() on Threads

I am having some problems to play the sounddevice on a Thread. I import the sounddevice as sd at the beginning. Then during running I want to play a tone on a thread using the ASIO sound card. All the configurations I need to do on the thread works well. However, when I want to play the tone I got the following Error:
sounddevice.PortAudioError: Error opening OutputStream: Unanticipated host API 2 error 0: u'Failed to load ASIO driver'
If I initialize the import at the thread everytime I need it, it work. But of course I do not want to do that. Any Idea hot to solve it?
Thanks!
Here a simple code example:
from threading import Thread
import numpy as np
import sounddevice as sd
class Test(Thread):
def __init__(self):
Thread.__init__(self)
#-- Configuration of the Tone to be played
self.fs = 44100 # sampling rate, in Hz, 44100 or 48000
duration = 1.05 # in seconds, may be float
f = 200.0 # sine frequency, Hz, may be float
self.tone_data = (np.sin(2*np.pi*np.arange(self.fs*duration)*f/self.fs)).astype(np.float32)
def run(self):
#-- Configuration of the ASIO sound card
#import sounddevice as sd
sd.default.channels = 2
sd.default.device = 14
print sd.query_devices(sd.default.device)['name']
#sd.default.latency = ('low','low')
#asio_out = sd.AsioSettings(channel_selectors=[1, 2])
#sd.default.extra_settings = asio_out
sd.default.samplerate = self.fs
sd.play(self.tone_data)
sd.wait()
w = Test()
w.start()

This seems to be a platform-specific problem. I just tried it with ALSA/Linux and it works fine. With ASIO, you probably have to do the library initialization (which happens during import time) in the same thread you are using later to create the stream (which play() does for you)?
If I initialize the import at the thread everytime I need it, it work. But of course I do not want to do that.
Why do you not want to do that? Are you aware that the use of import in Python is cached automatically? The second time you use import, only a dict lookup is done and nothing else.
But you are right, the repeated import still looks a bit strange.
Did you try to do the import only once in Test.__init__()? There you could also do all the sd.default stuff.
If you still have problems during the initialization (or if you insist on having all imports at the top), you can try to use the undocumented _initialize() and _terminate() functions, see issue #3.
If you want to use multiple Thread instances, you'll get problems with the play() function, which is meant for single-threaded use. But it probably makes more sense anyway to have only one Python thread that does the audio I/O. See also PortAudio Tips – Threading.
BTW, you don't need (...).astype(np.float32), this conversion is done automatically for you.
And while I'm at it, your line sd.query_devices(sd.default.device)['name'] will break if the default input and output devices are different.

Speeding up the sound processing algorithm

I use the following code to do some immediate sound processing/analyzing. It works, but really slow (compared to the planned speed). I have added some time markers to find out where the problem is and according to them there shouldn't be any. Typical duration (see below) is <0.01 s for all three computed times but it still takes around a second to complete the loop. Where is the problem?
Edit: Please note, that the time measurement is not the real issue here. To prove that: MyPeaks basically just finds the maximum of pretty short FFT - nothing expensive. And the problem persists even when these routines are commented out.
Should I use something different than lambda function to make the cycle?
Did I make some mistake when starting and recording the stream?
etc.
import pyaudio
import struct
import mute_alsa
import time
import numpy as np
from Tkinter import *
def snd_process(k=0):
if k<1000:
t0=time.clock()
data = stream.read(CHUNK)
t1=time.clock()
fl=CHUNK
int_data = struct.unpack("%sh" %str(fl),data)
ft=np.fft.fft(int_data)
ft=np.fft.fftshift(ft)
ft=np.abs(ft)
t2=time.clock()
pks=MyPeaks(np.log(ft))
freq_out.configure(text=str(pks))
t3=time.clock()
print t1-t0, t2-t1, t3-t2
master.after(1, lambda: snd_process(k+1))
CHUNK = 8000
FORMAT = pyaudio.paInt16
CHANNELS = 1
RATE = 4000
p = pyaudio.PyAudio()
stream = p.open(format=FORMAT,
channels=CHANNELS,
rate=RATE,
input=True,
frames_per_buffer=CHUNK)
#Tkinter stuff
master=Tk()
button_play=Button(master, command=snd_process, bg="yellow", text="Analyze")
button_play.grid(row=0, column=0)
freq_out = Label(master)
freq_out.grid(row=0, column=1)
freq_out.configure(text='base')
mainloop()

You are scheduling 1000 callback in tk main thread; for every callback you are using 1 ms delay (after()'s first argument). That means the last loop will start around after 1000 ms (1 second) the first one.
Maybe that is way the loop still takes around a second to complete.
So, try to use after_idle(). I don't think you really need to Speeding up the sound processing algorithm because np is already quite efficient.
[EDIT]
Surprise!! you are reading from audio channel at every iteration 1 second 8000 bytes in 16 bits format for a 4000 frame rate. You need a second to have it.

Squeezing I/O and calculations into the main loop like you are doing is the classical solution. But there are alternatives.
Do the audio gathering and calculations in a second thread. Since both I/O and numpy should release the GIL, it might be a good alternative here. There is a caveat here. Since GUI toolkits like TKinter are generally not multithread-safe, you should not make Tkinter calls from the second thread. But you could set up a function that is called with after to check the progress of the calculation and update the UI say every 100 ms.
Do the audio gathering and calculations in a different multiprocessing.Process. This makes it completely separate from your GUI. You will have to set up a communication channel like e.g. a Queue to send the pks back to the main process. You should use an after function to check if the Queue has data available and to update the display if so.

Depending on the OS you're running at you might nog be measuring actual 'wall-clock' time. See here http://pythoncentral.io/measure-time-in-python-time-time-vs-time-clock/ for some details. Note that for python 3.3 time.clock is deprecated and time.process_time() or time.perf_counter() is recommended.

Looping Efficiency in Python

I am looking for a way to execute my loop faster. With the current code the calculations takes forever. So I am looking for a way to make my code more efficient.
EDIT: I do not think either explain , I need to create a program that does all possible combinations of 8 digits , not forgetting to include uppercase , lowercase and numbers .. Then encrypt md5 these possible combinations and save them to a file.
But I have new questions , this process would take 63 years would weigh this file ?, As the end of the script? Latest buy a vps server for this task, but if it takes 63 years better not even try haha ..
I am new to coding and all help is appreciated
import hashlib
from random import choice
longitud = 8
valores = "0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ"
def enc(string):
m = hashlib.md5()
m.update(string.encode('utf-8'))
return m.hexdigest()
def code():
p = ""
p = p.join([choice(valores) for i in xrange(longitud)])
text = p
return text
i = 1
for i in xrange(2000000000000000000):
cod = code()
md = enc(cod)
print cod
print md
i += 1
print i
f=open('datos.txt','a')
f.write("%s " % cod)
f.write("%s" % md)
f.write('\n')
f.close()

You're not utilizing the full power of modern computers, that have multiple central processing units! This is by far the best optimization you can have here, since this is CPU bound. Note: for I/O bound operations multithreading (using the threading module) is suitable.
So let's see how python makes it easy to do so using multiprocessing module (read comments):
import hashlib
# you're sampling a string so you need sample, not 'choice'
from random import sample
import multiprocessing
# use a thread to synchronize writing to file
import threading
# open up to 4 processes per cpu
processes_per_cpu = 4
processes = processes_per_cpu * multiprocessing.cpu_count()
print "will use %d processes" % processes
longitud = 8
valores = "0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ"
# check on smaller ranges to compare before trying your range... :-)
RANGE = 200000
def enc(string):
m = hashlib.md5()
m.update(string.encode('utf-8'))
return m.hexdigest()
# we synchronize the results to be written using a queue shared by processes
q = multiprocessing.Manager().Queue()
# this is the single point where results are written to the file
# the file is opened ONCE (you open it on every iteration, that's bad)
def write_results():
with open('datos.txt', 'w') as f:
while True:
msg = q.get()
if msg == 'close':
break;
else:
f.write(msg)
# this is the function each process uses to calculate a single result
def calc_one(i):
s = ''.join(sample(valores, longitud))
md = enc(s)
q.put("%s %s\n" % (s, md))
# we start a process pool of workers to spread work and not rely on
# a single cpu
pool = multiprocessing.Pool(processes=processes)
# this is the thread that will write the results coming from
# other processes using the queue, so it's execution target is write_results
t = threading.Thread(target=write_results)
t.start()
# we use 'map_async' to not block ourselves, this is redundant here,
# but it's best practice to use this when you don't HAVE to block ('pool.map')
pool.map_async(calc_one, xrange(RANGE))
# wait for completion
pool.close()
pool.join()
# tell result-writing thread to stop
q.put('close')
t.join()
There are probably more optimizations to be done in this code, but a major optimization for any cpu-bound task like you present, is using multiprocessing.
note: A trivial optimization of file writes would be to aggregate some results from the queue and write them together (if you have many cpus that exceed the single writing thread's speed)
note 2: Since OP was looking to go over combinations/permutations of stuff, it should be noted that there is a module for doing just that, and it's called itertools.

Note that you should use
for cod in itertools.product(valores, longitud):
rather than picking the strings via random.sample as this will only ever visit a given string once.
Also note that for your given values this loop has 218340105584896 iterations. And the output file will occupy 9170284434565632 bytes or 8PB.

Profile your program first (with the cProfile module: https://docs.python.org/2/library/profile.html and http://ymichael.com/2014/03/08/profiling-python-with-cprofile.html), but I'm willing to bet your program is IO-bound (if your CPU usage never reaches 100% on one core, it means your hard drive is too slow to keep up with the execution speed of the rest of the program).
With that in mind, start by changing your program so that:
It opens and closes the file outside of the loop (opening and closing files is super slow).
It only makes one write call in each iteration (those each translate to a syscall, which are expensive), like so: f.write("%s %s\n" % (cod, md))

Although it helps with debugging, I have found printing makes a program run slower, so maybe don't quite print as much out. Also I'd move the "f=open('datos.txt', 'a') out from the loop as I can imagine Opening the same file over and over again might cause some time issues, and then move the "f.close()" out of the loop also to the end of the program.
CHANGED