I am using my raspberry Pi3 to create timelapse videos. I have a cron that runs a python script every minute that decides how many photos to take and then imports a function from another python script that takes the actual photos. The problem is that after running for about 4 hours the camera stops taking photos- if I try and take one manually it says it is out of memory, and top confirms this. If I watch top while the timelapse is running the memory usage steadily climbs.
I think I have narrowed the problem down to the python script that takes the photos. I can run this on its own, and if I start up the pi and run it a few times I see that the memory used climbs by about 10MB the first run and about 1MB every subsequent run (screenshot at the bottom of the post). This is the script
import time
import picamera
import os
def ShutterTS(dirname):
with picamera.PiCamera() as cam:
cam.resolution=(1920,1440)
cam.rotation=180
cam.hflip=True
# camera warm up time
time.sleep(2)
FNfmt = "%4d%02d%02d_%02d:%02d:%02d.JPG"
Fname = FNfmt % time.localtime()[0:6]
framename = os.path.join(dirname, Fname)
cam.capture(framename)
return
def main():
dirname = [insert path here, my path hidden]
ShutterTS(dirname)
return
if __name__ == '__main__':
import sys
sys.exit(main())
I'm not a good coder, I basically cobble stuff together from bits I find on the internet so I'm hoping this is something really simple that I've missed. The with is the raspberry pi recommended way of calling the camera. I know this should close the camera instance on exit but I'm guessing something is hanging around in memory? I tried adding close.cam() at the end of the function and it made no difference (didn't think it would). I've tried del on all the variables at the end of the function and it made no difference. I think the return at the end of the function is redundant but adding it made no difference.
This website https://www.linuxatemyram.com/ suggests that top showing the memory climbing is normal and free -m is a better gauge, and that shows plenty available- but the fact remains the camera stops working, saying it is out of memory. Any clues would be much appreciated!
This is the cron script (some other imports cropped)
from ShutterTimestamp import ShutterTS
from makedirectory import testmakedir
from SunTimesA import gettimes
def Timer(dirname,FRAMES_PER_MINUTE):
# I take a picture first and then loop so the program isn't
# sleeping pointlessly to the end of the minute
start = time.time()
ShutterTS(dirname)
if FRAMES_PER_MINUTE>1:
for frame in range(FRAMES_PER_MINUTE-1):
time.sleep(int(60 / FRAMES_PER_MINUTE) - (time.time() - start))
start = time.time()
ShutterTS(dirname)
return
def main():
dirfmt = []
dirname = dirfmt % time.localtime()[0:3]
FPM=gettimes()
if FPM > 0:
testmakedir(dirname)
Timer(dirname,FPM)
return
if __name__ == '__main__':
sys.exit(main())
Screenshot of memory use
I suppose you have a wrapping python script which import the script you provide in the question and call ShutterTS in a loop. This function does not return any output to main script (just return).
If you can observe a memory leak it probably is located in the picamera module.
A workaround it to call this script as a sub-process, not as a function call in the main process. It can be done in a shell script or in the python script using subprocess module.
Thus the memory will be released after each capture.
Related
I am trying to make one of my functions run asynchronous. In the code this function runs approximately 600-700 times. So I am trying to make it asynchronous so overall code will be faster.
I am using threadpool for it to run faster. But as I searched for solutions I saw something called GIL(Global Interpreter Lock). If I understood correctly because python is not a thread safe language, GIL doesn't let python use more than one thread at a time.
Here is the piece of code I am trying to run asynchronous. Function was too long so I pasted here the time consuming part.
for row in range(0, image_row-1 ,1):
if row in ignore_row:
continue
for column in range(0, image_col-1 ,1):
if image_binary[row][column] > 0:
#For the first triangle of a square
vertice1 = vertices[row][column]
vertice2 = vertices[row][column+1]
vertice3 = vertices[row+1][column+1]
face1 = np.array([vertice1, vertice2, vertice3])
#For the second triangle if a square
vertice1 = vertices[row][column]
vertice2 = vertices[row+1][column]
vertice3 = vertices[row+1][column+1]
face2 = np.array([vertice1,vertice2,vertice3])
faces.extend(face1)
faces.extend(face2)
At first I thought if one of the variables was global maybe the threads are waiting in the queue to access it. But non of the variables are global they all created in the function.
In the sequential code for 1 run all program takes about 0.2 seconds but in the parallel program in this snippet it takes about 4-9 seconds.
Here is the multiprocess code:
pool = ThreadPool(config.THREAD_COUNT)
pool.map(process_image, png_names)
pool.close()
pool.join()
png_names holds the name of the png's in a folder.
And for config.THREAD_COUNT i tried 2-17 threads. They all read the png files and start to process it but when they come to the part that i shared above they wait and execute that part in about 4-9 seconds.
I am using cv2 and numpy libraries for reading and editing the images.
I am using Python 3.10
I'm trying to create a monitoring script for a specific process that will run in a loop but I don't know how to make the app batch run on a new window without affecting the monitoring script.
This is what I have so far, but it runs on the same cmd window. Also, how can I make this a simple look so when it can check every half hour if the process is running and if it's not, execute the the app batch file.
This is what I have so far, but everything I find online is way over my head...
import subprocess
def process_exists(process_name):
call = 'TASKLIST', '/FI', 'imagename eq %s' % process_name
# use buildin check_output right away
output = subprocess.check_output(call).decode()
# check in last line for process name
last_line = output.strip().split('\r\n')[-1]
# because Fail message could be translated
return last_line.lower().startswith(process_name.lower())
print (process_exists('process.exe'))
if not process_exists('process.exe'):
subprocess.call([r'app.bat'])
There are a few ways that one might use:
[1] Threading:
import threading
def main_runner():
# some code
return
every_n_mins = 1
threading.Timer( 60*every_n_mins, main_runner, [loop_number, every_n_mins] ).start()
[2] for loop:
Just add a for loop with sleep in the code.
[3] Task Scheduler:
use a task scheduler in the operating system to keep running the code at specified periods. In windows it is calle dtask scheduler.
Each method has its merits and open for discussion.
Just found something that worked, at least for the new window issue. Just removed the brackets and added the following to subprocess.call:
subprocess.call('app.bat', creationflags=subprocess.CREATE_NEW_CONSOLE)
As for the loop, just thought I'd mention it here for the super beginners like myself:
This is the final script:
import subprocess
import time
from datetime import datetime
def process_exists(process_name):
call = 'TASKLIST', '/FI', 'imagename eq %s' % process_name
# use buildin check_output right away
output = subprocess.check_output(call).decode()
# check in last line for process name
last_line = output.strip().split('\r\n')[-1]
# because Fail message could be translated
return last_line.lower().startswith(process_name.lower())
print (process_exists('ethminer.exe'))
x=1
while x == 1:
if not process_exists('ethminer.exe'):
#subprocess.call(r'', creationflags=subprocess.CREATE_NEW_CONSOLE)
subprocess.call(r'C:\Users\User\Downloads\ethminer-0.19.0-alpha.0-cuda10.0-windows-amd64\bin\main.bat', creationflags=subprocess.CREATE_NEW_CONSOLE)
print(datetime.now())
time.sleep(300)
is there any way I can run python code in this way:
main code will run all the time ,
once every 5 min will run another function while running the main code.
my code is reading gps signal and send it to my server every 5 seconds.
I have to run another code that check the device cpu\file\temp every 5 min (this part take around 30 seconds )
can both of them run at the same time while still getting gps?
I have the GPS code ready and also the Check code ready - how do I combine them (if it's possiable)
Thanks ,
This might answer your question: Running Two Script at Once Using Bash
Based on the answer here, all you'd have to run is:
python script1.py &
python script2.py &
You can use the threading module, to do this task: it allows you to run functions as different processes seperate from the main program. Threading Documentation
I think you should get what you want with the following code:
import threading
def gpsJob():
threading.Timer(300.0, gpsJob).start()
# Here goes your GPS-code
gpsJob()
if __name__ == '__main__':
# main code
I got temperature, pressure, and altitude readings on my PI using a sensor:
The problem is, to see the results, I have to execute the code.py every time by myself. I am trying to automate it somehow so it will keep running itself for the time I want.
Once that is automated, would like to save the results and analyze the output after some time.
Is there a way I can write code for both the tasks?
Thank you.
There are two things required here. First a script i.e code.py to log the functional behavior like temperature, pressure, and altitude readings along with error/response during the process. Another is the script executions logs i.e a success or failure during the scheduled time and other system logs.
For first job, you have to do by your self but ensure to have a logger module in place to log the process flow.
For Second job, you can use OS provided scheduler crontab for Linux based os. For example to execute script every minutes
* * * * * python /home/script/code.py > /home/script/code.log 2>&1
For more about scheduler jobs, you can refer here
The time module is your friend here. You can set up an infinite loop with while True: and use time.sleep(secs) at the end of the loop (after output).
I'd use additional controller script like this:
import subprocess;
import time;
import sys;
x = True;
while x:
while exit_code!=0:
try:
exit_code = subprocess.check_call(['python', 'collect_data.py', '-cli_args_if_needed']);
except:
print(sys.exec_info()[1]);
print('Relaunching in 5 seconds');
time.sleep(5)
I am looking for a nice pattern for python hadoop streaming that involves loading an expensive resource, for example a pickled python object on the server. Here is what I came up with; I've tested by piping input files and slow running programs directly into the script in bash, but haven't yet run it on a hadoop cluster. For you hadoop wizards---am i handling io such that this will work as a python streaming job? I guess I'll go spin up something on amazon to test but it would be nice if someone knew off the top.
you can test it out via cat file.txt | the_script or ./a_streaming_program | the_script
#!/usr/bin/env python
import sys
import time
def resources_for_many_lines():
# load slow, shared resources here
# for example, a shared pickle file
# in this example we use a 1 second sleep to simulate
# a long data load
time.sleep(1)
# we will pretend the value zero is the product
# of our long slow running import
resource = 0
return resource
def score_a_line(line, resources):
# put fast code to score a single example line here
# in this example we will return the value of resource + 1
return resources + 1
def run():
# here is the code that reads stdin and scores the model over a streaming data set
resources = resources_for_many_lines()
while 1:
# reads a line of input
line = sys.stdin.readline()
# ends if pipe closes
if line == '':
break
# scores a line
print score_a_line(line, resources)
# prints right away instead of waiting
sys.stdout.flush()
if __name__ == "__main__":
run();
This looks fine to me. I often load up yaml or sqlite resources in my mappers.
You typically won't be running that many mappers in your job so even if you spend a couple of seconds in loading something from disk it's usually not a huge problem.