Develop Reference

How to write data to a file every 10 seconds

I'm a JS dev trying to learn a bit of Python while working on a Raspberry-Pi3 project that reads data from a Bluetooth temperature sensor.
I need to write the data to my file.txt every 10 seconds, how could I do that please? I found similar topic here (Run certain code every n seconds ), but I don't know how to make it work in my current scenario.
#!/usr/bin/env python3
import argparse
import re
import logging
import sys
import time
from btlewrap import available_backends, BluepyBackend, GatttoolBackend, PygattBackend
from mitemp_bt.mitemp_bt_poller import MiTempBtPoller, \
MI_TEMPERATURE, MI_HUMIDITY, MI_BATTERY
def valid_mitemp_mac(mac, pat=re.compile(r"4C:65:A8:[0-9A-F]{2}:[0-9A-F]{2}:[0-9A-F]{2}")):
"""Check for valid mac adresses."""
if not pat.match(mac.upper()):
raise argparse.ArgumentTypeError('The MAC address "{}" seems to be in the wrong format'.format(mac))
return mac
def poll(args):
"""Poll data from the sensor."""
backend = _get_backend(args)
poller = MiTempBtPoller(args.mac, backend)
line1 = "Temperature: {}".format(poller.parameter_value(MI_TEMPERATURE))
line2 = "Humidity: {}".format(poller.parameter_value(MI_HUMIDITY))
print("Getting data from Mi Temperature and Humidity Sensor")
print("FW: {}".format(poller.firmware_version()))
print("Name: {}".format(poller.name()))
print("Battery: {}".format(poller.parameter_value(MI_BATTERY)))
print(line1)
print(line2)
f = open('file.txt', 'w')
f.write("%s \n %s \n" % (line1, line2))
f.close()
def _get_backend(args):
"""Extract the backend class from the command line arguments."""
if args.backend == 'gatttool':
backend = GatttoolBackend
elif args.backend == 'bluepy':
backend = BluepyBackend
elif args.backend == 'pygatt':
backend = PygattBackend
else:
raise Exception('unknown backend: {}'.format(args.backend))
return backend
def list_backends(_):
"""List all available backends."""
backends = [b.__name__ for b in available_backends()]
print('\n'.join(backends))
def main():
parser = argparse.ArgumentParser()
parser.add_argument('--backend', choices=['gatttool', 'bluepy', 'pygatt'], default='gatttool')
parser.add_argument('-v', '--verbose', action='store_const', const=True)
subparsers = parser.add_subparsers(help='sub-command help', )
parser_poll = subparsers.add_parser('poll', help='poll data from a sensor')
parser_poll.add_argument('mac', type=valid_mitemp_mac)
parser_poll.set_defaults(func=poll)
parser_scan = subparsers.add_parser('backends', help='list the available backends')
parser_scan.set_defaults(func=list_backends)
args = parser.parse_args()
if args.verbose:
logging.basicConfig(level=logging.DEBUG)
if not hasattr(args, "func"):
parser.print_help()
sys.exit(0)
args.func(args)
if __name__ == '__main__':
main()

You can use the time module to pause the program for 10 seconds on each iteration;
from time import sleep
def func(n):
print(n+1)
for i in range(5):
func(i)
sleep(10)
>1
>2
>3
>4
>5
# (every 10 seconds)
However this will block the rest of the program running, although a simple multi-threading script to call the writing function would suffice.
And in relation to the code you are using, insert the sleep call within the poll function and wrap what you have there. If you want to loop the program 10 times then;
def poll(args):
"""Poll data from the sensor."""
for _ in range(10):
# code things
f = open('file.txt', 'a') # << Use append here or you will keep overwriting file contents
f.write('hello')
f.close()
sleep(10)
Or if you want it to run forever until you KeyboardInterrupt or exit somehow:
def poll(args):
"""Poll data from the sensor."""
while True:
# code things
f = open('file.txt', 'a') # << Use append here or you will keep overwriting file contents
f.write('hello')
f.close()
sleep(10)

you need some kind of loop that polls your sensor - I do not see one glancing over your code. You got while and for loops in JS as well - look them up in http://docs.python.org/3/tutorial if you are unsure about the syntax.
store the time you wrote to a variable , sleep a bit poll the next value, check if 10s passed, write if, else not. (or simply sleep 10s between polls if you do not want intermediate values printed
Readup about loops:
for statement
looping techniques
import time
def poll():
return time.time(), 42
last_write = None # when did we record to file last?
# loop for as long as you want - while True would loop endlessly
for _ in range(7):
t,c = poll() # call poll() to get time and temperature from mocked data reader
# check if enough time has passed
if last_write is None or (t-last_write) > 2: # check if new reading needed
with open("t.txt","a") as f:
f.write(f"{t} {c}\n")
last_write=t
print("in file ", t,c)
else:
print("just output ", t,c)
time.sleep(0.7) # sleep some
Output:
in file 1552978725.5224085 42 # ...25.5
just output 1552978726.2232893 42 # ...26.2 - not 2s passed
just output 1552978726.9241226 42 # ...26.9 - not 2s passed
in file 1552978727.6249442 42 # ...27.6 - 2.1s passed
just output 1552978728.3259027 42 # ...28.3 - not 2s passed
just output 1552978729.0267787 42 # ...29.0 - not 2s passed
in file 1552978729.7275977 42 # ...29.7 - 2.1s passed
More remarks:
use with open(filename,mode) as f: and scope the file opeations below it - it will autoclose your file after scope and handle exceptions by closing the filehandle as well.
Using mode "w" will truncate the file before writing to it - you might want to use append instead: reading and writing files

Why do only the first three lines come up in my .txt file -?! Raspberry Pi - temperature humidity readings

I am a completely inexperienced A level student trying to get to grips with python to complete an assignment. I have been given a week to complete it - I have very little knowledge of what to do and have no experience with coding - I am truly stuck and will probably seem very stupid to people on his forum.
I have to create a temperature and humidity logger with a raspberry pi and DHT22 sensor. I am to write a script that produces a loop that sleeps for 10 seconds - I will run the script for two days to collect enough data to produce graphs. So far the code I have is this and it's not working - probably for some obvious reasons. The data needs to come out in two columns in a leafpad file. Nothing seems to be happening when I sudo python execute the script - no .txt file has been created in my ls (there is one with just this in it:
indoors
51.58778
-0.15944
But no error message in the LX terminal.. Am I doing something very obviously wrong?
# Assign header details to STRING variables - change manually
txt_studentid = '999'
txt_pi_location = 'indoors'
txt_pi_latitude = '51.58778'
txt_pi_longitude = '-0.15944'
import Adafruit_DHT
pin = 4
sensor = Adafruit_DHT.DHT22
# Import Time module import time
# open file to write
f = open('/home/pi/my_data.txt','w')
f.write(txt_studentid)
f.write('\n')
f.write(txt_pi_location)
f.write('\n')
f.write(txt_pi_latitude)
f.write('\n')
f.write(txt_pi_longitude)
f.write('\n')
f.close()
while True:
# store off the date and time details for this
sample num_month = time.localtime().tm_mon
num_day = time.localtime().tm_mday
num_hour = time.localtime().tm_hour
num_min = time.localtime().tm_min
num_sec = time.localtime().tm_sec
num_humidity, num_temperature = Adafruit_DHT.read_retry(sensor, pin)
txt_month = str(num_month)
txt_day = str(num_day)
txt_hour = str(num_hour)
txt_min = str(num_min)
txt_sec = str(num_sec)
txt_humidity = str(num_humidity)
txt_temperature = str(num_temperature)
f = open('/home/pi/my_data.txt','a')
f.write(txt_month)
f.write(',')
f.write(txt_day)
f.write(',')
f.write(txt_hour)
f.write(',')
f.write(txt_min)
f.write(',')
f.write(txt_sec)
f.write(',')
# write the temperature and humidity to file
f,write(txt_humidity)
f.write(',')
f,write(txt_temperature)
f.write(',')
# write new line
f.write('\n')
# close the file
f.close()
# wait for ten seconds
time.sleep(10)

You definitely want to at least include the file writing in the while loop; or keep track somehow of the readings for later saving.
I've modified your code to help you get started:
import Adafruit_DHT
import time
from datetime import datetime
pin = 4
sensor = Adafruit_DHT.DHT22
# Import Time module import time
# open file to write
f = open('/home/pi/my_data.txt','w')
f.write(txt_studentid)
f.write('\n')
f.write(txt_pi_location)
f.write('\n')
f.write(txt_pi_latitude)
f.write('\n')
f.write(txt_pi_longitude)
f.write('\n')
f.close()
f = open('/home/pi/my_data.txt','a')
begintime = datetime.now()
while True:
# store off the date and time details for this
sample_time = datetime.now()
sample num_month = time.localtime().tm_mon
num_day = time.localtime().tm_mday
num_hour = time.localtime().tm_hour
num_min = time.localtime().tm_min
num_sec = time.localtime().tm_sec
num_humidity, num_temperature = Adafruit_DHT.read_retry(sensor, pin)
txt_month = str(num_month)
txt_day = str(num_day)
txt_hour = str(num_hour)
txt_min = str(num_min)
txt_sec = str(num_sec)
txt_humidity = str(num_humidity)
txt_temperature = str(num_temperature)
f.write(txt_month)
f.write(',')
f.write(txt_day)
f.write(',')
f.write(txt_hour)
f.write(',')
f.write(txt_humidity)
f.write(',')
f.write(num_temperature)
f.write('\n')
time.sleep(10) #sleep for 10 seconds
timedelta = sample_time - begintime
if timedelta.days >= 2:
break
f.close()
I would try setting the timedelta requirement to something like 30 seconds to make sure it works as expected before going up to 2 days. You can do that by changing if timedelta.days >= 2: to if timedelta.seconds >= 30:

I guess you indentation is wrong. You'll get stuck in the while loop and will never write anything to the file. Try indenting everything from num_month = time.localtime().tm_mon to time.sleep(10)

Python refresh file from disk

I have a python script that calls a system program and reads the output from a file out.txt, acts on that output, and loops. However, it doesn't work, and a close investigation showed that the python script just opens out.txt once and then keeps on reading from that old copy. How can I make the python script reread the file on each iteration? I saw a similar question here on SO but it was about a python script running alongside a program, not calling it, and the solution doesn't work. I tried closing the file before looping back but it didn't do anything.
EDIT:
I already tried closing and opening, it didn't work. Here's the code:
import subprocess, os, sys
filename = sys.argv[1]
file = open(filename,'r')
foo = open('foo','w')
foo.write(file.read().rstrip())
foo = open('foo','a')
crap = open(os.devnull,'wb')
numSolutions = 0
while True:
subprocess.call(["minisat", "foo", "out"], stdout=crap,stderr=crap)
out = open('out','r')
if out.readline().rstrip() == "SAT":
numSolutions += 1
clause = out.readline().rstrip()
clause = clause.split(" ")
print clause
clause = map(int,clause)
clause = map(lambda x: -x,clause)
output = ' '.join(map(lambda x: str(x),clause))
print output
foo.write('\n'+output)
out.close()
else:
break
print "There are ", numSolutions, " solutions."

You need to flush foo so that the external program can see its latest changes. When you write to a file, the data is buffered in the local process and sent to the system in larger blocks. This is done because updating the system file is relatively expensive. In your case, you need to force a flush of the data so that minisat can see it.
foo.write('\n'+output)
foo.flush()

I rewrote it to hopefully be a bit easier to understand:
import os
from shutil import copyfile
import subprocess
import sys
TEMP_CNF = "tmp.in"
TEMP_SOL = "tmp.out"
NULL = open(os.devnull, "wb")
def all_solutions(cnf_fname):
"""
Given a file containing a set of constraints,
generate all possible solutions.
"""
# make a copy of original input file
copyfile(cnf_fname, TEMP_CNF)
while True:
# run minisat to solve the constraint problem
subprocess.call(["minisat", TEMP_CNF, TEMP_SOL], stdout=NULL,stderr=NULL)
# look at the result
with open(TEMP_SOL) as result:
line = next(result)
if line.startswith("SAT"):
# Success - return solution
line = next(result)
solution = [int(i) for i in line.split()]
yield solution
else:
# Failure - no more solutions possible
break
# disqualify found solution
with open(TEMP_CNF, "a") as constraints:
new_constraint = " ".join(str(-i) for i in sol)
constraints.write("\n")
constraints.write(new_constraint)
def main(cnf_fname):
"""
Given a file containing a set of constraints,
count the possible solutions.
"""
count = sum(1 for i in all_solutions(cnf_fname))
print("There are {} solutions.".format(count))
if __name__=="__main__":
if len(sys.argv) == 2:
main(sys.argv[1])
else:
print("Usage: {} cnf.in".format(sys.argv[0]))

You take your file_var and end the loop with file_var.close().
for ... :
ga_file = open(out.txt, 'r')
... do stuff
ga_file.close()
Demo of an implementation below (as simple as possible, this is all of the Jython code needed)...
__author__ = ''
import time
var = 'false'
while var == 'false':
out = open('out.txt', 'r')
content = out.read()
time.sleep(3)
print content
out.close()
generates this output:
2015-01-09, 'stuff added'
2015-01-09, 'stuff added' # <-- this is when i just saved my update
2015-01-10, 'stuff added again :)' # <-- my new output from file reads
I strongly recommend reading the error messages. They hold quite a lot of information.
I think the full file name should be written for debug purposes.

Python 3.3: LAN speed test which calculates the read & write speeds and then returns the figure to Excel

I'm creating a LAN speed test which creates a data file in a specified location of a specified size and records the speed at which it is created/read. For the most part this is working correctly, there is just one problem: the read speed is ridiculously fast because all it's doing is timing how long it takes for the file to open, rather than how long it takes for the file to actually be readable (if that makes sense?).
So far I have this:
import time
import pythoncom
from win32com.client import Dispatch
import os
# create file - write speed
myPath = input('Where do you want to write the file?')
size_MB = int(input('What sized file do you want to test with? (MB)'))
size_B = size_MB * 1024 * 1024
fName = '\pydatafile'
#start timer
start = time.clock()
f = open(myPath + fName,'w')
f.write("\x00" * size_B)
f.close()
# how much time it took
elapsed = (time.clock() -start)
print ("It took", elapsed, "seconds to write the", size_MB, "MB file")
time.sleep(1)
writeMBps = size_MB / elapsed
print("That's", writeMBps, "MBps.")
time.sleep(1)
writeMbps = writeMBps * 8
print("Or", writeMbps, "Mbps.")
time.sleep(2)
# open file - read speed
startRead = time.clock()
f = open(myPath + fName,'r')
# how much time it took
elapsedRead = (time.clock() - startRead)
print("It took", elapsedRead,"seconds to read the", size_MB,"MB file")
time.sleep(1)
readMBps = size_MB / elapsedRead
print("That's", readMBps,"MBps.")
time.sleep(1)
readMbps = readMBps * 8
print("Or", readMbps,"Mbps.")
time.sleep(2)
f.close()
# delete the data file
os.remove(myPath + fName)
# record results on Excel
xl = Dispatch('Excel.Application')
xl.visible= 0
wb = xl.Workbooks.Add(r'C:\File\Location')
ws = wb.Worksheets(1)
# Write speed result
#
# loop until empty cell is found in column
col = 1
row = 1
empty = False
while not empty:
val = ws.Cells(row,col).value
print("Looking for next available cell to write to...")
if val == None:
print("Writing result to cell")
ws.Cells(row,col).value = writeMbps
empty = True
row += 1
# Read speed result
#
# loop until empty cell is found in column
col = 2
row = 1
empty = False
while not empty:
val = ws.Cells(row,col).value
print("Looking for next available cell to write to...")
if val == None:
print("Writing result to cell")
ws.Cells(row,col).value = readMbps
empty = True
row += 1
xl.Run('Save')
xl.Quit()
pythoncom.CoUninitialize()
How can I make this so the read speed is correct?
Thanks a lot

Try to actually read the file:
f = open(myPath + fName, 'r')
f.read()
Or (if the file is too large to fit in memory):
f = open(myPath + fName, 'r')
while f.read(1024 * 1024):
pass
But operating system could still make read fast by caching file content. You've just written it there! And even if you manage to disable caching, your measurement (in addition to network speed) could include the time to write data to file server disk.
If you want network speed only, you need to use 2 separate machines on LAN. E.g. run echo server on one machine (e.g. by enabling Simple TCP/IP services or writing and running your own). Then run Python echo client on another machine that sends some data to echo server, makes sure it receives the same data back and measures the turnaround time.

Python program crashing

So I've designed a program that runs on a computer, looks for particular aspects of files that have been plaguing us, and deletes the files if a flag is passed. Unfortunately the program seems to be almost-randomly shutting down/crashing. I say almost-randomly, because the program always exits after it deletes a file, though it will commonly stay up after a success.
I've run a parallel Python program that counts upwards in the same intervals, but does nothing else. This program does not crash/exit, and stays open.
Is there perhaps a R/W access issue? I am running the program as administrator, so I'm not sure why that would be the case.
Here's the code:
import glob
import os
import time
import stat
#logging
import logging
logging.basicConfig(filename='disabledBots.log')
import datetime
runTimes = 0
currentPhp = 0
output = 0
output2 = 0
while runTimes >= 0:
#Cycles through .php files
openedProg = glob.glob('*.php')
openedProg = openedProg[currentPhp:currentPhp+1]
progInput = ''.join(openedProg)
if progInput != '':
theBot = open(progInput,'r')
#Singles out "$output" on this particular line and closes the process
readLines = theBot.readlines()
wholeLine = (readLines[-4])
output = wholeLine[4:11]
#Singles out "set_time_limit(0)"
wholeLine2 = (readLines[0])
output2 = wholeLine2[6:23]
theBot.close()
if progInput == '':
currentPhp = -1
#Kills the program if it matches the code
currentTime = datetime.datetime.now()
if output == '$output':
os.chmod(progInput, stat.S_IWRITE)
os.remove(progInput)
logging.warning(str(currentTime) +' ' + progInput + ' has been deleted. Please search for a faux httpd.exe process and kill it.')
currentPhp = 0
if output2 == 'set_time_limit(0)':
os.chmod(progInput, stat.S_IWRITE)
os.remove(progInput)
logging.warning(str(currentTime) +' ' + progInput + ' has been deleted. Please search for a faux httpd.exe process and kill it.')
currentPhp = 0
else:
currentPhp = currentPhp + 1
time.sleep(30)
#Prints the number of cycles
runTimes = runTimes + 1
logging.warning((str(currentTime) + ' botKiller2.0 has scanned '+ str(runTimes) + ' times.'))
print('botKiller3.0 has scanned ' + str(runTimes) + ' times.')

Firstly, it'll be hell of a lot easier to work out what's going on if you base your code around something like this...
for fname in glob.glob('*.php'):
with open(fname) as fin:
lines = fin.readlines()
if '$output' in lines[-4] or 'set_time_limit(0)' in lines[0]:
try:
os.remove(fname)
except IOError as e:
print "Couldn't remove:", fname
And err, that's not actually a secondly at the moment, your existing code is just too tricky to follow fullstop, let alone all the bits that could cause a strange error that we don't know yet!

if os.path.exists(progInput):
os.chmod(progInput, stat.S_IWRITE)
os.remove(progInput)
ALSO:
You never reset the output or output2 variables in the loop?
is this on purpose?