I am trying to use a MPU-6000 accelerometer and Raspberry Pi Zero W to log vibration data in a windshield. I'm fairly new to Python so please bear with me.
I've written a python2 script that configures the MPU-6000 to communicate over I2C, with the clock configured to 400 kHz.
The MPU-6000 gives an interrupt when there is new data available in the accelerometer registers, which is read, converted to 2's complement and then written to a CSV file together with a timestamp. The output rate of the accelerometer is configured to be 1 kHz.
I'm experiencing that when sampling all three sensor axis the script isn't able to write all data points to the CSV file. Instead of a 1000 datapoints pr axis pr second I get approximately 650 datapoints pr axis pr second.
I've tried writing only one axis, which proved succesfull with 1000 datapoints pr second. I know that the MPU-6000 has a FIFO register available, which I probably can burst read to get 1000 samples/s without any problem. The problem will be obtaining a timestamp for each sample, so I haven't tried to implement reading from the FIFO register yet.
I will most likely do most of the post processing in Matlab, so the most important things the python script should do is to write sensor data in any form to a CSV file at the determined rate, with a timestamp.
Is there any way to further improve my Python script, so I can sample all three axis and write to a CSV file at a 1 kHz rate?
Parts of my script is depicted below:
#!/usr/bin/python
import smbus
import math
import csv
import time
import sys
import datetime
# Register addresses
power_mgmt_1 = 0x6b
power_mgmt_2 = 0x6c
samlerate_divider = 0x19
accel_config = 0x1C
INT_Enable = 0x38
def read_byte(reg):
return bus.read_byte_data(address, reg)
def read_word(reg):
h = bus.read_byte_data(address, reg)
l = bus.read_byte_data(address, reg+1)
value = (h <<8)+l
return value
def read_word_2c(reg):
val = read_word(reg)
if (val >= 0x8000):
return -((65535 - val) + 1)
else:
return val
csvwriter = None
def csv_open():
csvfile = open('accel-data.csv', 'a')
csvwriter = csv.writer(csvfile)
def csv_write(timedelta, accelerometerx, accelerometery, accelerometerz):
global csvwriter
csvwriter.writerow([timedelta, accelerometerx, accelerometery,
accelerometerz])
# I2C configs
bus = smbus.SMBus(1)
address = 0x69
#Power management configurations
bus.write_byte_data(address, power_mgmt_1, 0)
bus.write_byte_data(address, power_mgmt_2, 0x00)
#Configure sample-rate divider
bus.write_byte_data(address, 0x19, 0x07)
#Configure data ready interrupt:
bus.write_byte_data(address,INT_Enable, 0x01)
#Opening csv file and getting ready for writing
csv_open()
csv_write('Time', 'X_Axis', 'Y_Axis', 'Z_Axis')
print
print "Accelerometer"
print "---------------------"
print "Printing acccelerometer data: "
#starttime = datetime.datetime.now()
while True:
data_interrupt_read = bus.read_byte_data(address, 0x3A)
if data_interrupt_read == 1:
meas_time = datetime.datetime.now()
# delta_time = meas_time - starttime
accelerometer_xout = read_word_2c(0x3b)
accelerometer_yout = read_word_2c(0x3d)
accelerometer_zout = read_word_2c(0x3f)
# accelerometer_xout = read_word(0x3b)
# accelerometer_yout = read_word(0x3d)
# accelerometer_zout = read_word(0x3f)
# accelerometer_xout_scaled = accelerometer_xout / 16384.0
# accelerometer_yout_scaled = accelerometer_yout / 16384.0
# accelerometer_zout_scaled = accelerometer_zout / 16384.0
# csv_write(meas_time, accelerometer_xout_scaled,
accelerometer_yout_scaled, accelerometer_zout_scaled)
csv_write(meas_time, accelerometer_xout, accelerometer_yout,
accelerometer_zout)
continue
If the data you are trying to write is continuous, then the best approach is to minimise the amount of processing needed to write it and to also minimise the amount of data being written. To do this, a good approach would be to write the raw data into a binary formatted file. Each data word would then only require 2 bytes to be written. The datetime object can be converted into a timestamp which would need 4 bytes. So you would use a format such as:
[4 byte timestamp][2 byte x][2 byte y][2 byte z]
Python's struct library can be used to convert multiple variables into a single binary string which can be written to a file. The data appears to be signed, if this is the case, you could try writing the word as is, and then using the libraries built in support for signed values to read it back in later.
For example, the following could be used to write the raw data to a binary file:
#!/usr/bin/python
import smbus
import math
import csv
import time
import sys
import datetime
import struct
# Register addresses
power_mgmt_1 = 0x6b
power_mgmt_2 = 0x6c
samlerate_divider = 0x19
accel_config = 0x1C
INT_Enable = 0x38
def read_byte(reg):
return bus.read_byte_data(address, reg)
def read_word(reg):
h = bus.read_byte_data(address, reg)
l = bus.read_byte_data(address, reg+1)
value = (h <<8)+l
return value
# I2C configs
bus = smbus.SMBus(1)
address = 0x69
#Power management configurations
bus.write_byte_data(address, power_mgmt_1, 0)
bus.write_byte_data(address, power_mgmt_2, 0x00)
#Configure sample-rate divider
bus.write_byte_data(address, 0x19, 0x07)
#Configure data ready interrupt:
bus.write_byte_data(address, INT_Enable, 0x01)
print
print "Accelerometer"
print "---------------------"
print "Printing accelerometer data: "
#starttime = datetime.datetime.now()
bin_format = 'L3H'
with open('accel-data.bin', 'ab') as f_output:
while True:
#data_interrupt_read = bus.read_byte_data(address, 0x3A)
data_interrupt_read = 1
if data_interrupt_read == 1:
meas_time = datetime.datetime.now()
timestamp = time.mktime(meas_time.timetuple())
accelerometer_xout = read_word(0x3b)
accelerometer_yout = read_word(0x3d)
accelerometer_zout = read_word(0x3f)
f_output.write(struct.pack(bin_format, timestamp, accelerometer_xout, accelerometer_yout, accelerometer_zout))
Then later on, you could then convert the binary file to a CSV file using:
from datetime import datetime
import csv
import struct
bin_format = 'L3h' # Read data as signed words
entry_size = struct.calcsize(bin_format)
with open('accel-data.bin', 'rb') as f_input, open('accel-data.csv', 'wb') as f_output:
csv_output = csv.writer(f_output)
csv_output.writerow(['Time', 'X_Axis', 'Y_Axis', 'Z_Axis'])
while True:
bin_entry = f_input.read(entry_size)
if len(bin_entry) < entry_size:
break
entry = list(struct.unpack(bin_format, bin_entry))
entry[0] = datetime.fromtimestamp(entry[0]).strftime('%Y-%m-%d %H:%M:%S')
csv_output.writerow(entry)
If your data collection is not continuous, you could make use of threads. One thread would read your data into a special queue. Another thread could read items out of the queue onto the disk.
If it is continuous this approach will fail if the writing of data is slower than the reading of it.
Take a look at the special Format characters used to tell struct how to pack and unpack the binary data.
Related
This is a follow up question from: <class 'TypeError'>: LP_c_long instance instead of list but is a different issue so I am posting a new question here.
I am using ctypes to invoke a c function from python script. The c function which is being invoked is:
uint32_t crc32Word(uint32_t crc, const void *buffer, uint32_t size)
I have this python code:
import datetime
import os
import struct
import time
import pigpio
import spidev
import ctypes
lib = ctypes.CDLL('/home/pi/serial_communication/crc.so')
lib.crc32Word.argtypes = ctypes.c_uint32, ctypes.c_void_p, ctypes.c_uint32
lib.crc32Word.restype = ctypes.c_uint32
bus = 0
device = 0
spi = spidev.SpiDev()
spi.open(bus, device)
spi.max_speed_hz = 4000000
spi.mode = 0
pi.set_mode(12, pigpio.INPUT)
C=0
def output_file_path():
return os.path.join(os.path.dirname(__file__),
datetime.datetime.now().strftime("%dT%H.%M.%S") + ".csv")
def spi_process(gpio,level,tick):
print("Detected")
data = bytes([0]*1024)
spi.xfer([0x02])
with open(output_file_path(), 'w') as f:
t1=datetime.datetime.now()
for x in range(1):
spi.xfer2(data)
values = struct.unpack("<" +"I"*256, bytes(data))
C = lib.crc32Word(0xffffffff,data,len(data))
f.write("\n")
f.write("\n".join([str(x) for x in values]))
t2=datetime.datetime.now()
print(t2-t1)
print(C)
input("Press Enter to start the process ")
spi.xfer2([0x01])
cb1=pi.callback(INTERRUPT_GPIO, pigpio.RISING_EDGE, spi_process)
while True:
time.sleep(1)
Previously, I initialized data as data = [0]*1024, so I was receiving some error as mentioned in the previous post. But the error was resolved by initializing data to bytes. Now the issue that I have is even though the correct data is received on Raspberry Pi (checked using logic analyzer), only 0s are stored in the file.
I also tried initializing data as arrays instead of lists using numpy library as below, but I receive an error saying:
Tried code:
import numpy as np
#changed initialization of data to:
data= np.arange(1024) #also tried: np.zeros(1024,ctypes.c_uint8), also get same error
Error:
File "2crc_spi.py", line 48, in spi_process
spi.xfer2(data)
TypeError: Non-Int/Long value in arguments: b5410b00
But data only receives a byte at a time so not sure where is the issue when using array.
Can someone please help? Thanks~
EDIT:
Below is the original code that was working fine with the data collection and unpacking functions before I started integrating the CRC functionality:
import datetime
import os
import struct
import time
import pigpio
import spidev
bus = 0
device = 0
spi = spidev.SpiDev()
spi.open(bus, device)
spi.max_speed_hz = 4000000
spi.mode = 0
pi.set_mode(12, pigpio.INPUT)
a=0
def output_file_path():
return os.path.join(os.path.dirname(__file__),
datetime.datetime.now().strftime("%dT%H.%M.%S") + ".csv")
def spi_process(gpio,level,tick):
print("Detected")
data = [0]*2048
spi.xfer([0x02])
with open(output_file_path(), 'w') as f:
t1=datetime.datetime.now()
for x in range(1):
spi.xfer2(data)
values = struct.unpack("<" +"I"*256, bytes(data))
f.write("\n")
f.write("\n".join([str(x) for x in values]))
t2=datetime.datetime.now()
print(t2-t1)
input("Press Enter to start the process ")
spi.xfer2([0x01])
cb1=pi.callback(INTERRUPT_GPIO, pigpio.RISING_EDGE, spi_process)
while True:
time.sleep(1)
As #Mark's suggested in comments, you would have to store the data into another variable/list, for example: recv = spi.xfer2(data). Then you would need to use this recv in your unpack function.
Further, you could also use a python library called zlib instead of using a c library (there are other python libraries as well).
Another point, since zlib takes only bytes as input, you would need to convert recv into bytes (here recv is a list; check my code).
I modified some of your code.
import datetime
import os
import struct
import time
import pigpio
import spidev
import zlib
bus = 0
device = 0
spi = spidev.SpiDev()
spi.open(bus, device)
spi.max_speed_hz = 4000000
spi.mode = 0
pi.set_mode(12, pigpio.INPUT)
C=0
def output_file_path():
return os.path.join(os.path.dirname(__file__),
datetime.datetime.now().strftime("%dT%H.%M.%S") + ".csv")
def spi_process(gpio,level,tick):
print("Detected")
data = bytes([0]*1024)
spi.xfer2([0x02])
with open(output_file_path(), 'w') as f:
t1=datetime.datetime.now()
for x in range(1):
recv = spi.xfer2(data)
values = struct.unpack("<" +"I"*256, bytes(recv))
C = zlib.crc32(bytes(recv))
f.write("\n")
f.write("\n".join([str(x) for x in values]))
t2=datetime.datetime.now()
print(t2-t1)
print(C)
input("Press Enter to start the process ")
spi.xfer2([0x01])
cb1=pi.callback(INTERRUPT_GPIO, pigpio.RISING_EDGE, spi_process)
while True:
time.sleep(1)
i Try to set up my Raspberry pi with a fan. It should log the Temperature every 5 minutes and write it to a .csv-file with a Python Script. If the Temperature is hotter than 52 Degrees Celsius, it shoud turn on a USB-Hub, and turn it of if it is below that value. For now, i created a little website, to see my logged data. And here is my question: as you can see in the screenshot, it tells me sometimes [HOT] and sometimes [OK] for the same Values? Also it is not sepperating the Data like a want it to seperate (over/under 52 Degrees Celsius).
Screenshot from my little website (This only displays my .csv-File)
My .py-code:
from gpiozero import CPUTemperature
from time import sleep, strftime, time
from datetime import datetime, timedelta
from threading import Timer
import matplotlib.pyplot as plt
v=datetime.today()
w = v.replace(day=v.day, hour=1, minute=0, second=0, microsecond=0) + timedelta(days=1)
delta_t=w-v
secs=delta_t.total_seconds()
cpu = CPUTemperature()
plt.ion()
x = []
y = []
def write_tempHot(temp):
with open("/var/www/html/cpuTemp.csv", "a") as log:
log.write("{0}{1}\n".format(strftime("%Y-%m-%d %H:%M:%S = [HOT] "),str(temp)))
def write_tempLow(temp):
with open("/var/www/html/cpuTemp.csv", "a") as log:
log.write("{0}{1}\n".format(strftime("%Y-%m-%d %H:%M:%S = [OK] "),str(temp)))
def clearTemp():
filename = "/var/www/html/cpuTemp.csv"
# opening the file with w+ mode truncates the file
f = open(filename, "w+")
f.close()
while True:
temp = cpu.temperature
if temp > 52:
temp = cpu.temperature
write_tempHot(temp)
plt.pause(300)
t = Timer(secs, clearTemp)
t.start()
else:
temp = cpu.temperature
write_tempLow(temp)
plt.pause(300)
t = Timer(secs, clearTemp)
t.start()
(Also dont mind for the clearTemp()-function, i want to clear the .csv-file once a day)
Does anybody have n idea, why the results are that kind of strange?
Greetings & Thanks a lot!
Your comparison is
if temp > 52
when it should be
if temp >= 52.0
because otherwise your temperature Hi will only match if it is 53C or above.
I'd also use time.sleep() instead of plt.pause().
For your logfile, you've got two functions to write to your logfile, I'd use one instead:
def write_log(temp):
fmt = """{when} = [{option}] {temp}"""
datefmt = """%Y-%m-%d %H:%M:%S"""
option = "OK"
with open("/var/www/html/cpuTemp.csv", "a") as log:
when = datetime.now().strftime(datefmt)
if temp >= 52.0:
option = "HOT"
log.write(fmt.format(when=when, option=option, temp=temp))
Finally, I do not understand why you want to truncate your logfile every 5 minutes.
I am trying to read a huge binary file.
I have used chunks method and i am breaking my stream in chunks like:
with open("testUniform.bin", 'rb') as f:
for chunk in iter(lambda: f.read(4096), b''):
dostuff(chunk)
All I am getting in chunk is this:
.y.$.!...
like stream,which I have attached a screenshot of.
I don't get the way out to convert it back to binary stream and I still don't know that why is this stream being converted into something like this.
Please help:
I am trying to convert a binary stream to a decimal values, but since being a huge file I cannot apply
f.read()
Here is my code attached:
from math import log,ceil,pow
from flask import Flask, request
from flask_restful import Resource, Api
import struct
app = Flask(__name__)
api = Api(app)
def binaryToDecimal(n):
return int(n,2)
def dostuff(inputarray):
args = request.args
lower_end_range = args['lower_end_range']
higher_end_range = args['higher_end_range']
amount = args['amount']
lower_end_range =int(lower_end_range)
higher_end_range=int(higher_end_range)
amount =int(amount)
#range_input is the range
range_input=higher_end_range-lower_end_range+1
#taking the log of the range to generate offset
log_of_range=log(range_input,2)
log_of_range=int(ceil(log_of_range))
higher_end_range_represented_by_bits = 0
lower_end_range_represented_by_bits = 0
lst = []
FinalRandomArray = []
#creating the maximum of numbers which it can go to by saving,for ex: 2^3+2^2+2^1+2^0
for i in range(0,(log_of_range)):
higher_end_range_represented_by_bits+=pow(2,i)
while True:
i=range_input%2
range_input=range_input/2
lst.append(i)
if range_input==0:
break
length = len(lst)
#where length is equal to the window size
for file in range(0,len(inputarray),length):
print(inputarray[0])
number=binaryToDecimal((inputarray[file]+inputarray[file+1]+inputarray[file+2]))+lower_end_range
if(number>=lower_end_range and number<=higher_end_range):
if(amount!=0):
FinalRandomArray.append(number)
amount-=1
return {'finalrandomarray':FinalRandomArray}
class ReturnMainModule(Resource):
def get(self):
with open("testUniform.bin", 'rb') as f:
for chunk in iter(lambda: f.read(4096), b''):
dostuff(chunk)
api.add_resource(ReturnMainModule, '/main')
# Driver code
if __name__ == '__main__':
app.run(port='5004')
My goal is to read a wave file and edit the data of it by adding a random number to each bit of data in the range of -1 to 1 with the hope of creating some distortion and then saving it as an edited wave file. I read and edit the wave file like so:
riffTag = fileIn.read(4)
if riffTag != 'RIFF':
print 'not a valid RIFF file'
exit(1)
riffLength = struct.unpack('<L', fileIn.read(4))[0]
riffType = fileIn.read(4)
if riffType != 'WAVE':
print 'not a WAV file'
exit(1)
# now read children
while fileIn.tell() < 8 + riffLength:
tag = fileIn.read(4)
length = struct.unpack('<L', fileIn.read(4))[0]
if tag == 'fmt ': # format element
fmtData = fileIn.read(length)
fmt, numChannels, sampleRate, byteRate, blockAlign, bitsPerSample = struct.unpack('<HHLLHH', fmtData)
stHeaderFields['AudioFormat'] = fmt
stHeaderFields['NumChannels'] = numChannels
stHeaderFields['SampleRate'] = sampleRate
stHeaderFields['ByteRate'] = byteRate
stHeaderFields['BlockAlign'] = blockAlign
stHeaderFields['BitsPerSample'] = bitsPerSample
elif tag == 'data': # data element
rawData = fileIn.read(length)
else: # some other element, just skip it
fileIn.seek(length, 1)
numChannels = stHeaderFields['NumChannels']
# some sanity checks
assert(stHeaderFields['BitsPerSample'] == 16)
assert(numChannels * stHeaderFields['BitsPerSample'] == blockAlign * 8)
samples = []
edited_samples = []
for offset in range(0, len(rawData), blockAlign):
samples.append(struct.unpack('<h', rawData[offset:offset+blockAlign]))
for sample in samples:
edited_samples.append(sample[0] + random.randint(-1, 1))
After I've done this I try to save the data is a new edited wave file by doing the following:
foo = []
for sample in edited_samples:
foo.append(struct.pack('<h', int(sample)))
with open(fileIn.name + ' edited.wav', 'w') as file_out:
file_out.write('RIFF')
file_out.write(struct.pack('<L', riffLength))
file_out.write('WAVE')
file_out.write(ur'fmt\u0020')
file_out.write(struct.pack('<H', fmt))
file_out.write(struct.pack('<H', numChannels))
file_out.write(struct.pack('<L', sampleRate))
file_out.write(struct.pack('<L', byteRate))
file_out.write(struct.pack('<H', blockAlign))
file_out.write(struct.pack('<H', bitsPerSample))
file_out.write('data')
for item in foo:
file_out.write(item)
While it doesn't give me any errors I can't play the new wave file in a media player. When I try open my new wave file I get a crash on the line fmt, numChannels, sampleRate, byteRate, blockAlign, bitsPerSample = struct.unpack('<HHLLHH', fmtData) with the error error: unpack requires a string argument of length 16. I imagine I'm building the wave file wrong. How do I build it correctly?
Unless you're intent on writing the support for .wav files yourself for some other reason (getting experience with dealing with binary files, etc.), don't do this. Python comes with the wave module that handles all of the file format issues and lets you just work with the data.
I'm creating a LAN speed test which creates a data file in a specified location of a specified size and records the speed at which it is created/read. For the most part this is working correctly, there is just one problem: the read speed is ridiculously fast because all it's doing is timing how long it takes for the file to open, rather than how long it takes for the file to actually be readable (if that makes sense?).
So far I have this:
import time
import pythoncom
from win32com.client import Dispatch
import os
# create file - write speed
myPath = input('Where do you want to write the file?')
size_MB = int(input('What sized file do you want to test with? (MB)'))
size_B = size_MB * 1024 * 1024
fName = '\pydatafile'
#start timer
start = time.clock()
f = open(myPath + fName,'w')
f.write("\x00" * size_B)
f.close()
# how much time it took
elapsed = (time.clock() -start)
print ("It took", elapsed, "seconds to write the", size_MB, "MB file")
time.sleep(1)
writeMBps = size_MB / elapsed
print("That's", writeMBps, "MBps.")
time.sleep(1)
writeMbps = writeMBps * 8
print("Or", writeMbps, "Mbps.")
time.sleep(2)
# open file - read speed
startRead = time.clock()
f = open(myPath + fName,'r')
# how much time it took
elapsedRead = (time.clock() - startRead)
print("It took", elapsedRead,"seconds to read the", size_MB,"MB file")
time.sleep(1)
readMBps = size_MB / elapsedRead
print("That's", readMBps,"MBps.")
time.sleep(1)
readMbps = readMBps * 8
print("Or", readMbps,"Mbps.")
time.sleep(2)
f.close()
# delete the data file
os.remove(myPath + fName)
# record results on Excel
xl = Dispatch('Excel.Application')
xl.visible= 0
wb = xl.Workbooks.Add(r'C:\File\Location')
ws = wb.Worksheets(1)
# Write speed result
#
# loop until empty cell is found in column
col = 1
row = 1
empty = False
while not empty:
val = ws.Cells(row,col).value
print("Looking for next available cell to write to...")
if val == None:
print("Writing result to cell")
ws.Cells(row,col).value = writeMbps
empty = True
row += 1
# Read speed result
#
# loop until empty cell is found in column
col = 2
row = 1
empty = False
while not empty:
val = ws.Cells(row,col).value
print("Looking for next available cell to write to...")
if val == None:
print("Writing result to cell")
ws.Cells(row,col).value = readMbps
empty = True
row += 1
xl.Run('Save')
xl.Quit()
pythoncom.CoUninitialize()
How can I make this so the read speed is correct?
Thanks a lot
Try to actually read the file:
f = open(myPath + fName, 'r')
f.read()
Or (if the file is too large to fit in memory):
f = open(myPath + fName, 'r')
while f.read(1024 * 1024):
pass
But operating system could still make read fast by caching file content. You've just written it there! And even if you manage to disable caching, your measurement (in addition to network speed) could include the time to write data to file server disk.
If you want network speed only, you need to use 2 separate machines on LAN. E.g. run echo server on one machine (e.g. by enabling Simple TCP/IP services or writing and running your own). Then run Python echo client on another machine that sends some data to echo server, makes sure it receives the same data back and measures the turnaround time.