Python while loop not terminating after defined amount of time - python

For this project, I have been making a vibration measurement device that plots high speed vibrations that are controlled by a motor. Here is the code below:
# Import the modules
from __future__ import division
import spidev, datetime, time
from sys import exit
import RPi.GPIO as GPIO
# Setup SPI
spi = spidev.SpiDev()
spi.open(0, 0)
spi.mode = 3
# Constants
accres, accrate = 1, 15
# Set GPIO chip select pins
GPIO.setwarnings(False)
GPIO.setmode(GPIO.BCM)
cs1, cs2, motor = 23, 24, 18
GPIO.setup(cs1, GPIO.OUT)
GPIO.setup(cs2, GPIO.OUT)
GPIO.setup(motor, GPIO.OUT)
# Note: the duty cycle goes from 0.0 to 100.0, with 100.0 being no motor movement,
# and 0.0 being the maximum motor speed.
motor_output = GPIO.PWM(motor, 60)
# Initialize the ADXL345
def initadxl345():
# Set data rate (accrate=15 -> 3200 Hz, 14=1600 Hz, 13=800 Hz, 12=400 Hz, 11=200 Hz, 10=100 Hz etc.)
spi.xfer2([44, accrate])
# Enable full range (10 bits resolution) and +/- 16g 4 LSB
spi.xfer2([49, accres])
# Read the first ADXL x-y-z axes
def readadxl345_1():
# Chip select pin ensures that the first sensor is being read by grounding its pin
GPIO.output(cs1, 0)
GPIO.output(cs2 ,1)
rx = spi.xfer2([242, 0, 0, 0, 0, 0, 0])
out = [rx[1] | (rx[2] << 8), rx[3] | (rx[4] << 8), rx[5] | (rx[6] << 8)]
# Format x-axis
if (out[0] & (1 << 16 - 1 )):
out[0] = out[0] - (1 << 16)
# Format y-axis
if (out[1] & (1 << 16 - 1 )):
out[1] = out[1] - (1<<16)
# Format z-axis
if (out[2] & (1 << 16 - 1 )):
out[2] = out[2] - (1 << 16)
# Return human readable values
return out
# Read the second ADXL x-y-z axes
def readadxl345_2():
# Chip select pin ensures that the first sensor is being read by grounding its pin
GPIO.output(cs1, 1)
GPIO.output(cs2 ,0)
rx = spi.xfer2([242, 0, 0, 0, 0, 0, 0])
out = [rx[1] | (rx[2] << 8), rx[3] | (rx[4] << 8), rx[5] | (rx[6] << 8)]
# Format x-axis
if (out[0] & (1 << 16 - 1 )):
out[0] = out[0] - (1 << 16)
# Format y-axis
if (out[1] & (1 << 16 - 1 )):
out[1] = out[1] - (1<<16)
# Format z-axis
if (out[2] & (1 << 16 - 1 )):
out[2] = out[2] - (1 << 16)
# Return human readable values
return out
print("Vibration Reader Initializing...")
time.sleep(1)
print(GPIO.RPI_INFO)
time.sleep(1)
response = input("Proceed measurements? [Y, n] ")
if response == "Y" or "y":
filename = input("Filename: ")
pwm_speed = float(input("Motor PWM value: "))
# Initialize the ADXL345 accelerometer
print("Initializing ADXL345s...")
initadxl345()
motor_output.start(pwm_speed)
print("Motor is up and running at {}".format(pwm_speed))
time.sleep(1)
timeout = 0.0003125 / 2 # timeout=1/samplerate=>not sufficient measurements. Half the time is sufficient (don't know why!)
timetosend = 1
while True:
with open('/proc/uptime', 'r') as f: # get uptime
uptime_start = float(f.readline().split()[0])
uptime_last = uptime_start
active_file_first = filename + '.csv'
file = open('/var/log/sensor/' + active_file_first, 'w')
while uptime_last < uptime_start + timetosend:
time1 = str(datetime.datetime.now().strftime('%S.%f'))
sensor1 = readadxl345_1()
sensor2 = readadxl345_2()
file.write(str(sensor1[0]) + ',' + str(sensor1[1]) + ',' + str(sensor1[2]) + ',' + str(sensor2[0]) + ',' + str(sensor2[1]) + ',' + str(sensor2[2]) + ',' + time1 + '\n')
# Print data every "timeout" second
elapsed = time.process_time()
current = 0
while(current < timeout):
current = time.process_time() - elapsed
motor_output.stop
print("Motor shutting off and cleaning up GPIO.")
GPIO.cleanup()
elif response == "N" or "n":
print("Quitting...")
time.sleep(1)
quit()
From this I get into trouble. The issue is that once I do choose to proceed, the print statement Motor is up and running at 100.0 displays and nothing happens until I press a random key on the keyboard in which is says
Traceback (most recent call last):
File "accelerometer.py", line 116, in <module>
current = time.process_time() - elapsed
KeyboardInterrupt
Its as if the code following that print statement didn't even exist, or the interpreter just ignored it completely. Does anyone have a coherent explanation of these problems?


Let's take a step back and think about what this block of code is doing:
while True:
### Block 1 ###
with open('/proc/uptime', 'r') as f: # get uptime
uptime_start = float(f.readline().split()[0])
uptime_last = uptime_start
active_file_first = filename + '.csv'
file = open('/var/log/sensor/' + active_file_first, 'w')
### Block 2 ###
while uptime_last < uptime_start + timetosend:
time1 = str(datetime.datetime.now().strftime('%S.%f'))
sensor1 = readadxl345_1()
sensor2 = readadxl345_2()
file.write(str(sensor1[0]) + ',' + str(sensor1[1]) + ',' + str(sensor1[2]) + ',' + str(sensor2[0]) + ',' + str(sensor2[1]) + ',' + str(sensor2[2]) + ',' + time1 + '\n')
# Print data every "timeout" second
elapsed = time.process_time()
current = 0
### Block 3 ###
while(current < timeout):
current = time.process_time() - elapsed
Block 1 checks an uptime file, reads the time and then opens a file. However, if this uptime_start variable never changes, then we can just set it once outside of (and instead of) the top while loop. Block 2 then checks uptime_last < uptime_start + x, but in the previous few lines, you've defined uptime_last = uptime_start. This means that for as long as the timetosend is not negative, this loop will execute infinitely unless an internal break condition is met. Block 2 writes sensor data to file. Block 3 does a time calculation until a condition is met, but doesn't really do anything with this information.
My guess as to what could be a good answer is this:
### Block 1 ###
with open('/proc/uptime', 'r') as f: # get uptime
uptime_start = float(f.readline().split()[0])
file = open('/var/log/sensor/' + filename + '.csv', 'w')
time_elapsed = 0
time_start = time.process_time()
### Block 2 ###
while time_elapsed < uptime_start:
time1 = str(datetime.datetime.now().strftime('%S.%f'))
sensor1 = readadxl345_1()
sensor2 = readadxl345_2()
file.write(str(sensor1[0]) + ',' + str(sensor1[1]) + ',' + str(sensor1[2]) + ',' + str(sensor2[0]) + ',' + str(sensor2[1]) + ',' + str(sensor2[2]) + ',' + time1 + '\n')
time_elapsed = time.process_time() - time_start

Related

FIFO Cpu scheduling in python where Process ID, Arrival Time, Instruction Load gets collected from a xlsx worksheet

I want to create a Fifo Cpu Scheduling simulator in python 3 where I want to get the Process ID, Arrival Time, Instruction Load from an excel worksheet. I'll just put this line to run the code:
schedule_sheet=load workbook(filename='cpu-scheduling.xlsx')
Now my code is running the FIFO where I am giving the details inside the program but, I am unable to fetch from the excel sheet & send that to my program.
my code is:
def findWaitingTime(processes, n, bt, wt):
wt[0] = 0
for i in range(1, n ):
wt[i] = bt[i - 1] + wt[i - 1]
def findTurnAroundTime(processes, n,
bt, wt, tat):
for i in range(n):
tat[i] = bt[i] + wt[i]
def findavgTime( processes, n, bt):
wt = [0] * n
tat = [0] * n
total_wt = 0
total_tat = 0
findWaitingTime(processes, n, bt, wt)
findTurnAroundTime(processes, n,
bt, wt, tat)
print( "Processes Burst time " +
" Waiting time " +
" Turn around time")
for i in range(n):
total_wt = total_wt + wt[i]
total_tat = total_tat + tat[i]
print(" " + str(i + 1) + "\t\t" +
str(bt[i]) + "\t " +
str(wt[i]) + "\t\t " +
str(tat[i]))
print( "Average waiting time = "+
str(total_wt / n))
print("Average turn around time = "+
str(total_tat / n))
if __name__ =="__main__":
processes = [ 1, 2, 3]
n = len(processes)
burst_time = [10, 5, 8]
findavgTime(processes, n, burst_time)
How I can do that from the Excel sheet please help me.
This is sample picture how the excelsheet would be:

You've got your variable pointing to the Excel sheet with the data. You just need to put that variable in the __main__ function and run the code, as follows:
if __name__ =="__main__":
# Add these below lines of code to read data from the Excel file
schedule_sheet=load_workbook(filename='cpu-scheduling.xlsx')
sheets = schedule_sheet.sheetnames
# To work with the first sheet (by name)
ws = schedule_sheet[sheets[0]]
processes = [ 1, 2, 3]
n = len(processes)
burst_time = [10, 5, 8]
findavgTime(processes, n, burst_time)
Output produces:
Processes Burst time Waiting time Turn around time
1 10 0 10
2 5 10 15
3 8 15 23
Average waiting time = 8.333333333333334
Average turn around time = 16.0

Python multiprocessing shared memory without copy

I'm currently working on a project for fun, involving calculating way too many numbers of the Fibonacci sequence. Thing is, I want it to go fast and I also don't want to loose too much progress if I have to do an update or have a computer issues....
So I over-engineered that, and I am currently looking to implement it with multiprocessing, but everywhere I look the only way to share memory in that case is to make a duplicate of it and that takes about an hour for (Yeah I have very big numbers)
Is there any way to share a dictionary with multiprocessing without making a copy of it?
Here is the code, currently with my last implementation using threading instead of using multiprocessing. (ignore the bad timekeeping of this, i'll fix it later)
import threading
import time
def printProgressBar(iteration, total, prefix="", suffix="", decimals=1, length=100, fill="█", printEnd="\r"):
"""
Call in a loop to create terminal progress bar
#params:
iteration - Required : current iteration (Int)
total - Required : total iterations (Int)
prefix - Optional : prefix string (Str)
suffix - Optional : suffix string (Str)
decimals - Optional : positive number of decimals in percent complete (Int)
length - Optional : character length of bar (Int)
fill - Optional : bar fill character (Str)
printEnd - Optional : end character (e.g. "\r", "\r\n") (Str)
"""
percent = ("{0:." + str(decimals) + "f}").format(100 * (iteration / float(total)))
filledLength = int(length * iteration // total)
bar = fill * filledLength + "-" * (length - filledLength)
print(
f"\r{prefix} |{bar}| {percent}% {suffix} {time.strftime('%dd %H:%M:%S', time.gmtime(time.time() - start_time - 86400)).replace('31d', '0d')}",
end=printEnd,
)
# Print New Line on Complete
if iteration == total:
print()
dict47 = {0: 0, 1: 1, 2: 1}
dict47[0] = 2
def fibSequence(n):
if n not in dict47:
dict47[n] = fibSequence(n - 1) + fibSequence(n - 2)
if n > dict47[0]:
dict47[0] = n
if n > 5 and not ((n - 3) % (iterations // 100) == 0) and not ((n - 2) % (iterations // 100) == 0):
dict47.pop(n - 3, None)
return dict47[n]
def makeBackup(start, num, total):
x = threading.Thread(target=writeBackup, args=(num,), daemon=False)
y = threading.Thread(target=writeBackup, args=(num - 1,), daemon=False)
x.start()
y.start()
x.join()
y.join()
time.sleep(1)
print(
f'{num/10000000}% done after {time.strftime("%dd %H:%M:%S", time.gmtime(time.time() - start - 86400)).replace("31d", "0d")}'
)
timings = open("times.txt", "a")
timings.write(str(int(time.time() - start)) + "\n")
timings.close()
def writeBackup(num):
file = open(f".temp/fib{num}.txt", "a")
file.write(str(num) + " : " + str(dict47[num]))
file.close()
dict47.pop(num, None)
def loadDict():
from pathlib import Path
maximum = 0
for n in range(1, 100):
if Path(f".temp/fib{n*10000000}.txt").is_file():
maximum = n * 10000000
print("Maximum number found:", maximum)
if maximum != 0:
file = open(f".temp/fib{maximum}.txt", "r")
temp = "".join(file.readlines())
dict47[maximum] = int(temp.lstrip(str(maximum) + " : "))
file.close()
file = open(f".temp/fib{maximum - 1}.txt", "r")
temp = "".join(file.readlines())
dict47[maximum - 1] = int(temp.lstrip(str(maximum - 1) + " : "))
file.close()
dict47[0] = maximum
print("Dictionary loaded at ", maximum)
else:
print("No dictionary found, starting from scratch")
if __name__ == "__main__":
try:
timings = open("times.txt", "r")
lastTime = int(timings.readlines()[-1])
start_time = time.time() - lastTime
timings.close()
except:
start_time = time.time() + 86400
print("Start duration:", time.strftime("%dd %H:%M:%S", time.gmtime(time.time() - start_time)).replace("31d", "0d"))
try:
iterations = int(input("Enter the number of iterations: "))
except:
iterations = 1000000000
print(iterations, "iterations will be performed")
loadDict()
num = dict47[0]
while num < iterations:
if num == 2:
num += 248
else:
num += 250
fibSequence(num)
if num % 1000 == 0:
printProgressBar(num, iterations, prefix="Progress:", suffix="Complete", length=100)
if num % (iterations // 100) == 0:
save = threading.Thread(
target=makeBackup,
args=(start_time, num, iterations),
daemon=False,
)
save.start()
file = open("fib.txt", "a")
file.write(str(iterations) + " : " + str(fibSequence(iterations)) + "\n")
file.close()
try:
save.join()
except:
print("No save thread running, exiting...")

How to fix error TypeError: a bytes-like object is required, not 'str'?

#-*- coding: utf-8 -*-
import sha3
import pyopencl as cl
import Keccak
import time
#Initialize OpenCL
platforms = cl.get_platforms()
devices = platforms[0].get_devices()
context = cl.Context(devices[:2])
queue = cl.CommandQueue(context, context.devices[0],
properties=cl.command_queue_properties.PROFILING_ENABLE)
program = cl.Program(context, open('sha3.cl').read()).build(options='')
#Parameters for SHA 512
r = 576
c = 1024
n = 512
inputlist = []
inputlist.append("a" * 1000)
start = time.time()
result = sha3.Keccak(inputlist, n, r,c, program, context, queue)
print ("Hashing Result is")
print (result)
print ("Time taken is: " + str(time.time() - start))
This is error what give script when i try run it.
Traceback (most recent call last):
File "J:/PuzzleUtils/AllInOneWithPermutation/eeeee.py", line 24, in <module>
result = sha3.Keccak(inputlist, n, r,c, program, context, queue)
File "J:/PuzzleUtils/AllInOneWithPermutation\sha3.py", line 195, in Keccak
gpu_string = cl.Buffer(context, cl.mem_flags.READ_ONLY | cl.mem_flags.COPY_HOST_PTR, hostbuf=host_string)
TypeError: a bytes-like object is required, not 'str'
no mater what i do. I cannot find what give this error , when i try add encode() it's give list has no attribute 'encode'
I trying rework this opencl keccak implementaion script https://github.com/kevinkeyjkw/sha3_hash_parallel
for make it work at python 3.8
from __future__ import division
import sys
import pyopencl as cl
import numpy as np
import pylab
import pdb
import time
def pad10star1(M, n):
"""Pad M with the pad10*1 padding rule to reach a length multiple of r bits
M: message pair (length in bits, string of hex characters ('9AFC...')
n: length in bits (must be a multiple of 8)
Example: pad10star1([60, 'BA594E0FB9EBBD30'],8) returns 'BA594E0FB9EBBD93'
"""
[my_string_length, my_string]=M
# Check the parameter n
if n%8!=0:
raise KeccakError.KeccakError("n must be a multiple of 8")
# Check the length of the provided string
if len(my_string)%2!=0:
#Pad with one '0' to reach correct length (don't know test
#vectors coding)
my_string=my_string+'0'
if my_string_length>(len(my_string)//2*8):
raise KeccakError.KeccakError("the string is too short to contain the number of bits announced")
nr_bytes_filled=my_string_length//8
nbr_bits_filled=my_string_length%8
l = my_string_length % n
if ((n-8) <= l <= (n-2)):
if (nbr_bits_filled == 0):
my_byte = 0
else:
my_byte=int(my_string[nr_bytes_filled*2:nr_bytes_filled*2+2],16)
my_byte=(my_byte>>(8-nbr_bits_filled))
my_byte=my_byte+2**(nbr_bits_filled)+2**7
my_byte="%02X" % my_byte
my_string=my_string[0:nr_bytes_filled*2]+my_byte
else:
if (nbr_bits_filled == 0):
my_byte = 0
else:
my_byte=int(my_string[nr_bytes_filled*2:nr_bytes_filled*2+2],16)
my_byte=(my_byte>>(8-nbr_bits_filled))
my_byte=my_byte+2**(nbr_bits_filled)
my_byte="%02X" % my_byte
my_string=my_string[0:nr_bytes_filled*2]+my_byte
while((8*len(my_string)//2)%n < (n-8)):
my_string=my_string+'00'
my_string = my_string+'80'
return my_string
def KeccakF(to_hash, iterations, curr_iter, program, context, queue):
WORDLENGTH = 64
inputnum = int(to_hash.shape[0]/5)
#Set up Round constants
RC=[0x0000000000000001,
0x0000000000008082,
0x800000000000808A,
0x8000000080008000,
0x000000000000808B,
0x0000000080000001,
0x8000000080008081,
0x8000000000008009,
0x000000000000008A,
0x0000000000000088,
0x0000000080008009,
0x000000008000000A,
0x000000008000808B,
0x800000000000008B,
0x8000000000008089,
0x8000000000008003,
0x8000000000008002,
0x8000000000000080,
0x000000000000800A,
0x800000008000000A,
0x8000000080008081,
0x8000000000008080,
0x0000000080000001,
0x8000000080008008]
round_constants = np.array([np.uint64(x) for x in RC])
round_constants_gpu = cl.Buffer(context, cl.mem_flags.READ_ONLY, 8 * len(round_constants)) #Why * 8???
cl.enqueue_copy(queue, round_constants_gpu, round_constants, is_blocking=False)
#Set up rotation offsets
rotation_offsets = np.array([0, 36, 3, 41, 18, \
1 ,44 ,10 ,45 ,2, \
62, 6 ,43 ,15 ,61, \
28, 55, 25, 21, 56, \
27, 20, 39, 8 ,14])
rotation_offsets = np.array([np.uint64(x) for x in rotation_offsets])
rotation_gpu_buffer = cl.Buffer(context, cl.mem_flags.READ_ONLY, 8 * len(rotation_offsets))
cl.enqueue_copy(queue, rotation_gpu_buffer, rotation_offsets, is_blocking=False)
stuff_to_hash = cl.Buffer(context, cl.mem_flags.READ_ONLY, to_hash.size * 8)
cl.enqueue_copy(queue, stuff_to_hash, to_hash, is_blocking=False)#is_block=True means wait for completion
#Buffer for GPU to write final hash
gpu_final_hash = cl.Buffer(context, cl.mem_flags.READ_WRITE, to_hash.size * 8)
#control the number of iterations of each hash in Keccak
gpu_iterations = cl.Buffer(context, cl.mem_flags.READ_ONLY, len(iterations)*8)
cl.enqueue_copy(queue, gpu_iterations, np.array(iterations), is_blocking=False)#is_block=True means wait for completion
gpu_curr_iter = cl.Buffer(context, cl.mem_flags.READ_ONLY, 8)
cl.enqueue_copy(queue, gpu_curr_iter, np.array(curr_iter), is_blocking=False)#is_block=True means wait for completion
#Create 5x5 workgroup, local buffer
local_size, global_size = (5, 5) , (5,5*inputnum)
local_buf_w,local_buf_h = np.uint64(5),np.uint64(5)
A = cl.LocalMemory(8*25)
B = cl.LocalMemory(8*25)
C = cl.LocalMemory(8*25)
D = cl.LocalMemory(8*25)
#Hash input
final_hash = np.zeros((5*inputnum,5))
final_hash = np.array([np.uint64(x) for x in final_hash])
hash_event = program.sha_3_hash(queue, global_size, local_size,
stuff_to_hash, gpu_final_hash,rotation_gpu_buffer,round_constants_gpu, gpu_iterations, gpu_curr_iter,
B,A, C, D, local_buf_w,local_buf_h)
cl.enqueue_copy(queue, final_hash, gpu_final_hash, is_blocking=True)
return final_hash
def Keccak(inputlist, n,r,c, program, context, queue):
inputnum = len(inputlist)
input_str = inputlist[0]
#P is a storage for the padded inputs
P = []
#Z is a storage for the output hashes
Z = []
iterations = []
#start = time.time()
### Padding Phase
for i in range(inputnum):
tmpstr = pad10star1([len(inputlist[i])*4, inputlist[i]],r)
P.append(tmpstr)
Z.append("")
iterations.append((len(tmpstr)*8//2)//r)
#print ("Time to run padding: " + str(time.time() - start))
# Initialisation of state
S = np.zeros((5*inputnum,5))
#Testing
S = np.array([np.uint64(x) for x in S])
#Initialize workgroup sizes for the gpu
local_size, global_size = (5, 5) , (5,5*inputnum)
for i in range(max(iterations)):
host_string = ""
Pi = np.zeros((5*inputnum,5))
Pi = np.array([np.uint64(x) for x in Pi])
for j in range(inputnum):
if (iterations[j] > i):
#Absorbing Phase
host_string = host_string + str(P[j][i*(2*r//8):(i+1)*(2*r//8)]+'00'*(c//8))
else:
#Dummy variables. Won't be used.
host_string = host_string + "0"*400
gpu_string = cl.Buffer(context, cl.mem_flags.READ_ONLY | cl.mem_flags.COPY_HOST_PTR, hostbuf=host_string)
gpu_table = cl.Buffer(context, cl.mem_flags.READ_WRITE, 25*8 * inputnum)
part_of_string = cl.LocalMemory(1*16)
program.convert_str_to_table(queue,global_size,local_size, gpu_string, gpu_table, part_of_string, np.uint64(5),np.uint64(5),np.uint64(64))
cl.enqueue_copy(queue, Pi, gpu_table, is_blocking=True)
for x in range(5*inputnum):
for y in range(5):
#print 'type S:',type(S[x][y]),'type P:',type(Pi[x][y])
if (iterations[int(x/5)] > i):
S[x][y] = S[x][y]^Pi[x][y]
S = np.array([np.uint64(x) for x in S])
#start = time.time()
S = KeccakF(S, iterations, i, program, context, queue)
#print ("Time to run KeccakF: " + str(time.time() - start))
#print (S)
#Squeezing phase
outputstring = np.chararray(400 * inputnum)
gpu_table = cl.Buffer(context, cl.mem_flags.READ_ONLY | cl.mem_flags.COPY_HOST_PTR, hostbuf=S)
gpu_string = cl.Buffer(context, cl.mem_flags.READ_WRITE, 144*8 * inputnum)
program.convert_table_to_str(queue,global_size,local_size,gpu_table, gpu_string,np.uint64(5),np.uint64(5),np.uint64(64))
cl.enqueue_copy(queue, outputstring, gpu_string, is_blocking=True)
string = ''.join(outputstring)
for x in range(inputnum):
Z[x] = Z[x] + string[400 * x: 400 * x + r*2//8]
for x in range(inputnum):
Z[x] = Z[x][0:2*n//8]
#output the pre-set number of bits
return Z
if __name__ == '__main__':
# List our platforms
platforms = cl.get_platforms()
# Create a context with all the devices
devices = platforms[0].get_devices()
context = cl.Context(devices[:2])
#print ('This context is associated with ', len(context.devices), 'devices')
# Create a queue for transferring data and launching computations.
# Turn on profiling to allow us to check event times.
queue = cl.CommandQueue(context, context.devices[0],
properties=cl.command_queue_properties.PROFILING_ENABLE)
#print ('The queue is using the device:', queue.device.name)
program = cl.Program(context, open('sha3.cl').read()).build(options='')
#PARAMETERS for SHA 512
r = 576
c = 1024
n = 512
inputlist = []
inputlist.append("")
inputlist.append("abcd")
inputlist.append("abcd")
inputlist.append("abcd")
inputlist.append("a" * 1000)
start = time.time()
result = Keccak(inputlist, n, r,c, program, context, queue)
print ("Hashing Result is")
print (result)
print ("Time taken is: " + str(time.time() - start))
this is functions from sha3

Try this.
The inputlist shall be converted to a bytearray.
result = sha3.Keccak(bytearray(inputlist), n, r,c, program, context, queue)

Catching stdout and camera frames in realtime from subprocess

I have a wireless sensor node that transmits wirelessly to a receiver connected to my computer. I use this command to see the data in real time
sudo stdbuf -o0 ./pip_sense.v2 l l | stdbuf -o0 grep -P "TX:03(376|004)
I am new to Python, but I was able to develop a piece of code as show below to capture the stdout and takes frames from a camera connected to the same computer. That is when a packet (data) is received, at that timestamp, a frame will be captured.
It is working fine, except the fact that its running slow. I am running my sensor at a rate of 10 Hz (transmitting every 0.1 sec.) but the code does what is required at a slower rate. I know that is because I am using cv2.imwrite() (I commented out this part and that helped in reducing the delay). But also, as discussed in this post and its said (as I understand) that using shell=True can invoke extra shell process and thus increase the delay.
Also, looking at this, Popen() seems to be causing delays as well. A solution is proposed in the second link, but I do not really understand how to modify it to work for me.
#!/usr/bin/env python
from subprocess import Popen, PIPE
import time
import sys
import cv2
import os
import csv
import argparse
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.patches as mpatches
from collections import OrderedDict
from datetime import datetime
PIP_CMD = 'sudo stdbuf -o0 ./pip_sense.v2 l l | stdbuf -o0 grep -P "TX:03(376|004)"'
def run(command):
process = Popen(command, stdout=PIPE, shell=True)
while True:
line = process.stdout.readline().rstrip()
if not line:
break
yield line
print ('Starting...')
def createFolder(directory):
try:
if not os.path.exists(directory):
os.makedirs(directory)
except OSError:
print ('Error: Creating directory. ' + directory)
createFolder('/home/piptag/pip-test/capture_webcam/EXPERIMENTS')
file1 = open('/home/piptag/pip-test/capture_webcam/EXPERIMENTS/3376.txt','w')
file1.write("Time Stamp \t TX ID \t RSSI \tSEQ NO \tCAP \tX-Axis \tY-Axis \tZ-Axis \t X \t Y \t Z \n") # Header line
file1.write("-------------------------------------------------------------------------------------------------------------------------------------\n")
file2 = open('/home/piptag/pip-test/capture_webcam/EXPERIMENTS/3004.txt','w')
file2.write("Time Stamp \t TX ID \t RSSI \tSEQ NO \tX-Axis \tY-Axis \tZ-Axis \t X \t Y \t Z \n") # Header line
file2.write("-------------------------------------------------------------------------------------------------------------------------------------\n")
dirname = "/home/piptag/pip-test/capture_webcam/EXPERIMENTS/"
def save_webcam(dirname):
cam = cv2.VideoCapture(-1) # was getting error (V4L: can't open camera by index 0), thus changes it to -1
jpg_quality = 75
frame_number = 0
if cam.isOpened():
ret_val, img = cam.read()
else:
ret_val = False
timestamp = int(TS)
path = dirname + str(tx) + str("-") + str(timestamp) + ".jpg"
cv2.imwrite(path, img, [int(cv2.IMWRITE_JPEG_QUALITY), jpg_quality])
frame_number = frame_number + 1
##this is the end of the camera program
def twos_comp(val, bits):
"""compute the 2's compliment of int value val"""
if (val & (1 << (bits - 1))) != 0: # if sign bit is set e.g., 8bit: 128-255
val = val - (1 << bits) # compute negative value
return val # return positive value as is
#########################################MAIN###################################
if __name__ == "__main__":
for line in run(PIP_CMD):
raw_data = line.split('\t')
if len(raw_data) > 1:
TS = raw_data[0][3:]
tx = raw_data[3].split(':')[-1]
rssi = float(raw_data[4][5:])
crc_ok = True if (raw_data[5] != b'BAD CRC') else False
no_drop = True if (raw_data[1] == b'Drop:0') else False
# If CRC check is ok and no drop, process the packet
if crc_ok and no_drop:
data = raw_data[-1].split(':')[-1].split()
cat = ""
for i in data:
cat += str(i)
if tx == '03376':
save_webcam(dirname)
print data
CapStatus=data[1]
if CapStatus == '50':
Cap='0'
elif CapStatus == '51':
Cap='1'
SEQNO1=str(int((data[2]),16))
x_axis1=data[3]+data[4]
y_axis1=data[5]+data[6]
z_axis1=data[7]+data[8]
TX1=tx
x1 = twos_comp(int(x_axis1,16), 16) * 0.0039 #* 9.80665 #the value is multiplied by 0.004 as the ADXL345 reports data as 4mg per 1lsb
y1 = twos_comp(int(y_axis1,16), 16) * 0.0039 #* 9.80665 #the value is multiplied by 0.004 as the ADXL345 reports data as 4mg per 1lsb
z1 = twos_comp(int(z_axis1,16), 16) * 0.0039 #* 9.80665 #the value is multiplied by 0.004 as the ADXL345 reports data as 4mg per 1lsb
st1 = str(TS) + "\t "+ "{:<5}".format(str (TX1)) + "\t" + "{:<10}".format(str (rssi)) + "{:<5}".format(SEQNO1) + "\t" + "{:<5}".format(str (Cap)) + "\t" + "{:<5}".format(str (x_axis1)) + "\t" + "{:<5}".format(str (y_axis1)) + "\t"+ "{:<5}".format(str(z_axis1)) + "\t" + "{:<5}".format(str(x1)) + "\t" + "{:<5}".format(str(y1)) + "\t" + "{:<5}".format(str(z1)) +"\n"
file1.write(st1)
elif tx == '03004':
save_webcam(dirname)
print data
SEQNO2=str(int((data[1]),16))
x_axis2=data[2]+data[3]
y_axis2=data[4]+data[5]
z_axis2=data[6]+data[7]
TX2=tx
x2 = twos_comp(int(x_axis2,16), 16) * 0.0039 #* 9.80665 #the value is multiplied by 0.004 as the ADXL345 reports dataas 4mg per 1lsb
y2 = twos_comp(int(y_axis2,16), 16) * 0.0039 #* 9.80665 #the value is multiplied by 0.004 as the ADXL345 reports data as 4mg per 1lsb
z2 = twos_comp(int(z_axis2,16), 16) * 0.0039 #* 9.80665 #the value is multiplied by 0.004 as the ADXL345 reports data as 4mg per 1lsb
st2 = str(TS) + "\t "+ "{:<5}".format(str (TX2)) +"\t "+ "{:<10}".format(str (rssi)) + "{:<5}".format(SEQNO2) + "\t" + "{:<5}".format(str (x_axis2)) + "\t"+ "{:<5}".format(str (y_axis2)) + "\t"+ "{:<5}".format(str(z_axis2))+ "\t"+"{:<5}".format(str(x2)) + "\t" + "{:<5}".format(str(y2))+ "\t"+ "{:<5}".format(str(z2)) +"\n"
file2.write(st2)
file1.close()
file2.close()
I appreciate any help
Thank you

Update:
I got this advice from a friend and it worked.
Calling the cam = cv2.VideoCapture(-1) outside the save_webcam function (as a global variable) reduced the delays a lot.
In this case, the camera will be turned on one time and then capturing images will be done each time save_webcam is called...hence, turning on and off the camera was slowing the process.
However, I would like to enhance the program more, as it get stuck sometimes. So, anyone understands what is in this post and how my stdout part should be modified, then please comment here. I appreciate this.

BitString with python

I am trying to use bitstring for python to interpret an incoming data packet and break it up into readable sections. the packet will consist of a header( Source (8bits), Destination (8bits), NS(3bits), NR(3bits), RSV(1bit), LST(1bit), OPcode(8bits), LEN(8bits) ),
the Payload which is somewhere between 0 and 128 bytes (determined by the LEN in the header) and a CRC of 16bits.
The data will be arriving in a large packet over the com port. The data is originated from a micro controller that is packetizing the data and sending it to the user, which is where the python comes in to play.
Since i am unsure of how to store it before parsing I do not have any code for this.
I am new to python and need a little help getting this off the ground.
Thanks,
Erik
EDIT
I currently have a section of code up and running, but it is not producing exactly what i need.... Here is the section of code that i have up and running....
def packet_make(ser):
src = 10
p = 0
lst = 0
payload_make = 0
crc = '0x0031'
ns = 0
nr = 0
rsv = 0
packet_timeout = 0
top = 256
topm = 255
#os.system(['clear','cls'][os.name == 'nt'])
print("\tBatts: 1 \t| Berry: 2 \t| Bessler: 3")
print("\tCordell: 4 \t| Dave: 5 \t| Gold: 6")
print("\tYen: 7 \t| Erik: 8 \t| Tommy: 9")
print("\tParsons: 10 \t| JP: 11 \t| Sucess: 12")
dst = raw_input("Please select a destination Adderss: ")
message = raw_input("Please type a message: ")
#################### Start Making packet#################
p_msg = message
message = message.encode("hex")
ln = (len(message)/2)
#print (ln)
ln_hex = (ln * 2)
message = list(message)
num_of_packets = ((ln/128) + 1)
#print (num_of_packets)
message = "".join(message)
src = hex(src)
dst = hex(int(dst))
#print (message)
print("\n########Number of packets = "+str(num_of_packets) + " ############\n\n")
for p in range (num_of_packets):
Ack_rx = 0
if( (p + 1) == (num_of_packets)):
lst = 1
else:
lst = 0
header_info = 0b00000000
if ((p % 2) > 0):
ns = 1
else:
ns = 0
header_info = (header_info | (ns << 5))
header_info = (header_info | (nr << 2))
header_info = (header_info | (rsv << 1))
header_info = (header_info | (lst))
header_info = hex(header_info)
#print (header_info)
op_code = '0x44'
if (lst == 1):
ln_packet = ((ln_hex - (p * 256)) % 256)
if (p > 0):
ln_packet = (ln_packet + 2)
else:
ln_packet = ln_packet
ln_packet = (ln_packet / 2)
# print (ln_packet)
# print()
else:
ln_packet = 128
# print(ln_packet)
# print()
#ll = (p * 128)
#print(ll)
#ul = ((ln - ll) % 128)
#print(ul)
#print (message[ll:ul])
if ((p == 0)&(ln_hex > 256)):
ll = (p * 255)
# print(ll)
payload_make = (message[ll:256])
# print (payload_make)
elif ((p > 0) & ((ln_hex - (p*256)) > 256)):
ll = (p * 256)
# print(ll)
ll = (ll - 2)
ul = (ll + 256)
# print (ul)
payload_make = (message[ll:ul])
# print(payload_make)
elif ((p > 0) & ((ln_hex - (p*256)) < 257)):
ll = (p * 256)
# print(ll)
ll = (ll - 2)
ul = ((ln_hex - ll) % 256)
ul = (ll + (ul))
ul = ul + 2
print()
print(ul)
print(ln_hex)
print(ln_packet)
print()
# print(ul)
payload_make = (message[ll:ul])
# print(payload)
elif ((p == 0) & (ln_hex < 257)):
ll = (p * 255)
ul = ln_hex
payload_make = (message[ll:ul])
print(payload_make)
packet_m = BitStream()
########################HEADER#########################
packet_m.append('0x0')
packet_m.append(src) #src
packet_m.append('0x0')
packet_m.append(dst) #dst
if(int(header_info,16) < 16):
packet_m.append('0x0')
packet_m.append(header_info) # Ns, Nr, RSV, Lst
packet_m.append(op_code) #op Code
#if(ln_packet < 16):
#packet_m.append('0x0')
packet_m.append((hex(ln_packet))) #Length
###################END OF HEADER#######################
packet_m.append(("0x"+payload_make)) #Payload
#packet_m.append(BitArray(p_msg)) #Payload
packet_m.append(crc) #CRC
#print()
#print(packet)
temp_ack = '0x00'
print(packet_m)
print(ln_packet)
while((Ack_rx == 0) & (packet_timeout <= 5)):
try:
###### Send the packet
#ser.write(chr(0x31))
str_pack = list(str(packet_m)[2:])
"".join(str_pack)
ser.write(chr(0x02))
#ser.write((str(packet_m)[2:]))
for i in range (len(str_pack)):
t = ord(str_pack[i])
ser.write(chr(t))
print(chr(t))
ser.write(chr(0x04))
ser.write(chr(0x10))
ack_packet = BitStream(ser.read())
if((len(ack_packet) > 3)):
temp_ack = ACK_parse(ack_packet)
else:
packet_timeout = (packet_timeout + 1)
print "why so serious\n\n"
if(temp_ack == '0x41'):
Ack_rx = 1
elif (temp_ack == '0x4E'):
Ack_rx = 0
else:
Acl_rx = 0
except serial.SerialTimeoutException: #if timeout occurs increment counter and resend last packet
Ack_rx = 0
packet_timeout = (packet_timeout + 1)
except serial.SerialException:
print "Error ... is not Active!!!", port
The output that is printed to the terminal is as follows when source and payload are both 1:
#######Number of packets = 1 #######
31
0x0a0101441310031
1
0
.... etc..
The micro on the other end of the serial reads : 0a0101441310031
when it should read a 1 1 44 1 31 0031
Python is sending each value as a separate character rather than putting it as one char. when it was appended into the packet rather than storing into the proper length and data type it seems to have separated the hex into 2 8 bit locations rather than 1 8 bit location....
The section of python code where i am reading from the Micro works flawlessly when reading an acknowledgement packet. I have not tried it with data, but i don't think that will be an issue. The C side can not read the ACK from the python side since it is separating the hex values into 2 char rather than transmitting just the 8 bit value....
Any ideas??? Thanks

Your exact problem is a bit vague, but I should be able to help with the bitstring portion of it.
You've probably got your payload to analyse as a str (or possibly bytes if you're using Python 3 but don't worry - it works the same way). If you haven't got that far then you're going to have to ask a more basic question. I'm going to make up some data to analyse (all this is being done with an interactive Python session):
>>> from bitstring import BitStream
>>> packet_data = '(2\x06D\x03\x124V\x03\xe8'
>>> b = BitStream(bytes=packet_data)
Now you can unpack or use reads on your BitStream to extract the things you need. For example:
>>> b.read('uint:8')
40
>>> b.read('uint:8')
50
>>> b.readlist('uint:3, uint:3')
[0, 1]
>>> b.readlist('2*bool')
[True, False]
>>> b.readlist('2*uint:8')
[68, 3]
>>> b.read('bytes:3')
'\x124V'
This is just parsing the bytes and interpreting chunks as unsigned integers, bools or bytes. Take a look at the manual for more details.
If you just want the payload, then you can just extract the length then grab it by slicing:
>>> length = b[32:40].uint
>>> b[40:40 + length*8]
BitStream('0x123456')
and if you want it back as a Python str, then use the bytes interpretation:
>>> b[40:40 + 3*8].bytes
'\x124V'
There are more advance things you can do too, but a good way to get going in Python is often to open an interactive session and try some things out.

Categories

Resources