Develop Reference

Python writing twice but randomly?

So I am currently working on a program that was handed down to me from a previous coworker and I am working through a strange bug. When reading data output from 2 separate serial sources byte by byte, python will write to the same cell in the .csv file as well as the console.
import serial
from datetime import datetime
import os
pressure_passed = False
arduino_passed = False
file_passed = False
BAUD_RATE = 115200
GARBAGE_CYCLES = 3 # how many cycles to ignore before logging data
garbage_cycle = 0
# Save data to log file
def LogData(startTime, pressureData, arduinoData, file):
global garbage_cycle
if garbage_cycle < GARBAGE_CYCLES:
garbage_cycle += 1
else:
delta = datetime.now() - startTime
ms = delta.total_seconds() * 1000
dataString = "{:0.2f}, {}, {}\n".format(ms, pressureData, arduinoData)
file.write(dataString)
file.flush()
print(dataString, end = "")
# Get the COM port for the Mark-10 Series 5
while not pressure_passed:
try:
pressure_com = input("Enter Mark-10 Series 5 COM Port #: ")
pressure_ser = serial.Serial("COM" + str(pressure_com), BAUD_RATE)
pressure_passed = True
except:
print("Invalid COM Port, please enter a valid port.\n-----")
# Get the COM port for the Arduino
while not arduino_passed:
try:
arduino_com = input("Enter Ardunio COM Port #: ")
arduino_ser = serial.Serial("COM" + str(arduino_com), BAUD_RATE)
arduino_passed = True
except:
print("Invalid COM Port, please enter a valid port.\n-----")
# Get the name for the log file
while not file_passed:
try:
file_name = input("Enter log file name: ")
# Add extension if not already given
if "." not in file_name:
file_name += ".csv"
log_file = open(file_name, "a")
# Add header row to log file
if os.stat(log_file.name).st_size == 0:
log_file.write("time (ms), pressure, rate (deg/ms)")
file_passed = True
except:
print("Invalid file, or could not open the file specified.\n-----")
start = datetime.now()
# Variables to read serial input
pressure_data = ""
last_pressure = ""
arduino_data = ""
last_arduino = ""
# Main program loop
# Serial is read from byte by byte to better sync the two devices
while True:
try:
x_changed = False
y_changed = False
# Read from Mark-10 serial if available
# x is a byte read from the serial line, converted to ascii
if pressure_ser.in_waiting > 0:
x = pressure_ser.read().decode('ascii')
x_changed = True
# Read from Arduino serial if available
# y is a byte read from the serial line, converted to ascii
if arduino_ser.in_waiting > 0:
y = arduino_ser.read().decode('ascii')
y_changed = True
# If new data received, check if we should log it
if x_changed:
if x == '\n': # New line detected, log the accumulated data
if last_pressure != pressure_data:
LogData(start, last_pressure, last_arduino, log_file)
last_pressure = pressure_data
pressure_data = ""
elif x != '\r': # Otherwise, add the read character to the string
pressure_data += x
if y_changed:
if y == '\n': # New line detected, log the accumulated data
if last_arduino != arduino_data:
LogData(start, last_pressure, last_arduino, log_file)
last_arduino = arduino_data
arduino_data = ""
elif y != '\r': # Otherwise, add the read character to the string
arduino_data += y
except Exception as e:
print(e)
if arduino_ser.isOpen():
arduino_ser.close()
if pressure_ser.isOpen():
pressure_ser.close()
log_file.close()
break
Here is what the file is spitting out, IE the double printing to a single cell. Sample of the data
Any advice is much appreciated, thank you all!

It looks like when a new pressure is read in, but the value has not changed from last time, then it's not resetting the string that's accumulating all the characters and it doubles up. Then on the next pass, when the REAL pressure hasn't changed, it compares the doubled to the non-doubled and writes again, and vice-versa.
Try unindenting the line that resets the string to remove it from the if clause:
# If new data received, check if we should log it
if x_changed:
if x == '\n': # New line detected, log the accumulated data
if last_pressure != pressure_data:
LogData(start, last_pressure, last_arduino, log_file)
last_pressure = pressure_data
pressure_data = ""
elif x != '\r': # Otherwise, add the read character to the string
pressure_data += x
Then the same thing for the arduino value block.
Your logs will probably be much shorter now.

I like your username! My guess is that it is reading from the serial too quickly and going through the loop twice before the arduino has time to change the value of in_waiting.
At the top of your code add:
import time
And in the LogData function add:
time.sleep(0.1)
Give that a shot and let me know if it helps. 0.1s may be too long for your application but it is a good test to see if this is the issue. If it is, you can play around with the amount of time it sleeps.

Based on the sample output provided, I think it's not writing twice but rather the following specific condition is met occasionally which calls two identical LogData() lines.
Only when Condition 1 AND Condition 2 are met - the data is written "twice". Note that the LogData() call is same in both conditions.
Condition 1:
# If new data received, check if we should log it
if x_changed:
if x == '\n': # New line detected, log the accumulated data
if last_pressure != pressure_data:
LogData(start, last_pressure, last_arduino, log_file)
Condition 2:
if y_changed:
if y == '\n': # New line detected, log the accumulated data
if last_arduino != arduino_data:
LogData(start, last_pressure, last_arduino, log_file)

Having trouble parsing a .CSV file into a dict

I've done some simple .csv parsing in python but have a new file structure that's giving me trouble. The input file is from a spreadsheet converted into a .CSV file. Here is an example of the input:
Layout
Each set can have many layouts, and each layout can have many layers. Each layer has only one layer and name.
Here is the code I am using to parse it in. I suspect it's a logic/flow control problem because I've parsed things in before, just not this deep. The first header row is skipped via code. Any help appreciated!
import csv
import pprint
def import_layouts_schema(layouts_schema_file_name = 'C:\\layouts\\LAYOUT1.csv'):
class set_template:
def __init__(self):
self.set_name =''
self.layout_name =''
self.layer_name =''
self.obj_name =''
def check_layout(st, row, layouts_schema):
c=0
if st.layout_name == '':
st.layer_name = row[c+2]
st.obj_name = row[c+3]
layer = {st.layer_name : st.obj_name}
layout = {st.layout_name : layer}
layouts_schema.update({st.set_name : layout})
else:
st.layout_name = row[c+1]
st.layer_name = row[c+2]
st.obj_name = row[c+3]
layer = {st.layer_name : st.obj_name}
layout = {st.layout_name : layer}
layouts_schema.update({st.set_name : layout})
return layouts_schema
def layouts_schema_parsing(obj_list_raw1): #, location_categories, image_schema, set_location):
#------ init -----------------------------------
skipfirst = True
c = 0
firstrow = True
layouts_schema = {}
end_flag = ''
st = set_template()
#---------- start parsing here -----------------
print('Now parsing layouts schema list')
for row in obj_list_raw1:
#print ('This is the row: ', row)
if skipfirst==True:
skipfirst=False
continue
if row[c] != '':
st.set_name = row[c]
st.layout_name = row[c+1]
st.layer_name = row[c+2]
st.obj_name = row[c+3]
print('FOUND A NEW SET. SET details below:')
print('Set name:', st.set_name, 'Layout name:', st.layout_name, 'Layer name:', st.layer_name, 'Object name:', st.obj_name)
if firstrow == True:
print('First row of layouts import!')
layer = {st.layer_name : st.obj_name}
layout = {st.layout_name : layer}
layouts_schema = {st.set_name : layout}
firstrow = False
check_layout(st, row, layouts_schema)
continue
elif firstrow == False:
print('Not the first row of layout import')
layer = {st.layer_name : st.obj_name}
layout = {st.layout_name : layer}
layouts_schema.update({st.set_name : layout})
check_layout(st, row, layouts_schema)
return layouts_schema
#begin subroutine main
layouts_schema_file_name ='C:\\Users\\jason\\Documents\\RAY\\layout_schemas\\ANIBOT_LAYOUTS_SCHEMA.csv'
full_path_to_file = layouts_schema_file_name
print('============ Importing LAYOUTS schema from: ', full_path_to_file , ' ==============')
openfile = open(full_path_to_file)
reader_ob = csv.reader(openfile)
layout_list_raw1 = list(reader_ob)
layouts_schema = layouts_schema_parsing(layout_list_raw1)
print('=========== End of layouts schema import =========')
return layouts_schema
layouts_schema = import_layouts_schema()
Feel free to throw any part away that doesn't work. I suspect I've inside my head a little bit here. A for loop or another while loop may do the trick. Ultimately I just want to parse the file into a dict with the same key structure shown. i.e. the final dict's first line would look like:
{'RESTAURANT': {'RR_FACING1': {'BACKDROP': 'restaurant1'}}}
And the rest on from there. Ultimately I am goign to use this key structure and the dict for other purposes. Just can't get the parsing down!

Wouaw, that's a lot of code !
Maybe try something simpler :
with open('file.csv') as f:
keys = f.readline().split(';') # assuming ";" is your csv fields separator
for line in f:
vals = line.split(';')
d = dict(zip(keys, vals))
print(d)
Then either make a better data file (without blanks), or have the parser remembering the previous values.

While I agree with #AK47 that the code review site may be the better approach, I received so many help from SO that I'll try to give back a little: IMHO you are overthinking the problem. Please find below an approach that should get you in the right direction and doesn't even require converting from Excel to CSV (I like the xlrd module, it's very easy to use). If you already have a CSV, just exchange the loop in the process_sheet() function. Basically, I just store the last value seen for "SET" and "LAYOUT" and if they are different (and not empty), I set the new value. Hope that helps. And yes, you should think about a better data structure (redundancy is not always bad, if you can avoid empty cells :-) ).
import xlrd
def process_sheet(sheet : xlrd.sheet.Sheet):
curr_set = ''
curr_layout = ''
for rownum in range(1, sheet.nrows):
row = sheet.row(rownum)
set_val = row[0].value.strip()
layout_val = row[1].value.strip()
if set_val != '' and set_val != curr_set:
curr_set = set_val
if layout_val != '' and layout_val != curr_layout:
curr_layout = layout_val
result = {curr_set: {curr_layout: {row[2].value: row[3].value}}}
print(repr(result))
def main():
# open a workbook (adapt your filename)
# then get the first sheet (index 0)
# and call the process function
wbook = xlrd.open_workbook('/tmp/test.xlsx')
sheet = wbook.sheet_by_index(0)
process_sheet(sheet)
if __name__ == '__main__':
main()

opening shelf file / dbm file returns error dbmError created file using dbm.open and shelve.Shelf.open

python 3.4.2, on Linux
I'm pretty new to this language but I'm coding this project. It started as a simple program that displayed a dictionary . Well I'm trying to expand on it based on tutorials that i am reading. I came to one about shelving and being able to preserve info in a save file in a format much like a dictionary. So far i have a program that takes input and updates the dictionary based on the input. It's very basic but it works on a simple level but naturally I would want to save what I entered. Following is the code so far.
the updateSaveP1() function is what is giving me trouble. Although not coded like this currently, i would ultimately like the function to take 2 arguments, one to name the key in the shelf file, and one to reference the target dictionary/ list etc. Currently its not even saving to the file.
the loadINV() function is a place holder and doent work as coded currently. i need to figure out the dbm problem first as i get the same dbmError with the load function too.
I originally just .opened the file. found documentation here at Stack that i should open it with either so that it creates the right file. I have tried both to no avail.
NOTICE**** this code will create an empty file on your system at the working directory of python named savedata.db
Much Thanks and appreciation for any help.
`import pygame, math, sys, os, pprint, shelve, dbm,
SAVE_LOCATION = os.path.join(os.getcwd() + '/savedata')
SAVE_FILE_LIST = os.listdir(os.getcwd())
SOURCE = os.path.dirname('inventory.py')
YES = ["y","Y"]
NO = ["n","N"]
playerStash = {"rope": 77,
"giant's toe" : 1,
"gold" : 420}
'''def loadINV():
fileTOload = dbm.open('savedata.db', 'w')
print('opened savedata file')
#for eachLine in fileTOload:
playerStash.update(str(fileTOload.read())
)
print('updated dict')
fileTOload.close()'''
def checkSavesFile():
while True:
if os.path.exists(SAVE_LOCATION):
print('Save file found')
break
elif os.path.exists(SAVE_LOCATION + '.db'):
print('.db Save file found')
loadINV()
break
else:
updateSaveP1()
print('New Save Created')
break
def updateSaveP1():
with dbm.open('savedata', 'c') as save:
save['player1'] = str(playerStash)
save.close()
#print(SAVE_LOCATION) #debugging - file name format verification
#pprint.pprint(SAVE_FILE_LIST) debugging will pretty print list of files
checkSavesFile() # runs the save file check
def askAboutInv(player):
while True:
print("What item would you like to add? \n\
(leave blank and press enter to quit)")
name = input() # Reads input and checks for duplicates or non entries
if name == '':
break # Stop loop
elif name in playerStash.keys():
# the check to see if input was in dictionary
dict_quant = int(playerStash.get(name, 0))
# "dict_quant" represents the value in dictionary as an integer
dict_item = str(playerStash.get(name, 0))
# "dict_item represents the value in dictionary as a string
addedItem = dict_quant + 1
#handles adding the value of the input
print("You have " + dict_item + " already, \n\
would you like to add more Y/N?")
# prints " You have "dictionary number" already"
answer = input()
# checks for input if you want to add more to inventory
if answer in YES: #checks to see if y or Y is entered
playerStash[name] = addedItem
# adds +1 to the quantity of "name" per the dict_quant variable
print("you have " + str(addedItem) + " now")
# prints " you have "new dictionary number" now"
if answer in NO: #checks to see if n or N was entered
print("Nothing added") #prints
break #ends loop
else: #if none others statements are true
if name not in playerStash.keys():
#if "name" / input is not in the dictionary
playerStash[name] = playerStash.setdefault(name, 1)
# add the item to the dictionary with a value of 1
print('Inventory updated.')
# prints
updateSaveP1()
def inventoryDisp(player):# displays dictionary pointing towards argument
print("Inventory")
item_total = 0
for eachOne in playerStash.items():
print(eachOne) # looks at and prints each item/ key in dictionary
for i, q in playerStash.items():
item_total = item_total + q #adds all the quantities / values up
print("Total number of items: " + str(item_total))
# prints total number of items in inventory
def updatedInv(player): #same as above just reads "updated inventory"
print("Updated Inventory")
item_total = 0
for eachOne in playerStash.items():
print(eachOne)
for i, q in playerStash.items():
item_total = item_total + q
print("Total number of items: " + str(item_total))
inventoryDisp(playerStash)
askAboutInv(playerStash)
updateSaveP1()
updatedInv(playerStash)`
Update*****
after changing this:
`def updateSaveP1():
with dbm.open('savedata', 'c') as save:
save['player1'] = str(playerStash)
save.close()`
to this:
`def updateSaveP1():
save = openShelf()
#save = shelve.Shelf(dbm.open('savedata', 'c')) #old code
save['player1'] = str(playerStash)
print(save['player1'])
save.close()`
it would seem that the dictionary does get saved. now the loadINV function is giving me trouble. This:
def loadINV():
fileTOload = dbm.open('savedata.db', 'w')
print('opened savedata file')
#for eachLine in fileTOload:
playerStash.update(str(fileTOload.read())
)
print('updated dict')
fileTOload.close()
is now this:
def loadINV():
file = openShelf()
print('opened savedata file')
playerStash.update(file['player1'])
print('updated dict')
fileTOload.close()
but the .update() method returns this error: which i can't seem to find any info.
Traceback (most recent call last):
File "/home/pi/inventoryShelve.py", line 58, in <module>
checkSavesFile() # runs the save file check
File "/home/pi/inventoryShelve.py", line 40, in checkSavesFile
loadINV()
File "/home/pi/inventoryShelve.py", line 25, in loadINV
playerStash.update(file['player1'])
ValueError: dictionary update sequence element #0 has length 1; 2 is required

Turned out I was saving (into the shelf) the data for the inventory dictionary as something other than a dict:
def updateSaveP1():
save = openShelf()
save['player1'] = str(playerStash)
#print(save['player1'])
save.close()
became
def updateSaveP1():
save = openShelf()
save['player1'] = dict(playerStash)
#print(save['player1'])
save.close()

unknown error in jython when use startswith()

I'm using python to analyze some records bib and ris files. I made two functions for each type. The first function is the one you see below:
def limpiarlineasris(self, data):
cont = data
dic = cont.splitlines()
cont = ""
con = []
i = 0
for a in dic:
if len(a) != 0:
con.append(a)
for a in con:
cont = cont + a + "\n"
return cont
That works well and I can compile without problem. The problem arises when I write the second function see below:
def limpiarlineasbib(self, data):
cont = data
dic = cont.splitlines()
cont = ""
con = []
separador = "°-°-°"
for a in dic:
if len(a)!= 0:
if a.startswith('#'):
con.append(separador)
else:
con.append(a)
for a in con:
cont = cont + a + "\n"
return cont
When building the first function no problem. But when I compile the second compiler shows me an error but does not tell me exactly what or where it is because I am using plyjy a jar to create Jython objects, and the console only shows me an exception Plyjy without the line where it occurs. I'm using Netbeans to compile

python logparse search specific text

I am using this function in my code to return the strings i want from reading the log file, I want to grep the "exim" process and return the results, but running the code gives no error, but the output is limited to three lines, how can i just get the output only related to exim process..
#output:
{'date': '13', 'process': 'syslogd', 'time': '06:27:33', 'month': 'May'}
{'date': '13', 'process': 'exim[23168]:', 'time': '06:27:33', 'month': 'May'}
{'May': ['syslogd']}
#function:
def generate_log_report(logfile):
report_dict = {}
for line in logfile:
line_dict = dictify_logline(line)
print line_dict
try:
month = line_dict['month']
date = line_dict['date']
time = line_dict['time']
#process = line_dict['process']
if "exim" in line_dict['process']:
process = line_dict['process']
break
else:
process = line_dict['process']
except ValueError:
continue
report_dict.setdefault(month, []).append(process)
return report_dict

It's because you have a break statement inside the if that checks for "exim". As soon as you find a line with "exim", you will stop processing entirely, which sounds like the opposite of what you want!
I think you want to remove the break and put your printout inside the if. If your question is about the return value of the function, you need to make much more significant changes, probably removing report_dict entirely and simply creating a list of line_dicts that have exim in their process fields.

i changed the code to this, but it gives me just one line as output??? anything missing...
#!/usr/bin/env python
import sys
def generate_log_report(logfile):
for line in logfile:
line_split = line.split()
list = [line_split[0], line_split[1], line_split[2], line_split[4]]
if "exim" in list[3]:
l = [line_split[0], line_split[1], line_split[2], line_split[4]]
else:
li = [line_split[0], line_split[1], line_split[2], line_split[4]]
return l
if __name__ == "__main__":
if not len(sys.argv) > 1:
print __doc__
sys.exit(1)
infile_name = sys.argv[1]
try:
infile = open(infile_name, "r")
except IOError:
print "you must specify a valid file"
print __doc__
sys.exit(1)
log_report = generate_log_report(infile)
print log_report
infile.close()