I am doing a BeagleBone project that measures temperature from a sensor. I want to save the data into a CSV file. I am new to programming and Python and I am little lost. I have been google searching for a while now and found little that can help me. So I wanted to ask instead about this. How to I get new data written into the CSV file every second?
import Adafruit_BBIO.ADC as ADC
import time
import csv
sensor_pin = 'P9_40'
ADC.setup()
while True:
reading = ADC.read(sensor_pin)
millivolts = reading * 1800 # 1.8V reference = 1800 mV
temp_c = (millivolts - 500) / 10
temp_f = (temp_c * 9/5) + 32
print('mv=%d C=%d F=%d' % (millivolts, temp_c, temp_f))
time.sleep(1)
# field names
fields = ['Milivolts', 'Celsius']
# data rows of csv file
rows = [ [millivolts,"|", temp_c]]
# name of csv file
filename = "textfile.csv"
# writing to csv file
with open(filename, 'w') as csvfile:
# creating a csv writer object
csvwriter = csv.writer(csvfile)
# writing the fields
csvwriter.writerow(fields)
# writing the data rows
csvwriter.writerows(rows)
One fix to apply is to open the file in append mode so that the file content is not overwritten at every step; just change the 'w' to 'a' in this line:
with open(filename, 'a') as csvfile:
Please note that without any output and/or description of the problem you are encountering, it will be difficult to help you more.
Honestly, i don't see much wrong with the code other than the order.
The way you wrote it, every single time you are iterating the loop, you are opening the file in write mode, which erases the file. So if im guessing right, you probably only have one row in the CSV.
The below is just a re-ordering and it should work. Note that i put the fields in before the loop because you only want it once.
Keep in mind that every time you run the program it will start a fresh csv. If you want it to keep a history regardless of interrupts/ restarts, just remove the fields, and use open(filename, 'a') instead.
Since this is data stuff, if you are going for a long time, you might want to include the time.time() as part of each row. That way you can see downtime ect.
import Adafruit_BBIO.ADC as ADC
import time
import csv
sensor_pin = 'P9_40'
ADC.setup()
# name of csv file
filename = "textfile.csv"
with open(filename, 'w') as csvfile:
# creating a csv writer object
csvwriter = csv.writer(csvfile)
# writing the fields
csvwriter.writerow(['Milivolts', 'Celsius'])
while True:
reading = ADC.read(sensor_pin)
millivolts = reading * 1800 # 1.8V reference = 1800 mV
temp_c = (millivolts - 500) / 10
temp_f = (temp_c * 9/5) + 32
# writing the data rows
rows = [ [millivolts,"|", temp_c]]
csvwriter.writerows(rows)
time.sleep(1)
Related
I have one CSV file, and I want to extract the first column of it. My CSV file is like this:
Device ID;SysName;Entry address(es);IPv4 address;Platform;Interface;Port ID (outgoing port);Holdtime
PE1-PCS-RANCAGUA;;;192.168.203.153;cisco CISCO7606 Capabilities Router Switch IGMP;TenGigE0/5/0/1;TenGigabitEthernet3/3;128 sec
P2-CORE-VALPO.cisco.com;P2-CORE-VALPO.cisco.com;;200.72.146.220;cisco CRS Capabilities Router;TenGigE0/5/0/0;TenGigE0/5/0/4;128 sec
PE2-CONCE;;;172.31.232.42;Cisco 7204VXR Capabilities Router;GigabitEthernet0/0/0/14;GigabitEthernet0/3;153 sec
P1-CORE-CRS-CNT.entel.cl;P1-CORE-CRS-CNT.entel.cl;;200.72.146.49;cisco CRS Capabilities Router;TenGigE0/5/0/0;TenGigE0/1/0/6;164 sec
For that purpose I use the following code that I saw here:
import csv
makes = []
with open('csvoutput/topologia.csv', 'rb') as f:
reader = csv.reader(f)
# next(reader) # Ignore first row
for row in reader:
makes.append(row[0])
print makes
Then I want to replace into a textfile a particular value for each one of the values of the first column and save it as a new file.
Original textfile:
PLANNED.IMPACTO_ID = IMPACTO.ID AND
PLANNED.ESTADOS_ID = ESTADOS_PLANNED.ID AND
TP_CLASIFICACION.ID = TP_DATA.ID_TP_CLASIFICACION AND
TP_DATA.PLANNED_ID = PLANNED.ID AND
PLANNED.FECHA_FIN >= CURDATE() - INTERVAL 1 DAY AND
PLANNED.DESCRIPCION LIKE '%P1-CORE-CHILLAN%’;
Expected output:
PLANNED.IMPACTO_ID = IMPACTO.ID AND
PLANNED.ESTADOS_ID = ESTADOS_PLANNED.ID AND
TP_CLASIFICACION.ID = TP_DATA.ID_TP_CLASIFICACION AND
TP_DATA.PLANNED_ID = PLANNED.ID AND
PLANNED.FECHA_FIN >= CURDATE() - INTERVAL 1 DAY AND
PLANNED.DESCRIPCION LIKE 'FIRST_COLUMN_VALUE’;
And so on for every value in the first column, and save it as a separate file.
How can I do this? Thank you very much for your help.
You could just read the file, apply changes, and write the file back again. There is no efficient way to edit a file (inserting characters is not efficiently possible), you can only rewrite it.
If your file is going to be big, you should not keep the whole table in memory.
import csv
makes = []
with open('csvoutput/topologia.csv', 'rb') as f:
reader = csv.reader(f)
for row in reader:
makes.append(row)
# Apply changes in makes
with open('csvoutput/topologia.csv', 'wb') as f:
writer = csv.writer(f)
writer.writerows(makes);
I have a Python script that works well -- it creates QR codes from a CSV and stores them while adding the img file path to the end of the CSV.
There's just two updates I want to make that I can't crack:
Currently I have this line id, first, last = row in the reader loop which identifies the column headers from the csv. But it only works if there are three columns so they match. Is there any way to change this so it would work whether there are three columns or 20? (Should keep in mind that the script also writes two additional columns at the end of the columns.)
Also the QR codes are currently generated by whatever the first column in the CSV is because it's defined as having created them via the id column. Is there any way to have the user identify which column the QR codes should be based on as opposed to always assuming it's the first column?
Here's an example of the data that works (but wouldn't work if there were more columns):
ID First Last
144 Jerry Seinfeld
491 George Costanza
104 Elaine Benes
99 Cosmo Kramer
And here's the current script that works except these two tweaks (thanks to some others on StackOverflow to get me this far!):
#!/usr/bin/python
import csv
import qrcode
import os
import shutil
import time
import inquirer
# Identify Timestamp
timestr = time.strftime("%Y%m%d-%H%M%S")
local = 'NBC'
# Load csv
filename = "stackoverflowtest.csv"
# Path to new local folder
localfolder = local
localimagefolder = os.path.join(localfolder, 'image')
localfilefolder = os.path.join(localfolder, 'files')
# Check/create folders based on local
if not os.path.exists(localfolder):
os.makedirs(localfolder)
if not os.path.exists(localimagefolder):
os.makedirs(localimagefolder)
if not os.path.exists(localfilefolder):
os.makedirs(localfilefolder)
# Copy uploaded file to their local's file folder
target = os.path.join(localfilefolder, local+'-'+timestr+'.csv') # Target filename
# Read csv and generate QR code for local+first column of csv
with open(filename, 'rb') as csvfile, open(target, 'wb') as outfile:
reader = csv.reader(csvfile, delimiter=',', dialect=csv.excel_tab)
writer = csv.writer(outfile, delimiter=',', dialect=csv.excel_tab)
next(reader) # Skip header row.
for row in reader:
id, first, last = row # This is where I would like `id, first, last` to actually be just whatever the columns are in the CSV
qr = qrcode.QRCode(
version=1,
error_correction=qrcode.constants.ERROR_CORRECT_L,
box_size=10,
border=4,
)
qr.add_data(local+"-"+id) # This is where it would be great to not assume that the first column is always going to be their ID
qr.make()
img = qr.make_image()
imagepath = os.path.join(localimagefolder, local+"-"+id+".png")
img.save(imagepath) # Save image.
print "saving img:", imagepath
writer.writerow(row + [local+'-'+id, imagepath])
For #1, I attempted to substitute id, first, last = row with something like this but it didn't work:
with open(filename, "rb") as f:
reader = csv.reader(f)
i = reader.next()
And for #2, I attempted to do a raw_input request to ask the user what their column name is called, but I don't know if that's the best route -- and don't think it would work anyway until #1 is fixed.
Thank you!
First I suggest you use format on your strings to make your image name more readable:
e.g:
image_name = "{local_name}-{id}.png".format(local_name=local, id=id)
Open your file with 'r' instead of 'rb'.
# such that
with open(filename, "r") as f:
reader = csv.reader(f)
i = reader.next()
# I'd probably also change i to headers
headers = reader.next()
I have a python script that tell me what 'period' it is (for school) which then outputs into a csv file:
import datetime
import csv
format = "%H%M"
today = datetime.datetime.today()
s = today.strftime(format)
period = 0
ofile = open('Period.csv', "wb")
writer = csv.writer(ofile, delimiter=',')
if s > "0845" < "0945":
period = 1
if s >= "0945" < "1120":
period = 2
if s >= "1120" < "1220":
period = 3
if s >= "1220" < "1335":
period = 4
if s >= "1335" < "1430":
period = 5
print
print s
print period
writer.writerow([period])
ofile.close()
I have another python script that grabs a student's name once they swipe their student card through a magnetic card reader and then outputs that into a CSV file.
What I want to happen is when they swipe their card their name gets imported into the CSV then the period gets imported next to it (ex. john,4). Then the next student will swipe their card and their name and period will be placed under the last students
John,4
Marcus,4
But every time a card it swiped the csv file gets erased and then the current students name is the only one left in there.
import csv
import datetime
## EQ ID
eqid = "john" ## Output from magnetic card reader
## CSV writer
ofile = open('attendance.csv', "wb")
writer = csv.writer(ofile, delimiter=',')
writer.writerow([eqid])
ofile.close()
This is just a temp mock up the final code will have both codes in it.
TL;DR - how do I have python add data to a CSV file instead of just rewriting the whole file?
But every time a card it swiped the csv file gets erased and then the current students name is the only one left in there.
This is because you are writing ("wb") the file every time the card is swiped, which means a new file is created (an existing file with the same name will be erased). So You should use "ab" mode which is appending values in existing file.
So the line
ofile = open('attendance.csv', "wb")
should be changed to
ofile = open('attendance.csv', "ab")
Now everytime card is swiped the values will be appended.
you can check File Operations at https://docs.python.org/3/tutorial/inputoutput.html#reading-and-writing-files
I need to write data to a single row and column by where data is coming from the serial port. So I used to read and write row by row.
But the requirement is I need to write data to the next column of a single row In PYTHON.
I need to have filed at the end like this
1,2,33,43343,4555,344323
In such a way all data is to be in a single row and multiple columns, and not in one column and multiple rows.
So this writes data in one after one row.
1
12
2222
3234
1233
131
but I want
1 , 12 , 2222 , 3234 , 1233 , 131
like every single row and multiple columns.
import serial
import time
import csv
ser = serial.Serial('COM29', 57600)
timeout = time.time() + 60/6 # 5 minutes from now
while True:
test = 0
if test == 5 or time.time() > timeout:
break
ss=ser.readline()
print ss
s=ss.replace("\n","")
with open('C:\Users\Ivory Power\Desktop\EEG_Data\Othr_eeg\egg31.csv', 'ab') as csvfile:
spamwriter = csv.writer(csvfile,delimiter=',', lineterminator='\n')
spamwriter.writerow([ s ])
csvfile.close()
time.sleep(0.02)
The csv module writes rows - every time you call writerow a newline is written and a new row is started. So, you can't call it multiple times and expect to get columns. You can, however, collect the data into a list and then write that list when you are done. The csv module is overkill for this.
import serial
import time
import csv
ser = serial.Serial('COM29', 57600)
timeout = time.time() + 60/6 # 5 minutes from now
data = []
for _ in range(5):
data.append(ser.readline().strip())
time.sleep(0.02)
with open('C:\Users\Ivory Power\Desktop\EEG_Data\Othr_eeg\egg31.csv', 'ab') as csvfile:
spamwriter = csv.writer(csvfile,delimiter=',', lineterminator='\n')
spamwriter.writerow(data)
# csv is overkill here unless the data itself contains commas
# that need to be escaped. You could do this instead.
# csvfile.write(','.join(data) + '\n')
UPDATE
One of the tricks to question writing here to to supply a short, runnable example of the problem. That way, everybody runs the same thing and you can talk about whats wrong in terms of the code and output everyone can play with.
Here is the program updated with mock data. I changed the open to "wb" so that the file is deleted if it already exists when the program runs. Run it and let me know how its results different from what you want.
import csv
import time
filename = 'deleteme.csv'
test_row = '1,2,33,43343,4555,344323'
test_data = test_row.split(',')
data = []
for _ in range(6):
data.append(test_data.pop(0).strip())
time.sleep(0.02)
with open(filename, 'wb') as csvfile:
spamwriter = csv.writer(csvfile,delimiter=',', lineterminator='\n')
spamwriter.writerow(data)
print repr(open(filename).read())
assert open(filename).read().strip() == test_row, 'got one row'
Assuming your serial port data wouldn't overrun your main memory heap, the following would be the code that would suit your need.
import serial
import time
import csv
ser = serial.Serial('COM29', 57600)
timeout = time.time() + 60/6 # 5 minutes from now
result = []
while True:
test = 0
if test == 5 or time.time() > timeout:
break
ss=ser.readline()
print(ss)
s=ss.replace("\n","")
result.append(s)
time.sleep(0.02)
with open('C:\Users\Ivory Power\Desktop\EEG_Data\Othr_eeg\egg31.csv', 'w') as csvfile:
spamwriter = csv.writer(csvfile,delimiter=',', lineterminator='\n')
spamwriter.writerow([result])
csvfile.close()
I have a .csv file (see image):
In the image there is a time column with datetime strings, I have a program that takes this column and only reads the times H:M:S. Yet, not only in my program I am attempting to take the column to read only the time stamp H:M:S , but I am also attempting to overwrite the time column of the first file and replace it with only the H:M:S time stamp onto a the new .csv with the following code.
CODE:
import csv
import datetime as dt
import os
File = 'C:/Users/Alan Cedeno/Desktop/Test_Folder/HiSAM1_data_160215_164858.csv'
root, ext = os.path.splitext(File)
output = root + '-new.csv'
with open(File,'r') as csvinput,open(output, 'w') as csvoutput:
writer = csv.writer(csvoutput, lineterminator='\n')
reader = csv.reader(csvinput)
all = []
row = next(reader)
for line in reader:
row.append(dt.datetime.strptime(line[0],'%m/%d/%Y %H:%M:%S').time())
all.append(row)
for row in reader:
row.append(row[0])
all.append(row)
writer.writerows(all)
The program works, and takes the datetime strings and overwrites the string with the time stamp H:M:S in a new .csv file. However, here is the problem, the output file instead of replacing the time column it replaced every column obtaining an output file that looks like this. See 2nd image:
At this point I don' t really know how to make the new output file to look like the file of the first image, with the format H:M:S in the first column ONLY, not all scrambled like in the second image. Any suggestions?
SCREENSHOT FOR BAH:
See the K column, it should be column A of the first image, and columns B,C,D,E,F,G,I,and J should stay the same like in image 1.
Download LInk of .csv file: http://www.speedyshare.com/z2jwq/HiSAM1-data-160215-164858.csv
The main problem with your code seems that you're keeping appending to the first row the time of each of the line in the csv, which results in the second image posted in the question.
The idea is to keep track of the different lines and modify just the first element of each line. Also, if you want, you should keep the first line, which indicates the labels of the column. For solving the issue, the code would look like:
import csv
import datetime as dt
import os
File = 'C:/Users/Alan Cedeno/Desktop/Test_Folder/HiSAM1_data_160215_164858.csv'
root, ext = os.path.splitext(File)
output = root + '-new.csv'
with open(File,'r') as csvinput,open(output, 'w') as csvoutput:
writer = csv.writer(csvoutput, lineterminator='\n')
reader = csv.reader(csvinput)
rows = [next(reader)]
for line in reader:
line[0] = str(dt.datetime.strptime(line[0],'%m/%d/%Y %H:%M:%S').time())
rows.append(line)
writer.writerows(rows)
Note the list rows has the modified lines from the csvinput.
The resulting output csv file (tested with the first line in the question duplicated) would be
With some simplified data:
#!python3
import csv
import datetime as dt
import os
File = 'data.csv'
root, ext = os.path.splitext(File)
output = root + '-new.csv'
# csv module documents opening with `newline=''` mode in Python 3.
with open(File,'r',newline='') as csvinput,open(output, 'w',newline='') as csvoutput:
writer = csv.writer(csvoutput)
reader = csv.reader(csvinput)
# Copy the header
row = next(reader)
writer.writerow(row)
# Edit the first column of each row.
for row in reader:
row[0] = dt.datetime.strptime(row[0],'%m/%d/%Y %H:%M:%S').time()
writer.writerow(row)
Input:
Time,0.3(/L),0.5(/L)
02/15/2016 13:44:01,88452,16563
02/15/2016 13:44:02,88296,16282
Output:
Time,0.3(/L),0.5(/L)
13:44:01,88452,16563
13:44:02,88296,16282
If actually on Python 2, the csv module documents using binary mode. Replace the with line with:
with open(File,'rb') as csvinput,open(output, 'wb') as csvoutput:
You cannot overwrite a single row in the CSV file. You'll have to write all the rows you want to a new file and then rename it back to the original file name.
Your pattern of usage may fit a database better than a CSV file. Look into the sqlite3 module for a lightweight database.