Develop Reference

Read input data to csv file

I am doing a BeagleBone project that measures temperature from a sensor. I want to save the data into a CSV file. I am new to programming and Python and I am little lost. I have been google searching for a while now and found little that can help me. So I wanted to ask instead about this. How to I get new data written into the CSV file every second?
import Adafruit_BBIO.ADC as ADC
import time
import csv
sensor_pin = 'P9_40'
ADC.setup()
while True:
reading = ADC.read(sensor_pin)
millivolts = reading * 1800 # 1.8V reference = 1800 mV
temp_c = (millivolts - 500) / 10
temp_f = (temp_c * 9/5) + 32
print('mv=%d C=%d F=%d' % (millivolts, temp_c, temp_f))
time.sleep(1)
# field names
fields = ['Milivolts', 'Celsius']
# data rows of csv file
rows = [ [millivolts,"|", temp_c]]
# name of csv file
filename = "textfile.csv"
# writing to csv file
with open(filename, 'w') as csvfile:
# creating a csv writer object
csvwriter = csv.writer(csvfile)
# writing the fields
csvwriter.writerow(fields)
# writing the data rows
csvwriter.writerows(rows)

One fix to apply is to open the file in append mode so that the file content is not overwritten at every step; just change the 'w' to 'a' in this line:
with open(filename, 'a') as csvfile:
Please note that without any output and/or description of the problem you are encountering, it will be difficult to help you more.

Honestly, i don't see much wrong with the code other than the order.
The way you wrote it, every single time you are iterating the loop, you are opening the file in write mode, which erases the file. So if im guessing right, you probably only have one row in the CSV.
The below is just a re-ordering and it should work. Note that i put the fields in before the loop because you only want it once.
Keep in mind that every time you run the program it will start a fresh csv. If you want it to keep a history regardless of interrupts/ restarts, just remove the fields, and use open(filename, 'a') instead.
Since this is data stuff, if you are going for a long time, you might want to include the time.time() as part of each row. That way you can see downtime ect.
import Adafruit_BBIO.ADC as ADC
import time
import csv
sensor_pin = 'P9_40'
ADC.setup()
# name of csv file
filename = "textfile.csv"
with open(filename, 'w') as csvfile:
# creating a csv writer object
csvwriter = csv.writer(csvfile)
# writing the fields
csvwriter.writerow(['Milivolts', 'Celsius'])
while True:
reading = ADC.read(sensor_pin)
millivolts = reading * 1800 # 1.8V reference = 1800 mV
temp_c = (millivolts - 500) / 10
temp_f = (temp_c * 9/5) + 32
# writing the data rows
rows = [ [millivolts,"|", temp_c]]
csvwriter.writerows(rows)
time.sleep(1)

Parsing column from CSV and replace a value in text file with the new value

I have one CSV file, and I want to extract the first column of it. My CSV file is like this:
Device ID;SysName;Entry address(es);IPv4 address;Platform;Interface;Port ID (outgoing port);Holdtime
PE1-PCS-RANCAGUA;;;192.168.203.153;cisco CISCO7606 Capabilities Router Switch IGMP;TenGigE0/5/0/1;TenGigabitEthernet3/3;128 sec
P2-CORE-VALPO.cisco.com;P2-CORE-VALPO.cisco.com;;200.72.146.220;cisco CRS Capabilities Router;TenGigE0/5/0/0;TenGigE0/5/0/4;128 sec
PE2-CONCE;;;172.31.232.42;Cisco 7204VXR Capabilities Router;GigabitEthernet0/0/0/14;GigabitEthernet0/3;153 sec
P1-CORE-CRS-CNT.entel.cl;P1-CORE-CRS-CNT.entel.cl;;200.72.146.49;cisco CRS Capabilities Router;TenGigE0/5/0/0;TenGigE0/1/0/6;164 sec
For that purpose I use the following code that I saw here:
import csv
makes = []
with open('csvoutput/topologia.csv', 'rb') as f:
reader = csv.reader(f)
# next(reader) # Ignore first row
for row in reader:
makes.append(row[0])
print makes
Then I want to replace into a textfile a particular value for each one of the values of the first column and save it as a new file.
Original textfile:
PLANNED.IMPACTO_ID = IMPACTO.ID AND
PLANNED.ESTADOS_ID = ESTADOS_PLANNED.ID AND
TP_CLASIFICACION.ID = TP_DATA.ID_TP_CLASIFICACION AND
TP_DATA.PLANNED_ID = PLANNED.ID AND
PLANNED.FECHA_FIN >= CURDATE() - INTERVAL 1 DAY AND
PLANNED.DESCRIPCION LIKE '%P1-CORE-CHILLAN%’;
Expected output:
PLANNED.IMPACTO_ID = IMPACTO.ID AND
PLANNED.ESTADOS_ID = ESTADOS_PLANNED.ID AND
TP_CLASIFICACION.ID = TP_DATA.ID_TP_CLASIFICACION AND
TP_DATA.PLANNED_ID = PLANNED.ID AND
PLANNED.FECHA_FIN >= CURDATE() - INTERVAL 1 DAY AND
PLANNED.DESCRIPCION LIKE 'FIRST_COLUMN_VALUE’;
And so on for every value in the first column, and save it as a separate file.
How can I do this? Thank you very much for your help.

You could just read the file, apply changes, and write the file back again. There is no efficient way to edit a file (inserting characters is not efficiently possible), you can only rewrite it.
If your file is going to be big, you should not keep the whole table in memory.
import csv
makes = []
with open('csvoutput/topologia.csv', 'rb') as f:
reader = csv.reader(f)
for row in reader:
makes.append(row)
# Apply changes in makes
with open('csvoutput/topologia.csv', 'wb') as f:
writer = csv.writer(f)
writer.writerows(makes);

appending dates by columns in a csv file in python

I have a script who goes through all the files within a given path('C:\Users\Razvi\Desktop\italia') and reads the number of lines from each file found with the name "cnt.csv" and then it writes in another file named "counters.csv" the current date + the name of folder + the sum of lines found in the "cnt.csv".
Now the output file("counters.csv") looks like this:
30/9/2017
8dg5 5
8dg7 7
01/10/2017
8dg5 8
8dg7 10
In which the names 8dg5 and 8dg7 are the folders where the script found the file"cnt.csv" aand the numers 5,7,8,10 are the sum of lines found in csv files every time when i run the script and obviously the date when i run the script it's appended every time.
Now what i want it's to write those dates in csv, somehow appended on columns , not lines, something like this:
30/9/2017 01/10/2017
8dg5 5 8dg5 8
8dg7 7 8dg7 10
Here is the code:
import re
import os
from datetime import datetime
#italia =r'C:\Users\timraion\Desktop\italia'
italia =r'C:\Users\Razvi\Desktop\italia'
file = open(r'C:\Users\Razvi\Desktop\counters.csv', 'a')
now=datetime.now()
dd=str(now.day)
mm=str(now.month)
yyyy=str(now.year)
date=(dd + "/" + mm + "/" + yyyy)
file.write(date + '\n')
for filename in os.listdir(italia):
infilename = os.path.join(italia, filename)
for files in os.listdir(infilename):
if files.endswith("cnt.csv"):
result=os.path.join(infilename,"cnt.csv")
print(result)
infilename2 = os.path.join(infilename, files)
lines = 0
for line in open(infilename2):
lines += 1
file = open(r'C:\Users\Razvi\Desktop\counters.csv', 'a')
file.write(str(filename) +","+ str(lines)+"\n" )
file.close()
Thanks!

If you want to add the new entries as columns instead of rows further down, you'll have to read in the counters.csv file each time, append your columns, and rewrite it.
import os
from datetime import datetime
import csv
italia =r'C:\Users\Razvi\Desktop\italia'
counts = []
for filename in os.listdir(italia):
infilename = os.path.join(italia, filename)
for files in os.listdir(infilename):
if files.endswith("cnt.csv"):
result=os.path.join(infilename,"cnt.csv")
print(result)
infilename2 = os.path.join(infilename, files)
lines = 0
for line in open(infilename2):
lines += 1
counts.append((filename, lines)) # save the filename and count for later
if os.path.exists(r'C:\Users\Razvi\Desktop\counters.csv'):
with open(r'C:\Users\Razvi\Desktop\counters.csv', 'r') as csvfile:
counters = [row for row in csv.reader(csvfile)] # read the csv file
else:
counters = [[]]
now=datetime.now()
# Add the date and a blank cell to the first row
counters[0].append('{}/{}/{}'.format(now.day, now.month, now.year))
counters[0].append('')
for i, count in enumerate(counts):
if i + 1 == len(counters): # check if we need to add a row
counters.append(['']*(len(counters[0])-2))
while len(counters[i+1]) < len(counters[0]) - 2:
counters[i+1].append('') # ensure there are enough columns in the row we're on
counters[i+1].extend(count) # Add the count info as two new cells
with open(r'C:\Users\Razvi\Desktop\counters.csv', 'w') as csvfile:
writer = csv.writer(csvfile) # write the new csv file
for row in counters:
writer.writerow(row)
Another note, file is a builtin function in Python, so you shouldn't use that name as a variable, since unexpected stuff might happen further down in your code.

String Replace for Multiple Lines In A CSV

Below is a snippet from a csv file. The first column is the product number, 2 is the stock level, 3 is the target level, and 4 is the distance from target (target minus stock level.)
34512340,0,95,95
12395675,3,95,92
56756777,70,95,25
90673412,2,95,93
When the stock level gets to 5 or below, I want to have the stock levels updated from python when a user requests it.
I am currently using this piece of code which I have adapted from just updating one line in the CSV. It isn't working though. The first line is written back to the file as 34512340,0,95,95 and the rest of the file is deleted.
choice = input("\nTo update the stock levels of the above products, type 1. To cancel, enter anything else.")
if choice == '1':
with open('stockcontrol.csv',newline='') as f:
for line in f:
data = line.split(",")
productcode = int(data[0])
target = int(data[2])
stocklevel = int(data[1])
if stocklevel <= 5:
target = str(target)
import sys
import csv
data=[]
newval= target
newtlevel = "0"
f=open("stockcontrol.csv")
reader=csv.DictReader(f,fieldnames=['code','level', 'target', 'distancefromtarget'])
for line in reader:
line['level']= newval
line['distancefromtarget']= newtlevel
data.append('%s,%s,%s,%s'%(line['code'],line['level'],line['target'],line['distancefromtarget']))
f.close()
f=open("stockcontrol.csv","w")
f.write("\n".join(data))
f.close()
print("The stock levels were updated successfully")
else:
print("Goodbye")
Here is the code that I had changing one line in the CSV file and works:
with open('stockcontrol.csv',newline='') as f:
for line in f:
if code in line:
data = line.split(",")
target = (data[2])
newlevel = stocklevel - quantity
updatetarget = int(target) - int(newlevel)
stocklevel = str(stocklevel)
newlevel = str(newlevel)
updatetarget = str(updatetarget)
import sys
import csv
data=[]
code = code
newval= newlevel
newtlevel = updatetarget
f=open("stockcontrol.csv")
reader=csv.DictReader(f,fieldnames=['code','level', 'target', 'distancefromtarget'])
for line in reader:
if line['code'] == code:
line['level']= newval
line['distancefromtarget']= newtlevel
data.append('%s,%s,%s,%s'%(line['code'],line['level'],line['target'],line['distancefromtarget']))
f.close()
f=open("stockcontrol.csv","w")
f.write("\n".join(data))
f.close()
What can I change to make the code work? I basically want the program to loop through each line of the CSV file, and if the stock level (column 2) is equal to or less than 5, update the stock level to the target number in column 3, and then set the number in column 4 to zero.
Thanks,

The below code reads each line and checks the value of column 2. If it is less than or equal to 5, the value of column2 is changed to value of column3 and last column is changed to 0 else all the columns are left unchanged.
import sys
import csv
data=[]
f=open("stockcontrol.csv")
reader=csv.DictReader(f,fieldnames=['code','level','target','distancefromtarget'])
for line in reader:
if int(line['level']) <= 5:
line['level']= line['target']
line['distancefromtarget']= 0
data.append("%s,%s,%s,%s"%(line['code'],line['level'],line['target'],line['distancefromtarget']))
f.close()
f=open("stockcontrol.csv","w")
f.write("\n".join(data))
f.close()
Coming to issues in your code:
You are first reading the file without using the csv module and getting the values in each column by splitting the line. You are again using the DictReader method of csv module to read the values you already had.

delete rows by date and add file name column for multiple csv

I have multiple "," delimited csv files with recorded water pipe pressure sensor data, already sorted by date older-newer. For all original files, the first column always contains dates formated as YYYYMMDD. I have looked at similar discussion threads but couldn't find what I need.
Python script to add a new column to every csv file in the directory, where each row of the new column titled as "Pipe" would have a file name, omitting file extension string.
Have the option of specifying a cut off date as YYYYMMDD in order to delete rows in the orginal input file. For example, if some file has dates 20140101 to 20140630, I would like cut out rows of data if their date is < 20140401.
Have the option of either to overwrite the original files after having made these modifications or save each file to a different directory, with file names same as the originals.
Input: PipeRed.csv; Headers: Date,Pressure1,Pressure2,Temperature1,Temperature2 etc,
Output: PipeRed.csv; Headers: Pipe,Date,Pressure1,Pressure2,Temperature1, Temperature2,etc,
I have found some code and modified it a little, but it doesn't delete rows like was described above and adds the file name column last rather than 1st.
import csv
import sys
import glob
import re
for filename in glob.glob(sys.argv[1]):
#def process_file(filename):
# Read the contents of the file into a list of lines.
f = open(filename, 'r')
contents = f.readlines()
f.close()
# Use a CSV reader to parse the contents.
reader = csv.reader(contents)
# Open the output and create a CSV writer for it.
f = open(filename, 'wb')
writer = csv.writer(f)
# Process the header.
writer = csv.writer(f)
writer.writerow( ('Date','Pressure1','Pressure2','Pressure3','Pressure4','Pipe') )
header = reader.next()
header.append(filename.replace('.csv',""))
writer.writerow(header)
# Process each row of the body.
for row in reader:
row.append(filename.replace('.csv',""))
writer.writerow(row)
# Close the file and we're done.
f.close()

This function should be very close to what you want. I've tested it in both Python 2.7.9 and 3.4.2. The initial version I posted had some problems because — as I mention then — it was untested. I'm not sure if you're using Python 2 or 3, but this worked properly in either one.
Another change from the previous version is that the optional keyword date argument's name had been changed from cutoff_date to start_date to better reflect what it is. A cutoff date usually means the last date on which it is possible to do something—the opposite of the way you used it in your question. Also note that any date provided should a string, i.e. start_date='20140401', not as an integer.
One enhancement is that it will now create the output directory if one is specified but doesn't already exist.
import csv
import os
import sys
def open_csv(filename, mode='r'):
""" Open a csv file in proper mode depending on Python verion. """
return (open(filename, mode=mode+'b') if sys.version_info[0] == 2 else
open(filename, mode=mode, newline=''))
def process_file(filename, start_date=None, new_dir=None):
# Read the entire contents of the file into memory skipping rows before
# any start_date given (assuming row[0] is a date column).
with open_csv(filename, 'r') as f:
reader = csv.reader(f)
header = next(reader) # Save first row.
contents = [row for row in reader if start_date and row[0] >= start_date
or not start_date]
# Create different output file path if new_dir was specified.
basename = os.path.basename(filename) # Remove dir name from filename.
output_filename = os.path.join(new_dir, basename) if new_dir else filename
if new_dir and not os.path.isdir(new_dir): # Create directory if necessary.
os.makedirs(new_dir)
# Open the output file and create a CSV writer for it.
with open_csv(output_filename, 'w') as f:
writer = csv.writer(f)
# Add name of new column to header.
header = ['Pipe'] + header # Prepend new column name.
writer.writerow(header)
# Data for new column is the base filename without extension.
new_column = [os.path.splitext( os.path.split(basename)[1] )[0]]
# Process each row of the body by prepending data for new column to it.
writer.writerows((new_column+row for row in contents))