I am trying to capture data from an oscilloscope using a python script. The script saves it as in csv format. I need to add few lines of text describing the data at the beginning.
I looked at existing threads to see if there was a possible solution. I just started learning Python. I am using code that came with the instrument.
This is part of the script that saves the data as csv.
NewD = (np.insert(Wav_Data, 0, DataTime, axis = 0)).T
filename = BASE_DIRECTORY + BASE_FILE_NAME + ".csv"
now = time.time() # Only to show how long it takes to save
with open(filename, 'w') as filehandle:
np.savetxt(filename, NewD, delimiter = ',', header = column_titles)
I tried to use the section below from another code but am not sure how to append this to the csv file.
with open("notes.txt") as f:
NOTES = f.readlines()
NOTES = "".join(NOTES)
It is unable to find notes.txt which is located in the same directory as the script.
Eager to hear your feedback. Thanks in advance.
Updated to:
# Save data
NewD = (np.insert(Wav_Data, 0, DataTime, axis = 0)).T
filename = BASE_DIRECTORY + BASE_FILE_NAME + ".csv"
with open("notes.txt") as f:
NOTES = f.readlines()
NOTES = "".join(NOTES)
with open(filename, "a") as fh:
fh.write(NOTES)
now = time.time() # Only to show how long it takes to save
with open(filename, 'w') as filehandle:
np.savetxt(filename, NewD, delimiter = ',', header = column_titles)
Just open the file for appending or write
If you want to write CSV first then notes:
with open("notes.txt") as f:
NOTES = f.readlines()
NOTES = "".join(NOTES)
with open(filename, "w") as fh:
fh.write(NOTES)
# this time we give np the opened filehandle, not the filename
np.savetxt(fh, NewD, delimiter = ',', header = column_titles)
I can print results to terminal but unable to write to csv file
full file:https://1drv.ms/u/s!AizscpxS0QM4hJo5SnYOHAcjng-jww
datapath = '1.json'
data = json.load(open(datapath))
with open('1.json') as file:
data = json.load(file)
for element in data['RoleDetailList']:
if 'RoleName' in element.keys():
s = element['RoleName']
#print s
with open('roleassign.csv', 'wt') as file:
file.write('Role,Policy\n')
for policy in element['AttachedManagedPolicies']:
c = s + ',' + policy['PolicyName']
#print c
file.write(c + '\n')
In csv file i get only headers, when uncomment print c i see lines are printed into terminal (output)
some of the lines from output:
ADFS-amtest-ro,pol-amtest-ro
adfs-host-role,pol-amtest-ro
aws-elasticbeanstalk-ec2-role,AWSElasticBeanstalkWebTier
Please try code below:
with open('output.json') as file:
data = json.load(file)
with open('roleassign.csv', 'wt') as file:
file.write('Role,Policy\n')
for element in data['RoleDetailList']:
if 'RoleName' in element.keys():
s= element['RoleName']
for policy in element['AttachedManagedPolicies']:
c = s + ',' + policy['PolicyName']
file.write(c + '\n')
Your File writer is being opened in the loop and every time it was overwriting the file with only the headers. Simply moved it out.
You should use csv.writer from the csv built-in module
in your example:
with open('roleassign.csv', 'w') as csv_file:
writer = csv.writer(csv_file, delimiter=',')
writer.writerow(['Role','Policy'])
for policy in element['AttachedManagedPolicies']:
c = [s, policy['PolicyName']]
writer.writerow(c)
Additionally, to incorporate Rehan's answer, the loop that is updating you s variable shouldn't be out there. The code blow should work for you:
with open('roleassign.csv', 'w') as csv_file:
writer = csv.writer(csv_file, delimiter=',')
writer.writerow(['Role','Policy'])
for element in data['RoleDetailList']:
if 'RoleName' in element.keys():
s = element['RoleName']
for policy in element['AttachedManagedPolicies']:
c = [s, policy['PolicyName']]
writer.writerow(c)
I am transforming JSON like data to CSV and having a few issues.
The code is here:
import json
import csv
def parse_file(inputed_file):
with open(input_file, 'r') as inputed_file:
content = inputed_file.readlines()
split_file = open('test.csv', 'w')
for line in content:
lines = line.split('\t')
data = json.loads(lines[0])
writer = csv.DictWriter(split_file, fieldnames = ["title", "firstname"], delimiter = ',')
writer.writeheader()
The problem is this is adding a header on each row for the data, I want to only have the header displayed once. Then add this for the data to go below the headers:
writer.writerow(data)
I have looked at this and tried it but failed: How can I convert JSON to CSV?.
Create the DictWriter outside the loop, and just call writer.writeheader() there. Then call writer.writerow() inside the loop.
def parse_file(inputed_file):
with open(input_file, 'r') as inputed_file:
content = inputed_file.readlines()
split_file = open('test.csv', 'w')
writer = csv.DictWriter(split_file, fieldnames = ["title", "firstname"], delimiter = ',')
writer.writeheader()
for line in content:
lines = line.split('\t')
data = json.loads(lines[0])
writer.writerow(data)
Here's my code, really simple stuff...
import csv
import json
csvfile = open('file.csv', 'r')
jsonfile = open('file.json', 'w')
fieldnames = ("FirstName","LastName","IDNumber","Message")
reader = csv.DictReader( csvfile, fieldnames)
out = json.dumps( [ row for row in reader ] )
jsonfile.write(out)
Declare some field names, the reader uses CSV to read the file, and the filed names to dump the file to a JSON format. Here's the problem...
Each record in the CSV file is on a different row. I want the JSON output to be the same way. The problem is it dumps it all on one giant, long line.
I've tried using something like for line in csvfile: and then running my code below that with reader = csv.DictReader( line, fieldnames) which loops through each line, but it does the entire file on one line, then loops through the entire file on another line... continues until it runs out of lines.
Any suggestions for correcting this?
Edit: To clarify, currently I have: (every record on line 1)
[{"FirstName":"John","LastName":"Doe","IDNumber":"123","Message":"None"},{"FirstName":"George","LastName":"Washington","IDNumber":"001","Message":"Something"}]
What I'm looking for: (2 records on 2 lines)
{"FirstName":"John","LastName":"Doe","IDNumber":"123","Message":"None"}
{"FirstName":"George","LastName":"Washington","IDNumber":"001","Message":"Something"}
Not each individual field indented/on a separate line, but each record on it's own line.
Some sample input.
"John","Doe","001","Message1"
"George","Washington","002","Message2"
The problem with your desired output is that it is not valid json document,; it's a stream of json documents!
That's okay, if its what you need, but that means that for each document you want in your output, you'll have to call json.dumps.
Since the newline you want separating your documents is not contained in those documents, you're on the hook for supplying it yourself. So we just need to pull the loop out of the call to json.dump and interpose newlines for each document written.
import csv
import json
csvfile = open('file.csv', 'r')
jsonfile = open('file.json', 'w')
fieldnames = ("FirstName","LastName","IDNumber","Message")
reader = csv.DictReader( csvfile, fieldnames)
for row in reader:
json.dump(row, jsonfile)
jsonfile.write('\n')
You can use Pandas DataFrame to achieve this, with the following Example:
import pandas as pd
csv_file = pd.DataFrame(pd.read_csv("path/to/file.csv", sep = ",", header = 0, index_col = False))
csv_file.to_json("/path/to/new/file.json", orient = "records", date_format = "epoch", double_precision = 10, force_ascii = True, date_unit = "ms", default_handler = None)
import csv
import json
file = 'csv_file_name.csv'
json_file = 'output_file_name.json'
#Read CSV File
def read_CSV(file, json_file):
csv_rows = []
with open(file) as csvfile:
reader = csv.DictReader(csvfile)
field = reader.fieldnames
for row in reader:
csv_rows.extend([{field[i]:row[field[i]] for i in range(len(field))}])
convert_write_json(csv_rows, json_file)
#Convert csv data into json
def convert_write_json(data, json_file):
with open(json_file, "w") as f:
f.write(json.dumps(data, sort_keys=False, indent=4, separators=(',', ': '))) #for pretty
f.write(json.dumps(data))
read_CSV(file,json_file)
Documentation of json.dumps()
I took #SingleNegationElimination's response and simplified it into a three-liner that can be used in a pipeline:
import csv
import json
import sys
for row in csv.DictReader(sys.stdin):
json.dump(row, sys.stdout)
sys.stdout.write('\n')
You can try this
import csvmapper
# how does the object look
mapper = csvmapper.DictMapper([
[
{ 'name' : 'FirstName'},
{ 'name' : 'LastName' },
{ 'name' : 'IDNumber', 'type':'int' },
{ 'name' : 'Messages' }
]
])
# parser instance
parser = csvmapper.CSVParser('sample.csv', mapper)
# conversion service
converter = csvmapper.JSONConverter(parser)
print converter.doConvert(pretty=True)
Edit:
Simpler approach
import csvmapper
fields = ('FirstName', 'LastName', 'IDNumber', 'Messages')
parser = CSVParser('sample.csv', csvmapper.FieldMapper(fields))
converter = csvmapper.JSONConverter(parser)
print converter.doConvert(pretty=True)
I see this is old but I needed the code from SingleNegationElimination however I had issue with the data containing non utf-8 characters. These appeared in fields I was not overly concerned with so I chose to ignore them. However that took some effort. I am new to python so with some trial and error I got it to work. The code is a copy of SingleNegationElimination with the extra handling of utf-8. I tried to do it with https://docs.python.org/2.7/library/csv.html but in the end gave up. The below code worked.
import csv, json
csvfile = open('file.csv', 'r')
jsonfile = open('file.json', 'w')
fieldnames = ("Scope","Comment","OOS Code","In RMF","Code","Status","Name","Sub Code","CAT","LOB","Description","Owner","Manager","Platform Owner")
reader = csv.DictReader(csvfile , fieldnames)
code = ''
for row in reader:
try:
print('+' + row['Code'])
for key in row:
row[key] = row[key].decode('utf-8', 'ignore').encode('utf-8')
json.dump(row, jsonfile)
jsonfile.write('\n')
except:
print('-' + row['Code'])
raise
Add the indent parameter to json.dumps
data = {'this': ['has', 'some', 'things'],
'in': {'it': 'with', 'some': 'more'}}
print(json.dumps(data, indent=4))
Also note that, you can simply use json.dump with the open jsonfile:
json.dump(data, jsonfile)
Use pandas and the json library:
import pandas as pd
import json
filepath = "inputfile.csv"
output_path = "outputfile.json"
df = pd.read_csv(filepath)
# Create a multiline json
json_list = json.loads(df.to_json(orient = "records"))
with open(output_path, 'w') as f:
for item in json_list:
f.write("%s\n" % item)
How about using Pandas to read the csv file into a DataFrame (pd.read_csv), then manipulating the columns if you want (dropping them or updating values) and finally converting the DataFrame back to JSON (pd.DataFrame.to_json).
Note: I haven't checked how efficient this will be but this is definitely one of the easiest ways to manipulate and convert a large csv to json.
As slight improvement to #MONTYHS answer, iterating through a tup of fieldnames:
import csv
import json
csvfilename = 'filename.csv'
jsonfilename = csvfilename.split('.')[0] + '.json'
csvfile = open(csvfilename, 'r')
jsonfile = open(jsonfilename, 'w')
reader = csv.DictReader(csvfile)
fieldnames = ('FirstName', 'LastName', 'IDNumber', 'Message')
output = []
for each in reader:
row = {}
for field in fieldnames:
row[field] = each[field]
output.append(row)
json.dump(output, jsonfile, indent=2, sort_keys=True)
def read():
noOfElem = 200 # no of data you want to import
csv_file_name = "hashtag_donaldtrump.csv" # csv file name
json_file_name = "hashtag_donaldtrump.json" # json file name
with open(csv_file_name, mode='r') as csv_file:
csv_reader = csv.DictReader(csv_file)
with open(json_file_name, 'w') as json_file:
i = 0
json_file.write("[")
for row in csv_reader:
i = i + 1
if i == noOfElem:
json_file.write("]")
return
json_file.write(json.dumps(row))
if i != noOfElem - 1:
json_file.write(",")
Change the above three parameter, everything will be done.
import csv
import json
csvfile = csv.DictReader('filename.csv', 'r'))
output =[]
for each in csvfile:
row ={}
row['FirstName'] = each['FirstName']
row['LastName'] = each['LastName']
row['IDNumber'] = each ['IDNumber']
row['Message'] = each['Message']
output.append(row)
json.dump(output,open('filename.json','w'),indent=4,sort_keys=False)
I have a list like this(python 3)
my_list = [["xxx","moon",150],["wordq","pop",3]]
and i save it on a csv using this code
import csv
myfile = open("pppp.csv", 'wb')
with open("pppp.csv", "w", newline='') as myfile:
wr = csv.writer(myfile, quoting=csv.QUOTE_NONE)
wr.writerows(list_of_DVDsuppliers)
now i need to export this csv in to my program as a list and change the data .
please help me ?
Just convert the data you get from reader() to a list:
data = csv.reader(open('example.csv','r'))
data = list(data)
print data
Unless you have a reason why you are using newline='', you can skip that and below code works with python 2.7,
import csv
my_list = [["xxx","moon",150],["wordq","pop",3]]
myfile = open("pppp.csv", 'wb')
with open("pppp.csv", "w") as myfile:
wr = csv.writer(myfile, quoting=csv.QUOTE_NONE)
wr.writerows(my_list)
data = csv.reader(open('pppp.csv','r'))
for row in data:
print row