Maybe this question is too naive, but is giving me a hard time!. I want to write 2 float values and a list of int to a row in csv file in a loop. The file may or may not exist before an attempt is made to write in it. In case it does not, a new file should be created. This is what I am doing:
f = open('stat.csv','a')
try:
writer=csv.writer(f,delimiter=' ',quoting=csv.QUOTE_MINIMAL)
writer.writerow((some_float1,some_float2,alist))
finally:
f.close()
where alist = [2,3,4,5]. I am getting the following output:
some_float1 some_float2 "[2,3,4,5]"
What I want is this:
some_float1 some_float2 2 3 4 5
i.e. I would like to get rid of the "", the square brackets and make the delimiter consistent throughout. Any suggestions ?
How about:
writer.writerow([some_float1, some_float2] + alist)
Related
i am trying to add some file operation capabilities to a example program, i am strugling with the reading from the file. here is the code that is modified.
def read(fn):
fileout=open(f"{fn}","a+")
fileout.seek(0,0)
s=fileout.readlines()
if s==[]:
print("the file specified does not appear to exists or is empty. if the file does not exist, it will be created")
else:
last=s[-1]
print(last)
print(type(last))
convert(last)
def find(last):
tup=last.partition(".")
fi=tup[0:1]
return fi[0]
def convert(last):
tup=last.partition(".")
part=tup[2:]
print(part)
part=part[0]
print(part)
part=part.split("\n")
print(part)
part=part[0]
print(part)
print(type(part))
#__main__
file(fn)
the write functionality writes in the form of
(fileindex number).[(planned campaign)][(conducted campaign)]
example:- some random data writen to the file by the program(first two number are dates)
0.['12hell']['12hh']
1.['12hell']['12hh']
2.['121341']['132324']
but i am strugling to write the read function, i don't understand how i could convert the data back.
with the current read function i get back
['121341']['132324']
as a string type, i have brainstormed many ideas but could not figureout how to convert string to list(they need to be 2 separate lists)
edit: the flaw as actually in the format that i was writing in, i added a , between the two lists and used eval as suggested in an answer, thanks
Insert a ',' inbetween the brackets, then use eval. This will return a tuple of lists.
strLists = "['121341']['132324']['abcdf']"
strLists = strLists.replace('][', '],[')
evalLists = eval(strLists)
for each in evalLists:
print(each)
Output:
['121341']
['132324']
['abcdf']
I have read in data from a basic txt file. The data is time and date in this form "DD/HHMM" (meteorological date and time data). I have read this data into a list: time[]. It prints out as you would imagine like so: ['15/1056', '15/0956', '15/0856', .........]. Is there a way to alter the list so that it ends up just having the time, basically removing the date and the forward slash, like so: ['1056', '0956', '0856',.........]? I have already tried list.split but thats not how that works I don't think. Thanks.
I'm still learning myself and I haven't touched python in sometime, BUT, my solution if you really need one:
myList = ['15/1056', '15/0956', '15/0856']
newList = []
for x in mylist:
newList.append(x.split("/")[1])
# splits at '/'
# returns ["15", "1056"]
# then appends w/e is at index 1
print(newList) # for verification
I have some parameters and functions that I need to write to a file, but the functions and x-values have different lengths, i.e. domains and codomains, when compared to each other.
My current formatting, assuming e.g. two parameters A and B and two functions f1 and f2 is [A,B,x,f1,f2] where then x, f1, f2 are itself again lists or numpy arrays.
So my imagined data.txt could look like this:
[0, 0, [0,1,2,3], [1,2,3,4], [4,5,6,7]]
[0, 1, [0,1,2,3,4,5,6], [1,2,3,4,5,6,7], [4,5,6,7,8,9,10]]
[1, 10, [2,3,4,5,6], [1,2,3,4,5], [4,5,6,7,8]]
Then I could read in line by line, getting the parameters A and B and plot the functions f1 and f2, given the right x-values.
To write to a file I use the following code, which almost gives me what I described above.
OutSummary=[A,B,x,f1,f2]
Out=str(OutSummary).replace('\n', '')
f=open(filename,'a')
np.savetxt(f, [Out],fmt='%s')
f.close()
Currently, this produces entries like [0, 1, array([ 8. , 8.29229229, 8.58458458, ....
The issue is now that reading in does not work, due to the different lengths of the x-values and function arrays, i.e.
PrevEntris = np.genfromtxt(filename,dtype=str)
(with dtype=str, or dtype=None, or no dtype given) gives me e.g.
ValueError: Some errors were detected ! Line #7 (got 555 columns instead of 1105),
i.e. the x-values contained in the 7th line only had 555 entries, while the previous 6 had 1105.
I see that this is not good code, and I am saving arrays as strings, but I did not manage to find a better way. I'd be really interested to know if there is some advanced numpy way of handling this, or maybe even using a SQL database kind of thing rather than one .txt file? I spent the last few hours trying to make it work using json, but was not successful, yet (1st time user).
Thanks
You can use the builtin json module, since Python lists and JSON have the same syntax. Here is some example code if you want to store each of the lists in a bigger list:
import json
f=open("data.txt")
lines=f.read().splitlines()
data=[json.loads(line) for line in lines]
f.close() #remember to close your files
Edit: I realized I should have used list comprehensions instead so I changed my response. Still works the same way, but is neater
The answer form Icestrike411 works very well for me, especially for my requested formatting style. Additionally, I solved it another way in the meantime also using json, when slightly altering the format.
One data block could look like
new_data = {
"A": 1,
"B": 0.05,
"X": [0,1,2,3,4]
}
and then I append it to the outfile with the following function, running append('some.txt', new_data):
def append(filename, new_entry):
try:
with open(filename, "r") as filea:
content = json.load(filea)
except:
content=[]
#print("Likely empty file.")
content.append(new_entry)
out=json.dumps(content).replace("},", "},\n") #new line for each entry in file
with open(filename, "w") as fileb:
fileb.write(out)
and then reading it with
with open(filename, "r") as file:
contentr = json.load(file)
So we are making a Python program in class in which we extract data about road accidents.
The file we are extracting from is a table with each line giving information about the people involved in a given accident
usagers_2016 = open('usagers_2016.csv','w',encoding='utf8', errors='ignore', newline="\n")
usagers_2016.write("Num_Acc;place;catu;grav;sexe;trajet;secu;locp;actp;etatp;an_nais;num_veh\n
201600000001;1;1;1;2;0;11;0;0;0;1983;B02\n
201600000001;1;1;3;1;9;21;0;0;0;2001;A01\n
201600000002;1;1;3;1;5;11;0;0;0;1960;A01\n
201600000002;2;2;3;1;0;11;0;0;0;2000;A01\n
201600000002;3;2;3;2;0;11;0;0;0;1962;A01\n
201600000003;1;1;1;1;1;11;0;0;0;1997;A01\n")
next(usagers_2016)
dict_acc = {}
for ligne in usagers_2016.readlines():
ligne = ligne[:-2].split(";")
I chose to extract the info in a dictionnary, where the accident is the key, the value of each key is a list, whose first element is a list of the people involved, each person being represented by a list including their gender and birth year
if ligne[0] not in dict_acc.keys():
dict_acc[ligne[0]] = [[],0,0,0,0]
dict_acc[ligne[0]][0].append([ligne[4],ligne[10]])
usagers_2016.close()
for accident in dict_acc:
accident[1] = len(accident[0]) # TypeError: 'str' object does not support item assignment
My problem is the following: I want to add, as the second element of the main list (the value of the key), the number of people involved in each accident (which is the len() of the first element (list) of the list). However it was revealed during the running of the code that the first 0 (line 2 of the previous code extract) is considered a str and can't receive item assignment whatsoever. The problem is that it was supposed to be an int!!!! I thought that expliciting the int type as following dict_acc[ligne[0]] = [[],int(0),int(0),int(0),int(0)] would correct it, but no, my 0s are still considered strings. Would you know why?
OK so the problem was that I was calling accident[1] instead of dict_acc[accident][1]
solution is
for accident in dict_acc:
dict_acc[accident][1] = len(dict_acc[accident][0])
I thank #MisterMiyagi
The reason is you are reading the file with open you have an object of type _io.TextIOWrapper and each line is a string, which you are later splitting based on the delimeter ';'.
The line next(usagers_2016) makes me think you are dropping the first row with headers.
So what you could do instead is to open this csv as a Dataframe in Pandas like this:
import pandas as pd
df = pd.read_csv ('usagers_2016.csv', sep=';')
# To remove the tailing \n
df.columns=df.columns.str.replace(r'\n','')
df.replace(r'\\n','', regex=True)
# Now to calculate the number of people involve in each accident you can
df.groupby(['Num_Acc']).size()
This is and example of what my csv file looks like with 6 columns:
0.0028,0.008,0.0014,0.008,0.0014,0.008,
I want to create 6 variables to use later in my program using these numbers as the values; however, the number of columns WILL vary depending on exactly which csv file I open.
If I were to do this manually and the number of columns was always 6, I would just create the variables like this:
thickness_0 = (row[0])
thickness_1 = (row[1])
thickness_2 = (row[2])
thickness_3 = (row[3])
thickness_4 = (row[4])
thickness_5 = (row[5])
Is there a way to create these variables with a for loop so that it is not necessary to know the number of columns? Meaning it will create the same number of variables as there are columns?
There are ways to do what you want, but this is considered very bad practice, you better never mix your source code with data.
If your code depends on dynamic data from "outer world", use dictionary (or list in your case) to access your data programatically.
You can use a dictionary
mydict = {}
with open('StackupThick.csv', 'r') as infile:
reader = csv.reader(infile, delimiter=',')
for idx, row in enumerate(reader):
key = "thickness_" + str(idx)
mydict[key] = row
Call your values like this:
print(mydict['thickness_3'])
From your question, I understand that your csv files have only one line with the comma separated values. If so, and if you are not fine with dictionares (as in #Mike C. answers) you can use globals() to add variables to the global namespace, which is a dict.
import csv
with open("yourfile.csv", "r", newline='') as yourfile:
rd = csv.reader(yourfile, delimiter=',')
row = next(rd)
for i, j in enumerate(row):
globals()['thickness_' + str(i)] = float(j)
Now you have whatever number of new variables called thickness_i where i is a number starting from 0.
Please be sure that all the values are in the first line of the csv file, as this code will ignore any lines beyond the first.