I am trying to import a csv file as an array in python using the ""numpy.loadtxt"" method. It keeps returning ""ValueError: could not convert string to float: ''"" despite there not being any blank cells in the csv file. Here is my code
import csv
import torch
import numpy as np
import pandas as pd
array = np.loadtxt("HIP Only 2.csv", dtype=np.float32, delimiter=",", skiprows=1)
It seems that there are some Non-numerics in the cells, which may be related to specified strings or errors created due to unsuccessful formula in some cells e.g. #DIV/0! in excel files which is appeared when corresponding cells have not filled or numbers divided by zero. numpy.loadtxt is for using when no data is missed. If getting array is the main goal, not using numpy.loadtxt, numpy.genfromtxt is more flexible and could be used instead e.g.:
array = np.genfromtxt("HIP Only 2.csv", dtype=np.float32, delimiter=",", skip_header=1)
Hope it e helpful.
Related
So I've been tasked with creating a suitable 2D array to contain all of the data from a csv with data on rainfall from the whole year. In the csv file, the rows represent the weeks of the year and the columns represent the day of the week.
I'm able to display the date I want using the following code.
import csv
data = list(csv.reader(open("rainfall.csv")))
print(data[1][2])
My issue is I'm not sure how to store this data in a 2D array.
I'm not sure how to go about doing this. Help would be appreciated, thanks!
You could use numpy for that. It seems to me, that you have created a list of lists in data. With that you can directly create a 2D numpy-array:
import numpy as np
2d_data = np.array(data)
Or you could even try to directly read the file with numpy:
import numpy as np
# Use the appropriate delimiter here
2d_data = np.genfromtxt("rainfall.csv", delimiter=",")
With pandas:
import pandas as pd
# Use the appropriate delimiter here
2d_data = pd.read_csv("rainfall.csv")
I'm trying to import some values from a csv-file to a numpy array in python.
So far I've read the CSV-file with pandas but I can't succeed with creating a numpy array with the values from the csv columns.
Just found the answer. Just had to use DataFrame.values
I'm probably trying to reinvent the wheel here, but numpy has a fromfile() function that can read - I imagine - CSV files.
It appears to be incredibly fast - even compared to Pandas read_csv(), but I'm unclear on how it works.
Here's some test code:
import pandas as pd
import numpy as np
# Create the file here, two columns, one million rows of random numbers.
filename = 'my_file.csv'
df = pd.DataFrame({'a':np.random.randint(100,10000,1000000), 'b':np.random.randint(100,10000,1000000)})
df.to_csv(filename, index = False)
# Now read the file into memory.
arr = np.fromfile(filename)
print len(arr)
I included the len() at the end there to make sure it wasn't reading just a single line. But curiously, the length for me (will vary based on your random number generation) was 1,352,244. Huh?
The docs show an optional sep parameter. But when that is used:
arr = np.fromfile(filename, sep = ',')
...we get a length of 0?!
Ideally I'd be able to load a 2D array of arrays from this CSV file, but I'd settle for a single array from this CSV.
What am I missing here?
numpy.fromfile is not made to read .csv files, instead, it is made for reading data written with the numpy.ndarray.tofile method.
From the docs:
A highly efficient way of reading binary data with a known data-type, as well as parsing simply formatted text files. Data written using the tofile method can be read using this function.
By using it without a sep parameter, numpy assumes you are reading a binary file, hence the different lengths. When you specify a separator, I guess the function just breaks.
To read a .csv file using numpy, I think you can use numpy.genfromtext or numpy.loadtxt (from this question).
I'm using this code:
import arcpy
import numpy as np
f = open("F:\INTRO_PY\LAB_7\lab_7.csv","w")
array = np.random.rand(1000,1000)
f.write(array)
f.close
in order to create a 1000x1000 random array in arcpy.
This is what I get when I open the csv:
CSV
I have absolutely no idea why it's doing this, and I'm at my wit's end. Any advice would be really, really appreciated!
In order to save it to CSV, you need to can use numpy's numpy.savetxt [numpy-doc]:
np.savetxt(
r"F:\INTRO_PY\LAB_7\lab_7.csv",
np.random.rand(1000,1000),
delimiter=','
)
The `delimeter thus specifies what one uses to split the different values.
Note that you can only save 1D arrays or 2D arrays to a text file.
I think you are trying to store a numpy in a file, you should convert it to a string first.
Something like the following:
f = open("test.csv","w")
array = np.random.rand(1000,1000)
f.write(str(array))
f.close
I try get mean from csv line. I get data from csv in string list, further i convert it to array with numpy. Its work perfect when i try plot some graphics.
But when i calculate mean i get some errors with my data.
If i use NumPy i get:
TypeError: cannot perform reduce with flexible type
If i use statistics library i get:
TypeError: can't convert type 'string' to numerator/denominator
If i check my array with comand 'type' on iPython i see that it numpy.ndarray type.
Whats wrong with my array? Can you explain, why convert numpy.asarray for matplotlib work perfect, but get wrong type for different operation.
import csv
import numpy as np
import statistics as stat
life_exp=[]
with open('country.csv') as csvfile:
datareader = csv.reader(csvfile)
for row in datareader:
if datareader.line_num!=1:
life_exp.append(row[1])
array_life_exp = np.asarray(life_exp)
print(stat.mean(array_life_exp))
print(np.mean(array_life_exp))
Try this:
from pandas import read_csv
data = read_csv('country.csv')
print(data.iloc[:,1].mean())
This code will convert your csv to pandas dataframe with automatic type conversion and print mean of the second column.