I have been trying to convert values with commas in a pandas dataframe to floats with little success. I also tried .replace(",","") but it doesn't work? How can I go about changing the Close_y column to float and the Date column to date values so that I can plot them? Any help would be appreciated.
Convert 'Date' using to_datetime for the other use str.replace(',','.') and then cast the type:
df['Date'] = pd.to_datetime(df['Date'], format='%m/%d/%Y')
df['Close_y'] = df['Close_y'].str.replace(',','.').astype(float)
replace looks for exact matches, what you're trying to do is replace any match in the string
pandas.read_clipboard implements the same kwargs as pandas.read_table in which there are options for the thousands and parse_dates kwarg.s
Try loading your data with:
df = pd.read_clipboard(thousands=',', parse_dates=[0])
Assuming that the Dates column is in the 0 index. If you have a large amount of data you may also try using the infer_datetime_format kwarg to speed things up.
Related
I have a column in my DataFrame with values like '2022-06-03T00:00:00.000Z' and I want to convert these (in place) to pd.Timestamp. I see many answers he on how to convert to np.datetime64 and on how do convert arbitrary columns of DataFrames, but can't figure out how to apply these to covering to pd.Timestamp.
Use from pd.to_datetime method
I think this solve your problem
Just you need to active utc argument in your method
import pandas as pd
lst = {'a':['Geeks', 'For'],'b':['2022-06-03T00:00:00.000Z','2024-03-03T00:00:00.000Z']}
df = pd.DataFrame(lst)
df['b']=pd.to_datetime(df['b'],utc=True)
type(df['b'][0])
pandas._libs.tslibs.timestamps.Timestamp
I have a dataset in CSV which first column are dates (not datetimes, just dates).
The CSV is like this:
date,text
2005-01-01,"FOO-BAR-1"
2005-01-02,"FOO-BAR-2"
If I do this:
df = pd.read_csv('mycsv.csv')
I get:
print(df.dtypes)
date object
text object
dtype: object
How can I get column date by datetime.date?
Use:
df = pd.read_csv('mycsv.csv', parse_dates=[0])
This way the initial column will be of native pandasonic datetime type,
which is used in Pandas much more often than pythonic datetime.date.
It is a more natural approach than conversion of the column in question
after you read the DataFrame.
You can use pd.to_datetime function available in pandas.
For example in a dataset about scores of a cricket match. I can convert the Matchdate column to datatime object by applying pd.to_datetime function based on the data time format given in the data. ( Refer https://www.w3schools.com/python/python_datetime.asp to assign commands based on your data time formating )
cricket["MatchDate"]=pd.to_datetime(cricket["MatchDate"], format= "%m-%d-%Y")
I have a column in a data frame with dates in the format of “1/4/2021 0:00”. And I would like to get rid of everything after the first space, including the first space so that way it becomes “1/4/2021”.
How can I do that in Python? Also, does the column already have to be a specific data type in order to complete this task?
If you are using pandas you can try the following, assuming the entire column is following a similar datetime format.
Your dataframe is called df, and your column of dates is date.
df['date'] = df['date'].dt.date
or
df['date'] = pd.to_datetime(df['date'].dt.date)
or
df['date'] = df['date'].dt.normalize()
Depending on what you want the format of your date column to be.
Try this:
df['date'] = df['date'].apply(lambda x: x.split(' ')[0] if isinstance(x, str) else x)
Note that this code only works if your column in data frame has type string.
In order to check the data type, run: df.dtypes.
I am trying to change the format of the date in a pandas dataframe.
If I check the date in the beginning, I have:
df['Date'][0]
Out[158]: '01/02/2008'
Then, I use:
df['Date'] = pd.to_datetime(df['Date']).dt.date
To change the format to
df['Date'][0]
Out[157]: datetime.date(2008, 1, 2)
However, this takes a veeeery long time, since my dataframe has millions of rows.
All I want to do is change the date format from MM-DD-YYYY to YYYY-MM-DD.
How can I do it in a faster way?
You should first collapse by Date using the groupby method to reduce the dimensionality of the problem.
Then you parse the dates into the new format and merge the results back into the original DataFrame.
This requires some time because of the merging, but it takes advantage from the fact that many dates are repeated a large number of times. You want to convert each date only once!
You can use the following code:
date_parser = lambda x: pd.datetime.strptime(str(x), '%m/%d/%Y')
df['date_index'] = df['Date']
dates = df.groupby(['date_index']).first()['Date'].apply(date_parser)
df = df.set_index([ 'date_index' ])
df['New Date'] = dates
df = df.reset_index()
df.head()
In my case, the execution time for a DataFrame with 3 million lines reduced from 30 seconds to about 1.5 seconds.
I'm not sure if this will help with the performance issue, as I haven't tested with a dataset of your size, but at least in theory, this should help. Pandas has a built in parameter you can use to specify that it should load a column as a date or datetime field. See the parse_dates parameter in the pandas documentation.
Simply pass in a list of columns that you want to be parsed as a date and pandas will convert the columns for you when creating the DataFrame. Then, you won't have to worry about looping back through the dataframe and attempting the conversion after.
import pandas as pd
df = pd.read_csv('test.csv', parse_dates=[0,2])
The above example would try to parse the 1st and 3rd (zero-based) columns as dates.
The type of each resulting column value will be a pandas timestamp and you can then use pandas to print this out however you'd like when working with the dataframe.
Following a lead at #pygo's comment, I found that my mistake was to try to read the data as
df['Date'] = pd.to_datetime(df['Date']).dt.date
This would be, as this answer explains:
This is because pandas falls back to dateutil.parser.parse for parsing the strings when it has a non-default format or when no format string is supplied (this is much more flexible, but also slower).
As you have shown above, you can improve the performance by supplying a format string to to_datetime. Or another option is to use infer_datetime_format=True
When using any of the date parsers from the answers above, we go into the for loop. Also, when specifying the format we want (instead of the format we have) in the pd.to_datetime, we also go into the for loop.
Hence, instead of doing
df['Date'] = pd.to_datetime(df['Date'],format='%Y-%m-%d')
or
df['Date'] = pd.to_datetime(df['Date']).dt.date
we should do
df['Date'] = pd.to_datetime(df['Date'],format='%m/%d/%Y').dt.date
By supplying the current format of the data, it is read really fast into datetime format. Then, using .dt.date, it is fast to change it to the new format without the parser.
Thank you to everyone who helped!
I have a CSV file that looks like this:
Date,Close
16-Mar-17,848.78
15-Mar-17,847.2
Whenever I try to load it in and set the date as the index by doing:
df = pd.read_csv("new_data.csv")
df.set_index("Date")
I getValueError: could not convert string to float: '18-Mar-16'. Why is this happening? I thought you could set a date even if it was a string. I am a novice to pandas so it is most likely a simple misunderstanding.
EDIT:
I was reading the error on the wrong line, here is the chunck of code that throws the error.
df = pd.read_csv("new_data.csv")
Close = df.sort_index(ascending=True)
plt.plot(Close)
plt.gca().invert_xaxis()
plt.show()
Now, you need to convert string Date to datetime:
Close['Date'] = pd.to_datetime(Close['Date'])
I had similar issues, all related with taking good care of the date data.
good practice is while loading data, using pandas functionality to load the date info.
df = pd.read_csv("new_data.csv", parse_dates=[0], infer_datetime_format = True)
where column[0] is where date column located.
and then pandas will do the "magic" and handle xticks of the plot nicely.