Develop Reference

How to convert a date with a specific format without loosing the date type in Pandas dataframe

I have a list of columns I need to convert to date without loosing the date type format. To convert to date one could use df1[col] = df1[col].astype('datetime64[ns]') which gives an actual date type but if I want it to be of type '%m/%d/%Y' it is often suggested on here that one does this: df1[col] = df1[col].dt.strftime('%m/%d/%Y') but in excel this is now recognized as a string type and not a date type.
I have gone through many posts and searched online to find a solution to this problem and there must be one.
Here is my code I have that is giving the incorrect types that I do not want:
convert_date_cols = ['CutoffDate', 'ModEffectiveDate', 'IOExpirationDate', 'FirstRateAdjustmentDate', 'GF_Service_Xfer_Date',
'BPODate', 'FirstPaymentDate', 'MaturityDate', 'NextRateAdjustmentDate', 'NextPaymentAdjustmentDate',
'ModFirstPayDate', 'ModMaturityDate', 'REO Date', 'FCDate', 'BK Date', 'Fico Score Date']
for i, col in enumerate(convert_date_cols):
df1[col] = df1[col].astype('datetime64[ns]')
df1[col] = df1[col].dt.strftime('%m/%d/%Y')

You're trying to solve a contradiction in terms. The code you posted explicitly changes the datetime variable to a string representation. It's not reasonable to expect Excel to treat it as a date for you. Note, however, that most progress depends on wanting something "unreasonable", and making it happen.
I suspect that your actual problem is to make the spreadsheet display the date in the desired format. To this end, do not blindly accept the automatic set-up in your excel file. Write a little extra code to specify the column display format to be what you want. See the Excel documentation for details. Then look to see how much of that control you can grab through the Python interface.

Unable to convert index to date format

I have data which is in-64 in the Index with values like "01/11/2018" in the index. It is data that has been imported from a csv. I am unable to convert it to a "01-11-2018" format. How do I do this because I get an error message:
'time data 0 does not match format '%Y' (match)'
I got the data from the following website:
https://www.nasdaq.com/symbol/spy/historical
and you can find a ' Download this file in Excel Format ' icon at the bottom.
import datetime
spyderdat.index = pd.to_datetime(spyderdat.index, format='%Y')
spyderdat.head()
How do I format this correctly?
Thanks a lot.

Your format string must match exactly:
import datetime
spyderdat.index = pd.to_datetime(spyderdat.index, format='%d/%m/%Y')
spyderdat.head()
Example w/o spyder:
import datetime
date = "1/11/2018"
print(datetime.datetime.strptime(date,"%d/%m/%Y"))
Output:
2018-11-01 00:00:00
You can strftime this datetime then anyhow you like. See link for formats. Or you store datetimes.

Assuming your input is a string, simply converting the / to - won't fix the issue.
The real problem is that you've told to_datetime to expect the input string to be only a 4-digit year but you've handed it an entire date, days and months included.
If you meant to use only the year portion you should manually extract the year first with something like split.
If you meant to use the full date as a value, you'll need to change your format to something like %d/%m/%Y. (Although I can't tell if your input is days first or months first due to their values.)

The easy way is to try this
datetime.datetime.strptime("01/11/2018", '%d/%m/%Y').strftime('%d-%m-%Y')

Changing date format in python

I have a pandas dataframe with a column containing a date; the format of the original string is YYYY/DD/MM HH:MM:SS.
I am trying to convert the string into a datetime format, by using
df['Date']=pd.to_datetime(df['Data'], errors='coerce')
but plotting it I can see it doesn't recognize the correct format.
Can you help me to understand whether there is an option to give python the correct format to read the column?
I have seen the format tag for to_datetime function, but I can't use it correctly.
Thanks a lot for your help!

Try this:
df['Date'] = pd.to_datetime(df['Data'], format='%Y/%d/%m %H:%M:%S')

It looks like you're using a non-standard date format. It should be YYYY-MM-DD. Try formating with the strptime() method.
time.strptime('2016/15/07', '%Y/%d/%m')
If you need to get it to a string after that use time.strftime().

Save date time in filename with valid chars

To keep track of my when my files were backed up I want to have the filename of the backups as the datetime of when they were backed up. This will eventually be sorted and retrieved and sorted using python to allow me to get the most recent file based on the datetime filename.
The problem is, the automatic format of date time cant be saved like this:
2007-12-31 22:29:59
It can for example be saved like this:
2007-12-31 22-29-59
What is the best way to format the datetime so that I can easily sort by datetime on the name, and for bonus points, what is the python to show the datetime in that way.

You should have a look the documentation of the python time module: http://docs.python.org/2/library/time.html#module-time
If you go to the strftime() function, you will see that it accepts a string as input, which describes the format of the string you want to get as the return value.
Example (with hyphens between each date/time token):
>>> s = time.strftime('%Y-%m-%d-%H-%M-%S')
>>> print s
2012-12-08-14-55-44
The documentation contains a complete table of directives you can use to get different tokens.
What is the best way to format the datetime so that I can easily sort by datetime?
If you want to sort files according to datetimes names, you can consider that a biggest-to-lowest time specifier representation of a datetime (e.g.: YYYYMMDDhhmmss) preserves the same chronological and lexicographical order.

Convert DD/MM/YYYY HH:MM:SS into MySQL TIMESTAMP

I would like a simple way to find and reformat text of the format 'DD/MM/YYYY' into 'YYYY/MM/DD' to be compatible with MySQL TIMESTAMPs, in a list of text items that may or may not contain a date atall, under python. (I'm thinking RegEx?)
Basically i am looking for a way to inspect a list of items and correct any timestamp formats found.
Great thing about standards is that there are so many to choose from....

You can read the string into a datetime object and then output it back as a string using a different format. For e.g.
>>> from datetime import datetime
>>> datetime.strptime("31/12/2009", "%d/%m/%Y").strftime("%Y/%m/%d")
'2009/12/31'
Basically i am looking for a way to inspect a list of items and correct any timestamp formats found.
If the input format is inconsistent, can vary, then you are better off with dateutil.
>>> from dateutil.parser import parse
>>> parse("31/12/2009").strftime("%Y/%m/%d")
'2009/12/31'
Dateutil can handle a lot of input formats automatically. To operate on a list you can map the a wrapper over the parse function over the list and convert the values appropriately.

If you're using the MySQLdb (also known as "mysql-python") module, for any datetime or timestamp field you can provide a datetime type instead of a string. This is the type that is returned, also and is the preferred way to provide the value.
For Python 2.5 and above, you can do:
from datetime import datetime
value = datetime.strptime(somestring, "%d/%m/%Y")
For older versions of python, it's a bit more verbose, but not really a big issue.
import time
from datetime import datetime
timetuple = time.strptime(somestring, "%d/%m/%Y")
value = datetime(*timetuple[:6])
The various format-strings are taken directly from what's accepted by your C library. Look up man strptime on unix to find other acceptable format values. Not all of the time formats are portable, but most of the basic ones are.
Note datetime values can contain timezones. I do not believe MySQL knows exactly what to do with these, though. The datetimes I make above are usually considered as "naive" datetimes. If timezones are important, consider something like the pytz library.