Develop Reference

Converting string of lists of timestamps to lists of timestamps in pandas

I parsed data from s3 which is similar to this
ID departure
1 "[Timestamp('2021-05-25 09:00:00'), datetime.datetime(2021, 5, 25, 9, 21, 35, 769406)]"
2 "[Timestamp('2021-05-25 08:00:00'), datetime.datetime(2021, 5, 25, 11, 15), datetime.datetime(2021, 5, 25, 14, 15)]"
Is there any way to convert the departure into list
I tried this
samp['departure'] = samp['departure'].apply(lambda x: eval(x))
-> Error: eval() arg 1 must be a string, bytes or code object
and
samp['departure'] = samp['departure'].apply(lambda x: x[1:-1].split(','))
# Here datetime.datetime(2021, 5, 25, 11, 15) splited into many sub-parts
and
samp.departure = samp.departure.apply(ast.literal_eval)
error -> malformed node or string: ["Timestamp('2021-05-25 09:00:00')", ' datetime.datetime(2021', ' 5', ' 25', ' 9', ' 21', ' 35', ' 769406)']
Output should be
ID departure
1 [Timestamp('2021-05-25 09:00:00'), datetime.datetime(2021, 5, 25, 9, 21, 35, 769406)]
2 [Timestamp('2021-05-25 08:00:00'), datetime.datetime(2021, 5, 25, 11, 15), datetime.datetime(2021, 5, 25, 14, 15)]
(I tried converters while read_csv initially but getting an error too)

If you are trying to replace " present in your departure column:
Try via replace():
samp['departure']=samp['departure'].replace('"','',regex=True)
OR
try via strip():
samp['departure']=samp['departure'].str.strip('"')
If you are evaluating the values inside:
from pandas import Timestamp
import datetime
samp['departure']=samp['departure'].astype(str).apply(pd.eval)
OR
from pandas import Timestamp
import datetime
import ast
samp.departure = samp.departure.astype(str).apply(ast.literal_eval)

converting an array of unixtime to mmddyy on python

I have an array of unixtime timestamps. How do I convert that using
datetime.utcfromtimestamp().strftime("%Y-%M-%D %H:%M:%S")
? My array is saved under "time". How do I utilize that array in this conversion?

Assuming your times are of the format datetime, you can loop through the list and convert each one.
Here is a quick example:
import datetime
time = []
for i in range(10):
time.append(datetime.datetime.now())
print(time) # output: [datetime.datetime(2020, 7, 8, 10, 7, 4, 314614), datetime.datetime(2020, 7, 8, 10, 7, 4, 314622)....
formattedTime = []
for t in time:
formattedTime.append(t.strftime('%Y-%m-%d %H:%M:%S'))
print(formattedTime) # output: ['2020-07-07/08/20 10:07:04', '2020-07-07/08/20 10:07:04', ....
# the update to my answer:
newTimes = []
for date_time_str in formattedTime:
newTimes.append(datetime.datetime.strptime(date_time_str, '%Y-%m-%d %H:%M:%S'))
print(newTimes) # '%Y-%m-%d %H:%M:%S' [datetime.datetime(2020, 7, 8, 18, 56, 47), datetime.datetime(2020, 7, 8, 18, 56, 47), datetime.datetime(2020, 7, 8, 18, 56, 47),...]
Let me know if you have more questions.
I am also attaching this article for datetime which I found really helpful.
Here is the example in repl

Convert the date time stamp to local epoch datatime

Assume this is my date time stamp list:
[datetime.datetime(2017, 11, 17, 9, 33, 11), datetime.datetime(2017, 11, 17, 9, 33, 36), datetime.datetime(2017, 11, 17, 9, 33, 48)]
A lot of examples have been given for converting these values to epoch, but the values are in GMT.
How do we convert it to Epoch Local time?
To put it in simpler words. The general epoch conversion coverts the given date into epoch considering that the datetime given is in GMT. But the given date time is localtime!

>>> x = datetime.datetime(2017, 11, 17, 9, 33, 36)
>>> x.timestamp()
1510882416.0
>>> x.ctime()
'Fri Nov 17 09:33:36 2017'

You can use this:
l = [datetime.datetime(2017, 11, 17, 9, 33, 11), datetime.datetime(2017, 11, 17, 9, 33, 36), datetime.datetime(2017, 11, 17, 9, 33, 48)]
epo = [x.strftime('%s') for x in l]
print(epo)
# ['1510898591', '1510898616', '1510898628']

You can try arrow lib:
import arrow
import datetime
x = datetime.datetime(2017, 11, 17, 9, 33, 36)
time = arrow.get(x, 'local').shift(hours=-6) // here you can change hours to what you want
print time
>>2017-11-17T03:33:36+00:00
time = time.format('YYYY-MM-DD HH:mm:ss')
print time
>>2017-11-17 03:33:36

Convert datetime to UTC

I have a date string like - 2015-01-05T10:30:47-0800,
It looks to me that this is some timezone because of the offset. How can I get a date string which is in the UTC timezone from the above date string.
I tried the following -
datestring = '2015-01-05T10:30:47-0800'
from dateutil import parser
d = parser.parse(datestring) # datetime.datetime(2015, 1, 5, 10, 30, 47, tzinfo=tzoffset(None, -28800))
import pytz
d.astimezone(pytz.timezone('UTC')) # datetime.datetime(2015, 1, 5, 18, 30, 47, tzinfo=<UTC>)
EDIT -
The above code returns the correct answer. My bad!

Try this:
>>> import dateutil.parser
>>> d = dateutil.parser.parse('2015-01-05T10:30:47-0800')
>>> d.astimezone(dateutil.tz.tzutc())
datetime.datetime(2015, 1, 5, 18, 30, 47, tzinfo=tzutc())

Datetime - 10 Hours

Consider:
now = datetime.datetime.now()
now
datetime.datetime(2009, 11, 6, 16, 6, 42, 812098)
How would I create a new datetime object (past) and minus n values from the hours?

Use timedelta in the datetime module:
import datetime
now = datetime.datetime.now()
past = now - datetime.timedelta(hours=10)

Use a timedelta object.
>>> now = datetime.datetime.now()
>>> now
datetime.datetime(2009, 11, 6, 16, 35, 50, 593000)
>>> ten_hours = datetime.timedelta(hours=10)
>>> now + ten_hours
datetime.datetime(2009, 11, 7, 2, 35, 50, 593000)
>>> now - ten_hours
datetime.datetime(2009, 11, 6, 6, 35, 50, 593000)

Use a timedelta object.
from datetime import datetime
back = datetime.now() - timedelta(hours=10)