My issue is ValueError which is rising, because of bad formatting of my datetime string.
# month is given. Sometimes with seconds as '.Z' (int) and sometimes as '.0000Z' (float)
try:
next_month = datetime.datetime.strptime(month, '%Y-%m-%dT%H:%M:%S.%fZ')
except ValueError:
print('asd')
next_month = datetime.datetime.strptime(month, '%Y-%m-%dT%H:%M:%SZ')
next_month += relativedelta.relativedelta(months=1)
Django error trace stack is hard to copy, but here is the raise:
raise ValueError("time data %r does not match format %r" %(data_string, format))
With this, I'm pretty sure, it's about datetime. (about datetime, hehe)
Found some questions on this topic, but none of proposed solutions worked. Most of them where about not placing the exact piece of code that generates this error in try-except.
The question is: why does ValueError is not being handled ?
EDIT: reporoduction snippet:
Below code run in PyCharm (python3.7, same as in above use), does not reproduce this error.
import datetime
if __name__ == "__main__":
d1 = '2021-03-24T11:20:33Z'
d2 = "2021-03-10T20:45:01.036000Z"
d = d2 # for quick change
try:
next_month = datetime.datetime.strptime(
d, '%Y-%m-%dT%H:%M:%S.%fZ')
print('try: ', next_month)
except ValueError:
next_month = datetime.datetime.strptime(
d, '%Y-%m-%dT%H:%M:%SZ')
print('except: ', next_month)
EDIT2:
In django error page: ValueError at /api/sessions_stats_csv/
Traceback (most recent call last):
django_server_dev | File "/usr/local/lib/python3.9/site-packages/django/core/handlers/exception.py", line 47, in inner
django_server_dev | response = get_response(request)
django_server_dev | File "/usr/local/lib/python3.9/site-packages/django/core/handlers/base.py", line 179, in _get_response
django_server_dev | response = wrapped_callback(request, *callback_args, **callback_kwargs)
django_server_dev | File "/app/label_it/rest_api/statistics_view.py", line 334, in AllSessionsDetailsCSV
django_server_dev | rows = ExtractSessionsForUser(item, params['for_month'])
django_server_dev | File "/app/label_it/rest_api/statistics_view.py", line 485, in ExtractSessionsForUser
django_server_dev | format_date = datetime.datetime.strptime(
django_server_dev | File "/usr/local/lib/python3.9/_strptime.py", line 568, in _strptime_datetime
django_server_dev | tt, fraction, gmtoff_fraction = _strptime(data_string, format)
django_server_dev | File "/usr/local/lib/python3.9/_strptime.py", line 349, in _strptime
django_server_dev | raise ValueError("time data %r does not match format %r" %
django_server_dev | ValueError: time data '2021-03-24T11:20:33Z' does not match format '%Y-%m-%dT%H:%M:%S.%fZ'
Related
import sys
import requests
date = str(sys.argv[1])
from_currency=str(sys.argv[2]).upper()
to_currency= str(sys.argv[3]).upper()
amount=float(sys.argv[4])
response = requests.get(f"https://api.frankfurter.app/{date}?amount={amount}&from={from_currency}&to={to_currency}")
print(f"{amount} {from_currency} is {response.json()['rates'][to_currency]} {to_currency} on {date}")
I tried running the following command on the terminal
python main.py '2020-01-01' 'USD' 'GBP' 2
I was expecting "3 USD is 86.77 GBP on 2020-01-01", but instead I get this error, which I do not understand
Traceback (most recent call last):
File "C:\Users\samee\anaconda3\lib\site-packages\requests\models.py", line 910, in json
return complexjson.loads(self.text, **kwargs)
File "C:\Users\samee\anaconda3\lib\json\__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "C:\Users\samee\anaconda3\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Users\samee\anaconda3\lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\samee\OneDrive\Desktop\PAI\Assignments\Assignment 1b\main.py", line 34, in <module>
print(f"{amount} {from_currency} is {response.json()['rates'][to_currency]} {to_currency} on {date}")
File "C:\Users\samee\anaconda3\lib\site-packages\requests\models.py", line 917, in json
raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: [Errno Expecting value] : 0
I ran your code, I am able to get the response. This error comes when sometime the website sends empty response.
By running your code, I got the response : 2.0 USD is 1.5147 GBP on 2020-01-01
To handle such empty or other response add Error Handling:
date = str(sys.argv[1])
from_currency=str(sys.argv[2]).upper()
to_currency= str(sys.argv[3]).upper()
amount=float(sys.argv[4])
try:
response = requests.get(f"https://api.frankfurter.app/{date}?amount={amount}&from={from_currency}&to={to_currency}")
print(f"{amount} {from_currency} is {response.json()['rates'][to_currency]} {to_currency} on {date}")
except ValueError: # includes simplejson.decoder.JSONDecodeError
print('Decoding JSON has failed')
I have a DataFrame of 320000 rows and 18 columns.
Two of the columns are the project start date and project end date.
I simply want to add a column with the duration of the project in days.
df['proj_duration'] = df['END_FORMATED'] - df['START_FORMATED']
The data is imported from a SQL Server.
The dates are formated (yyyy-mm-dd).
When I run the code above, I get this error:
Traceback (most recent call last):
File "pandas_libs\tslibs\timedeltas.pyx", line 234, in
pandas._libs.tslibs.timedeltas.array_to_timedelta64
TypeError: Expected unicode, got datetime.timedelta
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "", line 1, in
df['proj_duration'] = df['END_FORMATED'] - df['START_FORMATED']
File
"C:\Users\77797\Anaconda3\lib\site-packages\pandas\core\ops\common.py",
line 64, in new_method
return method(self, other)
File
"C:\Users\77797\Anaconda3\lib\site-packages\pandas\core\ops_init_.py",
line 502, in wrapper
return _construct_result(left, result, index=left.index, name=res_name)
File
"C:\Users\77797\Anaconda3\lib\site-packages\pandas\core\ops_init_.py",
line 475, in _construct_result
out = left._constructor(result, index=index)
File
"C:\Users\77797\Anaconda3\lib\site-packages\pandas\core\series.py",
line 305, in init
data = sanitize_array(data, index, dtype, copy, raise_cast_failure=True)
File
"C:\Users\77797\Anaconda3\lib\site-packages\pandas\core\construction.py",
line 424, in sanitize_array
subarr = _try_cast(data, dtype, copy, raise_cast_failure)
File
"C:\Users\77797\Anaconda3\lib\site-packages\pandas\core\construction.py",
line 537, in _try_cast
subarr = maybe_cast_to_datetime(arr, dtype)
File
"C:\Users\77797\Anaconda3\lib\site-packages\pandas\core\dtypes\cast.py",
line 1346, in maybe_cast_to_datetime
value = maybe_infer_to_datetimelike(value)
File
"C:\Users\77797\Anaconda3\lib\site-packages\pandas\core\dtypes\cast.py",
line 1198, in maybe_infer_to_datetimelike
value = try_timedelta(v)
File
"C:\Users\77797\Anaconda3\lib\site-packages\pandas\core\dtypes\cast.py",
line 1187, in try_timedelta
return to_timedelta(v)._ndarray_values.reshape(shape)
File
"C:\Users\77797\Anaconda3\lib\site-packages\pandas\core\tools\timedeltas.py",
line 102, in to_timedelta
return _convert_listlike(arg, unit=unit, errors=errors)
File
"C:\Users\77797\Anaconda3\lib\site-packages\pandas\core\tools\timedeltas.py",
line 140, in _convert_listlike
value = sequence_to_td64ns(arg, unit=unit, errors=errors, copy=False)[0]
File
"C:\Users\77797\Anaconda3\lib\site-packages\pandas\core\arrays\timedeltas.py",
line 943, in sequence_to_td64ns
data = objects_to_td64ns(data, unit=unit, errors=errors)
File
"C:\Users\77797\Anaconda3\lib\site-packages\pandas\core\arrays\timedeltas.py",
line 1052, in objects_to_td64ns
result = array_to_timedelta64(values, unit=unit, errors=errors)
File "pandas_libs\tslibs\timedeltas.pyx", line 239, in
pandas._libs.tslibs.timedeltas.array_to_timedelta64
File "pandas_libs\tslibs\timedeltas.pyx", line 198, in
pandas._libs.tslibs.timedeltas.convert_to_timedelta64
File "pandas_libs\tslibs\timedeltas.pyx", line 143, in
pandas._libs.tslibs.timedeltas.delta_to_nanoseconds
OverflowError: int too big to convert
I suspect that there is a problem in the formatting of the dates. I tried:
a = df.head(50000)['END_FORMATED']
b = df.head(50000)['START_FORMATED']
c = a-b
and got the same error. However, when I ran it for the last 50000 rows, it worked perfectly:
x = df.tail(50000)['END_FORMATED']
y = df.tail(50000)['START_FORMATED']
z = x-y
This shows that the problem does not exist in all of the dataset and only in some of the rows.
Any idea how I can solve the problem?
Thanks!
Seems like you have a date in your SQL dataset set as 1009-01-06. pandas only understand dates between 1677-09-21 and 2262-04-11, as per this oficial documentation.
Try to cast each Series into a datetime object to catch if some entry is not in the expected format, with infer_datetime_format = True and errors = 'coerce' as follows:
df['START_FORMATED'] = ['2020-05-05', '2020-05-06', '2020-05-07', 1009-01-06]
df['END_FORMATED'] = ['2020-06-05', '2020-06-06', '2020-06-07', '2020-06-08']
df['proj_duration'] = pd.to_datetime(df['END_FORMATED'], infer_datetime_format = True, errors = 'coerce') - pd.to_datetime(df['START_FORMATED'], infer_datetime_format=True, errors = 'coerce')
This will set NaT value when impossible to use pd.to_datetime(), which resulted in this df:
START_FORMATED END_FORMATED proj_duration
0 2020-05-05 2020-06-05 31 days
1 2020-05-06 2020-06-06 31 days
2 2020-05-07 2020-06-07 31 days
3 1009-01-06 2020-06-08 NaT
There is a source column of values from Excel that I would like to convert to dates (python).
df['DATE_'] = pd.to_datetime(df['DATE_'], format='%Y%m%d.0')
Source column from Excel:
'2010-06-16 00:00:00'
Unfortunately, system generates the following error.
Traceback (most recent call last):
File "C:/Users/103925alf1/PycharmProjects/p03/venv/Include/p03.py", line 5, in <module>
df['DATE_'] = pd.to_datetime(df['DATE_'], format='%Y%m%d.0')
File "C:\Users\103925alf1\PycharmProjects\p03\venv\lib\site-packages\pandas\core\tools\datetimes.py", line 728, in to_datetime
values = convert_listlike(arg._values, format)
File "C:\Users\103925alf1\PycharmProjects\p03\venv\lib\site-packages\pandas\core\tools\datetimes.py", line 435, in _convert_listlike_datetimes
raise e
File "C:\Users\103925alf1\PycharmProjects\p03\venv\lib\site-packages\pandas\core\tools\datetimes.py", line 400, in _convert_listlike_datetimes
arg, format, exact=exact, errors=errors
File "pandas\_libs\tslibs\strptime.pyx", line 142, in pandas._libs.tslibs.strptime.array_strptime
ValueError: time data '2010-06-16 00:00:00' does not match format '%Y%m%d.0' (match)
Process finished with exit code 1
I have a dataset in the following format:
date_time,open,close,...
2012-02-01,1307.25,1320.5,...
2012-02-03,1322.5,1339.5,...
....
These data are in a file called Dataset.csv. I read it as follows:
#This is the whole data. I will use it only later
self.data= pd.read_csv('./dataset/Dataset.csv')
#I will get only the indices from date_time columns. This is what I
#want now
self.sp = pd.read_csv('./dataset/Dataset.csv')
#Set index
self.sp = self.sp.set_index('date_time')
#save indices
self.sp = self.sp.index
if I print self.sp, here is what I get
Index(['2012-02-01', '2012-02-02', '2012-02-03', '2012-02-06', '2012-02-07',
'2012-02-08', '2012-02-09', '2012-02-10', '2012-02-13', '2012-02-14',
...
'2019-08-19', '2019-08-20', '2019-08-21', '2019-08-22', '2019-08-23',
'2019-08-26', '2019-08-27', '2019-08-28', '2019-08-29', '2019-08-30'],
dtype='object', name='date_time', length=1960)
I would like to get an interval of values, according to the date_time column considering a beginning date and interval of dates as the following:
#The initial date
begin=datetime.datetime(2012,2,1,0,0,0,0)
#The total number of days I will get from the dataset is 360, starting
#from the date in the
#begin variable
trainSize=datetime.timedelta(days=360*1).days
#The TrainMinLimit will be loaded as the initial date
#If the initial date cannot be used, add 1 day to the initial date and
#consider it as the initial date
trainMinLimit=None
while(trainMinLimit is None):
try:
trainMinLimit = self.sp.get_loc(begin)
except:
begin+=datetime.timedelta(1,0,0,0,0,0,0)
#The TrainMaxLimit will be loaded as the interval between the initial date plus the training
#size. If the initial date cannot be used, add 1 day to the initial date and consider it the
#initial date
trainMaxLimit=None
while(trainMaxLimit is None):
try:
trainMaxLimit = self.sp.get_loc(begin+trainSize)
except:
begin+=datetime.timedelta(1,0,0,0,0,0,0)
When I run this code, I have the following error:
trainMinLimit = self.sp.get_loc(begin)
File "/usr/local/lib/python3.5/dist-packages/pandas/core/indexes/base.py", line 2659, in
get_loc
return self._engine.get_loc(self._maybe_cast_indexer(key))
File "pandas/_libs/index.pyx", line 108, in pandas._libs.index.IndexEngine.get_loc
File "pandas/_libs/index.pyx", line 127, in pandas._libs.index.IndexEngine.get_loc
File "pandas/_libs/index.pyx", line 153, in
pandas._libs.index.IndexEngine._get_loc_duplicates
File "pandas/_libs/index.pyx", line 170, in
pandas._libs.index.IndexEngine._maybe_get_bool_indexer
KeyError: datetime.date(2012, 2, 1)
Here Python is not understanding in the get_loc() how the begin variable format (datetime) can index the dataframe of dates. How can I use a datetime variable to get the position it is located in a dates pandas dataframe?
edit: As suggested, I tried to convert the index to datetime format as the following:
self.sp = pd.read_csv('./dataset/Dataset.csv')
self.sp = self.sp.set_index('date_time')
self.sp = pd.to_datetime(self.sp.index)
And I have the following error:
self.sp = pd.to_datetime(self.sp.index)
File "/usr/local/lib/python3.5/dist-
packages/pandas/core/tools/datetimes.py", line 603, in to_datetime
result = convert_listlike(arg, box, format)
File "/usr/local/lib/python3.5/dist-
packages/pandas/core/tools/datetimes.py", line 302, in
_convert_listlike_datetimes
allow_object=True)
File "/usr/local/lib/python3.5/dist-
packages/pandas/core/arrays/datetimes.py", line 1866, in
objects_to_datetime64ns
raise e
File "/usr/local/lib/python3.5/dist-
packages/pandas/core/arrays/datetimes.py", line 1857, in
objects_to_datetime64ns
require_iso8601=require_iso8601
File "pandas/_libs/tslib.pyx", line 460, in
pandas._libs.tslib.array_to_datetime
File "pandas/_libs/tslib.pyx", line 685, in
pandas._libs.tslib.array_to_datetime
File "pandas/_libs/tslib.pyx", line 809, in
pandas._libs.tslib.array_to_datetime_object
File "pandas/_libs/tslib.pyx", line 803, in
pandas._libs.tslib.array_to_datetime_object
File "pandas/_libs/tslibs/parsing.pyx", line 99, in
pandas._libs.tslibs.parsing.parse_datetime_string
File "/usr/local/lib/python3.5/dist-
packages/dateutil/parser/_parser.py", line 1374, in parse
return DEFAULTPARSER.parse(timestr, **kwargs)
File "/usr/local/lib/python3.5/dist-
packages/dateutil/parser/_parser.py", line 649, in parse
raise ParserError("Unknown string format: %s", timestr)
dateutil.parser._parser.ParserError: Unknown string format: date_time
how must I change an element from a list that is in this form: 05/06/2020, to get a date object like:
date_object = datetime.strptime(listelement, '%m/%d/%y') ?
Here is my Code:
daten = {}
with open("Observed_Monthly_Rain_Gauge_Accumulations_-_Oct_2002_to_May_2017.csv", 'r') as csvfile:
regen_csv = csv.reader(csvfile)
next(regen_csv, None)
for rows in regen_csv:
keys = rows[:1]
strt= str(keys[0]) in
date_object = datetime.strptime(strt, '%m/%d/%y')
values = rows[1:]
daten[strt] = values
print(daten)
Traceback (most recent call last):
File *directory*, line 16, in <module>
date_object = datetime.strptime(strt, '%m/%d/%y')
File "C:\Program Files\Python37\lib\_strptime.py", line 577, in _strptime_datetime
tt, fraction, gmtoff_fraction = _strptime(data_string, format)
File "C:\Program Files\Python37\lib\_strptime.py", line 362, in _strptime
data_string[found.end():])
ValueError: unconverted data remains: 02
Heres a link to the list with the data:
https://data.seattle.gov/api/views/rdtp-hzy3/rows.csv?accessType=DOWNLOAD
You need to use %Y for years in the form 2020. A full list of format codes can be found here.
date_object = datetime.strptime(strt, '%m/%d/%Y')