I would like to create a date_range() with using pandas. I am kinda sure it worked before I updated pandas package.
with following line of code, I am trying to create the date_range():
date_time_index = pd.date_range(start='1/1/2018', periods=8760, freq='H')
and here is the error message:
ValueError: Error parsing datetime string "1/1/2018" at position 1
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "main.py", line 36, in <module>
date_time_index = pd.date_range(start='1/1/2018', periods=8760, freq='H')
File "/usr/local/lib/python3.6/dist-packages/pandas/tseries/index.py", line 2024, in date_range
closed=closed, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/pandas/util/decorators.py", line 91, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/pandas/tseries/index.py", line 301, in __new__
ambiguous=ambiguous)
File "/usr/local/lib/python3.6/dist-packages/pandas/tseries/index.py", line 403, in _generate
start = Timestamp(start)
File "pandas/tslib.pyx", line 406, in pandas.tslib.Timestamp.__new__ (pandas/tslib.c:9940)
File "pandas/tslib.pyx", line 1401, in pandas.tslib.convert_to_tsobject (pandas/tslib.c:25239)
File "pandas/tslib.pyx", line 1516, in pandas.tslib.convert_str_to_tsobject (pandas/tslib.c:26859)
File "pandas/src/datetime.pxd", line 141, in datetime._string_t
SystemError: <class 'str'> returned a result with an error set
What am I doing wrong?
Pandas version 0.19.1 date_range() does not work with the input I gave. I updated pandas to 0.23.4 now everything is fine.
Meanwhile:
pip3 install --upgrade pandas
Related
Could someone help me figure out why my files dont open.
import pandas as pd
file = "C://Dev//20211103_logfile Box 2.8.xlsx"
temp=pd.read_excel(file)
Here is the full error!
PS C:\Dev> & C:/Users/keyur/AppData/Local/Programs/Python/Python39/python.exe c:/Dev/test_excel.py
C:\Users\keyur\AppData\Local\Programs\Python\Python39\lib\site-packages\openpyxl\reader\workbook.py:88:
UserWarning: File contains an invalid specification for 20211103_logfile. This will be removed
warn(msg)
Traceback (most recent call last):
File "c:\Dev\test_excel.py", line 6, in <module>
temp=pd.read_excel(file)
File "C:\Users\keyur\AppData\Local\Programs\Python\Python39\lib\site-packages\pandas\util\_decorators.py", line 311, in wrapper
return func(*args, **kwargs)
File "C:\Users\keyur\AppData\Local\Programs\Python\Python39\lib\site-packages\pandas\io\excel\_base.py", line 372, in read_excel
data = io.parse(
File "C:\Users\keyur\AppData\Local\Programs\Python\Python39\lib\site-packages\pandas\io\excel\_base.py", line 1272, in parse
return self._reader.parse(
File "C:\Users\keyur\AppData\Local\Programs\Python\Python39\lib\site-packages\pandas\io\excel\_base.py", line 537, in parse
sheet = self.get_sheet_by_index(asheetname)
File "C:\Users\keyur\AppData\Local\Programs\Python\Python39\lib\site-packages\pandas\io\excel\_openpyxl.py", line 546, in get_sheet_by_index
self.raise_if_bad_sheet_by_index(index)
File "C:\Users\keyur\AppData\Local\Programs\Python\Python39\lib\site-packages\pandas\io\excel\_base.py", line 468, in raise_if_bad_sheet_by_index
raise ValueError(
ValueError: Worksheet index 0 is invalid, 0 worksheets found
PS C:\Dev>
There are problem with your excel,
try make a new excel and copy pase all data ,then try again ,this method works for me.
This error just started to pop up in our pipelines.
I'm moving a dataframe that's about 1.5mil rows using the pandas.to_gbq method.
Any help would be greatly appreciated!
Code:
output.to_gbq('table_name',
'project-id',
chunksize=50000,
private_key='ga_auth.json',
if_exists='replace'
)
Error:
Traceback (most recent call last):
File ".\rfm_bigquery.py", line 175, in <module>
send_rfm_to_gbq()
File ".\rfm_bigquery.py", line 152, in send_rfm_to_gbq
if_exists='replace',
File "C:\Users\yyu\Desktop\env\rfm_bigquery\lib\site-packages\pandas\core\frame.py", line 1187, in to_gbq
table_schema=table_schema)
File "C:\Users\yyu\Desktop\env\rfm_bigquery\lib\site-packages\pandas\io\gbq.py", line 119, in to_gbq
table_schema=table_schema)
File "C:\Users\yyu\Desktop\env\rfm_bigquery\lib\site-packages\pandas_gbq\gbq.py", line 1036, in to_gbq
progress_bar=progress_bar,
File "C:\Users\yyu\Desktop\env\rfm_bigquery\lib\site-packages\pandas_gbq\gbq.py", line 513, in load_data
self.process_http_error(ex)
File "C:\Users\yyu\Desktop\env\rfm_bigquery\lib\site-packages\pandas_gbq\gbq.py", line 376, in process_http_error
raise GenericGBQException("Reason: {0}".format(ex))
pandas_gbq.gbq.GenericGBQException: Reason: 404 POST https://www.googleapis.com/upload/bigquery/v2/projects/hidden-moon-164616/jobs?uploadType=resumable: Not Found
I had the same problem with the latest version, I moved back to pandas==0.23.3 and pandas-gbq==0.5.0 and it is finally working ...
Using pandas version 0.19.1 (with py27-tables-3.2.2_1 and hdf5-1.10.0 installed on my system), I am trying to save a pandas dataframe as a .h5 with:
import pandas as pd
df = pd.DataFrame(dict(A=range(5), B=range(5)))
df.to_hdf('savefile.h5', 'table', mode='w')
However the following error results:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python2.7/site-packages/pandas/core/generic.py", line 1138, in to_hdf
return pytables.to_hdf(path_or_buf, key, self, **kwargs)
File "/usr/local/lib/python2.7/site-packages/pandas/io/pytables.py", line 270, in to_hdf
f(store)
File "/usr/local/lib/python2.7/site-packages/pandas/io/pytables.py", line 264, in <lambda>
f = lambda store: store.put(key, value, **kwargs)
File "/usr/local/lib/python2.7/site-packages/pandas/io/pytables.py", line 873, in put
self._write_to_group(key, value, append=append, **kwargs)
File "/usr/local/lib/python2.7/site-packages/pandas/io/pytables.py", line 1315, in _write_to_group
s.write(obj=value, append=append, complib=complib, **kwargs)
File "/usr/local/lib/python2.7/site-packages/pandas/io/pytables.py", line 2864, in write
self.attrs.ndim = data.ndim
File "/usr/local/lib/python2.7/site-packages/tables/attributeset.py", line 461, in __setattr__
self._g__setattr(name, value)
File "/usr/local/lib/python2.7/site-packages/tables/attributeset.py", line 403, in _g__setattr
self._g_setattr(self._v_node, name, stvalue)
File "tables/hdf5extension.pyx", line 696, in tables.hdf5extension.AttributeSet._g_setattr (tables/hdf5extension.c:7549)
tables.exceptions.HDF5ExtError: HDF5 error back trace
File "H5A.c", line 634, in H5Awrite
not an attribute
End of HDF5 error back trace
Can't set attribute 'ndim' in node:
/table (Group) ''.
Could someone provide a simple working example of how to save a pandas dataframe in hdf5 format.
pytables is currently not compatible with hdf5-1.10 as reported in this issue on github, downgrading to hdf5-0.8 is the recommended solution.
I have mpi4py 1.3 installed with openMPI 1.6. Simple mpi4py demos like helloworld.py are working. In my code I call MPI.Comm.Allreduce(MPI.IN_PLACE, obj) , where obj is a numpy array.
This results in the error message:
ValueError: message: cannot guess count, datatype extent 0 (lb:0, ub:0)" in message_simple
Any clues on what I've done wrong?
Full traceback:
Traceback (most recent call last):
File "ptycho_508.py", line 124, in <module>
p = ptycho.ptycho_DM(p, numit=numit_DM, subpix=False)
File "/afs/psi.ch/project/cxs/users/braun_p/epd/lib/python2.7/site-packages/ptycho/core_object_modes.py", line 1357, in ptycho_DM comm.Allreduce(MPI.IN_PLACE, obj)
File "Comm.pyx", line 544, in mpi4py.MPI.Comm.Allreduce (src/mpi4py.MPI.c:63736)
File "message.pxi", line 579, in mpi4py.MPI._p_msg_cco.for_allreduce (src/mpi4py.MPI.c:23465)
File "message.pxi", line 532, in mpi4py.MPI._p_msg_cco.for_cro_recv (src/mpi4py.MPI.c:23068)
File "message.pxi", line 144, in mpi4py.MPI.message_simple (src/mpi4py.MPI.c:19614)
ValueError: message: cannot guess count, datatype extent 0 (lb:0, ub:0)
message_simple in <http://mpi4py.googlecode.com/svn/trunk/src/MPI/message.pxi> is failing
I created a file by using:
store = pd.HDFStore('/home/.../data.h5')
and stored some tables using:
store['firstSet'] = df1
store.close()
I closed down python and reopened in a fresh environment.
How do I reopen this file?
When I go:
store = pd.HDFStore('/home/.../data.h5')
I get the following error.
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/misc/apps/linux/python-2.6.1/lib/python2.6/site-packages/pandas-0.10.0-py2.6-linux-x86_64.egg/pandas/io/pytables.py", line 207, in __init__
self.open(mode=mode, warn=False)
File "/misc/apps/linux/python-2.6.1/lib/python2.6/site-packages/pandas-0.10.0-py2.6-linux-x86_64.egg/pandas/io/pytables.py", line 302, in open
self.handle = _tables().openFile(self.path, self.mode)
File "/apps/linux/python-2.6.1/lib/python2.6/site-packages/tables/file.py", line 230, in openFile
return File(filename, mode, title, rootUEP, filters, **kwargs)
File "/apps/linux/python-2.6.1/lib/python2.6/site-packages/tables/file.py", line 495, in __init__
self._g_new(filename, mode, **params)
File "hdf5Extension.pyx", line 317, in tables.hdf5Extension.File._g_new (tables/hdf5Extension.c:3039)
tables.exceptions.HDF5ExtError: HDF5 error back trace
File "H5F.c", line 1582, in H5Fopen
unable to open file
File "H5F.c", line 1373, in H5F_open
unable to read superblock
File "H5Fsuper.c", line 334, in H5F_super_read
unable to find file signature
File "H5Fsuper.c", line 155, in H5F_locate_signature
unable to find a valid file signature
End of HDF5 error back trace
Unable to open/create file '/home/.../data.h5'
What am I doing wrong here? Thank you.
In my hands, following approach works best:
df = pd.DataFrame(...)
"write"
with pd.HDFStore('test.h5', mode='w') as store:
store.append('df', df, data_columns= df.columns, format='table')
"read"
with pd.HDFStore('test.h5', mode='r') as newstore:
df_restored = newstore.select('df')
You could try doing instead:
store = pd.io.pytables.HDFStore('/home/.../data.h5')
df1 = store['firstSet']
or use the read method directly:
df1 = pd.read_hdf('/home/.../data.h5', 'firstSet')
Either way, you should have pandas 0.12.0 or higher...
I had the same problem and finally fixed it by installing the pytables module (next to the pandas modules which I was using):
conda install pytables
which got me numexpr-2.4.3 and pytables-3.2.0
After that it worked. I am using pandas 0.16.2 under python 2.7.9