Up until last week, I had a piece of code for calculating the odds of something occurring for a series of events, but starting this week the code doesn't work.
The code is the following:
import numpy as np
import pysal
import os
dbf = pysal.open('file.csv','r')
y = np.array([dbf.by_col('result')]).T
names_to_extract = ['dist', 'angle']
x = np.array([dbf.by_col(name) for name in names_to_extract]).T
id = np.array([dbf.by_col('incidenceId')]).T
model = pysal.spreg.probit.Probit(y, x, w=None, name_y='result', name_x=['dist','angle'], name_w=None, name_ds=None)
fin = np.column_stack((id, model.y, model.predy))
os.chdir("/destinyfolder")
np.savetxt('xg.csv', fin, delimiter=',', fmt='%d, %d, %f')
And I get this error:
/usr/local/lib/python3.7/site-packages/pysal/__init__.py:65: VisibleDeprecationWarning: PySAL's API will be changed on 2018-12-31. The last release made with this API is version 1.14.4. A preview of the next API version is provided in the `pysalnext` package. The API changes and a guide on how to change imports is provided at https://migrating.pysal.org
), VisibleDeprecationWarning)
Traceback (most recent call last):
File "xg.py", line 14, in <module>
model = pysal.spreg.probit.Probit(y, x, w=None, name_y='result', name_x=['dist','angle'], name_w=None, name_ds=None)
File "/usr/local/lib/python3.7/site-packages/pysal/spreg/probit.py", line 807, in __init__
n = USER.check_arrays(y, x)
File "/usr/local/lib/python3.7/site-packages/pysal/spreg/user_output.py", line 359, in check_arrays
if not spu.spisfinite(i):
File "/usr/local/lib/python3.7/site-packages/pysal/spreg/sputils.py", line 267, in spisfinite
return np.isfinite(a.sum())
File "/usr/local/lib/python3.7/site-packages/numpy/core/_methods.py", line 36, in _sum
return umr_sum(a, axis, dtype, out, keepdims, initial)
TypeError: cannot perform reduce with flexible type
Usually, I only got the DeprecationWarning, but the code worked. Since today, the code doesn't work. Consider that I am currently using pysal 1.14.4, numpy 1.16.2 and scipy 1.2.1. I have not updated my code to pysal 2.0 because I couldn't figure out how to port this code to the new version (and that's why I got the DeprecationWarning in the first place).
Here's the file: file.csv
Can you help me to make this work?
Entries in your x are strings, convert them to float:
x = np.array(...).T.astype(np.float)
This will fail at first because of "NULL" values in file.csv, you need to either filter these out or specify what float value they should be converted to.
Before posting a question, try first a simple google search for an immediate answer. If you google this error along with "pysal 1.14.4", you will immediately find this solution:
TypeError: cannot perform reduce with flexible type
Related
I'm trying to call a MATLAB function from python and continue with it's result in python. My script is as simple as this:
import librosa
import numpy as np
import matlab
import matlab.engine
eng = matlab.engine.start_matlab()
eng.addpath('TFDs/', '-end')
audio = librosa.chirp(0, 5000, duration=1, linear=True)
audio = matlab.double(audio.tolist())
choi_williams = eng.dtfd_nonsep(audio ,'cw',{30})
print(choi_williams)
I'm creating a linear chirp and expect to calculate it's time frequency distribution with the help of the fast_TFDs libary. The fast_TFDs folder is part of the current directory I'm executing the script from. But I'm recieving this error message:
Undefined function 'dtfd_nonsep' for input arguments of type 'cell'.
Traceback (most recent call last): File "cw.py", line 12, in
choi_williams = eng.dtfd_nonsep(audio ,'cw',{30}) File "/home/dunkeljo/tmp/anaconda3/envs/keras-gpu/lib/python3.8/site-packages/matlab/engine/matlabengine.py",
line 70, in call
return FutureResult(self._engine(), future, nargs, _stdout, File "/home/dunkeljo/tmp/anaconda3/envs/keras-gpu/lib/python3.8/site-packages/matlab/engine/futureresult.py",
line 67, in result
return self.__future.result(timeout) File "/home/dunkeljo/tmp/anaconda3/envs/keras-gpu/lib/python3.8/site-packages/matlab/engine/fevalfuture.py",
line 82, in result
self._result = pythonengine.getFEvalResult(self._future,self._nargout, None,
out=self._out, err=self._err) matlab.engine.MatlabExecutionError:
Undefined function 'dtfd_nonsep' for input arguments of type 'cell'.#
So what am I doing wrong? I'm not even sure whether the issue is the input arguments or that the function cannot be found. Is the use of the function wrong? Maybe even the import of them? Or is there a problem with the input parameters? Anyone got a clue?
The solution was as wwweagle suggested: Simply add cell2mat an it works:
import librosa
import numpy as np
import matlab
import matlab.engine
eng = matlab.engine.start_matlab()
eng.addpath('TFDs/', '-end')
audio = librosa.chirp(0, 5000, duration=1, linear=True)
audio = eng.cell2mat(audio.tolist())
audio = eng.double(audio)
choi_williams = eng.dtfd_nonsep(audio, 'cw', {eng.double(30)})
print(choi_williams)
I've created a standalone exe Windows service written in Python and built with pyInstaller. When I try to import wmi, an exception is thrown.
What's really baffling is that I can do it without a problem if running the code in a foreground exe, or a foreground python script, or a python script running as a background service via pythonservice.exe!
Why does it fail under this special circumstance of running as a service exe?
import wmi
Produces this error for me:
com_error: (-2147221020, 'Invalid syntax', None, None)
Here's the traceback:
Traceback (most recent call last):
File "<string>", line 43, in onRequest
File "C:\XXX\XXX\XXX.pyz", line 98, in XXX
File "C:\XXX\XXX\XXX.pyz", line 31, in XXX
File "C:\XXX\XXX\XXX.pyz", line 24, in XXX
File "C:\XXX\XXX\XXX.pyz", line 34, in XXX
File "C:\Program Files (x86)\PyInstaller-2.1\PyInstaller\loader\pyi_importers.py", line 270, in load_module
File "C:\XXX\XXX\out00-PYZ.pyz\wmi", line 157, in <module>
File "C:\XXX\XXX\out00-PYZ.pyz\win32com.client", line 72, in GetObject
File "C:\XXX\XXX\out00-PYZ.pyz\win32com.client", line 87, in Moniker
wmi.py line 157 has a global call to GetObject:
obj = GetObject ("winmgmts:")
win32com\client__init.py__ contains GetObject(), which ends up calling Moniker():
def GetObject(Pathname = None, Class = None, clsctx = None):
"""
Mimic VB's GetObject() function.
ob = GetObject(Class = "ProgID") or GetObject(Class = clsid) will
connect to an already running instance of the COM object.
ob = GetObject(r"c:\blah\blah\foo.xls") (aka the COM moniker syntax)
will return a ready to use Python wrapping of the required COM object.
Note: You must specifiy one or the other of these arguments. I know
this isn't pretty, but it is what VB does. Blech. If you don't
I'll throw ValueError at you. :)
This will most likely throw pythoncom.com_error if anything fails.
"""
if clsctx is None:
clsctx = pythoncom.CLSCTX_ALL
if (Pathname is None and Class is None) or \
(Pathname is not None and Class is not None):
raise ValueError("You must specify a value for Pathname or Class, but not both.")
if Class is not None:
return GetActiveObject(Class, clsctx)
else:
return Moniker(Pathname, clsctx)
The first line in Moniker(), i.e. MkParseDisplayName() is where the exception is encountered:
def Moniker(Pathname, clsctx = pythoncom.CLSCTX_ALL):
"""
Python friendly version of GetObject's moniker functionality.
"""
moniker, i, bindCtx = pythoncom.MkParseDisplayName(Pathname)
dispatch = moniker.BindToObject(bindCtx, None, pythoncom.IID_IDispatch)
return __WrapDispatch(dispatch, Pathname, clsctx=clsctx)
Note: I tried using
pythoncom.CoInitialize()
which apparently solves this import problem within a thread, but that didn't work...
I also face the same issue and I figure out this issue finally,
import pythoncom and CoInitialize pythoncom.CoInitialize (). They import wmi
import pythoncom
pythoncom.CoInitialize ()
import wmi
I tried solving this countless ways. In the end, I threw in the towel and had to just find a different means of achieving the same goals I had with wmi.
Apparently that invalid syntax error is thrown when trying to create an object with an invalid "moniker name", which can simply mean the service, application, etc. doesn't exist on the system. Under this circumstance "winmgmts" just can't be found at all it seems! And yes, I tried numerous variations on that moniker with additional specs, and I tried running the service under a different user account, etc.
Honestly I didn't dig in order to understand why this occurs.
Anyway, the below imports solved my problem - which was occurring only when ran from a Flask instance:
import os
import pythoncom
pythoncom.CoInitialize()
from win32com.client import GetObject
import wmi
The error "com_error: (-2147221020, 'Invalid syntax', None, None)" is exactly what popped up in my case so I came here after a long time of searching the web and voila:
Under this circumstance "winmgmts" just can't be found at all it
seems!
This was the correct hint for because i had just a typo , used "winmgmt:" without trailing 's'. So invalid sythax refers to the first methods parameter, not the python code itself. o_0 Unfortunately I can't find any reference which objects we can get with win32com.client.GetObject()... So if anybody has a hint to which params are "allowed" / should work, please port it here. :-)
kind regards
ChrisPHL
I'm trying to launch AWS EMR cluster using boto library, everything works well.
Because of that I need to install required python libraries, tried to add bootstrap action step using boto.emr.bootstrap_action
But It gives error below;
Traceback (most recent call last):
File "run_on_emr_cluster.py", line 46, in <module>
steps=[step])
File "/usr/local/lib/python2.7/dist-packages/boto/emr/connection.py", line 552, in run_jobflow
bootstrap_action_args = [self._build_bootstrap_action_args(bootstrap_action) for bootstrap_action in bootstrap_actions]
File "/usr/local/lib/python2.7/dist-packages/boto/emr/connection.py", line 623, in _build_bootstrap_action_args
bootstrap_action_params['ScriptBootstrapAction.Path'] = bootstrap_action.path AttributeError: 'str' object has no attribute 'path'
Code below;
from boto.emr.connection import EmrConnection
conn = EmrConnection('...', '...')
from boto.emr.step import StreamingStep
step = StreamingStep(name='mapper1',
mapper='s3://xxx/mapper1.py',
reducer='s3://xxx/reducer1.py',
input='s3://xxx/input/',
output='s3://xxx/output/')
from boto.emr.bootstrap_action import BootstrapAction
bootstrap_action = BootstrapAction(name='install related packages',path="s3://xxx/bootstrap.sh", bootstrap_action_args=None)
job = conn.run_jobflow(name='emr_test',
log_uri='s3://xxx/logs',
master_instance_type='m1.small',
slave_instance_type='m1.small',
num_instances=1,
action_on_failure='TERMINATE_JOB_FLOW',
keep_alive=False,
bootstrap_actions='[bootstrap_action]',
steps=[step])
What's the proper way of passing bootstrap arguments?
You are passing the bootstrap_actions argument as a literal string rather than as a list containing the BootstrapAction object you just created. Try this:
job = conn.run_jobflow(name='emr_test',
log_uri='s3://xxx/logs',
master_instance_type='m1.small',
slave_instance_type='m1.small',
num_instances=1,
action_on_failure='TERMINATE_JOB_FLOW',
keep_alive=False,
bootstrap_actions=[bootstrap_action],
steps=[step])
Notice that the ``bootstrap_action` argument is different here.
I have an error that occurs only when I call lrtest (from the lmtest package) from a user-defined function via rpy2.
R:
continuous.test <- function(dat) {
require('lmtest')
options(warn=-1)
model <- lm(formula='pheno ~ .', data=dat)
anova <- lrtest(model,'interaction')
pval <- anova$"Pr(>Chisq)"[2]
}
When I call this function from the R interpreter, everything runs correctly. However, I receive an error when calling from the following snippet of python code. Note, this particular python file makes many other calls to rpy2 with success.
Python:
...
kway_dat = R.DataFrame(dataframe) # this is a valid dataframe, it's used in other calls.
...
R.r("source('/path/to/user/defined/file/perm_test.r')")
continuous_test = R.r['continuous.test']
pval = continuous_test(kway_dat)
Error:
Error in is.data.frame(data) : object 'dat' not found
Traceback (most recent call last):
File "./test_r_.py", line 83, in <module>
pval = continuous_test(kway_dat)
File "/usr/lib/python2.6/site-packages/rpy2-2.2.6dev_20120806-py2.6-linux-x86_64.egg/rpy2/robjects/functions.py", line 82, in __call__
return super(SignatureTranslatedFunction, self).__call__(*args, **kwargs)
File "/usr/lib/python2.6/site-packages/rpy2-2.2.6dev_20120806-py2.6-linux-x86_64.egg/rpy2/robjects/functions.py", line 34, in __call__
res = super(Function, self).__call__(*new_args, **new_kwargs)
rpy2.rinterface.RRuntimeError: Error in is.data.frame(data) : object 'dat' not found
Troubleshooting:
I have tested the code in R and everything works fine.
I have passed a dataframe from python to R through rpy2 and called is.data.frame(dat) from an R function, and it returns true, so the issue is with lmtest or lrtest + rpy2.
Any help would be great. Thanks all!
It would be easier to help with a self-contained example (so one can reproduce exactly what you are experiencing).
A possible answer still: you might want to check that the content of the file
/path/to/user/defined/file/perm_test.r is really what you think it is.
I am also adding a stub for a self-contained example:
r_code = """
require('lmtest')
options(warn=-1)
continuous.test <- function(dat) {
model <- lm(formula='pheno ~ .', data=dat)
anova <- lmtest::lrtest(model,'interaction')
pval <- anova$"Pr(>Chisq)"[2]
}
"""
from rpy2.robjects import packages
my_r_pack = packages.SignatureTranslatedAnonymousPackage(r_code, "my_r_pack")
# [build a demo kway_dat here]
my_r_pack.continuous_test(kway_dat)
Answer Found
The issue was lrtest's internal call to update the model. Once inside lrtest, dat was out of scope. By updating the model manually and using lrtest's alternative call lrtest(model0,model1), the issue is entirely avoided.
Thanks to Achim Zeileis who replied incredibly promptly.
I'm running python2.5 and trying to use the astLib library to analyse WCS information in astronomical images. I try and get the object instanciated with the following skeleton code:
from astLib import astWCS
w = astWCS.WCS('file.fits') # error here
where file.fits is a string pointing to a valid fits file.
I have tried using the alternate method of passing a pyfits header object and this fails also:
import pyfits
from astLib import astWCS
f = pyfits.open('file.fits')
header = f[0].header
f.close()
w = astWCS.WCS(header, mode='pyfits') # error here also
The error is this:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/home/astro/phrfbf/build/lib/python2.6/site-packages/astLib/astWCS.py", line 79, in __init__
self.updateFromHeader()
File "/home/astro/phrfbf/build/lib/python2.6/site-packages/astLib/astWCS.py", line 119, in updateFromHeader
self.WCSStructure=wcs.wcsinit(cardstring)
File "/home/astro/phrfbf/build/lib/python2.6/site-packages/PyWCSTools/wcs.py", line 70, in wcsinit
return _wcs.wcsinit(*args)
TypeError: in method 'wcsinit', argument 1 of type 'char *'
When I run in ipython, I get the full error here on the pastebin
I know the astWCS module is a wrapped version of WCStools but i'd prefer to use the Python module as the rest of my code is in Python
Can anyone help with this problem?
Just found out the updated version of this library has fixed the problem, thanks for everyone's help
Oh sorry, I should have seen. Looking at the pastebin in more detail, the only error I can think of is that, for some reason the header has unicode in it. It can't be converted to char *, and you get the error. I tried searching for something in the header, but everything looks okay. Can you do this and post the output in another pastebin?
import pyfits
f = pyfits.open('file.fits')
header = f[0].header
f.close()
for x, i in enumerate(header.iteritems()):
if len(str(i[1])) >= 70:
print x, str(i[1])
cardlist = header.ascardlist()
cardstring = ""
for card in cardlist:
cardstring = cardstring + str(card)
print repr(cardstring)
Or, if you can check the header of your fits file for "funny" characters, getting rid of them should solve the issue.