Develop Reference

using qgis and shaply error: GEOSGeom_createLinearRing_r returned a NULL pointer

I tried to create a polygon shapefile in QGIS and read it in python by shapely. An example code looks like this:
import fiona
from shapely.geometry import shape
multipolys = fiona.open(somepath)
multi = multipolys[0]
coord = shape(multi['geometry'])
The EOSGeom_createLinearRing_r returned a NULL pointer
I checked if the polygon is valid in QGIS and no error was reported. Actually, it does not work for even a simple triangle generated in QGIS. Anyone knows how to solve it?
Thank you

Like J. P., I had this issue with creating LineStrings as well. There is an old issue (2016) in the Shapely github repository that seems related. Changing the order of the imports solved the problem for me:
from shapely.geometry import LineString
import fiona
LineString([[0, 0], [1, 1]]).to_wkt()
# 'LINESTRING (0.0000000000000000 0.0000000000000000, 1.0000000000000000 1.0000000000000000)'
whereas
import fiona
from shapely.geometry import LineString
LineString([[0, 0], [1, 1]]).to_wkt()
# Traceback (most recent call last):
# File "<stdin>", line 1, in <module>
# File "C:\Users\xxxxxxx\AppData\Roaming\Python\Python37\site-packages\shapely\geometry\linestring.py", line 48, in __init__
# self._set_coords(coordinates)
# File "C:\Users\xxxxxxx\AppData\Roaming\Python\Python37\site-packages\shapely\geometry\linestring.py", line 97, in _set_coords
# ret = geos_linestring_from_py(coordinates)
# File "shapely\speedups\_speedups.pyx", line 208, in shapely.speedups._speedups.geos_linestring_from_py
# ValueError: GEOSGeom_createLineString_r returned a NULL pointer
Some other issues in the Shapely repository to look at
553 for import order issues on a Mac
887 (same reverse-import-order trick with osgeo and shapely)
919

I had a similar problem but with the shapely.geometry.LineString. The error I got was
ValueError: GEOSGeom_createLineString_r returned a NULL pointer
I don't know the reason behind this message, but there are two ways, how to avoid it:
Do the following:
...
from shapely import speedups
...
speedups.disable()
Import the speedups module and disable the speedups. This needs to be done, since they are enabled by default.
From shapelys speedups init method:
"""
The shapely.speedups module contains performance enhancements written in C.
They are automaticaly installed when Python has access to a compiler and
GEOS development headers during installation, and are enabled by default.
"""
If you disable them, you won't get the NULL Pointer Exception, because you don't use the C implementation, rather than the usual implementation.
If you call python in a command shell, type:
from shapely.geometry import shape
this loads your needed shape. Then load your program
import yourscript
then run your script.
yourscript.main()
This should also work. I think in this variant the C modules get properly loaded and therefore you don't get the NULL Pointer Exception. But this only works, if you open a python terminal by hand and import the needed shape by hand. If you import the shape with your program, you will run into the same error again.

Face the same issue and this work for me
import shapely
shapely.speedups.disable()

Error when using scipy.misc.imread inside the loop

I have a case similar to this question.
Faced it when training deep learning model and reading images inside the loop.
scipy.misc.imread(rawImagePath).astype(numpy.float32)
The code above is working perfectly the vast majority of the time, but sometimes I get the error:
batch 90, loading batch data...
x path: /path_to_dataset/train/9.png
2017-10-31 20:23:34.531221: W tensorflow/core/kernels/queue_base.cc:294] _0_input_producer: Skipping cancelled enqueue attempt with queue not closed
Traceback (most recent call last):
File "train.py", line 452, in <module>
batch_X = load_data(batch_X_paths)
File "/path_to_my_script/utilities.py", line 85, in load_data
x = misc.imread(batch_X_paths[i]).astype(np.float32)
TypeError: float() argument must be a string or a number
The difference is: I have a fixed set of files and I do not write new ones to the directory.
I made a little research and figured out that error comes not from
numpy.astype()
but from
PIL.Image.open()
scipy.misc uses PIL under the hood.
Also, I managed to reproduce this bug in Jupyter Notebook. I made a loop with reading/astype code inside. After few seconds (the size of my test .png is about 10MB) I interrupt cell's execution and voila!
error reproduced in Jupyter Notebook
Inside scipy.misc sources there is a code like this:
def imread(name, flatten=False, mode=None):
im = Image.open(name)
return fromimage(im, flatten=flatten, mode=mode)
Image here is a PIL object. And Image.open() gives us 0-dimension list when interrupting cell.
I have tried Evert's advice and made a loop with try/except, but sometimes it fails even after 5 attempts.
Also I tried OpenCV imread() method instead of scipy.misc and sometimes it fails too. Error text:
libpng error: incorrect data check
So... I ran out of ideas.

Loading Compiled Matlab Shared Library in Python Using Ctypes

I am trying to do Incomplete Cholesky Decomposition in Python, but no direct Python package I can find.
Since the most available codes I can find online are written in Matlab, I want to take a detour by
compiling the matlab code to a shared library (I am using Mac OS and MATLAB_R2014a, so it should produce .dylib file)
load library in Python by using Ctypes
The following lists the detailed steps:
0. Download Matlab Source Code
The code can be downloaded from F. Bach's webpage link to zip file, which contains the following files:
panc:csi-1.0 panc25$ ls
center.m csi.dll csi.mexglx csi_gaussian.dll csi_gaussian.mexglx readme.txt
csi.c csi.m csi_gaussian.c csi_gaussian.m demo_csi.m sqdist.m
1. Compiling the matlab code to a shared library
Then by following this post, I run the command:
mcc -v -W cpplib:libcsi -T link:lib csi
After around a minute, the terminal prints MEX completed successfully and in my folder there are
panc:csi-1.0 panc25$ ls
center.m csi.m csi_gaussian.dll demo_csi.m libcsi.exports readme.txt
csi.c csi.mexglx csi_gaussian.m libcsi.cpp libcsi.h sqdist.m
csi.dll csi_gaussian.c csi_gaussian.mexglx libcsi.dylib mccExcludedFiles.log
where libcsi.dylib is the shared library I want.
2. Loading library in Python
Then I open IPython and try to load the library:
In [1]: import ctypes
In [2]: ctypes.C
ctypes.CDLL ctypes.CFUNCTYPE
In [2]: ctypes.CDLL('libcsi.dylib')
---------------------------------------------------------------------------
OSError Traceback (most recent call last)
<ipython-input-2-b6d0c1a91651> in <module>()
----> 1 ctypes.CDLL('libcsi.dylib')
/Users/panc25/anaconda/lib/python2.7/ctypes/__init__.pyc in __init__(self, name, mode, handle, use_errno, use_last_error)
363
364 if handle is None:
--> 365 self._handle = _dlopen(self._name, mode)
366 else:
367 self._handle = handle
OSError: dlopen(libcsi.dylib, 6): Library not loaded: #rpath/libmwmclmcrrt.8.3.dylib
Referenced from: /Users/panc25/Downloads/csi-1.0/libcsi.dylib
Reason: image not found
This problem persists even after I replace file name in ctypes.CDLL('libcsi.dylib') with the full path.
So I am confused. The shared library is there, but why Python says "image not found"?
BTW
SInce the source code also provide C implementation through mex.h, I also tried to first create a .mex file, then compile the .mex to a shared library as follows:
panc:csi-1.0 panc25$ mex csi.c
which created the csi.mexmaci64 file. Then according to this link, I called:
panc:csi-1.0 panc25$ mcc -B csharedlib:csi2 csi.mexmaci64
which produced csi2.dylib file.
But when I tried to load it in Python, I had the same error.
Could anyone let me know what is wrong?

I would avoid Matlab altogether, and instead use the Incomplete Cholesky Decomposition available in PyMC2:
from pymc.gp.incomplete_chol import ichol_full
The f2py wrapped Fortran code, that was actually adapted from a MEX file, can be found here. So you could use this independently of PyMC2 if need be.
If you are interested, you could also propose to add this function to scipy (see this githib issue ).

GDAL reprojection error: in method 'Geometry_Transform', argument 2 of type 'OSRCoordinateTransformationShadow *'

Using Python 2.7.9 with GDAL 1.11.1, with miniconda for package management --
Performing this a simple reprojection of a coordinate point causes the error described below.
I am relatively new to GDAL, so I checked to see if the code from the Python GDAL/OGR 1.0 Cookbook produces the same issue, and it does:
from osgeo import ogr
from osgeo import osr
source = osr.SpatialReference()
source.ImportFromEPSG(2927)
target = osr.SpatialReference()
target.ImportFromEPSG(4326)
transform = osr.CoordinateTransformation(source, target)
point = ogr.CreateGeometryFromWkt("POINT (1120351.57 741921.42)")
point.Transform(transform)
print point.ExportToWkt()
This is the error:
/opt/miniconda/envs/pygeo/lib/python2.7/site-packages/osgeo/ogr.pyc in Transform(self, *args)
4880 OGRERR_NONE on success or an error code.
4881 """
-> 4882 return _ogr.Geometry_Transform(self, *args)
4883
4884 def GetSpatialReference(self, *args):
TypeError: in method 'Geometry_Transform', argument 2 of type 'OSRCoordinateTransformationShadow *'
CoordinateTransform is a proxy for the C++ OSRCoordinateTransformationShadow class, generated by SWIG.
Per the source code for osgeo.ogr.Geometry (what Point is), the correct types were passed to the Transform method.
Best guess: Could this be caused by using a version of _ogr that is too old, and so the implementation of _ogr.Geometry_Transform(self, *args) is expecting a different?
_ogr is another SWIG-generated proxy, I'm guessing for the OGR class?

What everyone new to GDAL must learn: assign an error handler. (example: http://pcjericks.github.io/py-gdalogr-cookbook/gdal_general.html#install-gdal-ogr-error-handler)
With an error handler assigned, the output includes the explanation for the error. In this case, it was: "Unable to load PROJ.4 library (libproj.so), creation of OGRCoordinateTransformation failed."
Hopefully, imparting the knowledge of enabling GDAL error handling will help others who may stumble upon this very issue.

Similar information can be found on a rasterio FAQ and in unable to load "gcs.csv" file in gdal.
I encountered this problem when running GDAL transformations in my Anaconda3 QGIS environment. The problem is that the coordinate system informations were not loading through the GDAL_DATA environment variable.
To remedy, locate where the directory containing gcs.csv exists within your system (potentially ".../Library/share/gdal"). Add this to your environment prior to importing GDAL & other dependents.
import os
os.environ['GDAL_DATA'] = r'/path/to/dir/'

With help of the answer of jeremy that GDAL fails to load its information, I just edited the code to specify directly the PROJ.4 parameters from the EPSG webside and it runs
#target.ImportFromEPSG(4326)
target.ImportFromProj4('+proj=longlat +datum=WGS84 +no_defs')

Tracing Python warnings/errors to a line number in numpy and scipy

I am getting the error:
Warning: invalid value encountered in log
From Python and I believe the error is thrown by numpy (using version 1.5.0). However, since I am calling the "log" function in several places, I'm not sure where the error is coming from. Is there a way to get numpy to print the line number that generated this error?
I assume the warning is caused by taking the log of a number that is small enough to be rounded to 0 or smaller (negative). Is that right? What is the usual origin of these warnings?

Putting np.seterr(invalid='raise') in your code (before the errant log call)
will cause numpy to raise an exception instead of issuing a warning.
That will give you a traceback error message and tell you the line Python was executing when the error occurred.

If you have access to the numpy source, you should be able to find the line that prints that warning (using grep, etc) and edit the corresponding file to force an error (using an assertion, for example) when an invalid value is passed. That will give you a stack trace pointing to the place in your code that called log with the improper value.
I had a brief look in my numpy source, and couldn't find anything that matches the warning you described though (my version of numpy is older than yours, though).
>>> import numpy
>>> numpy.log(0)
-inf
>>> numpy.__version__
'1.3.0'
Is it possible that you're calling some other log function that isn't in numpy? For example, here is one that actually throws an exception when given invalid input.
>>> import math
>>> math.log(0)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: math domain error