Develop Reference

Installing GDAL for python in Google Cloud Functions -- error when deploying

I've been unsuccessful in using GDAL in a Google Cloud Function with Python 3.9. I've included gdal in the "requirements.txt" file:
numpy
pygrib
requests
google-cloud-storage
gdal
But get the following error when deploying the function:
Build failed: .../setuptools/command/egg_info.py", line 541, in run
self.add_defaults()
File "/layers/google.python.pip/pip/lib/python3.9/site-packages/setuptools/command/egg_info.py", line 578, in add_defaults
sdist.add_defaults(self)
File "/layers/google.python.pip/pip/lib/python3.9/site-packages/setuptools/_distutils/command/sdist.py", line 228, in add_defaults
self._add_defaults_ext()
File "/layers/google.python.pip/pip/lib/python3.9/site-packages/setuptools/_distutils/command/sdist.py", line 311, in _add_defaults_ext
build_ext = self.get_finalized_command('build_ext')
File "/layers/google.python.pip/pip/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 299, in get_finalized_command
cmd_obj.ensure_finalized()
File "/layers/google.python.pip/pip/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 107, in ensure_finalized
self.finalize_options()
File "/tmp/pip-install-d9gag00y/gdal_e01a421a21914f03a3c89fd3914501b0/setup.py", line 255, in finalize_options
gdaldir = self.get_gdal_config('prefix')
File "/tmp/pip-install-d9gag00y/gdal_e01a421a21914f03a3c89fd3914501b0/setup.py", line 194, in get_gdal_config
raise gdal_config_error(traceback_string + '\n' + msg)
__main__.gdal_config_error: Traceback (most recent call last):
File "/tmp/pip-install-d9gag00y/gdal_e01a421a21914f03a3c89fd3914501b0/setup.py", line 87, in fetch_config
p = subprocess.Popen([command, args], stdout=subprocess.PIPE)
File "/layers/google.python.runtime/python/lib/python3.9/subprocess.py", line 951, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
File "/layers/google.python.runtime/python/lib/python3.9/subprocess.py", line 1821, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'gdal-config'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/tmp/pip-install-d9gag00y/gdal_e01a421a21914f03a3c89fd3914501b0/setup.py", line 188, in get_gdal_config
return fetch_config(option, gdal_config=self.gdal_config)
File "/tmp/pip-install-d9gag00y/gdal_e01a421a21914f03a3c89fd3914501b0/setup.py", line 90, in fetch_config
raise gdal_config_error(e)
gdal_config_error: [Errno 2] No such file or directory: 'gdal-config'
Could not find gdal-config. Make sure you have installed the GDAL native library and development headers.
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed
× Encountered error while generating package metadata.
╰─> See above for output.
note: This is an issue with the package mentioned above, not pip.
hint: See above for details.; Error ID: c84b3231
Reviewing other similar posts, e.g. GDAL package missing, it seems that the GDAL library/headers needs to be installed, sudo apt-get install libgdal-dev, before the pip install gdal will work. Given that I can't run apt-get for a cloud function, does anyone have a fix or workaround?

Posting this as a community wiki to help other community members that will encounter this issue:
GDAL Libraries is not supported by Google Cloud Function at the moment. However, you can use Cloud Run for special cases where you have full control over the container.

When Installing polyglot with pip exception: Command "python setup.py egg_info" failed with error code 1 in

I am trying to install polyglot. I run Windows10 and have installed python3 using Anaconda3.
I have updated the setuptools.
The command and the error message are the following:
(base) C:\Users\Alienware\Documents>pip install -U git+https://github.com/aboSamoor/polyglot.git#master
Collecting git+https://github.com/aboSamoor/polyglot.git#master
Cloning https://github.com/aboSamoor/polyglot.git (to revision master) to c:\users\alienw~1\appdata\local\temp\pip-req-build-qt6pounr
Collecting PyICU>=1.8 (from polyglot==16.7.4)
Downloading https://files.pythonhosted.org/packages/c2/15/0af20b540c828943b6ffea5677c86e908dcac108813b522adebb75c827c1/PyICU-2.2.tar.gz (211kB)
100% |████████████████████████████████| 215kB 789kB/s
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "C:\Users\ALIENW~1\AppData\Local\Temp\pip-install-8fgenlih\PyICU\setup.py", line 43, in <module>
ICU_VERSION = os.environ['ICU_VERSION']
File "c:\programdata\anaconda3\lib\os.py", line 669, in __getitem__
raise KeyError(key) from None
KeyError: 'ICU_VERSION'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\ALIENW~1\AppData\Local\Temp\pip-install-8fgenlih\PyICU\setup.py", line 46, in <module>
ICU_VERSION = check_output(('icu-config', '--version')).strip()
File "c:\programdata\anaconda3\lib\subprocess.py", line 336, in check_output
**kwargs).stdout
File "c:\programdata\anaconda3\lib\subprocess.py", line 403, in run
with Popen(*popenargs, **kwargs) as process:
File "c:\programdata\anaconda3\lib\subprocess.py", line 707, in __init__
restore_signals, start_new_session)
File "c:\programdata\anaconda3\lib\subprocess.py", line 992, in _execute_child
startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Users\ALIENW~1\AppData\Local\Temp\pip-install-8fgenlih\PyICU\setup.py", line 53, in <module>
''')
RuntimeError:
Please set the ICU_VERSION environment variable to the version of
ICU you have installed.
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in C:\Users\ALIENW~1\AppData\Local\Temp\pip-install-8fgenlih\PyICU\

Error Installing Python.net on Python 3.7

I am attempting to get Python.net (pythonnet) working on Python 3.7 and when running the setup.py it fails with the following error. I'm assuming that it has to do with a missing entry in the path lines but I coulnd't fine any reference to it. It's basically a pretty plain install of Python 3.7 on Windows 7. the pip install also fails with the same results. Does anyone have a suggestion on what path or file might be missing?
PS C:\Users\user\pythonnet-master\pythonnet-master> python setup.py bdist_wheel
running bdist_wheel
running build
running build_ext
Checking for updates from https://www.nuget.org/api/v2/.
Currently running NuGet.exe 4.6.2.
NuGet.exe is up to date.
MSBuild auto-detection: using msbuild version '15.5.180.51428' from 'C:\Program Files (x86)\Microsoft Visual Studio\2017
\Professional\MSBuild\15.0\bin'.
All packages listed in packages.config are already installed.
Traceback (most recent call last):
File "tools\geninterop\geninterop.py", line 292, in <module>
sys.exit(main())
File "tools\geninterop\geninterop.py", line 272, in main
python_h = preprocess_python_headers()
File "tools\geninterop\geninterop.py", line 192, in preprocess_python_headers
for line in _check_output(cmd).splitlines():
File "tools\geninterop\geninterop.py", line 41, in _check_output
output = subprocess.check_output(*args, **kwargs)
File "C:\Users\i102604\AppData\Local\Programs\Python\Python37\lib\subprocess.py", line 376, in check_output
**kwargs).stdout
File "C:\Users\i102604\AppData\Local\Programs\Python\Python37\lib\subprocess.py", line 453, in run
with Popen(*popenargs, **kwargs) as process:
File "C:\Users\i102604\AppData\Local\Programs\Python\Python37\lib\subprocess.py", line 756, in __init__
restore_signals, start_new_session)
File "C:\Users\i102604\AppData\Local\Programs\Python\Python37\lib\subprocess.py", line 1155, in _execute_child
startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified

Cannot import jsonstreamer / cannot pip install yajl-py

I'm trying to use JSON streamer to parse a large 11gb JSON file. JSONstreamer appears to be the only package that streams the file without having to load the entire file into memory first.
When I try to import jsonstreamer in python I get this error.
OSError: Yajl cannot be found.
When I try to pip install yajl I get this error:
File "C:\Python34\lib\subprocess.py", line 1112, in _execute_child startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified
However all there files are there
I downloaded the yajl.dll file from their webpage and added it into my C:\Windows\SysWOW64 dir.
Don't really have any ideas how to proceed because it seems like I am stuck.

I had the same problem, it's fixed installing git on the machine: there is a dependency for it in pip3 scripts.
The key was on the log, you can see how a git subprocess is executed:
Running setup.py (path:/tmp/pip-build-01hs0gev/yajl/setup.py) egg_info for package yajl
Traceback (most recent call last):
File "<string>", line 17, in <module>
File "/tmp/pip-build-01hs0gev/yajl/setup.py", line 64, in <module>
subprocess.call(['git', 'submodule', 'update',])
File "/usr/lib/python3.4/subprocess.py", line 537, in call
with Popen(*popenargs, **kwargs) as p:
File "/usr/lib/python3.4/subprocess.py", line 859, in __init__
restore_signals, start_new_session)
File "/usr/lib/python3.4/subprocess.py", line 1457, in _execute_child
raise child_exception_type(errno_num, err_msg)
FileNotFoundError: [Errno 2] No such file or directory: 'git'

Getting an error when trying to install pyaardvark

I am using the canopy distribution of python (version 2.7.6) on win8-64, and when trying to use pip to install the module pyaardvark, I get the following error:
!pip install pyaardvark
Collecting pyaardvark
Using cached pyaardvark-0.3.tar.gz
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "<string>", line 20, in <module>
File "c:\...\appdata\local\temp\pip-build-5tvubw\pyaardvark\setup.py", line 13, in <module>
stderr=subprocess.STDOUT).rstrip()
File "c:\Program Files\Enthought\Canopy\App\appdata\canopy-1.4.0.1938.win-x86_64\lib\subprocess.py", line 566, in check_output
process = Popen(stdout=PIPE, *popenargs, **kwargs)
File "c:\Program Files\Enthought\Canopy\App\appdata\canopy-1.4.0.1938.win-x86_64\lib\subprocess.py", line 709, in __init__
errread, errwrite)
File "c:\Program Files\Enthought\Canopy\App\appdata\canopy-1.4.0.1938.win-x86_64\lib\subprocess.py", line 957, in _execute_child
startupinfo)
WindowsError: [Error 2] The system cannot find the file specified