We are using Azure Batch Apps which will create multiple VMs which can be used to run our tasks parallelly. We are using python for data fetching tasks.
We have mentioned in the batch apps to install anaconda on the VMs when they start up. Anaconda is installed properly. We have listed out the packages(requirements.txt) we need to install to run the tasks.
pip install -r requirements.txt
Some packages get installed correclty, but some packages result in the following error,
Error [Error 6] The handle is invalid while executing command python setup.py egg_info
Exception:
Traceback (most recent call last):
File "C:\user\tasks\shared\anaconda2\lib\site-packages\pip\basecommand.py", line 209, in main
status = self.run(options, args)
File "C:\user\tasks\shared\anaconda2\lib\site-packages\pip\commands\install.py", line 310, in run
wb.build(autobuilding=True)
File "C:\user\tasks\shared\anaconda2\lib\site-packages\pip\wheel.py", line 748, in build
self.requirement_set.prepare_files(self.finder)
File "C:\user\tasks\shared\anaconda2\lib\site-packages\pip\req\req_set.py", line 360, in prepare_files
ignore_dependencies=self.ignore_dependencies))
File "C:\user\tasks\shared\anaconda2\lib\site-packages\pip\req\req_set.py", line 591, in _prepare_file
abstract_dist.prep_for_dist()
File "C:\user\tasks\shared\anaconda2\lib\site-packages\pip\req\req_set.py", line 127, in prep_for_dist
self.req_to_install.run_egg_info()
File "C:\user\tasks\shared\anaconda2\lib\site-packages\pip\req\req_install.py", line 430, in run_egg_info
command_desc='python setup.py egg_info')
File "C:\user\tasks\shared\anaconda2\lib\site-packages\pip\utils\__init__.py", line 678, in call_subprocess
cwd=cwd, env=env)
File "C:\user\tasks\shared\anaconda2\lib\subprocess.py", line 702, in __init__
errread, errwrite), to_close = self._get_handles(stdin, stdout, stderr)
File "C:\user\tasks\shared\anaconda2\lib\subprocess.py", line 823, in _get_handles
p2cread = _subprocess.GetStdHandle(_subprocess.STD_INPUT_HANDLE)
WindowsError: [Error 6] The handle is invalid
When we open the VM and give the same command, all packages get installed correctly.
I just wonder where the issue is.
It seems that the issue was caused by some limits for Azure Batch service, you can see these limits here.
According to the error information, it seems that the installation process needs to fork the subprocess, but the maximum number of tasks per computer node is 4, please see below.
Related
I've been unsuccessful in using GDAL in a Google Cloud Function with Python 3.9. I've included gdal in the "requirements.txt" file:
numpy
pygrib
requests
google-cloud-storage
gdal
But get the following error when deploying the function:
Build failed: .../setuptools/command/egg_info.py", line 541, in run
self.add_defaults()
File "/layers/google.python.pip/pip/lib/python3.9/site-packages/setuptools/command/egg_info.py", line 578, in add_defaults
sdist.add_defaults(self)
File "/layers/google.python.pip/pip/lib/python3.9/site-packages/setuptools/_distutils/command/sdist.py", line 228, in add_defaults
self._add_defaults_ext()
File "/layers/google.python.pip/pip/lib/python3.9/site-packages/setuptools/_distutils/command/sdist.py", line 311, in _add_defaults_ext
build_ext = self.get_finalized_command('build_ext')
File "/layers/google.python.pip/pip/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 299, in get_finalized_command
cmd_obj.ensure_finalized()
File "/layers/google.python.pip/pip/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 107, in ensure_finalized
self.finalize_options()
File "/tmp/pip-install-d9gag00y/gdal_e01a421a21914f03a3c89fd3914501b0/setup.py", line 255, in finalize_options
gdaldir = self.get_gdal_config('prefix')
File "/tmp/pip-install-d9gag00y/gdal_e01a421a21914f03a3c89fd3914501b0/setup.py", line 194, in get_gdal_config
raise gdal_config_error(traceback_string + '\n' + msg)
__main__.gdal_config_error: Traceback (most recent call last):
File "/tmp/pip-install-d9gag00y/gdal_e01a421a21914f03a3c89fd3914501b0/setup.py", line 87, in fetch_config
p = subprocess.Popen([command, args], stdout=subprocess.PIPE)
File "/layers/google.python.runtime/python/lib/python3.9/subprocess.py", line 951, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
File "/layers/google.python.runtime/python/lib/python3.9/subprocess.py", line 1821, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'gdal-config'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/tmp/pip-install-d9gag00y/gdal_e01a421a21914f03a3c89fd3914501b0/setup.py", line 188, in get_gdal_config
return fetch_config(option, gdal_config=self.gdal_config)
File "/tmp/pip-install-d9gag00y/gdal_e01a421a21914f03a3c89fd3914501b0/setup.py", line 90, in fetch_config
raise gdal_config_error(e)
gdal_config_error: [Errno 2] No such file or directory: 'gdal-config'
Could not find gdal-config. Make sure you have installed the GDAL native library and development headers.
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed
× Encountered error while generating package metadata.
╰─> See above for output.
note: This is an issue with the package mentioned above, not pip.
hint: See above for details.; Error ID: c84b3231
Reviewing other similar posts, e.g. GDAL package missing, it seems that the GDAL library/headers needs to be installed, sudo apt-get install libgdal-dev, before the pip install gdal will work. Given that I can't run apt-get for a cloud function, does anyone have a fix or workaround?
Posting this as a community wiki to help other community members that will encounter this issue:
GDAL Libraries is not supported by Google Cloud Function at the moment. However, you can use Cloud Run for special cases where you have full control over the container.
Following the directions on the datatable page https://datatable.readthedocs.io/en/latest/install.html
So I ran this:
$pip3 install git+https://github.com/h2oai/datatable
The result is :
Collecting git+https://github.com/h2oai/datatable
Cloning https://github.com/h2oai/datatable to /private/var/folders/d6/m67jyndd7h754m3810cl3bpm0000gp/T/pip-req-build-hv991zd2
Running command git clone -q https://github.com/h2oai/datatable /private/var/folders/d6/m67jyndd7h754m3810cl3bpm0000gp/T/pip-req-build-hv991zd2
Getting requirements to build wheel ... done
ERROR: Exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 178, in main
status = self.run(options, args)
File "/usr/local/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 352, in run
resolver.resolve(requirement_set)
File "/usr/local/lib/python3.7/site-packages/pip/_internal/resolve.py", line 131, in resolve
self._resolve_one(requirement_set, req)
File "/usr/local/lib/python3.7/site-packages/pip/_internal/resolve.py", line 294, in _resolve_one
abstract_dist = self._get_abstract_dist_for(req_to_install)
File "/usr/local/lib/python3.7/site-packages/pip/_internal/resolve.py", line 242, in _get_abstract_dist_for
self.require_hashes
File "/usr/local/lib/python3.7/site-packages/pip/_internal/operations/prepare.py", line 362, in prepare_linked_requirement
abstract_dist.prep_for_dist(finder, self.build_isolation)
File "/usr/local/lib/python3.7/site-packages/pip/_internal/operations/prepare.py", line 169, in prep_for_dist
self.install_backend_dependencies(finder=finder)
File "/usr/local/lib/python3.7/site-packages/pip/_internal/operations/prepare.py", line 123, in install_backend_dependencies
reqs = req.pep517_backend.get_requires_for_build_wheel()
File "/usr/local/lib/python3.7/site-packages/pip/_vendor/pep517/wrappers.py", line 71, in get_requires_for_build_wheel
'config_settings': config_settings
File "/usr/local/lib/python3.7/site-packages/pip/_vendor/pep517/wrappers.py", line 162, in _call_hook
raise BackendUnavailable
pip._vendor.pep517.wrappers.BackendUnavailable
This is not actionable for me: I'm not certain what to fix/ how to proceed. Hints?
Note: in the meantime I have installed version 0.10.1 as follows:
sudo pip3 install 'datatable==0.10.1'
This has important by (expression based groupby) and sortvalues() that I need
Most likely this error occurs because you're trying to build datatable using an old version of pip. In order to build datatable from source, pip version 20.0 or higher is required.
Unfortunately, there is no way for a project to specify that it needs a specific minimum version of pip. And I presume if such setting would eventually be added, only the newest versions of pip would know about it anyways, defeating the purpose.
I am attempting to get Python.net (pythonnet) working on Python 3.7 and when running the setup.py it fails with the following error. I'm assuming that it has to do with a missing entry in the path lines but I coulnd't fine any reference to it. It's basically a pretty plain install of Python 3.7 on Windows 7. the pip install also fails with the same results. Does anyone have a suggestion on what path or file might be missing?
PS C:\Users\user\pythonnet-master\pythonnet-master> python setup.py bdist_wheel
running bdist_wheel
running build
running build_ext
Checking for updates from https://www.nuget.org/api/v2/.
Currently running NuGet.exe 4.6.2.
NuGet.exe is up to date.
MSBuild auto-detection: using msbuild version '15.5.180.51428' from 'C:\Program Files (x86)\Microsoft Visual Studio\2017
\Professional\MSBuild\15.0\bin'.
All packages listed in packages.config are already installed.
Traceback (most recent call last):
File "tools\geninterop\geninterop.py", line 292, in <module>
sys.exit(main())
File "tools\geninterop\geninterop.py", line 272, in main
python_h = preprocess_python_headers()
File "tools\geninterop\geninterop.py", line 192, in preprocess_python_headers
for line in _check_output(cmd).splitlines():
File "tools\geninterop\geninterop.py", line 41, in _check_output
output = subprocess.check_output(*args, **kwargs)
File "C:\Users\i102604\AppData\Local\Programs\Python\Python37\lib\subprocess.py", line 376, in check_output
**kwargs).stdout
File "C:\Users\i102604\AppData\Local\Programs\Python\Python37\lib\subprocess.py", line 453, in run
with Popen(*popenargs, **kwargs) as process:
File "C:\Users\i102604\AppData\Local\Programs\Python\Python37\lib\subprocess.py", line 756, in __init__
restore_signals, start_new_session)
File "C:\Users\i102604\AppData\Local\Programs\Python\Python37\lib\subprocess.py", line 1155, in _execute_child
startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified
I'm trying to start a Pinax Project for Django.
I setup my Python Virtual Environment.
Upgraded setup tools.
Upgraded pip.
Installed Pinax.
Then when I tried 'pinax start blog my_site', it gave the following error:
Installing Django...
Traceback (most recent call last):
File "/usr/local/bin/pinax", line 11, in <module>
sys.exit(main())
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 700, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 680, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 1027, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 873, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 508, in invoke
return callback(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/click/decorators.py", line 63, in new_func
return ctx.invoke(f, obj, *args[1:], **kwargs)
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 508, in invoke
return callback(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/pcli.py", line 97, in start
pip_install("Django")
File "/usr/local/lib/python2.7/dist-packages/pcli.py", line 23, in pip_install
command.run(opts, [package])
File "/usr/local/lib/python2.7/dist-packages/pip/commands/install.py", line 335, in run
wb.build(autobuilding=True)
File "/usr/local/lib/python2.7/dist-packages/pip/wheel.py", line 749, in build
self.requirement_set.prepare_files(self.finder)
File "/usr/local/lib/python2.7/dist-packages/pip/req/req_set.py", line 380, in prepare_files
ignore_dependencies=self.ignore_dependencies))
File "/usr/local/lib/python2.7/dist-packages/pip/req/req_set.py", line 634, in _prepare_file
abstract_dist.prep_for_dist()
File "/usr/local/lib/python2.7/dist-packages/pip/req/req_set.py", line 129, in prep_for_dist
self.req_to_install.run_egg_info()
File "/usr/local/lib/python2.7/dist-packages/pip/req/req_install.py", line 439, in run_egg_info
command_desc='python setup.py egg_info')
File "/usr/local/lib/python2.7/dist-packages/pip/utils/__init__.py", line 707, in call_subprocess
% (command_desc, proc.returncode, cwd))
pip.exceptions.InstallationError: Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-Jr1szu/Django/
I've tried making a python virtual environment inside my conda environment. Also tried making a standalone python environment. Neither solved it. Also tried installing django using pip.
Gave the following error:
sudo pip install django
The directory '/home/scarlet/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/scarlet/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting django
Downloading Django-2.0.tar.gz (8.0MB)
100% |████████████████████████████████| 8.0MB 62kB/s
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-build-SNP9W5/django/setup.py", line 32, in <module>
version = __import__('django').get_version()
File "django/__init__.py", line 1, in <module>
from django.utils.version import get_version
File "django/utils/version.py", line 61, in <module>
#functools.lru_cache()
AttributeError: 'module' object has no attribute 'lru_cache'
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-SNP9W5/django/
What would you suggest?
Thank you for the read!
It appears like you are using python 2; if that is correct Django version 2 only supports python 3. Please refer to https://docs.djangoproject.com/en/2.0/releases/2.0/#python-compatibility
You could try installing a lower version of Django:
pip install "Django<2"
you may try:
pip3 install django
I am trying to deploy python flask application using apache2 and mod-wsgi.
I am following this link.
While installing mog-wsgi using the command sudo apt-get install libapache2-mod-wsgi ,I see these lines at the end
apache2_invoke: Enable module wsgi
Action 'configtest' failed.
The Apache error log may have more information.
apache2_reload: Your configuration is broken. Not restarting Apache 2
Is this a smooth installation?
Also, in step 1 of the link when I tried to create a virtual environment using the command
sudo virtualenv venv
This gives me error sudo: virtualenv: command not found
So, I tried to install virtualenv using sudo pip install virtualenv, I am getting this huge error in red(exception).
Downloading/unpacking virtualenv
Downloading virtualenv-15.0.1-py2.py3-none-any.whl (1.8MB): 733kB downloaded
Cleaning up...
Exception:
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 122, in main
status = self.run(options, args)
File "/usr/lib/python2.7/dist-packages/pip/commands/install.py", line 278, in run
requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
File "/usr/lib/python2.7/dist-packages/pip/req.py", line 1198, in prepare_files
do_download,
File "/usr/lib/python2.7/dist-packages/pip/req.py", line 1376, in unpack_url
self.session,
File "/usr/lib/python2.7/dist-packages/pip/download.py", line 572, in unpack_http_url
download_hash = _download_url(resp, link, temp_location)
File "/usr/lib/python2.7/dist-packages/pip/download.py", line 433, in _download_url
for chunk in resp_read(4096):
File "/usr/lib/python2.7/dist-packages/pip/download.py", line 421, in resp_read
chunk_size, decode_content=False):
File "/usr/share/python-wheels/urllib3-1.7.1-py2.py3-none-any.whl/urllib3/response.py", line 225, in stream
data = self.read(amt=amt, decode_content=decode_content)
File "/usr/share/python-wheels/urllib3-1.7.1-py2.py3-none-any.whl/urllib3/response.py", line 174, in read
data = self._fp.read(amt)
File "/usr/lib/python2.7/httplib.py", line 573, in read
s = self.fp.read(amt)
File "/usr/lib/python2.7/socket.py", line 380, in read
data = self._sock.recv(left)
File "/usr/lib/python2.7/ssl.py", line 341, in recv
return self.read(buflen)
File "/usr/lib/python2.7/ssl.py", line 260, in read
return self._sslobj.read(len)
SSLError: The read operation timed out
Storing debug log for failure in /home/myname/.pip/pip.log
What could be the problem?
Please use apt-get install virtualenv or apt-get install python-virtualenv (depending on your OS version) instead of polluting the system with packages installed by sudo pip install.