Related
I would like to be able to install R packages from GitHub in a R conda environment created by Snakemake, as well as python libraries via pip in a python environment. I'll use these environments in a whole set of rules thereafter.
My initial thought was to create a rule running a script to install the specified packages.
For instance, my initial run was: snakemake -j1 --use-conda -R create_r_environment.
My Snakefile:
rule create_r_environment:
conda:
"envs/r.yaml"
script:
"scripts/r-dependencies.R"
rule create_python_environment:
conda:
"envs/python.yaml"
script:
"scripts/python-dependencies.py"
My envs/r.yaml file:
channels:
- conda-forge
dependencies:
- r=4.0
My r-dependencies.R file:
remotes::install_github("ramiromagno/gwasrapidd", upgrade = "never")
My envs/pyton.yaml file:
channels:
- conda-forge
dependencies:
- python=3.8.2
My python-dependencies.py file:
!pip install gseapy
The log output:
Building DAG of jobs...
Creating conda environment envs/r.yaml...
Downloading and installing remote packages.
Environment for envs/r.yaml created (location: .snakemake/conda/388,repos = "http://cran.us.r-project.org")f7df8)
Using shell: /usr/bin/bash
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Job counts:
count jobs
1 create_r_environment
1
[Fri Oct 30 22:38:56 2020]
rule create_r_environment:
jobid: 0
Activating conda environment: /home/cmcouto-silva/cmcouto.silva#usp.br/lab_files/phd_data/SO/.snakemake/conda/388f7df8
[Fri Oct 30 22:38:57 2020]
Error in rule create_r_environment:
jobid: 0
conda-env: /home/cmcouto-silva/cmcouto.silva#usp.br/lab_files/phd_data/SO/.snakemake/conda/388f7df8
RuleException:
CalledProcessError in line 5 of /home/cmcouto-silva/cmcouto.silva#usp.br/lab_files/phd_data/SO/Snakefile:
Command 'source /home/cmcouto-silva/miniconda3/bin/activate '/home/cmcouto-silva/cmcouto.silva#usp.br/lab_files/phd_data/SO/.snakemake/conda/388f7df8'; set -euo pipefail; Rscript --vanilla /home/cmcouto-silva/cmcouto.silva#usp.br/lab_files/phd_data/SO/.snakemake/scripts/tmpa6jdxovx.r-dependencies.R' returned non-zero exit status 1.
File "/home/cmcouto-silva/miniconda3/envs/snakemake/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 2168, in run_wrapper
File "/home/cmcouto-silva/cmcouto.silva#usp.br/lab_files/phd_data/SO/Snakefile", line 5, in __rule_create_r_environment
File "/home/cmcouto-silva/miniconda3/envs/snakemake/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 529, in _callback
File "/home/cmcouto-silva/miniconda3/envs/snakemake/lib/python3.8/concurrent/futures/thread.py", line 57, in run
File "/home/cmcouto-silva/miniconda3/envs/snakemake/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 515, in cached_or_run
File "/home/cmcouto-silva/miniconda3/envs/snakemake/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 2199, in run_wrapper
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /home/cmcouto-silva/cmcouto.silva#usp.br/lab_files/phd_data/SO/.snakemake/log/2020-10-30T223743.852983.snakemake.log
My folder structure:
.
├── envs
│ ├── python.yaml
│ └── r.yaml
├── scripts
│ ├── python-dependencies.py
│ └── r-dependencies.R
└── Snakefile
It successfully creates the environment but fails when running the script, and I don't know why. I've changed the envs/r.yaml file content to install.packages("data.table") to see if there was an issue with the github package, but it's not. It fails anyway. The same occurs when I run the rule create_python_environment (output not showed here).
Any help?
Edit after the accepted answer
As #dariober pointed out, I forgot to install the remotes package before calling it in the script. I did it in the .yaml file, and it worked well. Also, I installed the pip libraries using shell instead of a python file.
I would like to highlight some points though, just in case anyone's facing the same or similar problem:
First, I could successfully install further packages I needed to, but some of them require specific libraries (e.g. libcurl), which is installed in my system, but it's not recognized inside the Snakemake conda environment, forcing me to either install it in the Snakemake conda environment (which is good for reproducibility, although I don't know how to do that yet) or specify the path library. Maybe a better option would be using a container just like #merv commented out.
Second, I figured out that Snakemake already provides a way to install pip libraries using the .yaml file. From the documentation, it looks like this:
name: stats2
channels:
- javascript
dependencies:
- python=3.6 # or 2.7
- bokeh=0.9.2
- numpy=1.9.*
- nodejs=0.10.*
- flask
- pip:
- Flask-Testing
I think there are quite a few wrong things:
remotes::install_github("ramiromagno/gwasrapidd", upgrade = "never"): In your r.yaml you should include the remotes package.
!pip install gseapy is not valid python code. If anything, it is code to be executed by shell but I'm not sure that leading ! is correct. Also, gseapy is available from bioconda I don;t see why you should install it with pip.
Before OP edited the question
My envs/r.yaml file:
remotes::install_github("ramiromagno/gwasrapidd", upgrade = "never")
It's odd that you get the conda environment correctly created since that r.yaml is not a valid environment file.
This is what I tried to recreate your issue:
r.yaml
cat r.yaml
remotes::install_github("ramiromagno/gwasrapidd", upgrade = "never")
Snakefile:
cat Snakefile
rule create_r_environment:
conda:
"r.yaml"
script:
"r-dependencies.R"
Execute:
snakemake -j1 --use-conda -R create_r_environment
Building DAG of jobs...
Creating conda environment r.yaml...
Downloading and installing remote packages.
CreateCondaEnvironmentException:
Could not create conda environment from /home/dario/Downloads/r.yaml:
# >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<
Traceback (most recent call last):
File "/home/dario/miniconda3/lib/python3.7/site-packages/conda/exceptions.py", line 1079, in __call__
return func(*args, **kwargs)
File "/home/dario/miniconda3/lib/python3.7/site-packages/conda_env/cli/main.py", line 80, in do_call
exit_code = getattr(module, func_name)(args, parser)
File "/home/dario/miniconda3/lib/python3.7/site-packages/conda_env/cli/main_create.py", line 80, in execute
directory=os.getcwd())
File "/home/dario/miniconda3/lib/python3.7/site-packages/conda_env/specs/__init__.py", line 40, in detect
if spec.can_handle():
File "/home/dario/miniconda3/lib/python3.7/site-packages/conda_env/specs/yaml_file.py", line 18, in can_handle
self._environment = env.from_file(self.filename)
File "/home/dario/miniconda3/lib/python3.7/site-packages/conda_env/env.py", line 151, in from_file
return from_yaml(yamlstr, filename=filename)
File "/home/dario/miniconda3/lib/python3.7/site-packages/conda_env/env.py", line 137, in from_yaml
data = validate_keys(data, kwargs)
File "/home/dario/miniconda3/lib/python3.7/site-packages/conda_env/env.py", line 35, in validate_keys
new_data = data.copy() if data else {}
AttributeError: 'str' object has no attribute 'copy'
`$ /home/dario/miniconda3/bin/conda-env create --file /home/dario/Downloads/.snakemake/conda/095b0ca2.yaml --prefix /home/dario/Downloads/.snakemake/conda/095b0ca2`
environment variables:
CIO_TEST=<not set>
CMAKE_PREFIX_PATH=/home/dario/miniconda3/envs/tritume:/home/dario/miniconda3/envs/tritum
e/x86_64-conda-linux-gnu/sysroot/usr
CONDA_AUTO_UPDATE_CONDA=false
CONDA_BUILD_SYSROOT=/home/dario/miniconda3/envs/tritume/x86_64-conda-linux-gnu/sysroot
CONDA_DEFAULT_ENV=tritume
CONDA_EXE=/home/dario/miniconda3/bin/conda
CONDA_PREFIX=/home/dario/miniconda3/envs/tritume
CONDA_PROMPT_MODIFIER=(tritume)
CONDA_PYTHON_EXE=/home/dario/miniconda3/bin/python
CONDA_ROOT=/home/dario/miniconda3
CONDA_SHLVL=1
DEFAULTS_PATH=/usr/share/gconf/ubuntu.default.path
MANDATORY_PATH=/usr/share/gconf/ubuntu.mandatory.path
PATH=/home/dario/miniconda3/envs/tritume/bin:/home/dario/miniconda3/condabi
n:/opt/gradle/gradle-5.2/bin:/home/dario/.local/share/umake/bin:/home/
dario/.local/bin:/home/dario/bin:/opt/gradle/gradle-5.2/bin:/usr/local
/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/loc
al/games:/snap/bin:/usr/lib/jvm/java-10-oracle/bin:/usr/lib/jvm/java-1
0-oracle/db/bin
REQUESTS_CA_BUNDLE=<not set>
SSL_CERT_FILE=<not set>
WINDOWPATH=2
active environment : tritume
active env location : /home/dario/miniconda3/envs/tritume
shell level : 1
user config file : /home/dario/.condarc
populated config files : /home/dario/.condarc
conda version : 4.8.3
conda-build version : not installed
python version : 3.7.6.final.0
virtual packages : __glibc=2.27
base environment : /home/dario/miniconda3 (writable)
channel URLs : https://conda.anaconda.org/conda-forge/linux-64
https://conda.anaconda.org/conda-forge/noarch
https://conda.anaconda.org/bioconda/linux-64
https://conda.anaconda.org/bioconda/noarch
https://repo.anaconda.com/pkgs/main/linux-64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/r/linux-64
https://repo.anaconda.com/pkgs/r/noarch
package cache : /home/dario/miniconda3/pkgs
/home/dario/.conda/pkgs
envs directories : /home/dario/miniconda3/envs
/home/dario/.conda/envs
platform : linux-64
user-agent : conda/4.8.3 requests/2.22.0 CPython/3.7.6 Linux/4.15.0-91-generic ubuntu/18.04.4 glibc/2.27
UID:GID : 1001:1001
netrc file : None
offline mode : False
An unexpected error has occurred. Conda has prepared the above report.
If submitted, this report will be used by core maintainers to improve
future releases of conda.
Would you like conda to send this report to the core maintainers?
[y/N]:
Timeout reached. No report sent.
File "/home/dario/miniconda3/envs/tritume/lib/python3.6/site-packages/snakemake/deployment/conda.py", line 320, in create
Anyway, your error says:
... r-dependencies.R' returned non-zero exit status 1
What do you have in r-dependencies.R?
Python newbie here. I have encountered a permission problem with anaconda. Everything runs ok, but I do not seem to be able to update conda, create new environments or install new packages.
When I try to update (conda update conda) it I get:
Fetching package metadata ..... An unexpected error has occurred.
Please consider posting the following information to the
conda GitHub issue tracker at:
https://github.com/conda/conda/issues
Current conda version:
platform : osx-64
conda version : 4.3.29
conda is private : False
conda-env version : 4.3.29
conda-build version : not installed
python version : 2.7.11.final.0
requests version : 2.14.2
root environment : /anaconda (writable)
default environment : /anaconda
envs directories : /anaconda/Users/Tina/.conda/envs
package cache : /anaconda/Users/Tina/.conda/pkgs
channel URLs : https://conda.anaconda.org/anaconda-fusion/osx-64
https://conda.anaconda.org/anaconda-fusion/noarch
https://repo.continuum.io/pkgs/main/osx-64
https://repo.continuum.io/pkgs/main/noarch
https://repo.continuum.io/pkgs/free/osx-64
https://repo.continuum.io/pkgs/free/noarch
https://repo.continuum.io/pkgs/r/osx-64
https://repo.continuum.io/pkgs/r/noarch
https://repo.continuum.io/pkgs/pro/osx-64
https://repo.continuum.io/pkgs/pro/noarch
config file : /Users/Tina/.condarc
netrc file : None
offline mode : False
user-agent : conda/4.3.29 requests/2.14.2 CPython/2.7.11 Darwin/15.5.0 OSX/10.11.5
UID:GID : 501:20
$ /anaconda/bin/conda update conda
Traceback (most recent call last):
File "/anaconda/lib/python2.7/site-packages/conda/exceptions.py", line 640, in conda_exception_handler
return_value = func(*args, **kwargs)
File "/anaconda/lib/python2.7/site-packages/conda/cli/main.py", line 140, in _main
exit_code = args.func(args, p)
File "/anaconda/lib/python2.7/site-packages/conda/cli/main_update.py", line 65, in execute
install(args, parser, 'update')
File "/anaconda/lib/python2.7/site-packages/conda/cli/install.py", line 231, in install
unknown=index_args['unknown'], prefix=prefix)
File "/anaconda/lib/python2.7/site-packages/conda/core/index.py", line 101, in get_index
index = fetch_index(channel_priority_map, use_cache=use_cache)
File "/anaconda/lib/python2.7/site-packages/conda/core/index.py", line 120, in fetch_index
repodatas = collect_all_repodata(use_cache, tasks)
File "/anaconda/lib/python2.7/site-packages/conda/core/repodata.py", line 75, in collect_all_repodata
repodatas = _collect_repodatas_serial(use_cache, tasks)
File "/anaconda/lib/python2.7/site-packages/conda/core/repodata.py", line 485, in _collect_repodatas_serial
for url, schan, pri in tasks]
File "/anaconda/lib/python2.7/site-packages/conda/core/repodata.py", line 115, in func
res = f(*args, **kwargs)
File "/anaconda/lib/python2.7/site-packages/conda/core/repodata.py", line 467, in fetch_repodata
touch(cache_path)
File "/anaconda/lib/python2.7/site-packages/conda/gateways/disk/update.py", line 64, in touch
utime(path, None)
OSError: [Errno 13] Permission denied: '/anaconda/pkgs/cache/9cd9d6b5.json'```
I get the same error when trying to install seaborn or creating an environment. I am reluctant to use sudo because I do not want to break things.
I do not understand what is going on here, so any help would be highly appreciated.
Thanks so much;
T
The user that you are using to run conda update conda does not have write permission on /anaconda/pkgs/cache/.
If you don't want to manage anaconda as the superuser, I would recommend that you create a new user group (i.e. anaconda_admin) and run:
sudo groupadd anaconda_admin
sudo chown -R :anaconda_admin /anaconda
Then you will need to ensure that permissions are something like:
sudo chmod -R 775 /anaconda
And finally that your user is in the anaconda_admin group:
sudo adduser <<<your_user>>> anaconda_admin
For humble Windows users that cannot use sudo: You have to open the conda console as Administrator by right clicking on the console icon and then select run as administrator. Then conda update conda should work fine.
You ought to use sudo in order to write certain files into system. It is perfectly fine and will not break you OS, unless you work with sophisticated and rudimentary packages and installers (conda and python libraries are absolutely fine).
sudo conda update conda should do the thing not only with updating conda, but also with other dependencies and packages you wish to install.
In short, the installer tries to write a file into a certain directory (or modify a file in a directory) that it has not got an access to. With sudo you make them do that as you run it with appended priviliges.
Using pip install for any module apparently on my Ubuntu 16.04 system with python 2.7.11+ throws this error:
TypeError: unsupported operand type(s) for -=: 'Retry' and 'int'
What is wrong with pip? How could I reinstall it, if necessary?
Update: Full traceback is below
sunny#sunny:~$ pip install requests
Collecting requests
Exception:
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 209, in main
status = self.run(options, args)
File "/usr/lib/python2.7/dist-packages/pip/commands/install.py", line 328, in run
wb.build(autobuilding=True)
File "/usr/lib/python2.7/dist-packages/pip/wheel.py", line 748, in build
self.requirement_set.prepare_files(self.finder)
File "/usr/lib/python2.7/dist-packages/pip/req/req_set.py", line 360, in prepare_files
ignore_dependencies=self.ignore_dependencies))
File "/usr/lib/python2.7/dist-packages/pip/req/req_set.py", line 512, in _prepare_file
finder, self.upgrade, require_hashes)
File "/usr/lib/python2.7/dist-packages/pip/req/req_install.py", line 273, in populate_link
self.link = finder.find_requirement(self, upgrade)
File "/usr/lib/python2.7/dist-packages/pip/index.py", line 442, in find_requirement
all_candidates = self.find_all_candidates(req.name)
File "/usr/lib/python2.7/dist-packages/pip/index.py", line 400, in find_all_candidates
for page in self._get_pages(url_locations, project_name):
File "/usr/lib/python2.7/dist-packages/pip/index.py", line 545, in _get_pages
page = self._get_page(location)
File "/usr/lib/python2.7/dist-packages/pip/index.py", line 648, in _get_page
return HTMLPage.get_page(link, session=self.session)
File "/usr/lib/python2.7/dist-packages/pip/index.py", line 757, in get_page
"Cache-Control": "max-age=600",
File "/usr/share/python-wheels/requests-2.9.1-py2.py3-none-any.whl/requests/sessions.py", line 480, in get
return self.request('GET', url, **kwargs)
File "/usr/lib/python2.7/dist-packages/pip/download.py", line 378, in request
return super(PipSession, self).request(method, url, *args, **kwargs)
File "/usr/share/python-wheels/requests-2.9.1-py2.py3-none-any.whl/requests/sessions.py", line 468, in request
resp = self.send(prep, **send_kwargs)
File "/usr/share/python-wheels/requests-2.9.1-py2.py3-none-any.whl/requests/sessions.py", line 576, in send
r = adapter.send(request, **kwargs)
File "/usr/share/python-wheels/CacheControl-0.11.5-py2.py3-none-any.whl/cachecontrol/adapter.py", line 46, in send
resp = super(CacheControlAdapter, self).send(request, **kw)
File "/usr/share/python-wheels/requests-2.9.1-py2.py3-none-any.whl/requests/adapters.py", line 376, in send
timeout=timeout
File "/usr/share/python-wheels/urllib3-1.13.1-py2.py3-none-any.whl/urllib3/connectionpool.py", line 610, in urlopen
_stacktrace=sys.exc_info()[2])
File "/usr/share/python-wheels/urllib3-1.13.1-py2.py3-none-any.whl/urllib3/util/retry.py", line 228, in increment
total -= 1
TypeError: unsupported operand type(s) for -=: 'Retry' and 'int'
Ubuntu comes with a version of PIP from precambrian and that's how you have to upgrade it if you do not want to spend hours and hours debugging pip related issues.
apt-get remove python-pip python3-pip
wget https://bootstrap.pypa.io/get-pip.py
python get-pip.py
python3 get-pip.py
As you observed I included information for both Python 2.x and 3.x
If you are behind a proxy, you must do some extra configuration steps before starting the installation. You must set the environment variable http_proxy to the proxy address. Using bash this is accomplished with the command
export http_proxy="http://user:pass#my.site:port/"
You can also provide the
--proxy=[user:pass#]url:port
parameter to pip. The [user:pass#] portion is optional.
Updating setuptools has worked out fine for me.
sudo pip install --upgrade setuptools
First of all, this problem exists because of network issues, and uninstalling and re-installing everything won't be of much help. Probably you are behind proxy, and in that case you need to set proxy.
But in my case, I was facing the problem because I wasn't behind proxy. Generally, I work behind proxy, but when working from home, I set the proxy to None in Network settings.
But I was still getting the same errors even after removing the proxy settings.
So, when I did type
env | grep proxy
I found something like this :
http_proxy=http://127.0.0.1:1234/
And this was the reason I was still getting the very same error, even when I thought I had removed the proxy settings.
To unset this proxy, type
unset http_proxy
Follow the same approach for all the other entries, such as https_proxy.
What happens here is that the the vendored versions of request/urllib3 clash when imported in two different places (same code, but different names). If you then have a network error, it doesn't retry to get the wheel, but fails with the above error. See here for a deeper dive into this error.
For the solution with system pip, see above.
If you have this problem in a virtualenv built by python -m venv (which still copies the wheels from /usr/share/python-wheels, even if you have pip installed separately), the easiest way to "fix" it seems to be:
create the virtualenv: /usr/bin/python3.6 -m venv ...
install requests into the environment (this might raise the above error): <venv>/bin/pip install requests
remove the copied versions of requests which would be used by pip: rm <venv>/share/python-wheels/{requests,chardet,urllib3}-*.whl
Now a <venv>/bin/pip uses the installed version of requests which has urllib3 vendored.
port 443 is not open, just allow custom tcp port 443 if on AWS else open the port 443 for the outbound connections ...
Just upgrade pip worked for me:
pip install --upgrade pip
I have the same problem when installing a RaspberryPI TFT from Adafruit with pitft.sh / adafruit-pitft.sh.
I am not happy about coding-styles with errors from somewhere to be interpreted somehow - as could be seen by the previous answers.
Remark: The type error exception of retry.py is obviously a bug, caused by an unappropriate assignement and calculation of an instance of the class Reply to an int with the default value of 10 - somewhere in the code...
Should be fixed either by adding an inplace-operator, or fixing the erroneous assignment.
So tried to analyse and patch the error itself first. The actual error in my case case is the same - retry.py called by pip.
The installation script adafruit-pitft.sh / pitft.sh tries to apply urllib3 which itself tries to install nested dependencies by pip, so the same error.
https://github.com/adafruit/Raspberry-Pi-Installer-Scripts/blob/master/adafruit-pitft.sh
https://github.com/adafruit/Raspberry-Pi-Installer-Scripts
adafruit-pitft.sh # or pitft.sh
...
_stacktrace=sys.exc_info()[2])
File "/usr/share/python-wheels/urllib3-1.13.1-py2.py3 none-any.whl/urllib3/util/retry.py", line 228, in increment
total -= 1
TypeError: unsupported operand type(s) for -=: 'Retry' and 'int'
For the current distribution(based on debian-9.6.0/stretch):
File "/usr/share/python-wheels/urllib3-1.19.1-py2.py3-none-any.whl/urllib3/util/retry.py", line 315, in increment
total -= 1
TypeError: unsupported operand type(s) for -=: 'Retry' and 'int'
The following - dirty *:) - patch enables a sounding error trace:
# File: retry.py - in *def increment(self, ..* about line 315
# original: total = self.total
# patch: quick-and-dirty-fix
# START:
if isinstance(self.total, Retry):
self.total = self.total.total
if type(self.total) is not int:
self.total = 2 # default is 10
# END:
# continue with original:
total = self.total
if total is not None:
total -= 1
connect = self.connect
read = self.read
redirect = self.redirect
cause = 'unknown'
status = None
redirect_location = None
if error and self._is_connection_error(error):
# Connect retry?
if connect is False:
raise six.reraise(type(error), error, _stacktrace)
elif connect is not None:
connect -= 1
The sounding output with the temporary patch is(displayed twice...?):
Retrying (Retry(total=1, connect=None, read=None, redirect=None)) after connection broken by 'ConnectTimeoutError(<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at/
Retrying (Retry(total=0, connect=None, read=None, redirect=None)) after connection broken by
'ConnectTimeoutError(<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at/
Could not find a version that satisfies the requirement evdev (from versions: )
No matching distribution found for evdev
WARNING : Pip failed to install software!
So in my case actually two things cause the error, this may vary in other environments:
Missing evdev => try to install
Failed to connect a repo/dist containing evdev in order to download. => finally give it up
My installation environment is offline from an internal debian+raspbian mirror, thus
do not want to set the proxy...
So I proceeded by manual installation of the missing component evdev:
download evdev from PyPI(or e.g. from github.com):
https://pypi.org/project/evdev/
https://files.pythonhosted.org/packages/7e/53/374b82dd2ccec240b7388c65075391147524255466651a14340615aabb5f/evdev-1.1.2.tar.gz
Unpack and install manually as root user - for all local accounts, so detected as installed:
sudo su -
tar xf evdev-1.1.2.tar.gz
cd evdev-1.1.2
python setup.py install
Call install script again:
adafruit-pitft.sh # or pitft.sh
...Answer dialogues...
...that's it.
If you proceed online by direct PyPI access:
check your routing + firewall for access to pypi.org
set a proxy if required (http_proxy/https_proxy)
And it works..
Hope this helps in other cases too.
Arno-Can Uestuensoez
----------------------------------------------
See also: issue - 35334: https://bugs.python.org/issue35334
----------------------------------------------
See now also: issue - 1486: https://github.com/urllib3/urllib3/issues/1486
for file: https://github.com/urllib3/urllib3/blob/master/src/urllib3/util/retry.py
check for network issues, to bypass the exception case code
In my case, I was using a custom index, that index had no route and such would trigger the exception case code. The exception case bug still exists and still masks the real issue, however I was able to work around this by testing the connectivity with other tools such as nc -vzw1 myindex.example.org 443 and retrying when the network was up.
I was facing similar issue while trying to install awscli tool on ec2 instance.
I changed security group to allow port 443 inbound and outbound access and that solved the issue for me.
I got this error when I was trying to create a virtualenv with command virtualenv myVirtualEnv. I just added a sudo before the command; it solved everything.
Solution:
1. sudo apt remove python-pip
2. pip3 install pip (or install pip by get-pip.py)
Why:
This error occurred on pip 8.0.1 which installed by apt-get. And happened only when your network is unstable.
If you have a pip installed with apt, it hides the pip you installed by other ways, so you should remove the apt one first.
I disconnected the network and tested 8.0.1, 9.0.3, 10.x the 3 versions installed with pip3 or get-pip.py, no error occurred. So, I think only the apt version of pip 8.0.1 has that bug, the others is ok.
I tried the solution answered above:
apt-get remove python-pip python3-pip
wget https://bootstrap.pypa.io/get-pip.py
python get-pip.py
python3 get-pip.py
When I tried
python get-pip.py
python3 get-pip.py
I got this message
Could not install packages due to an EnvironmentError:
[Errno 13] Permission denied: /usr/bin/pip3 Consider using the --user
option or check the permissions.
I did the following and it works
python3 -m venv env
source ./env/bin/activate
Sudo apt-get update
apt-get remove python-pip python3-pip
wget https://bootstrap.pypa.io/get-pip.py
python get-pip.py
python3 get-pip.py
pip3 install pip
sudo easy_install pip
pip install --upgrade pip
In my case, i had opened Pycharm in sudo mode, and was running pip install nltk in pycharm terminal which showed this error. running with sudo pip install solves the error.
I also had this issue. Initially, a proxy was set and work fine. Then I connected to a network where it doesn't go through a proxy. After unsetting proxy pip again get works.
unset http_proxy; unset http_prox; unset HTTP_PROXY; unset HTTPS_PROXY
Bizarrely if I remove the proxy from the environment and add it to the command line it works for me. For example to upgrade pip itself:
env http_proxy= https_proxy= pip install pip --upgrade --proxy 'http://proxy-url:80'
My issue was having the proxy in the environment. It seems that pip only honors the one in argument.
This is the working solution to this problem I found.
sudo apt-get clean
cd /var/lib/apt
sudo mv lists lists.old
sudo mkdir -p lists/partial
sudo apt-get clean
sudo apt-get update
For myself, it turns out that wlan0 was down, which resulted in me being unable to connect out. So, ensuring that wlan0 was up, allowed pip / pip3 to work without issue.
fixed it temporary:
pip install requests -i http://a.b.com/pypi/simple --trusted-host a.b.com
fixed it permanent:
linux OS: add these in ~/.pip/pip.conf(create it if no exist)
[global]
index-url = http://a.b.com/pypi/simple
[install]
trusted-host = a.b.com
ps: http://a.b.com/pypi/simple your proxy_http_address
On RedHat Enterprise 7, trying to install node.js inside of a nodeenv (0.13.6) in a Python virtual environment (Python 2.7). When I do nodeenv -p, I get OSError: Command make --jobs=2 failed with error code 2...googling, the only reference to this is here. Not super useful for me, because I am already trying to install the newest version of node (4.2.1). Full trace of this is below:
$ nodeenv -p
* Install node (4.2.1..Traceback (most recent call last):
File "/usr/local/pythonenvs/producer/bin/nodeenv", line 11, in <module>
sys.exit(main())
File "/usr/local/pythonenvs/producer/lib/python2.7/site-packages/nodeenv.py", line 891, in main
create_environment(env_dir, opt)
File "/usr/local/pythonenvs/producer/lib/python2.7/site-packages/nodeenv.py", line 732, in create_environment
install_node(env_dir, src_dir, opt)
File "/usr/local/pythonenvs/producer/lib/python2.7/site-packages/nodeenv.py", line 608, in install_node
build_node_from_src(env_dir, src_dir, node_src_dir, opt)
File "/usr/local/pythonenvs/producer/lib/python2.7/site-packages/nodeenv.py", line 577, in build_node_from_src
callit([make_cmd] + make_opts, opt.verbose, True, node_src_dir, env)
File "/usr/local/pythonenvs/producer/lib/python2.7/site-packages/nodeenv.py", line 461, in callit
% (cmd_desc, proc.returncode))
OSError: Command make --jobs=2 failed with error code 2
I then tried to install from prebuilt, using the instructions in this GitHub issue.
nodeenv -p --prebuilt
That seemed to work...
* Install node (4.2.1... done.
* Appending data to /usr/local/pythonenvs/producer/bin/activate
Except nothing actually installed -- tab completing shows no node or npm install (I have deactivated and re-activated the virtual environment):
$ no
nodeenv nohup nologin notify-send
$ np
$ nproc
My other installs worked with the same instructions, so I'm at a loss for debugging this. Any hints or suggestions? If this is a permissions issue, where do I need to set / change them? The user already owns the virtual environment directory...
Okay, so I don't have a solution to the root cause (I suspect some sort of issue / conflict with make on my server), but I managed to get it installed via --prebuilt. I had to manually delete the node.js source from /usr/local/pythonenvs/producer/src/node-v4.2.1/, because the --prebuilt option was trying to copy those as if they were binaries. After deleting the directory, I downloaded / extracted from nodejs.org into the virtual environment's src directory. Then, the nodeenv -p --prebuilt command works fine.
This is the follow up on my previous question.
I have an OSX Mavericks machine where I installed Python 2.7 and Babel 1.3 using MacPorts.
The problem is that it seems I have at least two versions of babel on that machine and except for the one I installed using MacPort, the other does not have the CLDR locale data files installed, so when I execute the following command in my application directory :
pybabel extract -o ./locale/messages.pot ./
I get this error message :
[...]
writing PO template file to ./locale/messages.pot
Traceback (most recent call last):
File "/usr/local/bin/pybabel", line 8, in <module>
load_entry_point('Babel==2.0-dev-20131218', 'console_scripts', 'pybabel')()
File "/Library/Python/2.7/site-packages/Babel-2.0_dev_20131218-py2.7.egg/babel/messages/frontend.py", line 1151, in main
return CommandLineInterface().run(sys.argv)
File "/Library/Python/2.7/site-packages/Babel-2.0_dev_20131218-py2.7.egg/babel/messages/frontend.py", line 665, in run
return getattr(self, cmdname)(args[1:])
File "/Library/Python/2.7/site-packages/Babel-2.0_dev_20131218-py2.7.egg/babel/messages/frontend.py", line 947, in extract
sort_by_file=options.sort_by_file)
File "/Library/Python/2.7/site-packages/Babel-2.0_dev_20131218-py2.7.egg/babel/messages/pofile.py", line 427, in write_po
messages = list(catalog)
File "/Library/Python/2.7/site-packages/Babel-2.0_dev_20131218-py2.7.egg/babel/messages/catalog.py", line 552, in __iter__
for name, value in self.mime_headers:
File "/Library/Python/2.7/site-packages/Babel-2.0_dev_20131218-py2.7.egg/babel/messages/catalog.py", line 337, in _get_mime_headers
locale='en')))
File "/Library/Python/2.7/site-packages/Babel-2.0_dev_20131218-py2.7.egg/babel/dates.py", line 609, in format_datetime
locale = Locale.parse(locale)
File "/Library/Python/2.7/site-packages/Babel-2.0_dev_20131218-py2.7.egg/babel/core.py", line 284, in parse
language = get_global('language_aliases').get(language, language)
File "/Library/Python/2.7/site-packages/Babel-2.0_dev_20131218-py2.7.egg/babel/core.py", line 53, in get_global
_raise_no_data_error()
File "/Library/Python/2.7/site-packages/Babel-2.0_dev_20131218-py2.7.egg/babel/core.py", line 25, in _raise_no_data_error
raise RuntimeError('The babel data files are not available. '
RuntimeError: The babel data files are not available. This usually happens because you are using a source checkout from Babel and you did not build the data files. Just make sure to run "python setup.py import_cldr" before installing the library.
I found Babel installed at these locations on my machine :
> cd /Library/Python/2.7/site-packages/Babel-2.0_dev_20131218-py2.7.egg/
> ls
EGG-INFO babel
> cd babel
> ls
__init__.py core.py localedata.py numbers.py support.py
__init__.pyc core.pyc localedata.pyc numbers.pyc support.pyc
_compat.py dates.py localtime plural.py util.py
_compat.pyc dates.pyc messages plural.pyc util.pyc
> cd /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/babel
> ls
__init__.py core.py localedata numbers.py support.py
__init__.pyc core.pyc localedata.py numbers.pyc support.pyc
_compat.py dates.py localedata.pyc plural.py util.py
_compat.pyc dates.pyc localtime plural.pyc util.pyc
global.dat messages
These are the various paths defined in my environment :
> echo $PATH
/opt/local/bin:/opt/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/local/git/bin:/Users/XXXXX/.rvm/bin
> echo $PYTHONPATH
> echo $DYLD_FALLBACK_LIBRARY_PATH
> echo $DYLD_LIBRARY_PATH
> echo $DYLD_FALLBACK_FRAMEWORK_PATH
> echo $DYLD_FRAMEWORK_PATH
>
pybabel is an alias i created using the following command :
> alias pybabel='pybabel-2.7'
How can I make babel see the Babel version in /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/babel?
I tried setting the PYTHONPATH but made a mistake ...
I created a symbolic link in the egg archive pybabel was using to the localedata/ in the MacPorts version and it seems to have worked :
> cd /Library/Python/2.7/site-packages/Babel-2.0_dev_20131218-py2.7.egg/babel
> ln -s /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/babel/localedata localedata
ln: localedata: Permission denied
> sudo ln -s /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/babel/localedata localedata
Password:
>
If someone has better suggestions or advices, please leave a comment or another answer. Thank you.
[follow-up 2014-01-09]--------------------------------------------------------------------------------------------------------------
I still could not get the "{% trans %}" tags translated, I found here that you need to explicitly use the -F option on pybabel like that (even if the babel.cfg file is in the current directory!):
> pybabel -v extract -F babel.cfg -o ./locale/messages.pot ./
And to help here is the content of my babel.cfg file :
[jinja2: **/templates/**.html]
encoding = utf-8
[python: python/*.py]
[extractors]
jinja2 = jinja2.ext:babel_extract
But I got "ImportError: No module named ..." error messages because I did not have jinja2 and markupsafe properly configured. So, I added two symlinks :
> sudo ln -s /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/jinja2 /Library/Python/2.7/site-packages/jinja2
> sudo ln -s /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/markupsafe /Library/Python/2.7/site-packages/markupsafe
And it finally worked! I got the content of the "{% trans %}" tags in my messages.pot file!
Yet, I wonder if it is he best solution ...