Develop Reference

pip fails with SSLError on MacOS because of outdated certificate - how to recover?

When searching or installing packages with pip on MacOS (10.11.6), I am getting an SSL error similar to the one reproduced below.
The problem occurs for both python2 and python3. I acquired the versions (2.7.14 and 3.6.5) via www.python.org as Frameworks. The version of pip was 9.0.1. The problem is new (April 2018) and seems related to an update of OSX.
How to recover from this problem?
$ pip2 search numpy
Exception:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip/basecommand.py", line 215, in main
status = self.run(options, args)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip/commands/search.py", line 45, in run
pypi_hits = self.search(query, options)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip/commands/search.py", line 62, in search
hits = pypi.search({'name': query, 'summary': query}, 'or')
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xmlrpclib.py", line 1243, in __call__
return self.__send(self.__name, args)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xmlrpclib.py", line 1602, in __request
verbose=self.__verbose
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip/download.py", line 775, in request
headers=headers, stream=True)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 522, in post
return self.request('POST', url, data=data, json=json, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip/download.py", line 386, in request
return super(PipSession, self).request(method, url, *args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 475, in request
resp = self.send(prep, **send_kwargs)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 596, in send
r = adapter.send(request, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip/_vendor/cachecontrol/adapter.py", line 47, in send
resp = super(CacheControlAdapter, self).send(request, **kw)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip/_vendor/requests/adapters.py", line 497, in send
raise SSLError(e, request=request)
SSLError: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:661)

There are two ways to recover from this problem.
Solution 1. As can be read here, Python.org sites stopped the support for older TLS versions, which breaks pip < 9.0.3. A new release of pip was provided, but because one cannot simply update via the broken pip, one has to bootstrap the installation. This worked for me:
# For python2
curl https://bootstrap.pypa.io/get-pip.py | python2
# For python3 (haven't tested this myself)
curl https://bootstrap.pypa.io/get-pip.py | python3
Credits for these commands go back to this SO post.
Solution 2. For Python 3.6 frameworks, a script Install Certificates.command is deployed and installed under /Applications/Python 3.6/. Executing it resolved the problem, but only for pip3.

You just need to reinstall Python
brew reinstall python

Connection reset by peer w/ Python's Request Module. Curl and Postman work fine.

Python 2.7.10
I'm making a very basic API call to my gitlab repo.
I can successfully curl the endpoint and I can succesfully call it with Postman
However it fails when trying to use Python's request module.
Curl Works:
curl https://gitlab.example.com/api/v4/projects
Outputs the expected JSON response from the endpoint (no auth needed)
Postman I also get the expected result:
However when I submit via requests:
[Cert is self signed, hence the verify=False]
r = requests.get("https://gitlab.example.com/api/v4/projects", verify=False)
Error thrown:
Traceback (most recent call last):
File "./repo_analyzer.py", line 36, in <module>
r = requests.get("https://xxxxxxxxxxxx/api/v4/projects", verify=False)
File "/Library/Python/2.7/site-packages/requests/api.py", line 72, in get
return request('get', url, params=params, **kwargs)
File "/Library/Python/2.7/site-packages/requests/api.py", line 58, in request
return session.request(method=method, url=url, **kwargs)
File "/Library/Python/2.7/site-packages/requests/sessions.py", line 508, in request
resp = self.send(prep, **send_kwargs)
File "/Library/Python/2.7/site-packages/requests/sessions.py", line 618, in send
r = adapter.send(request, **kwargs)
File "/Library/Python/2.7/site-packages/requests/adapters.py", line 490, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', error(54, 'Connection reset by peer'))
I'm a bit at a loss on why it's not working with python, but works otherwise.

After searching a ton of blogs and posts about this, it seemed that I needed to update requests[security] package through pip.
I was getting errors running both pip install -U requests[security] and sudo pip install -U requests[security] so I started from scratch:
Downloaded Python 2.7 and reinstalled from the official site
pip install requests
pip install -U requests
pip install -U requests[security]
After that, the calls were successful.

Robot Framework test scripts fail with SSLError

I have written Robot Framework test scripts in .tsv format to test web-services/APIs. Everything was working fine until today (probably because of the new updates of Robot Framework) when I started to get the following error:
SSLError: ("bad handshake: SysCallError(-1, 'Unexpected EOF')",)
This error keeps popping up for the following code in a test script:
${headers}= Create Dictionary Content-Type application/json Accept application/json
RequestsKeywords.Get Request httpbin ${url} headers=${headers} //ERROR SHOWS FOR THIS STATEMENT
I did get a detailed traceback for this error which is as follows:
Traceback (most recent call last):
File "/Library/Python/2.7/site-packages/RequestsLibrary/RequestsKeywords.py", line 298, in get_request
session, uri, params, headers, redir, timeout)
File "/Library/Python/2.7/site-packages/RequestsLibrary/RequestsKeywords.py", line 801, in _get_request
cookies=self.cookies)
File "/Library/Python/2.7/site-packages/requests/sessions.py", line 480, in get
return self.request('GET', url, **kwargs)
File "/Library/Python/2.7/site-packages/requests/sessions.py", line 468, in request
resp = self.send(prep, **send_kwargs)
File "/Library/Python/2.7/site-packages/requests/sessions.py", line 576, in send
r = adapter.send(request, **kwargs)
File "/Library/Python/2.7/site-packages/requests/adapters.py", line 447, in send
raise SSLError(e, request=request)
My system configuration:
Mac OS X (10.11.3)
Python (2.7.10)
openssl (1.0.2f)
requests (2.9.1)
robotframework (3.0)
robotframework-httplibrary (0.4.2)
robotframework-requests (0.4.4)
robotframework-ride (1.5.2.1)
robotframework-sshlibrary (2.1.2)
pyOpenSSL (0.15.1)
How do I resolve this issue?

You're all up to date, so there are two possibilities:
The handshake isn't going smoothly because of a break in the trust chain. Start from the bottom and work up. Are you testing this on a local area network? Is the certificate up to date? Can you access the site from your location and others without an error? SSL labs showing anything up? What about firewall rules?
There's a bug in the recent update. OpenSSL reports in my experience are caused by connection issues rather than obscure software problems - this is the less likely of the two.

Try installing requests[security] instead of requests. It uses PyOpenSSL, which is better than OpenSSL: pip install requests[security] vs pip install requests: Difference

'error: Error -5 while decompressing data: incomplete or truncated stream' when installing pip package

I'm having the following error when running pip install Pillow==2.9.0 in a virtualenv: error: Error -5 while decompressing data: incomplete or truncated stream
Other packages install/uninstall fine, it just seems to affects Pillow 2.9.0. It doesn't seem to matter what virtualenv I'm in (or not).
Downloading a source tarball and installing from that worked, but since this is on a build server that's not an ideal workaround as I want to rely on pip install -r requirements.txt
Versions:
pip --version: pip 7.1.0 from /usr/local/lib/python2.7/site-packages (python 2.7)
python --version: Python 2.7.10
The full traceback is:
Collecting Pillow==2.9.0
/mnt/jenkins/jobA/workspace/.pyenv/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
Exception:
Traceback (most recent call last):
File "/mnt/jenkins/jobA/workspace/.pyenv/local/lib/python2.7/site-packages/pip/basecommand.py", line 223, in main
status = self.run(options, args)
File "/mnt/jenkins/jobA/workspace/.pyenv/local/lib/python2.7/site-packages/pip/commands/install.py", line 282, in run
requirement_set.prepare_files(finder)
File "/mnt/jenkins/jobA/workspace/.pyenv/local/lib/python2.7/site-packages/pip/req/req_set.py", line 334, in prepare_files
functools.partial(self._prepare_file, finder))
File "/mnt/jenkins/jobA/workspace/.pyenv/local/lib/python2.7/site-packages/pip/req/req_set.py", line 321, in _walk_req_to_install
more_reqs = handler(req_to_install)
File "/mnt/jenkins/jobA/workspace/.pyenv/local/lib/python2.7/site-packages/pip/req/req_set.py", line 491, in _prepare_file
session=self.session)
File "/mnt/jenkins/jobA/workspace/.pyenv/local/lib/python2.7/site-packages/pip/download.py", line 825, in unpack_url
session,
File "/mnt/jenkins/jobA/workspace/.pyenv/local/lib/python2.7/site-packages/pip/download.py", line 673, in unpack_http_url
from_path, content_type = _download_http_url(link, session, temp_dir)
File "/mnt/jenkins/jobA/workspace/.pyenv/local/lib/python2.7/site-packages/pip/download.py", line 857, in _download_http_url
stream=True,
File "/mnt/jenkins/jobA/workspace/.pyenv/local/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 477, in get
return self.request('GET', url, **kwargs)
File "/mnt/jenkins/jobA/workspace/.pyenv/local/lib/python2.7/site-packages/pip/download.py", line 373, in request
return super(PipSession, self).request(method, url, *args, **kwargs)
File "/mnt/jenkins/jobA/workspace/.pyenv/local/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 465, in request
resp = self.send(prep, **send_kwargs)
File "/mnt/jenkins/jobA/workspace/.pyenv/local/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 573, in send
r = adapter.send(request, **kwargs)
File "/mnt/jenkins/jobA/workspace/.pyenv/local/lib/python2.7/site-packages/pip/_vendor/cachecontrol/adapter.py", line 36, in send
cached_response = self.controller.cached_request(request)
File "/mnt/jenkins/jobA/workspace/.pyenv/local/lib/python2.7/site-packages/pip/_vendor/cachecontrol/controller.py", line 102, in cached_request
resp = self.serializer.loads(request, self.cache.get(cache_url))
File "/mnt/jenkins/jobA/workspace/.pyenv/local/lib/python2.7/site-packages/pip/_vendor/cachecontrol/serialize.py", line 108, in loads
return getattr(self, "_loads_v{0}".format(ver))(request, data)
File "/mnt/jenkins/jobA/workspace/.pyenv/local/lib/python2.7/site-packages/pip/_vendor/cachecontrol/serialize.py", line 164, in _loads_v2
cached = json.loads(zlib.decompress(data).decode("utf8"))
error: Error -5 while decompressing data: incomplete or truncated stream

Turns out that there was a corrupt entry in pip's local cache (located in my case, and by default I believe, in ~/.cache/pip).
I tested that by trying pip install --no-cache-dir Pillow==2.9.0 and lo and behold, it worked.
To confirm it was the cache, I ran:
pip uninstall Pillow
rm -rf ~/.cache/pip/*
pip install Pillow==2.9.0
which succeeded where it had failed before.
I don't know how there came to be a problem with the cache, but my guess is that pip got interrupted mid-download causing the cached data for Pillow to be corrupted

I found my issue to be with memory, of the disk.
Running df showed I had used 92% memory. After deleting and cleaning the hard drive (using Disk Usage Analyzer) I was able to successfully decompress data

for anyone having similar situation as mine. It's run out of space during installation --> cache still there but corrupted.
so ?
remove this folder ~/.cache/pip and pip works again.

Error downloading a file using python package requests with https

I'm a begginer in Python and have been struggling with downloading a file using the package requests. My code works for an url with http, but not for https. The code is the following:
import requests
url = 'https://firms.modaps.eosdis.nasa.gov/active_fire/shapes/zips/Central_America_24h.zip'
print "downloading with requests"
r = requests.get(url)
with open("Central_America_24h.zip", "wb") as code:
code.write(r.content)
I get the following errors:
Warning (from warnings module):
File "C:\Python27\ArcGIS10.1\lib\site-packages\requests\packages\urllib3\util\ssl_.py", line 90
InsecurePlatformWarning
InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
Traceback (most recent call last):
File "C:\Users\lmixco\Downloads\descarga.py", line 8, in <module>
r = requests.get(url)
File "C:\Python27\ArcGIS10.1\lib\site-packages\requests\api.py", line 68, in get
return request('get', url, **kwargs)
File "C:\Python27\ArcGIS10.1\lib\site-packages\requests\api.py", line 50, in request
response = session.request(method=method, url=url, **kwargs)
File "C:\Python27\ArcGIS10.1\lib\site-packages\requests\sessions.py", line 465, in request
resp = self.send(prep, **send_kwargs)
File "C:\Python27\ArcGIS10.1\lib\site-packages\requests\sessions.py", line 594, in send
history = [resp for resp in gen] if allow_redirects else []
File "C:\Python27\ArcGIS10.1\lib\site-packages\requests\sessions.py", line 196, in resolve_redirects
**adapter_kwargs
File "C:\Python27\ArcGIS10.1\lib\site-packages\requests\sessions.py", line 573, in send
r = adapter.send(request, **kwargs)
File "C:\Python27\ArcGIS10.1\lib\site-packages\requests\adapters.py", line 431, in send
raise SSLError(e, request=request)
SSLError: [Errno 8] _ssl.c:503: EOF occurred in violation of protocol
I have been looking for a solution, but have not found anything. Any help will be appreciated. Thanks in advance.

you need to install a package , use this
pip install requests[security]
for debain / ubuntu , install this before installing above package
apt-get install python-dev libffi-dev libssl-dev