Develop Reference

Connection reset by peer w/ Python's Request Module. Curl and Postman work fine.

Python 2.7.10
I'm making a very basic API call to my gitlab repo.
I can successfully curl the endpoint and I can succesfully call it with Postman
However it fails when trying to use Python's request module.
Curl Works:
curl https://gitlab.example.com/api/v4/projects
Outputs the expected JSON response from the endpoint (no auth needed)
Postman I also get the expected result:
However when I submit via requests:
[Cert is self signed, hence the verify=False]
r = requests.get("https://gitlab.example.com/api/v4/projects", verify=False)
Error thrown:
Traceback (most recent call last):
File "./repo_analyzer.py", line 36, in <module>
r = requests.get("https://xxxxxxxxxxxx/api/v4/projects", verify=False)
File "/Library/Python/2.7/site-packages/requests/api.py", line 72, in get
return request('get', url, params=params, **kwargs)
File "/Library/Python/2.7/site-packages/requests/api.py", line 58, in request
return session.request(method=method, url=url, **kwargs)
File "/Library/Python/2.7/site-packages/requests/sessions.py", line 508, in request
resp = self.send(prep, **send_kwargs)
File "/Library/Python/2.7/site-packages/requests/sessions.py", line 618, in send
r = adapter.send(request, **kwargs)
File "/Library/Python/2.7/site-packages/requests/adapters.py", line 490, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', error(54, 'Connection reset by peer'))
I'm a bit at a loss on why it's not working with python, but works otherwise.

After searching a ton of blogs and posts about this, it seemed that I needed to update requests[security] package through pip.
I was getting errors running both pip install -U requests[security] and sudo pip install -U requests[security] so I started from scratch:
Downloaded Python 2.7 and reinstalled from the official site
pip install requests
pip install -U requests
pip install -U requests[security]
After that, the calls were successful.

Python/Requests: requests.exceptions.SSLError: [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE]

In Python using MacOS X, made an attempt to make a POST request to a website but I got the following error post_response = session.post(post_url, data=post_payload, headers=post_headers):
Traceback (most recent call last):
File "web_requests.py", line 424, in <module>
main()
File "web_requests.py", line 340, in main
post_response = session.post(post_url, data=post_payload, headers=post_headers)
File "/Library/Python/2.7/site-packages/requests/sessions.py", line 535, in post
return self.request('POST', url, data=data, json=json, **kwargs)
File "/Library/Python/2.7/site-packages/requests/sessions.py", line 488, in request
resp = self.send(prep, **send_kwargs)
File "/Library/Python/2.7/site-packages/requests/sessions.py", line 609, in send
r = adapter.send(request, **kwargs)
File "/Library/Python/2.7/site-packages/requests/adapters.py", line 497, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:590)
What could be the issue?
Thank you in advance and will be sure to upvote/accept answer
EDIT
Specific to POST requests in Python

According to the addititional information in the comment SSLLabs reports for this site that it support TLS 1.2 only:
TLS 1.2 Yes
TLS 1.1 No
TLS 1.0 No
SSL 3 No
And according to another information in the comments the OpenSSL version is 0.9.8:
.. OpenSSL 0.9.8zg 14 July 2015
Since OpenSSL 0.9.8 only support up to TLS 1.0 and especially not TLS 1.2 there is no common TLS protocol spoken between the client and the server. Thus, the handshake fails.
The way to fix the problem is to upgrade the version of OpenSSL as used by Python away from the very old version to at least OpenSSL 1.0.1. See for example Updating openssl in python 2.7 for more information or use Anaconda Python which comes preinstalled with a lot of modules and also includes a current version of OpenSSL.

Python requests - Attempt to get presto documentation returns sslv3 alert handshake failure

I'm trying to get the content of Presto public documentation for some analysis using Python requests:
import requests
requests.get('https://prestodb.io/docs/current/functions/logical.html')
This returns the following error:
File "/Users/my_user/PycharmProjects/untitled/test.py", line 3, in <module>
requests.get('https://prestodb.io/docs/current/functions/logical.html')
File "/Library/Python/2.7/site-packages/requests/api.py", line 71, in get
return request('get', url, params=params, **kwargs)
File "/Library/Python/2.7/site-packages/requests/api.py", line 57, in request
return session.request(method=method, url=url, **kwargs)
File "/Library/Python/2.7/site-packages/requests/sessions.py", line 475, in request
resp = self.send(prep, **send_kwargs)
File "/Library/Python/2.7/site-packages/requests/sessions.py", line 585, in send
r = adapter.send(request, **kwargs)
File "/Library/Python/2.7/site-packages/requests/adapters.py", line 477, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: ("bad handshake: Error([('SSL routines', 'SSL23_GET_SERVER_HELLO', 'sslv3 alert handshake failure')],)",)
Tried diagnose using openssl and I found that the site requires Server Name Indication:
openssl s_client -connect prestodb.io:443 -servername prestodb.io // works
openssl s_client -connect prestodb.io:443 // doesn't work
CONNECTED(00000003)
140735204868176:error:14077438:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert internal error:s23_clnt.c:769:
I have done a lot of search and did try something else:
I have tried using a custom HTTPAdapter as described here to specify the protocol to be TLSv1 but that doesn't help.
I have also tried directly use SSL context and connection objects and set_tlsext_host_name but it doesn't help either.
It seems to me the requirement is to use TLSv1 and SNI. But I'm not able to get that through requests. Could someone please help me solve this issue? Thanks in advance.

Thanks to Lukas Graf and I resolved this based on https://github.com/kennethreitz/requests/issues/2022. What I actually did is to install OpenSSL from Homebrew, then install a new version of Python 2 from Homebrew which will automatically link against the Homebrew-provided OpenSSL:
brew install openssl
brew install python
Using the newly installed python I'm able to get the presto document page.

Why pip doesn't work in virtual enviroment?

I am trying to use pip from virtual enviroment on Linux Ubuntu 14.04 x64.
While using pip from shell as:
pip search bla
it works fine.
But using from virtualenv it shows error:
(proj_testing)uxu#box:~$ pip search bla
/home/uxu/.virtualenvs/proj_testing/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
/home/uxu/.virtualenvs/proj_testing/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
Exception:
Traceback (most recent call last):
File "/home/uxu/.virtualenvs/proj_testing/local/lib/python2.7/site-packages/pip/basecommand.py", line 223, in main
status = self.run(options, args)
File "/home/uxu/.virtualenvs/proj_testing/local/lib/python2.7/site-packages/pip/commands/search.py", line 43, in run
pypi_hits = self.search(query, options)
File "/home/uxu/.virtualenvs/proj_testing/local/lib/python2.7/site-packages/pip/commands/search.py", line 60, in search
hits = pypi.search({'name': query, 'summary': query}, 'or')
File "/usr/lib/python2.7/xmlrpclib.py", line 1233, in __call__
return self.__send(self.__name, args)
File "/usr/lib/python2.7/xmlrpclib.py", line 1587, in __request
verbose=self.__verbose
File "/home/uxu/.virtualenvs/proj_testing/local/lib/python2.7/site-packages/pip/download.py", line 785, in request
headers=headers, stream=True)
File "/home/uxu/.virtualenvs/proj_testing/local/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 508, in post
return self.request('POST', url, data=data, json=json, **kwargs)
File "/home/uxu/.virtualenvs/proj_testing/local/lib/python2.7/site-packages/pip/download.py", line 373, in request
return super(PipSession, self).request(method, url, *args, **kwargs)
File "/home/uxu/.virtualenvs/proj_testing/local/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 465, in request
resp = self.send(prep, **send_kwargs)
File "/home/uxu/.virtualenvs/proj_testing/local/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 573, in send
r = adapter.send(request, **kwargs)
File "/home/uxu/.virtualenvs/proj_testing/local/lib/python2.7/site-packages/pip/_vendor/cachecontrol/adapter.py", line 46, in send
resp = super(CacheControlAdapter, self).send(request, **kw)
File "/home/uxu/.virtualenvs/proj_testing/local/lib/python2.7/site-packages/pip/_vendor/requests/adapters.py", line 431, in send
raise SSLError(e, request=request)
SSLError: [Errno 1] _ssl.c:510: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
It doesn't make sense, anyone got idea?

Try execute pip search bla like superuser
For ubuntu the command is:
sudo pip search bla

From https://urllib3.readthedocs.org/en/latest/security.html:
Certain Python platforms (specifically, versions of Python earlier than 2.7.9) have restrictions in their ssl module that limit the configuration that urllib3 can apply. In particular, this can cause HTTPS requests that would succeed on more featureful platforms to fail, and can cause certain security features to be unavailable. If you encounter this warning, it is strongly recommended you upgrade to a newer Python version, or that you use pyOpenSSL as described in the OpenSSL / PyOpenSSL section.
You probably have a less recent version of python in your virtual environment. Perhaps this is why this error only occurs in the virtual environment.

requests package raises SSLError when connecting to github

I want to access https://api.github.com/ with the requests package. I tried requests.get('https://api.github.com/'), but got the following SSL error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/requests/api.py", line 55, in get
return request('get', url, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/requests/api.py", line 44, in request
return session.request(method=method, url=url, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/requests/sessions.py", line 456, in request
resp = self.send(prep, **send_kwargs)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/requests/sessions.py", line 559, in send
r = adapter.send(request, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/requests/adapters.py", line 382, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: [Errno 1] _ssl.c:507: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
I'm using OSX Yosemite Beta 3, python 2.7.6 and requests 2.3.0.
I know that I could pass verify=False to get to ignore the error. However, I can't do that because I can't modify the code that requests is being called from.
Using openssl, openssl s_client -connect github.com:443, does not seem to cause a problem.
Should requests work for SSL no matter what? Is my operating system causing the issue? How do I make requests trust GitHub's certificate?

Requests hard codes the default list of trusted certificates. It doesn't matter what your os trusts, only what requests trusts. If the site you're trying to access is not signed by one of the defaults, you need to tell requests what to trust manually.
You can tell requests what to trust by passing verify=<some file>, or by setting the env var REQUESTS_CA_BUNDLE=<some file> if you do not have access to the library code.
So call your program with REQUESTS_CA_BUNDLE=my_verify.pem python my_program.py
or some other way to set the env var.
If you're trying to access, for example, https://api.github.com/ and you can reach it with curl but not requests, you can pass requests the bundle used by your os. On most Linux systems it is found at /etc/ssl/certs/ca-certificates.crt. Of you can pass it a file containing just GitHub's public cert.
To get the cert for a given server use the command openssl s_client -showcerts -connect api.github.com:443 < /dev/null | openssl x509 -outform PEM > github.crt. See https://stackoverflow.com/a/16797458/400617.
Note that although pip uses requests, it also hard codes the list of trusted defaults. To override pip, you need to set the env var PIP_CERT=<some file>. So be aware that whatever library you're using might have it's own way to override this.