I want to access https://api.github.com/ with the requests package. I tried requests.get('https://api.github.com/'), but got the following SSL error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/requests/api.py", line 55, in get
return request('get', url, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/requests/api.py", line 44, in request
return session.request(method=method, url=url, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/requests/sessions.py", line 456, in request
resp = self.send(prep, **send_kwargs)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/requests/sessions.py", line 559, in send
r = adapter.send(request, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/requests/adapters.py", line 382, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: [Errno 1] _ssl.c:507: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
I'm using OSX Yosemite Beta 3, python 2.7.6 and requests 2.3.0.
I know that I could pass verify=False to get to ignore the error. However, I can't do that because I can't modify the code that requests is being called from.
Using openssl, openssl s_client -connect github.com:443, does not seem to cause a problem.
Should requests work for SSL no matter what? Is my operating system causing the issue? How do I make requests trust GitHub's certificate?
Requests hard codes the default list of trusted certificates. It doesn't matter what your os trusts, only what requests trusts. If the site you're trying to access is not signed by one of the defaults, you need to tell requests what to trust manually.
You can tell requests what to trust by passing verify=<some file>, or by setting the env var REQUESTS_CA_BUNDLE=<some file> if you do not have access to the library code.
So call your program with REQUESTS_CA_BUNDLE=my_verify.pem python my_program.py
or some other way to set the env var.
If you're trying to access, for example, https://api.github.com/ and you can reach it with curl but not requests, you can pass requests the bundle used by your os. On most Linux systems it is found at /etc/ssl/certs/ca-certificates.crt. Of you can pass it a file containing just GitHub's public cert.
To get the cert for a given server use the command openssl s_client -showcerts -connect api.github.com:443 < /dev/null | openssl x509 -outform PEM > github.crt. See https://stackoverflow.com/a/16797458/400617.
Note that although pip uses requests, it also hard codes the list of trusted defaults. To override pip, you need to set the env var PIP_CERT=<some file>. So be aware that whatever library you're using might have it's own way to override this.
Related
Trying to use Python 3 urlopen on many HTTPS sites on recent (>=Vista) Windows machines I get "SSL: CERTIFICATE_VERIFY_FAILED" errors when trying to do an urllib.request.urlopen on many sites (on some build machines even https://www.google.com/, but curiously never on https://www.microsoft.com/).
>>> import urllib.request
>>> urllib.request.urlopen("https://www.google.com/")
Traceback (most recent call last):
File "C:\Python35\lib\urllib\request.py", line 1254, in do_open
h.request(req.get_method(), req.selector, req.data, headers)
File "C:\Python35\lib\http\client.py", line 1106, in request
self._send_request(method, url, body, headers)
File "C:\Python35\lib\http\client.py", line 1151, in _send_request
self.endheaders(body)
File "C:\Python35\lib\http\client.py", line 1102, in endheaders
self._send_output(message_body)
File "C:\Python35\lib\http\client.py", line 934, in _send_output
self.send(msg)
File "C:\Python35\lib\http\client.py", line 877, in send
self.connect()
File "C:\Python35\lib\http\client.py", line 1260, in connect
server_hostname=server_hostname)
File "C:\Python35\lib\ssl.py", line 377, in wrap_socket
_context=self)
File "C:\Python35\lib\ssl.py", line 752, in __init__
self.do_handshake()
File "C:\Python35\lib\ssl.py", line 988, in do_handshake
self._sslobj.do_handshake()
File "C:\Python35\lib\ssl.py", line 633, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c
:645)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Python35\lib\urllib\request.py", line 163, in urlopen
return opener.open(url, data, timeout)
File "C:\Python35\lib\urllib\request.py", line 466, in open
response = self._open(req, data)
File "C:\Python35\lib\urllib\request.py", line 484, in _open
'_open', req)
File "C:\Python35\lib\urllib\request.py", line 444, in _call_chain
result = func(*args)
File "C:\Python35\lib\urllib\request.py", line 1297, in https_open
context=self._context, check_hostname=self._check_hostname)
File "C:\Python35\lib\urllib\request.py", line 1256, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certifica
te verify failed (_ssl.c:645)>
Most infuriatingly, this happens almost only on the build/CI servers, and often these errors disappear after trying to investigate the issue (e.g. checking the connectivity to the given site, which responds correctly when tried through a browser):
>>> import urllib.request
>>> urllib.request.urlopen("https://www.google.com/")
<http.client.HTTPResponse object at 0x0000000002D930B8>
I heard many suggestions about disabling the certificate validation by messing with SSL contexts, but I'd like to avoid this - I want to keep my HTTPS security intact!
What could be the cause of this issue? How can I fix it?
Unfortunately, it's a sad story still without a happy ending, and is detailed in https://bugs.python.org/issue20916.
Python 3.3 added the cadefault parameter to urllib.request.urlopen, defaulting to True (https://bugs.python.org/issue14780), which made HTTPS requests validate the server certificates using the system certificates store by default.
Python 3.4 made SSLContext.set_default_verify_paths kind-of-work on Windows (https://bugs.python.org/issue19292), enabling Python to use the Windows certificate store.
Previously, Microsoft pushed root certificates updates through Windows Update, which ensured that the system root certificates store was always updated (as long as the user installed the updates). So far, so good.
However, since Windows Vista, Windows is bundled with just few "core" certificates in the store (less than 20, IIRC), and whenever the CryptoAPI is asked to validate a certificate for which it cannot find a trusted root in the local store, the Microsoft servers are contacted to check if they have a trusted root for it. If so, the root certificate is provided and gets automatically installed to the system certificates store.
Unfortunately, Python doesn't use Windows SChannel/CryptoAPI, so it cannot benefit from this automatic mechanism; instead, it asks for all the certificates in the system certificates store and tries to use them - but this means that all it is getting is the handful of certificates shipped with Windows, the manually-installed certificates, plus all the certificates that happened to have been installed automatically, typically when browsing the Internet with Internet Explorer or Edge.
This makes the issue particularly insidious, as the sites which will exhibit a problem will vary between different machines (depending mostly on their browsing history!), and will generally disappear (for that site, and all sites depending from its same root certificate) if you check if you can connect to the site through a browser using SChannel. New Windows installations, build machines and servers in general (which do not see much interactive Internet browsing) for this reason are particularly subject to this problem, while developers may never encounter this issue on their "normal" desktop machines.
How to fix this? Unfortunately, there's no simple solution.
for simple cases, such as a CI server, where some tests needs to access some specific domains that pretty much never changes, a trivial workaround can be to open Internet Explorer and open a page on such domains. This will make it fetch the needed root certificate to the local certificates store, and Python won't have problems with it until it expires (notice: we are talking about the root certificate here, which generally has a duration of many years); on modern Windows versions that ship by default a curl version that uses SChannel as SSL backend, it can be used as well
you can disable certificate validation tout-court; this has been already covered on in many different answers, such as this. However, this is generally undesirable, as you are giving up the MITM protection provided by SSL;
you may manually install all the currently trusted root certificates to the Windows certificate store; here is a site that explains how (disclaimer: the explained procedure looks sensible, but I never tried it); unfortunately, it's a manual procedure and you would need to repeat it periodically to make sure you get the new root certificates;
you may install the certifi package, which provides its own certificate store (IIRC it's a copy of the Mozilla certificate store); you can then use it like this:
import certifi
import urllib.request
r = urllib.request.urlopen(url_website, cafile=certifi.where())
This is the road taken by the popular requests module, which indeed generally works "out of the box"; unfortunately, this is yet another certificate store, which has to be kept updated, so you have to make sure to periodically update the certifi package through pip or however you installed it.
Many thanks to the author of this blog article, that was the first that I managed to find that explained correctly this issue.
Python 2.7.10
I'm making a very basic API call to my gitlab repo.
I can successfully curl the endpoint and I can succesfully call it with Postman
However it fails when trying to use Python's request module.
Curl Works:
curl https://gitlab.example.com/api/v4/projects
Outputs the expected JSON response from the endpoint (no auth needed)
Postman I also get the expected result:
However when I submit via requests:
[Cert is self signed, hence the verify=False]
r = requests.get("https://gitlab.example.com/api/v4/projects", verify=False)
Error thrown:
Traceback (most recent call last):
File "./repo_analyzer.py", line 36, in <module>
r = requests.get("https://xxxxxxxxxxxx/api/v4/projects", verify=False)
File "/Library/Python/2.7/site-packages/requests/api.py", line 72, in get
return request('get', url, params=params, **kwargs)
File "/Library/Python/2.7/site-packages/requests/api.py", line 58, in request
return session.request(method=method, url=url, **kwargs)
File "/Library/Python/2.7/site-packages/requests/sessions.py", line 508, in request
resp = self.send(prep, **send_kwargs)
File "/Library/Python/2.7/site-packages/requests/sessions.py", line 618, in send
r = adapter.send(request, **kwargs)
File "/Library/Python/2.7/site-packages/requests/adapters.py", line 490, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', error(54, 'Connection reset by peer'))
I'm a bit at a loss on why it's not working with python, but works otherwise.
After searching a ton of blogs and posts about this, it seemed that I needed to update requests[security] package through pip.
I was getting errors running both pip install -U requests[security] and sudo pip install -U requests[security] so I started from scratch:
Downloaded Python 2.7 and reinstalled from the official site
pip install requests
pip install -U requests
pip install -U requests[security]
After that, the calls were successful.
In Python using MacOS X, made an attempt to make a POST request to a website but I got the following error post_response = session.post(post_url, data=post_payload, headers=post_headers):
Traceback (most recent call last):
File "web_requests.py", line 424, in <module>
main()
File "web_requests.py", line 340, in main
post_response = session.post(post_url, data=post_payload, headers=post_headers)
File "/Library/Python/2.7/site-packages/requests/sessions.py", line 535, in post
return self.request('POST', url, data=data, json=json, **kwargs)
File "/Library/Python/2.7/site-packages/requests/sessions.py", line 488, in request
resp = self.send(prep, **send_kwargs)
File "/Library/Python/2.7/site-packages/requests/sessions.py", line 609, in send
r = adapter.send(request, **kwargs)
File "/Library/Python/2.7/site-packages/requests/adapters.py", line 497, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:590)
What could be the issue?
Thank you in advance and will be sure to upvote/accept answer
EDIT
Specific to POST requests in Python
According to the addititional information in the comment SSLLabs reports for this site that it support TLS 1.2 only:
TLS 1.2 Yes
TLS 1.1 No
TLS 1.0 No
SSL 3 No
And according to another information in the comments the OpenSSL version is 0.9.8:
.. OpenSSL 0.9.8zg 14 July 2015
Since OpenSSL 0.9.8 only support up to TLS 1.0 and especially not TLS 1.2 there is no common TLS protocol spoken between the client and the server. Thus, the handshake fails.
The way to fix the problem is to upgrade the version of OpenSSL as used by Python away from the very old version to at least OpenSSL 1.0.1. See for example Updating openssl in python 2.7 for more information or use Anaconda Python which comes preinstalled with a lot of modules and also includes a current version of OpenSSL.
I'm trying to get the content of Presto public documentation for some analysis using Python requests:
import requests
requests.get('https://prestodb.io/docs/current/functions/logical.html')
This returns the following error:
File "/Users/my_user/PycharmProjects/untitled/test.py", line 3, in <module>
requests.get('https://prestodb.io/docs/current/functions/logical.html')
File "/Library/Python/2.7/site-packages/requests/api.py", line 71, in get
return request('get', url, params=params, **kwargs)
File "/Library/Python/2.7/site-packages/requests/api.py", line 57, in request
return session.request(method=method, url=url, **kwargs)
File "/Library/Python/2.7/site-packages/requests/sessions.py", line 475, in request
resp = self.send(prep, **send_kwargs)
File "/Library/Python/2.7/site-packages/requests/sessions.py", line 585, in send
r = adapter.send(request, **kwargs)
File "/Library/Python/2.7/site-packages/requests/adapters.py", line 477, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: ("bad handshake: Error([('SSL routines', 'SSL23_GET_SERVER_HELLO', 'sslv3 alert handshake failure')],)",)
Tried diagnose using openssl and I found that the site requires Server Name Indication:
openssl s_client -connect prestodb.io:443 -servername prestodb.io // works
openssl s_client -connect prestodb.io:443 // doesn't work
CONNECTED(00000003)
140735204868176:error:14077438:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert internal error:s23_clnt.c:769:
I have done a lot of search and did try something else:
I have tried using a custom HTTPAdapter as described here to specify the protocol to be TLSv1 but that doesn't help.
I have also tried directly use SSL context and connection objects and set_tlsext_host_name but it doesn't help either.
It seems to me the requirement is to use TLSv1 and SNI. But I'm not able to get that through requests. Could someone please help me solve this issue? Thanks in advance.
Thanks to Lukas Graf and I resolved this based on https://github.com/kennethreitz/requests/issues/2022. What I actually did is to install OpenSSL from Homebrew, then install a new version of Python 2 from Homebrew which will automatically link against the Homebrew-provided OpenSSL:
brew install openssl
brew install python
Using the newly installed python I'm able to get the presto document page.
I have written Robot Framework test scripts in .tsv format to test web-services/APIs. Everything was working fine until today (probably because of the new updates of Robot Framework) when I started to get the following error:
SSLError: ("bad handshake: SysCallError(-1, 'Unexpected EOF')",)
This error keeps popping up for the following code in a test script:
${headers}= Create Dictionary Content-Type application/json Accept application/json
RequestsKeywords.Get Request httpbin ${url} headers=${headers} //ERROR SHOWS FOR THIS STATEMENT
I did get a detailed traceback for this error which is as follows:
Traceback (most recent call last):
File "/Library/Python/2.7/site-packages/RequestsLibrary/RequestsKeywords.py", line 298, in get_request
session, uri, params, headers, redir, timeout)
File "/Library/Python/2.7/site-packages/RequestsLibrary/RequestsKeywords.py", line 801, in _get_request
cookies=self.cookies)
File "/Library/Python/2.7/site-packages/requests/sessions.py", line 480, in get
return self.request('GET', url, **kwargs)
File "/Library/Python/2.7/site-packages/requests/sessions.py", line 468, in request
resp = self.send(prep, **send_kwargs)
File "/Library/Python/2.7/site-packages/requests/sessions.py", line 576, in send
r = adapter.send(request, **kwargs)
File "/Library/Python/2.7/site-packages/requests/adapters.py", line 447, in send
raise SSLError(e, request=request)
My system configuration:
Mac OS X (10.11.3)
Python (2.7.10)
openssl (1.0.2f)
requests (2.9.1)
robotframework (3.0)
robotframework-httplibrary (0.4.2)
robotframework-requests (0.4.4)
robotframework-ride (1.5.2.1)
robotframework-sshlibrary (2.1.2)
pyOpenSSL (0.15.1)
How do I resolve this issue?
You're all up to date, so there are two possibilities:
The handshake isn't going smoothly because of a break in the trust chain. Start from the bottom and work up. Are you testing this on a local area network? Is the certificate up to date? Can you access the site from your location and others without an error? SSL labs showing anything up? What about firewall rules?
There's a bug in the recent update. OpenSSL reports in my experience are caused by connection issues rather than obscure software problems - this is the less likely of the two.
Try installing requests[security] instead of requests. It uses PyOpenSSL, which is better than OpenSSL: pip install requests[security] vs pip install requests: Difference