The title really is the question — how do I get urllib3 to download the SSL cert from the remote server when trying to make an HTTPS connection?
Background
Over in ServerVault land, I'm trying to get to the bottom of a problem with AWS autoscaling groups failing to be able to download phpMyAdmin as part of their init process. As part of trying to diagnose that, I'm now over in StackOverflow land, working with python scripts. Here's where I am:
#! /usr/bin/python3
import cfnbootstrap
from cfnbootstrap.packages import requests
from requests import utils
from requests.utils import DEFAULT_CA_BUNDLE_PATH
from requests.packages import urllib3
conn = urllib3.connection_from_url("https://dev.mysql.com/", retries=False)
conn.cert_reqs = 'CERT_REQUIRED'
conn.ca_certs = DEFAULT_CA_BUNDLE_PATH
response = conn.request("GET", "/get/")
conn.close()
print(response.status)
That works fine:
$ ./urltest.py
404
However, if I change it to be
conn = urllib3.connection_from_url("https://www.phpmyadmin.net", retries=False)
response = conn.request("GET", "/downloads/")
I run into my problem:
$ ./urltest.py
Traceback (most recent call last):
File "/usr/lib/python3.7/site-packages/cfnbootstrap/packages/requests/packages/urllib3/connectionpool.py", line 544, in urlopen
body=body, headers=headers)
File "/usr/lib/python3.7/site-packages/cfnbootstrap/packages/requests/packages/urllib3/connectionpool.py", line 341, in _make_request
self._validate_conn(conn)
File "/usr/lib/python3.7/site-packages/cfnbootstrap/packages/requests/packages/urllib3/connectionpool.py", line 762, in _validate_conn
conn.connect()
File "/usr/lib/python3.7/site-packages/cfnbootstrap/packages/requests/packages/urllib3/connection.py", line 238, in connect
ssl_version=resolved_ssl_version)
File "/usr/lib/python3.7/site-packages/cfnbootstrap/packages/requests/packages/urllib3/util/ssl_.py", line 265, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "/usr/lib64/python3.7/ssl.py", line 423, in wrap_socket
session=session
File "/usr/lib64/python3.7/ssl.py", line 870, in _create
self.do_handshake()
File "/usr/lib64/python3.7/ssl.py", line 1139, in do_handshake
self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1091)
It seems very unlikely that the certificate has expired, as if I go check manually by using a web browser, all is good with the world; however, I'd like to see exactly what certificate urllib3 is seeing as having expired. How do I do this?
NB: I'm not calling certify.where() directly, as I'm not sure what shananigans AWS is performing in the background, but since their scripts are auto-installed as part of booting the instance, there's not much I can do about it, so I'm trying to use their internal processes.
The problem is the expiration of the Let's Encrypt DST Root CA X3 that happened on Sept.30 -- https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/. The requests package doesn't trust the ISRG Root X1 cert and is trying to verify it with the expired cert.
Download the ISRG X1 PEM from https://letsencrypt.org/certificates/ and add it to the cfnbootstrap library's cacert.pem file.
cat isgrootx1.pem >> /path/to/your/python/env/lib/python3.9/site-packages/cfnbootstrap/packages/requests/cacert.pem
You can find the location of your cacert.pem with the following:
from cfnbootstrap.packages.requests.certs import where
print(where())
Also... I would change up your code to make sure you're actually using the cfn packaged requests module rather than the standalone requests. Otherwise, you probably will have to update it in both spots.
from cfnbootstrap.packages.requests.utils import DEFAULT_CA_BUNDLE_PATH
from cfnbootstrap.packages.requests.packages import urllib3
conn = urllib3.connection_from_url("https://www.phpmyadmin.net/", retries=False)
conn.cert_reqs = 'CERT_REQUIRED'
conn.ca_certs = DEFAULT_CA_BUNDLE_PATH
response = conn.request("GET", "/get/")
conn.close()
print(response.status)
If you do want to update the standalone requests and the one packaged with cfnbootstrap, use the following to find and update both cacert paths:
import requests
import cfnbootstrap.packages.requests
print(requests.certs.where())
print(cfnbootstrap.packages.requests.certs.where())
Related
How do you fix an SSL error caused by a CNAME DNS record?
I have an API hosted on AWS accessible from a URL like https://sd098fs0f98s9f0s.execute-api.us-east-1.amazonaws.com.
To shorten this and rebrand it a little, I setup a CNAME record to map myapi.mydomain.com to sd098fs0f98s9f0s.execute-api.us-east-1.amazonaws.com.
Using the original URL, this simple Python is able to access the API perfectly:
import requests
r = requests.get(url='https://sd098fs0f98s9f0s.execute-api.us-east-1.amazonaws.com')
However, using the new URL:
import requests
r = requests.get(url='https://myapi.mydomain.com')
results in the error:
Traceback (most recent call last):
File ".env/lib/python3.7/site-packages/urllib3/connectionpool.py", line 603, in urlopen
chunked=chunked)
File ".env/lib/python3.7/site-packages/urllib3/connectionpool.py", line 344, in _make_request
self._validate_conn(conn)
File ".env/lib/python3.7/site-packages/urllib3/connectionpool.py", line 843, in _validate_conn
conn.connect()
File ".env/lib/python3.7/site-packages/urllib3/connection.py", line 370, in connect
ssl_context=context)
File ".env/lib/python3.7/site-packages/urllib3/util/ssl_.py", line 355, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "/usr/lib/python3.7/ssl.py", line 423, in wrap_socket
session=session
File "/usr/lib/python3.7/ssl.py", line 870, in _create
self.do_handshake()
File "/usr/lib/python3.7/ssl.py", line 1139, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1076)
What's causing this and how do I fix it?
I see that you're using https. The server certificate is issued to specific hostname (Common name attribute). This name has to match the address you're trying to access with HTTPS. If you added DNS CNAME you have to update the certificate. If I recall correctly you have to add CNAME as Alternative Subject Name attribute, so you'll need a new certificate.
The error "WRONG_VERSION_NUMBER" might be somewhat misleading. It means that the server presented a wrong TLS version. In the TLS handshake, client and the server negotiate the TLS version. If the server supports only 1.0 and your client accept only 1.2+ then "WRONG_VERSION_NUMBER" will appear. But it might also happen if the client gets some unexpected data, like a plain HTTP instead of HTTPS. To check what exactly was the unexpected data, you'll have to capture wireshark / tcpdump network traffic logs. I wasted a lot of time debugging wrong_version_number recently. Only after looking at wireshark logs it became clear that it was the China firewall, because we got HTTP Forbidden to the initial Client Hello.
I am attempting to use python sockets to make an Extensible Provisioning Protocol (EPP) request to a domain registrar, which only accepts requests over ssl.
Certificate file: www.myDomain.se.crt
Key File: mydomain.pem
openssl s_client -connect epptestv3.iis.se:700 -cert www.myDomain.se.crt -key mydomain.pem
When I try making request using openssl client I successfully get greeting response from registrar, but when I use following code in python i get ssl certificate error.
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.settimeout(15)
sock.connect(('epptestv3.iis.se', 700))
sock.settimeout(60) # regular timeout
ssl_keyfile='myDomain.pem'
ssl_certfile='www.myDomain.se.crt'
ssl_ciphers='AES256-GCM-SHA384'
ssl_version=ssl.PROTOCOL_TLSv1_2
sock = ssl.wrap_socket(sock,
ssl_keyfile,
ssl_certfile,
ssl_version=ssl_version,
ciphers=ssl_ciphers,
server_side=False,
cert_reqs=ssl.CERT_REQUIRED,
ca_certs=None
)
After executing script I get following error:
Traceback (most recent call last):
File "server_connect.py", line 54, in <module>
ca_certs=ssl_keyfile
File "/usr/lib/python2.7/ssl.py", line 933, in wrap_socket
ciphers=ciphers)
File "/usr/lib/python2.7/ssl.py", line 601, in __init__
self.do_handshake()
File "/usr/lib/python2.7/ssl.py", line 830, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)
Any idea what's wrong here?
From your code:
cert_reqs=ssl.CERT_REQUIRED,
ca_certs=None
From the documentation of wrap_socket:
If the value of this parameter is not CERT_NONE, then the ca_certs parameter must point to a file of CA certificates.
Essentially you are asking in your code to validate the certificate from the server (CERT_REQUIRED) but specify at the same time that you have no trusted root (ca_certs=None). But without trusted root certificates no validation can be done.
Note that changing your code to use CERT_NONE instead would be a bad idea. It would probably work since no certificate validation will be done but it would be open to man in the middle attacks.
I want to open and read https://yande.re/ with urllib.request, but I'm getting an SSL error. I can open and read the page just fine using http.client with this code:
import http.client
conn = http.client.HTTPSConnection('www.yande.re')
conn.request('GET', 'https://yande.re/')
resp = conn.getresponse()
data = resp.read()
However, the following code using urllib.request fails:
import urllib.request
opener = urllib.request.build_opener()
resp = opener.open('https://yande.re/')
data = resp.read()
It gives me the following error: ssl.SSLError: [Errno 1] _ssl.c:392: error:1411809D:SSL routines:SSL_CHECK_SERVERHELLO_TLSEXT:tls invalid ecpointformat list. Why can I open the page with HTTPSConnection but not opener.open?
Edit: Here's my OpenSSL version and the traceback from trying to open https://yande.re/
>>> import ssl; ssl.OPENSSL_VERSION
'OpenSSL 1.0.0a 1 Jun 2010'
>>> import urllib.request
>>> urllib.request.urlopen('https://yande.re/')
Traceback (most recent call last):
File "<pyshell#3>", line 1, in <module>
urllib.request.urlopen('https://yande.re/')
File "C:\Python32\lib\urllib\request.py", line 138, in urlopen
return opener.open(url, data, timeout)
File "C:\Python32\lib\urllib\request.py", line 369, in open
response = self._open(req, data)
File "C:\Python32\lib\urllib\request.py", line 387, in _open
'_open', req)
File "C:\Python32\lib\urllib\request.py", line 347, in _call_chain
result = func(*args)
File "C:\Python32\lib\urllib\request.py", line 1171, in https_open
context=self._context, check_hostname=self._check_hostname)
File "C:\Python32\lib\urllib\request.py", line 1138, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 1] _ssl.c:392: error:1411809D:SSL routines:SSL_CHECK_SERVERHELLO_TLSEXT:tls invalid ecpointformat list>
>>>
What a coincidence! I'm having the same problem as you are, with an added complication: I'm behind a proxy. I found this bug report regarding https-not-working-with-urllib. Luckily, they posted a workaround.
import urllib.request
import ssl
##uncomment this code if you're behind a proxy
##https port is 443 but it doesn't work for me, used port 80 instead
##proxy_auth = '{0}://{1}:{2}#{3}'.format('https', 'username', 'password',
## 'proxy:80')
##proxies = { 'https' : proxy_auth }
##proxy = urllib.request.ProxyHandler(proxies)
##proxy_auth_handler = urllib.request.HTTPBasicAuthHandler()
##opener = urllib.request.build_opener(proxy, proxy_auth_handler,
## https_sslv3_handler)
https_sslv3_handler =
urllib.request.HTTPSHandler(context=ssl.SSLContext(ssl.PROTOCOL_SSLv3))
opener = urllib.request.build_opener(https_sslv3_handler)
urllib.request.install_opener(opener)
resp = opener.open('https://yande.re/')
data = resp.read().decode('utf-8')
print(data)
Btw, thanks for showing how to use http.client. I didn't know that there's another library that can be used to connect to the internet. ;)
This is due to a bug in the early 1.x OpenSSL implementation of elliptic curve cryptography. Take a closer look at the relevant part of the exception:
_ssl.c:392: error:1411809D:SSL routines:SSL_CHECK_SERVERHELLO_TLSEXT:tls invalid ecpointformat list
This is an error from the underlying OpenSSL library code which is a result of mishandling the EC point format TLS extension. One workaround is to use the SSLv3 instead of SSLv23 method, the other workaround is to use a cipher suite specification which disables all ECC cipher suites (I had good results with ALL:-ECDH, use openssl ciphers for testing). The fix is to update OpenSSL.
The problem is due to the hostnames that your giving in the two examples:
import http.client
conn = http.client.HTTPSConnection('www.yande.re')
conn.request('GET', 'https://yande.re/')
and...
import urllib.request
urllib.request.urlopen('https://yande.re/')
Note that in the first example, you're asking the client to make a connection to the host: www.yande.re and in the second example, urllib will first parse the url 'https://yande.re' and then try a request at the host yande.re
Although www.yande.re and yande.re may resolve to the same IP address, from the perspective of the web server these are different virtual hosts. My guess is that you had an SNI configuration problem on your web server's side. Seeing as that the original question was posted on May 21, and the current cert at yande.re starts May 28, I'm thinking that you already fixed this problem?
Try this:
import connection #imports connection
import url
url = 'http://www.google.com/'
webpage = url.open(url)
try:
connection.receive(webpage)
except:
webpage = url.text('This webpage is not available!')
connection.receive(webpage)
I am attempting to use the requests.py library for calls to a rest web service. I wrote a quick prototype for my usage under windows and everything worked fine, but when I attempted to run the same prototype under linux I get a "requests.exceptions.Timeout: Request timed out" error. Does anyone know why this might be happening? If I try to use the library to access a non https url it works fine under both windows and linux.
import requests
url = 'https://path.to.rest/svc/?options'
r = requests.get(url, auth=('uid','passwd'), verify=False)
print(r.content)
I did notice that if I leave off the verify=False parameter from the get call, I get a different exception, namely "requests.exceptions.SSLError: Can't connect to HTTPS URL because the SSL module is not available". This appears to be a possible underlying cause, though I dont know why they would change the errorcode, but I cant find any reference to an ssl module and I verified that certifi was installed. Interestingly, if I leave off the verify parameter in windows I get a different exception, "requests.exceptions.SSLError: [Errno 1] _ssl.c:503: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed"
EDIT:
Tracebacks for all cases/scenarios mentioned
Full code as shown above:
Traceback(most recent call last):
File "testRequests.py", line 15, in <module>
r = requests.get(url, auth=('uid','passwd'), verify=False)
File "build/bdist.linux-x86_64/egg/requests/api.py", line 52, in get
File "build/bdist.linux-x86_64/egg/requests/api.py", line 40, in request
File "build/bdist.linux-x86_64/egg/requests/sessions.py", line 208, in request
File "build/bdist.linux-x86_64/egg/requests/models.py", line 586, in send
requests.exceptions.Timeout: Request timed out
Code as shown above minus the "verify=False" paramter:
Traceback(most recent call last):
File "testRequests.py", line 15, in <module>
r = requests.get(url, auth=('uid','passwd'))
File "build/bdist.linux-x86_64/egg/requests/api.py", line 52, in get
File "build/bdist.linux-x86_64/egg/requests/api.py", line 40, in request
File "build/bdist.linux-x86_64/egg/requests/sessions.py", line 208, in request
File "build/bdist.linux-x86_64/egg/requests/models.py", line 584, in send
requests.exceptions.SSLError: Can't connect to HTTPS URL because the SSL module is not available
Code as show above minus the "verify=False" parameter and run under windows:
Traceback(most recent call last):
File "testRequests.py", line 59, in <module>
r = requests.get(url, auth=('uid','passwd'))
File "c:\Python27\lib\site-packages\requests\api.py", line 52, in get
return request('get', url, **kwargs)
File "c:\Python27\lib\site-packages\requests\api.py", line 40, in request
return s.request(method=method, url=url, **kwargs)
File "c:\Python27\lib\site-packages\requests\sessions.py", line 208, in request
r.send(prefetch=prefetch)
File "c:\Python27\lib\site-packages\requests\models.py", line 584, in send
raise SSLError(e)
requests.exceptions.SSLError: [Errno 1] _ssl.c:503: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
I'm not an expert on the matter but it looks like the certificate from the server can't be verified correctly. I don't know how Python and ssl handles certificate verification but the first option is to try ignoring the exception, or maybe change https to http in an attempt to see if the web-service allows non-secure service calls.
If the issue is revolving around an import error for ssl, the module is part of CPython and you may need to ensure that the Python interpreter is compiled with SSL support (from openssl). Look into removing the package for python (be careful) and compiling it with openssl support, personally I would strongly advise you looking into a virtualenv before removing anything, compiling Python is not too difficult and it would give you a finer grain of control for what you aim to do.
I'm getting a getaddress error and after doing some sleuthing, it looks like it might be my corporate intranet not allowing the connection (I'm assuming due to security, although it is strange that IE works but won't allow Python to open a url). Is there a safe way to get around this?
Here's the exact error:
Traceback (most recent call last):
File "<pyshell#1>", line 1, in <module>
b = urllib.urlopen('http://www.google.com')
File "C:\Python26\lib\urllib.py", line 87, in urlopen
return opener.open(url)
File "C:\Python26\lib\urllib.py", line 203, in open
return getattr(self, name)(url)
File "C:\Python26\lib\urllib.py", line 342, in open_http
h.endheaders()
File "C:\Python26\lib\httplib.py", line 868, in endheaders
self._send_output()
File "C:\Python26\lib\httplib.py", line 740, in _send_output
self.send(msg)
File "C:\Python26\lib\httplib.py", line 699, in send
self.connect()
File "C:\Python26\lib\httplib.py", line 683, in connect
self.timeout)
File "C:\Python26\lib\socket.py", line 498, in create_connection
for res in getaddrinfo(host, port, 0, SOCK_STREAM):
IOError: [Errno socket error] [Errno 11001] getaddrinfo failed
More info: I also get this error with urllib2.urlopen
You probably need to fill in proxy information.
import urllib2
proxy_handler = urllib2.ProxyHandler({'http': 'http://yourcorporateproxy:12345/'})
proxy_auth_handler = urllib2.HTTPBasicAuthHandler()
proxy_auth_handler.add_password('realm', 'host', 'username', 'password')
opener = urllib2.build_opener(proxy_handler, proxy_auth_handler)
opener.open('http://www.stackoverflow.com')
Check you are using the correct proxy.
You can get the proxy information by using urllib.getproxies (note: getproxies does not work with dynamic proxy configuration, like when using PAC).
Update As per information about empty proxy list, I would suggest using an urlopener, with the proxy name and information.
Some good information about how use proxies urlopeners:
Urllib manual
Michael Foord's introduction to urllib
Possibly this is a DNS issue, try urlopen with the IP address of the web server you're accessing, i.e.
import urllib
URL="http://66.102.11.99" # www.google.com
f = urllib.urlopen(URL)
f.read()
If this succeeds, then it's probably a DNS issue rather than a proxy issue (but you should also check your proxy setup).
Looks like a DNS problem.
Since you are using Windows, you can try run this command
nslookup www.google.com
To check if the web address can be resolved successfully.
If not, it is a network setting issue
If OK, then we have to look at possible alternative causes
I was facing the same issue.
In my system the proxy configuration is through a .PAC file.
So i opended that file, took out the default proxy url, for me it was http://168.219.61.250:8080/
Following test code worked for me :
import urllib2
proxy_support = urllib2.ProxyHandler({'http': 'http://168.219.61.250:8080/'})
opener = urllib2.build_opener(proxy_support)
urllib2.install_opener(opener)
response = urllib2.urlopen('http://python.org/')
html = response.read()
print html
You might need to add some more code, if your proxy requires authentication
Hope this helps!!