I'm using urllib3 against private services that have self signed certificates. Is there any way to have urllib3 ignore the certificate errors and make the request anyways?
import urllib3
c = urllib3.HTTPSConnectionPool('10.0.3.168', port=9001)
c.request('GET', '/')
When using the following:
import urllib3
c = urllib3.HTTPSConnectionPool('10.0.3.168', port=9001, cert_reqs='CERT_NONE')
c.request('GET', '/')
The following error is raised:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3/dist-packages/urllib3/request.py", line 67, in request
**urlopen_kw)
File "/usr/lib/python3/dist-packages/urllib3/request.py", line 80, in request_encode_url
return self.urlopen(method, url, **urlopen_kw)
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 415, in urlopen
body=body, headers=headers)
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 267, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/lib/python3.3/http/client.py", line 1061, in request
self._send_request(method, url, body, headers)
File "/usr/lib/python3.3/http/client.py", line 1099, in _send_request
self.endheaders(body)
File "/usr/lib/python3.3/http/client.py", line 1057, in endheaders
self._send_output(message_body)
File "/usr/lib/python3.3/http/client.py", line 902, in _send_output
self.send(msg)
File "/usr/lib/python3.3/http/client.py", line 840, in send
self.connect()
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 103, in connect
match_hostname(self.sock.getpeercert(), self.host)
File "/usr/lib/python3/dist-packages/urllib3/packages/ssl_match_hostname/__init__.py", line 32, in match_hostname
raise ValueError("empty or no certificate")
ValueError: empty or no certificate
Using cURL I'm able to get the expected response from the service
$ curl -k https://10.0.3.168:9001/
Please read the documentation for API endpoints
Try following code:
import urllib3
c = urllib3.HTTPSConnectionPool('10.0.3.168', port=9001, cert_reqs='CERT_NONE',
assert_hostname=False)
c.request('GET', '/')
See Setting assert_hostname to False will disable SSL hostname verification
In this question I see many answers but, IMHO, too much unnecessary information that can lead to confusion.
Just add the cert_reqs='CERT_NONE' parameter
import urllib3
http = urllib3.PoolManager(cert_reqs='CERT_NONE')
I found the answer to my problem. The urllib3 documentation does not, in fact, completely explain how to suppress SSL certificate validation. What is missing is a reference to ssl.CERT_NONE.
My code has a boolean, ssl_verify, to indicate whether or not I want SSL validation. The code now looks like this:
import ssl
import urllib3
#
#
#
if (ssl_verify):
cert_reqs = ssl.CERT_REQUIRED
else:
cert_reqs = ssl.CERT_NONE
urllib3.disable_warnings()
http = urllib3.PoolManager(cert_reqs = cert_reqs)
auth_url = f'https://{fmc_ip}/api/fmc_platform/v1/auth/generatetoken'
type = {'Content-Type': 'application/json'}
auth = urllib3.make_headers(basic_auth=f'{username}:{password}')
headers = { **type, **auth }
resp = http.request('POST',
auth_url,
headers=headers,
timeout=10.0)
Try to instanciate your connection pool this way:
HTTPSConnectionPool(self.host, self.port, cert_reqs=ssl.CERT_NONE)
or this way:
HTTPSConnectionPool(self.host, self.port, cert_reqs='CERT_NONE')
Source: https://github.com/shazow/urllib3/blob/master/test/with_dummyserver/test_https.py
EDIT (after seeing your edit):
It looks like the remote host didn't send a certificate (is it possible?).
This is the code (from urllib3) which raised an exception:
def match_hostname(cert, hostname):
"""Verify that *cert* (in decoded format as returned by
SSLSocket.getpeercert()) matches the *hostname*. RFC 2818 rules
are mostly followed, but IP addresses are not accepted for *hostname*.
CertificateError is raised on failure. On success, the function
returns nothing.
"""
if not cert:
raise ValueError("empty or no certificate")
So it looks like cert is empty, which means that self.sock.getpeercert() returned an empty string.
Related
I have the code below which works fine and brings back what I need
import requests
from requests.auth import HTTPBasicAuth
response = requests.get('https://example/answers/331', auth=HTTPBasicAuth('username', 'password'),json={"solution": "12345"})
print response.content
However when I change it to a patch method, which is accepted by the server, I get the following errors. Any idea on why?
Traceback (most recent call last):
File "auth.py", line 8, in <module>
response = requests.patch('https://example/answers/331', auth=HTTPBasicAuth('username', 'password'),json={"solution": "12345"})
File "C:\Python27\lib\site-packages\requests-2.12.0-py2.7.egg\requests\api.py", line 138, in patch
return request('patch', url, data=data, **kwargs)
File "C:\Python27\lib\site-packages\requests-2.12.0-py2.7.egg\requests\api.py", line 56, in request
return session.request(method=method, url=url, **kwargs)
File "C:\Python27\lib\site-packages\requests-2.12.0-py2.7.egg\requests\sessions.py", line 488, in request
resp = self.send(prep, **send_kwargs)
File "C:\Python27\lib\site-packages\requests-2.12.0-py2.7.egg\requests\sessions.py", line 609, in send
r = adapter.send(request, **kwargs)
File "C:\Python27\lib\site-packages\requests-2.12.0-py2.7.egg\requests\adapters.py", line 473, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', BadStatusLine("''",))
Thanks
Try using a POST request with the following header: X-HTTP-Method-Override: PATCH
This is unique to the Oracle Service Cloud REST API implementation and is documented.
In cases where the browser or client application does not support PATCH requests, or network intermediaries block PATCH requests, HTTP tunneling can be used with a POST request by supplying an X-HTTP-Method-Override header.
Example:
import requests
restURL = <Your REST URL>
params = {'field': 'val'}
headers = {'X-HTTP-Method-Override':'PATCH'}
try:
resp = requests.post(restURL, json=params, auth=('<uname>', '<pwd>'), headers=headers)
print resp
except requests.exceptions.RequestException as err:
errMsg = "Error: %s" % err
print errMsg
I am having trouble using pyVmomi with python 2.7.5. I get SSL certificate errors when trying to run the sample scripts from the SDK. I tried all the solutions mentioned on this post but none of them worked for me.
Below is the complete console output.
/usr/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:315: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#snimissingwarning. SNIMissingWarning /usr/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:120: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning. InsecurePlatformWarning
Traceback (most recent call last):
File "hello_world_vcenter.py", line 105, in <module>
main()
File "hello_world_vcenter.py", line 80, in main
port=int(args.port))
File "/usr/lib/python2.7/site-packages/pyVim/connect.py", line 663, in SmartConnect
sslContext)
File "/usr/lib/python2.7/site-packages/pyVim/connect.py", line 552, in __FindSupportedVersion
sslContext)
File "/usr/lib/python2.7/site-packages/pyVim/connect.py", line 472, in __GetServiceVersionDescription
tree = __GetElementTreeFromUrl(url, sslContext)
File "/usr/lib/python2.7/site-packages/pyVim/connect.py", line 440, in __GetElementTreeFromUrl
sock = requests.get(url)
File "/usr/lib/python2.7/site-packages/requests/api.py", line 67, in get
return request('get', url, params=params, **kwargs)
File "/usr/lib/python2.7/site-packages/requests/api.py", line 53, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 468, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 576, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 447, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: [Errno 1] _ssl.c:504: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
That looks like you're using a self-signed certificate. While connecting via SmartConnect use your own sslContext and disable certificate verification.
from pyVim.connect import SmartConnect
import ssl
context = ssl.SSLContext(ssl.PROTOCOL_TLSv1)
context.verify_mode = ssl.CERT_NONE
si = SmartConnect(host=somehost.com, port=443, user=someone, pwd=secret, sslContext=context)
... or use a signed ssl certificate.
There are a few similiar questions on here (e.g. here).
Good Evening,
I cannot get my https request to go through. I'm having to use SSLv3, so I'm specifying the protocol with:
import requests
from requests.adapters import HTTPAdapter
from urllib3.poolmanager import PoolManager
import ssl
class MyAdapter(HTTPAdapter):
def init_poolmanager(self, connections, maxsize, block=False):
self.poolmanager = PoolManager(num_pools=connections,
maxsize=maxsize,
block=block,
ssl_version=ssl.PROTOCOL_SSLv3)
username = 'username'
password = 'password'
email = 'email#example.com'
url = 'https://api.example.com/'
headers = {'Accept': 'application/json', 'content-type': 'application/json'}
params = {'emailaddress': email}
auth = (username, password)
s = requests.Session()
s.mount(url, MyAdapter())
r = s.get(url+'customer.svc/search', params=params, auth=auth, headers=headers)
When I run my get request I get the following error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/ubuntu/workspace/app/venv/lib/python2.7/site-packages/requests/sessions.py", line 473, in get
return self.request('GET', url, **kwargs)
File "/home/ubuntu/workspace/app/venv/lib/python2.7/site-packages/requests/sessions.py", line 461, in request
resp = self.send(prep, **send_kwargs)
File "/home/ubuntu/workspace/app/venv/lib/python2.7/site-packages/requests/sessions.py", line 573, in send
r = adapter.send(request, **kwargs)
File "/home/ubuntu/workspace/app/venv/lib/python2.7/site-packages/requests/adapters.py", line 370, in send
timeout=timeout
File "/home/ubuntu/workspace/app/venv/lib/python2.7/site-packages/urllib3/connectionpool.py", line 517, in urlopen
timeout_obj = self._get_timeout(timeout)
File "/home/ubuntu/workspace/app/venv/lib/python2.7/site-packages/urllib3/connectionpool.py", line 283, in _get_timeout
return Timeout.from_float(timeout)
File "/home/ubuntu/workspace/app/venv/lib/python2.7/site-packages/urllib3/util/timeout.py", line 152, in from_float
return Timeout(read=timeout, connect=timeout)
File "/home/ubuntu/workspace/app/venv/lib/python2.7/site-packages/urllib3/util/timeout.py", line 95, in __init__
self._connect = self._validate_timeout(connect, 'connect')
File "/home/ubuntu/workspace/app/venv/lib/python2.7/site-packages/urllib3/util/timeout.py", line 125, in _validate_timeout
"int or float." % (name, value))
ValueError: Timeout value connect was Timeout(connect=None, read=None, total=None), but it must be an int or float.
Any ideas? I can't figure it out.
Additional Context: I'm on an Amazon EC2 Ubuntu Instance, running requests 2.5.1 and python 2.7.6
Looks like requests doesn't always play nice when you bring in an outside urllib3 distribution. What I ended up doing was calling requests internal urllib3 instead.
from requests.packages.urllib3.poolmanager import PoolManager
instead of:
from urllib3.poolmanager import PoolManager
No issues since I started doing this.
I'm trying to use the requests module to make a post request to an endpoint that requires ssl auth. My pem file is in the specified path and contains the client cert, and private key. However, I keep getting the Certificate Verified Failed exception. I see in the nginx logs that the request never even made it there. Anyone have any ideas why? I know the certs should work.
params = {
"param_2" : "32100",
"param_1" : "abc"
}
headers = {
"Content-Type" : "application/json"
}
body = json.dumps(params)
r = requests.post(
https://somesite.com/somepath,
data=body,
headers=headers,
timeout=10,
verify="/path/to/cert.pem"
)
Traceback (most recent call last):
File "./somefile.py", line 264, in <module>
start()
File "./somefile.py", line 149, in start
verify="/path/to/cert.pem"
File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 88, in post
return request('post', url, data=data, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 44, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 448, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 554, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 417, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: [Errno 1] _ssl.c:504: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
From the requests docs:
Requests can also ignore verifying the SSL certificate if you set
verify to False.
requests.get('https://kennethreitz.com', verify=False)
By default, verify is set to True. Option verify only applies to host
certs.
You can also specify a local cert to use as client side certificate,
as a single file (containing the private key and the certificate) or
as a tuple of both file’s path:
requests.get('https://kennethreitz.com', cert=('/path/server.crt', '/path/key'))
So it seems like you've just got the args wrong. Try it with 'cert' instead of verify.
I'm trying to access a journal article hosted by an academic service provider (SP), using a Python script.
The server authenticates using a Shibboleth login. I read Logging into SAML/Shibboleth authenticated server using python and tried to implement a login with Python Requests.
The script starts by querying the SP for the link leading to my IDP institution, and is supposed then to authenticate automatically with the IDP. The first part works, but when following the link to the IDP, it chokes on an SSL error.
Here is what I used:
import requests
import lxml.html
LOGINLINK = 'https://www.jsave.org/action/showLogin?redirectUri=%2F'
USERAGENT = 'Mozilla/5.0 (X11; Linux x86_64; rv:28.0) Gecko/20100101 Firefox/28.0'
s = requests.session()
s.headers.update({'User-Agent' : USERAGENT})
# getting the page where you can search for your IDP
# need to get the cookies so we can continue
response = s.get(LOGINLINK)
rtext = response.text
print('Don\'t see your school?' in rtext) # prints True
# POSTing the name of my institution
data = {
'institutionName' : 'tubingen',
'submitForm' : 'Search',
'currUrl' : '%2Faction%2FshowBasicSearch',
'redirectUri' : '%2F',
'activity' : 'isearch'
}
response = s.post(BASEURL + '/action/showLogin', data=data)
rtext = response.text
print('university of tubingen' in rtext) # prints True
# get the link that leads to the IDP
tree = lxml.html.fromstring(rtext)
loginlinks = tree.cssselect('a.extLogin')
if (loginlinks):
loginlink = loginlinks[0].get('href')
else:
exit(1)
print('continuing to IDP')
response = s.get(loginlink)
rtext = response.text
print('zentrale Anmeldeseite' in rtext)
This yields:
continuing to IDP...
2014-04-04 10:04:06,010 - INFO - Starting new HTTPS connection (1): idp.uni-tuebingen.de
Traceback (most recent call last):
File "/usr/lib/python3.4/site-packages/requests/packages/urllib3/connectionpool.py", line 480, in urlopen
body=body, headers=headers)
File "/usr/lib/python3.4/site-packages/requests/packages/urllib3/connectionpool.py", line 285, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/lib/python3.4/http/client.py", line 1066, in request
self._send_request(method, url, body, headers)
File "/usr/lib/python3.4/http/client.py", line 1104, in _send_request
self.endheaders(body)
File "/usr/lib/python3.4/http/client.py", line 1062, in endheaders
self._send_output(message_body)
File "/usr/lib/python3.4/http/client.py", line 907, in _send_output
self.send(msg)
File "/usr/lib/python3.4/http/client.py", line 842, in send
self.connect()
File "/usr/lib/python3.4/site-packages/requests/packages/urllib3/connection.py", line 164, in connect
ssl_version=resolved_ssl_version)
File "/usr/lib/python3.4/site-packages/requests/packages/urllib3/util.py", line 639, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "/usr/lib/python3.4/ssl.py", line 344, in wrap_socket
_context=self)
File "/usr/lib/python3.4/ssl.py", line 540, in __init__
self.do_handshake()
File "/usr/lib/python3.4/ssl.py", line 767, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: TLSV1_ALERT_INTERNAL_ERROR] tlsv1 alert internal error (_ssl.c:598)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.4/site-packages/requests/adapters.py", line 330, in send
timeout=timeout
File "/usr/lib/python3.4/site-packages/requests/packages/urllib3/connectionpool.py", line 504, in urlopen
raise SSLError(e)
requests.packages.urllib3.exceptions.SSLError: [SSL: TLSV1_ALERT_INTERNAL_ERROR] tlsv1 alert internal error (_ssl.c:598)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "./try.py", line 154, in <module>
response = s.get(loginlink)
File "/usr/lib/python3.4/site-packages/requests/sessions.py", line 395, in get
return self.request('GET', url, **kwargs)
File "/usr/lib/python3.4/site-packages/requests/sessions.py", line 383, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python3.4/site-packages/requests/sessions.py", line 486, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python3.4/site-packages/requests/adapters.py", line 385, in send
raise SSLError(e)
requests.exceptions.SSLError: [SSL: TLSV1_ALERT_INTERNAL_ERROR] tlsv1 alert internal error (_ssl.c:598)
Using s.get(loginlink, verify=False) yields exactly the same error. Simply using urllib.request.urlopen(loginlink) does so, too.
Printing and pasting the link into Firefox, on the other hand, works fine.
After trying with openssl s_client it looks like the destination idp.uni-tuebingen.de:443 is only support SSLv3 and misbehaving on anything newer. With forcing SSLv3 one gets:
$ openssl s_client -connect idp.uni-tuebingen.de:443 -ssl3
CONNECTED(00000003)
depth=3 C = DE, O = Deutsche Telekom AG, OU = T-TeleSec Trust Center, CN = Deutsche Telekom Root CA 2
...
But with default setup or forcing TLv1 (-tls1) it only returns an alert:
openssl s_client -connect idp.uni-tuebingen.de:443
CONNECTED(00000003)
140493591938752:error:14077438:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert internal error:s23_clnt.c:741:
So you need to find a way to force SSLv3 for this connection. I'm not familiar with the python at this point but maybe http://docs.python-requests.org/en/latest/user/advanced/ chapter "Example: Specific SSL Version" helps.
And why it works with firefox: the browsers usually retry with a downgraded SSL version if the connects with the safer versions fail. E.g. everybody is trying to work around broken stuff so that the owner of the broken stuff has no intention to fix it :(