I'm trying to sent a request on https url to get data, the domain needs security certificate when I try to execute it on the browser.
But my issue is how to call the url on my python code to get the response data?
I've write the follwing code:
conn = HTTPSConnectionPool(BETTING_CONFG['api_url'],
maxsize = BETTING_CONFG['connection_max_size'])
response = conn.request_encode_body('POST', service_uri, headers= headers,
encode_multipart=False, body = body)
and I get the following response:
Response: status = 200, payload = {"_status":"error","payload":{"_code":"0-2","_message":"invalid_app_key"}} .
and this warning on the terminal:
/usr/local/lib/python2.7/dist-packages/urllib3/util/ssl_.py:318: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#snimissingwarning.
SNIMissingWarning
/usr/local/lib/python2.7/dist-packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
/usr/local/lib/python2.7/dist-packages/urllib3/connectionpool.py:821: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
InsecureRequestWarning)
[555WIN] 2016-05-30 14:02:06,043 - INFO - Betting Response: status = 200, payload = {"_status":"error","payload":{"_code":"0-2","_message":"invalid_app_key"}} .
Traceback (most recent call last):
File "/usr/lib/python2.7/logging/handlers.py", line 76, in emit
if self.shouldRollover(record):
File "/usr/lib/python2.7/logging/handlers.py", line 156, in shouldRollover
msg = "%s\n" % self.format(record)
File "/usr/lib/python2.7/logging/__init__.py", line 724, in format
return fmt.format(record)
File "/usr/lib/python2.7/logging/__init__.py", line 464, in format
record.message = record.getMessage()
File "/usr/lib/python2.7/logging/__init__.py", line 324, in getMessage
msg = str(self.msg)
TypeError: __str__ returned non-string (type dict)
Logged from file jsonapi.py, line 137
Traceback (most recent call last):
File "/usr/lib/python2.7/logging/__init__.py", line 851, in emit
msg = self.format(record)
File "/usr/lib/python2.7/logging/__init__.py", line 724, in format
return fmt.format(record)
File "/usr/lib/python2.7/logging/__init__.py", line 464, in format
record.message = record.getMessage()
File "/usr/lib/python2.7/logging/__init__.py", line 324, in getMessage
msg = str(self.msg)
TypeError: __str__ returned non-string (type dict)
when I added the certificate on the chrome and tried to send the request on the postman, it works fine?
any help how to fix it?
Please understand the your Chrome certificate store is not the same certificate store that is used by your Python application.
It would be much easier if you could only get a valid SSL certificate instead of trying to make self-signed ones to work.
Also, be sure you do upgrade your Python and urllib. Those warning messages are not to be ignored! Resolve them first!
SSL certificates used to be expensive but now you can get valid, fully supported certificates for free from LetsEncrypt. I run my own website using their certificates and I can assure you that Python does have no problem loading their certificates.
Related
How do you fix an SSL error caused by a CNAME DNS record?
I have an API hosted on AWS accessible from a URL like https://sd098fs0f98s9f0s.execute-api.us-east-1.amazonaws.com.
To shorten this and rebrand it a little, I setup a CNAME record to map myapi.mydomain.com to sd098fs0f98s9f0s.execute-api.us-east-1.amazonaws.com.
Using the original URL, this simple Python is able to access the API perfectly:
import requests
r = requests.get(url='https://sd098fs0f98s9f0s.execute-api.us-east-1.amazonaws.com')
However, using the new URL:
import requests
r = requests.get(url='https://myapi.mydomain.com')
results in the error:
Traceback (most recent call last):
File ".env/lib/python3.7/site-packages/urllib3/connectionpool.py", line 603, in urlopen
chunked=chunked)
File ".env/lib/python3.7/site-packages/urllib3/connectionpool.py", line 344, in _make_request
self._validate_conn(conn)
File ".env/lib/python3.7/site-packages/urllib3/connectionpool.py", line 843, in _validate_conn
conn.connect()
File ".env/lib/python3.7/site-packages/urllib3/connection.py", line 370, in connect
ssl_context=context)
File ".env/lib/python3.7/site-packages/urllib3/util/ssl_.py", line 355, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "/usr/lib/python3.7/ssl.py", line 423, in wrap_socket
session=session
File "/usr/lib/python3.7/ssl.py", line 870, in _create
self.do_handshake()
File "/usr/lib/python3.7/ssl.py", line 1139, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1076)
What's causing this and how do I fix it?
I see that you're using https. The server certificate is issued to specific hostname (Common name attribute). This name has to match the address you're trying to access with HTTPS. If you added DNS CNAME you have to update the certificate. If I recall correctly you have to add CNAME as Alternative Subject Name attribute, so you'll need a new certificate.
The error "WRONG_VERSION_NUMBER" might be somewhat misleading. It means that the server presented a wrong TLS version. In the TLS handshake, client and the server negotiate the TLS version. If the server supports only 1.0 and your client accept only 1.2+ then "WRONG_VERSION_NUMBER" will appear. But it might also happen if the client gets some unexpected data, like a plain HTTP instead of HTTPS. To check what exactly was the unexpected data, you'll have to capture wireshark / tcpdump network traffic logs. I wasted a lot of time debugging wrong_version_number recently. Only after looking at wireshark logs it became clear that it was the China firewall, because we got HTTP Forbidden to the initial Client Hello.
I'm trying to connect to an internal site using urlopen. And it's failing repeatedly with SSL error irrespective of providing cafile.
I tried all the various ways explained in the stackoverflow answers. But no luck.
urllib2.urlopen(url,cafile=certifi.where())
Second Way:
context = ssl.create_default_context(cafile=certifi.where())
urllib2.urlopen(url,context=context)
Third Way:
ctx = ssl.create_default_context()
ctx.load_verify_locations(cafile = certifi.where())
urllib2.urlopen(url,context=ctx)
Whichever way, I try, I get the following error.
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "urllib2.py", line 154, in urlopen
return opener.open(url, data, timeout)
File "urllib2.py", line 429, in open
response = self._open(req, data)
File "urllib2.py", line 447, in _open
'_open', req)
File "urllib2.py", line 407, in _call_chain
result = func(*args)
File "urllib2.py", line 1241, in https_open
context=self._context)
File "urllib2.py", line 1198, in do_open
raise URLError(err)
urllib2.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:661)>
I've added the entry of CA certificate to the file certifi.where() too. I use the python version 2.7.14.
Could someone tell me whether I miss something here? Or the python version, that I use doesn't support it? Also, let me know the way to debug this, that is, to find out whether there is any issue with the CA certificate.
Thanks.
EDIT: I don't want to opt out SSL verification as suggested in one of the answers in urllib and "SSL: CERTIFICATE_VERIFY_FAILED" Error. The other answer tells me to use cafile with urlopen which doesn't work in my case. I've tried the solutions given in the answers of this question; but no luck.
Also, openssl throws the following error.
[root#host1 ~]# openssl s_client -connect url -CAfile "cacert.pem"
...
Certificate chain
...
Server certificate
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
...
No client certificate CA names sent
...
SSL handshake has read 2098 bytes and written 415 bytes
...
Verify return code: 2 (unable to get issuer certificate)
---
HTTP/1.0 408 Request Time-out
Cache-Control: no-cache
Connection: close
Content-Type: text/html
<html><body><h1>408 Request Time-out</h1>
Your browser didn't send a complete request in time.
</body></html>
closed
I am having trouble using pyVmomi with python 2.7.5. I get SSL certificate errors when trying to run the sample scripts from the SDK. I tried all the solutions mentioned on this post but none of them worked for me.
Below is the complete console output.
/usr/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:315: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#snimissingwarning. SNIMissingWarning /usr/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:120: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning. InsecurePlatformWarning
Traceback (most recent call last):
File "hello_world_vcenter.py", line 105, in <module>
main()
File "hello_world_vcenter.py", line 80, in main
port=int(args.port))
File "/usr/lib/python2.7/site-packages/pyVim/connect.py", line 663, in SmartConnect
sslContext)
File "/usr/lib/python2.7/site-packages/pyVim/connect.py", line 552, in __FindSupportedVersion
sslContext)
File "/usr/lib/python2.7/site-packages/pyVim/connect.py", line 472, in __GetServiceVersionDescription
tree = __GetElementTreeFromUrl(url, sslContext)
File "/usr/lib/python2.7/site-packages/pyVim/connect.py", line 440, in __GetElementTreeFromUrl
sock = requests.get(url)
File "/usr/lib/python2.7/site-packages/requests/api.py", line 67, in get
return request('get', url, params=params, **kwargs)
File "/usr/lib/python2.7/site-packages/requests/api.py", line 53, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 468, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 576, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 447, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: [Errno 1] _ssl.c:504: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
That looks like you're using a self-signed certificate. While connecting via SmartConnect use your own sslContext and disable certificate verification.
from pyVim.connect import SmartConnect
import ssl
context = ssl.SSLContext(ssl.PROTOCOL_TLSv1)
context.verify_mode = ssl.CERT_NONE
si = SmartConnect(host=somehost.com, port=443, user=someone, pwd=secret, sslContext=context)
... or use a signed ssl certificate.
There are a few similiar questions on here (e.g. here).
I recently got an SSL certificate for my site:
https://ram.rachum.com/
It works great in browsers. But it fails for requests:
>>> import requests
>>> requests.get('https://ram.rachum.com')
Traceback (most recent call last):
File "<pyshell#1>", line 1, in <module>
requests.get('https://ram.rachum.com')
File "C:\Python27\lib\site-packages\requests\api.py", line 55, in get
return request('get', url, **kwargs)
File "C:\Python27\lib\site-packages\requests\api.py", line 44, in request
return session.request(method=method, url=url, **kwargs)
File "C:\Python27\lib\site-packages\requests\sessions.py", line 354, in request
resp = self.send(prep, **send_kwargs)
File "C:\Python27\lib\site-packages\requests\sessions.py", line 460, in send
r = adapter.send(request, **kwargs)
File "C:\Python27\lib\site-packages\requests\adapters.py", line 250, in send
raise SSLError(e)
SSLError: hostname 'ram.rachum.com' doesn't match either of '*.webfaction.com', 'webfaction.com'
Why? Why does requests look at the webfaction certificate rather than my own certificate, which is valid for ram.rachum.com?
You are using a requests library without support for SNI (server name indication), but you have multiple SSL certificates behind the same IP address which requires SNI. You can verify this with openssl s_client. Without given a name for SNI the server just gives the default certificate for this IP, which is *.webfaction.com:
openssl s_client -connect ram.rachum.com:443
...
0 ...CN=*.webfaction.com
But if you specify a hostname for SNI it returns the expected certificate:
openssl s_client -connect ram.rachum.com:443 -servername ram.rachum.com
...
0 ...CN=ram.rachum.com...
Maybe you need to upgrade your requests library and other modules too, see using requests with TLS doesn't give SNI support
I am attempting to use the requests.py library for calls to a rest web service. I wrote a quick prototype for my usage under windows and everything worked fine, but when I attempted to run the same prototype under linux I get a "requests.exceptions.Timeout: Request timed out" error. Does anyone know why this might be happening? If I try to use the library to access a non https url it works fine under both windows and linux.
import requests
url = 'https://path.to.rest/svc/?options'
r = requests.get(url, auth=('uid','passwd'), verify=False)
print(r.content)
I did notice that if I leave off the verify=False parameter from the get call, I get a different exception, namely "requests.exceptions.SSLError: Can't connect to HTTPS URL because the SSL module is not available". This appears to be a possible underlying cause, though I dont know why they would change the errorcode, but I cant find any reference to an ssl module and I verified that certifi was installed. Interestingly, if I leave off the verify parameter in windows I get a different exception, "requests.exceptions.SSLError: [Errno 1] _ssl.c:503: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed"
EDIT:
Tracebacks for all cases/scenarios mentioned
Full code as shown above:
Traceback(most recent call last):
File "testRequests.py", line 15, in <module>
r = requests.get(url, auth=('uid','passwd'), verify=False)
File "build/bdist.linux-x86_64/egg/requests/api.py", line 52, in get
File "build/bdist.linux-x86_64/egg/requests/api.py", line 40, in request
File "build/bdist.linux-x86_64/egg/requests/sessions.py", line 208, in request
File "build/bdist.linux-x86_64/egg/requests/models.py", line 586, in send
requests.exceptions.Timeout: Request timed out
Code as shown above minus the "verify=False" paramter:
Traceback(most recent call last):
File "testRequests.py", line 15, in <module>
r = requests.get(url, auth=('uid','passwd'))
File "build/bdist.linux-x86_64/egg/requests/api.py", line 52, in get
File "build/bdist.linux-x86_64/egg/requests/api.py", line 40, in request
File "build/bdist.linux-x86_64/egg/requests/sessions.py", line 208, in request
File "build/bdist.linux-x86_64/egg/requests/models.py", line 584, in send
requests.exceptions.SSLError: Can't connect to HTTPS URL because the SSL module is not available
Code as show above minus the "verify=False" parameter and run under windows:
Traceback(most recent call last):
File "testRequests.py", line 59, in <module>
r = requests.get(url, auth=('uid','passwd'))
File "c:\Python27\lib\site-packages\requests\api.py", line 52, in get
return request('get', url, **kwargs)
File "c:\Python27\lib\site-packages\requests\api.py", line 40, in request
return s.request(method=method, url=url, **kwargs)
File "c:\Python27\lib\site-packages\requests\sessions.py", line 208, in request
r.send(prefetch=prefetch)
File "c:\Python27\lib\site-packages\requests\models.py", line 584, in send
raise SSLError(e)
requests.exceptions.SSLError: [Errno 1] _ssl.c:503: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
I'm not an expert on the matter but it looks like the certificate from the server can't be verified correctly. I don't know how Python and ssl handles certificate verification but the first option is to try ignoring the exception, or maybe change https to http in an attempt to see if the web-service allows non-secure service calls.
If the issue is revolving around an import error for ssl, the module is part of CPython and you may need to ensure that the Python interpreter is compiled with SSL support (from openssl). Look into removing the package for python (be careful) and compiling it with openssl support, personally I would strongly advise you looking into a virtualenv before removing anything, compiling Python is not too difficult and it would give you a finer grain of control for what you aim to do.