Develop Reference

SSL: Certificate verification error on Ubuntu

Bear with me here guys, I am using Flask and I have two different API services running on different IP's and ports on Ubuntu 20.04.3 LTS. All I want to do is send a POST or GET request from one to another using an SSL connection.
I am able to create cert and key files with using "sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/ssl/name_of_key.key -out /etc/ssl/name_of_crt.crt" command, also
added these files here /etc/nginx/sites-enabled/myapi.service in order to have SSL connection.
My NGINX file is given below:
server {
listen 60000 ssl;
ssl_certificate /etc/ssl/name_of_cert.crt;
ssl_certificate_key /etc/ssl/name_of_key.key;
server_name IP_ADDRESS;
location / {
include uwsgi_params;
uwsgi_pass unix:/home/first_api/first_api.sock;
}}
The problem occurs when I try to send a POST request from one API to another. My code for first API is given below:
from flask import Flask, jsonify
import requests
app = Flask(__name__)
#app.route('/sample_api_one', methods=['POST', 'GET'])
def service():
data = {}
data["message"] = "first message has arrived"
response = requests.post('https://SECOND_API_IP_ADDRESS:57949/second_api', json=data, verify="/etc/ssl/name_of_crt.crt")
return jsonify(response.json())
I know that I can send my request with changing "verify=False" but that's not the thing I want.
I would like to send my request with using SSL connection. Almost tried everything here but still not able to do it.
Lastly, here is my error code when I try to run my app and send a request:
File "/home/venv_first_api/lib/python3.8/site-packages/requests/adapters.py", line 514, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='SECOND_API_IP_ADDRESS', port=57949): Max retries exceeded with url: /second_api (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:1131)')))

Can't send GET to HTTPS service with cert, error: certificate verify failed: self signed certificate

I'm trying to send a GET request to a host with (supposedly) correct certificates.
It's a university task, and they gave me these certificates. (which are only valid for 30 seconds)
But the code below gives me the error that certificate verify failed: self signed certificate
The package I got from the host in response says that Fatal Error: Unknown CA.
What could cause the issue? Thanks!
context = ssl.SSLContext(ssl.PROTOCOL_TLSv1_2)
context.load_cert_chain('clientcert.pem', keyfile='clientkey.pem')
connection = http.client.HTTPSConnection(IP)
connection.request("GET", "/")
response = connection.getresponse()
print("response:", response)

The error message seems to be self-explanatory. Self-signed SSL certificates always cause security warnings/errors. You will either need to add your self-signed SSL as an exception or add the self-signed CA to OS trusted certificates pool.
You may also try using something identical to --insecure option in curl.

SSL Error (bad handshake) with valid certificate | Elasticsearch & Python

I am passing data to Elasticsearch (ES) through a Python script. First, I secured ES with a self-signed certificate and everything works as expected. Then, I switched to a more trusted certificate (Let's Encrypt). Note, that I can reach my ES cluster without any problems. The Let's Encrypt cert is trusted by my browser + by an application that is talking to ES, no problem. But when I try to pass data from my Python script to ES with the new certificate, I get following error:
urllib3.exceptions.SSLError: ("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",)
I would have expected this error with a self signed cert but not with Let's encrypt. The only way I can avoid it, is by changing the settings to verify=False, which is no long-term solution.
Before I received the error message mentioned above, I got following error:
elasticsearch.exceptions.SSLError: ConnectionError([SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777)) caused by: SSLError([SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777))
I found a workaround for this by doing pip install requests. However, afterwards I receive the first error I mentioned (bad handshake). I know that this means that the certificate is not trusted. But how can this be, if it works for a self-signed cert but not for a Let's Encrypt cert that is trusted by a browser and an app? E.g. if I call ES on https://my-IP:9200, no warning is given by my browser, while a warning will be given with the self-signed cert.
Some additional info
python3
urllib3 1.25.7
certifi 2019.9.11
Ubuntu 18.04
So, basically everything is up-to-date. I also tried a suggested solution by downgrading certifi and/or urllib3, but it doesn't work. One suggestion is to downgrade urllib3 below version 1.25 (but as I said, it doesn't work in my case).
Any ideas?

You need to install certificate first, and then use it in connection to Elasticsearch with Python
host = 'mydomain.com:9200'
client = Elasticsearch(host, http_auth=('admin', 'pass'), scheme="https", use_ssl=True, ca_certs='C:/my_path/CertificateFile.cer.pem', port=443)
try:
info = json.dumps(client.info(), indent=4)
print ("Elasticsearch client info():", info)
except exceptions.ConnectionError as err:
print ("\nElasticsearch info() ERROR:", err)
print ("\nThe client host:", host, "is invalid or cluster is not running")
client = None
Response:
> Elasticsearch client info(): {
> "name": "my_name",
> "cluster_name": "my_cluster_name",
> "cluster_uuid": "fBRShbkSRy2vcfQJZsojGA",
> "version": {
> "number": "7.3.0",
> "build_flavor": "default",
> "build_type": "tar",
> "build_hash": "de777fa",
> "build_date": "2019-07-24T18:30:11.767338Z",
> "build_snapshot": false,
> "lucene_version": "8.1.0",
> "minimum_wire_compatibility_version": "6.8.0",
> "minimum_index_compatibility_version": "6.0.0-beta1"
> },
> "tagline": "You Know, for Search" }
elasticsearch.yml:
xpack.security.enabled: true
xpack.ml.enabled: false
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: elastic-certificates.p12
xpack.security.http.ssl.truststore.path: elastic-certificates.p12

Python jenkinsapi ignore certificate

Recently, I turned jenkins to https.
This is my code to use jenkinsapi :
import jenkinsapi
from jenkinsapi.jenkins import Jenkins
from jenkinsapi.utils.requester import Requester
import requests
url = 'https://jenkinsd:443'
username = 'MyUser'
password = '123'
requests.packages.urllib3.disable_warnings()
jenkins = Jenkins(url, username, password)
jobs = jenkins.get_jobs()
for jobName in jobs:
print(jobName)
I get this error:
requests.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:600)
I tried:
jenkins = Jenkins(url, requester=Requester(username, password, baseurl=url, ssl_verify=False))
OR:
jenkins = jenkinsapi.jenkins.Jenkins(url, username, password, requester =Requester(username, password, ssl_verify=False))
I get this error:
File "D:\Python34\lib\site-packages\requests-2.7.0-py3.4.egg\requests\adapters.py", line 415, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionRefusedError(10061, 'No connection could be made because the target machine actively refused it', None, 10061, None))
Advice please :)

Solved..
jenkins = Jenkins(url, username, password, ssl_verify=False)
After correcting url : Manage Jenkins > Configure system > Jenkins URL
The jenkinsapi now works.
https://github.com/pycontribs/jenkinsapi/blob/master/doc/source/ssl_certificate_verification
Thanks!

It appears the authors included an environment variable to disable SSL verification. This approach also has the benefit of suppressing the insecure request warnings that urllib3 will print.
Simply set the variable below in your shell before executing your python script.
In Bash:
export PYTHONHTTPSVERIFY=0
python myscript.py

SSL bad Handshake Error 10054 "WSAECONNRESET"

Notes:
versions
Python 2.7.11 and my requests version is '2.10.0'
'OpenSSL 1.0.2d 9 Jul 2015'
Please read the below comment by Martijn Pieters before reproducing
Initially I tried to get pdf from https://www.neco.navy.mil/necoattach/N6945016R0626_2016-06-20__INFO_NAS_Pensacola_Base_Access.docx using code as below
code1:
>>> import requests
>>> requests.get("https://www.neco.navy.mil/necoattach/N6945016R0626_2016-06-20__INFO_NAS_Pensacola_Base_Access.docx",verify=False)
Error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Users\mob140003207\AppData\Local\Enthought\Canopy\User\lib\site-packa
ges\requests\api.py", line 67, in get
return request('get', url, params=params, **kwargs)
File "C:\Users\mob140003207\AppData\Local\Enthought\Canopy\User\lib\site-packa
ges\requests\api.py", line 53, in request
return session.request(method=method, url=url, **kwargs)
File "C:\Users\mob140003207\AppData\Local\Enthought\Canopy\User\lib\site-packa
ges\requests\sessions.py", line 468, in request
resp = self.send(prep, **send_kwargs)
File "C:\Users\mob140003207\AppData\Local\Enthought\Canopy\User\lib\site-packa
ges\requests\sessions.py", line 576, in send
r = adapter.send(request, **kwargs)
File "C:\Users\mob140003207\AppData\Local\Enthought\Canopy\User\lib\site-packa
ges\requests\adapters.py", line 447, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: ("bad handshake: SysCallError(10054, 'WSAECONNRESE
T')",)
After googling and searching I found that you have use SSL verification and using session with adapters can solve the problem. But I still got error's please find the code and error's below
Code2:
import requests
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.poolmanager import PoolManager
import ssl
import traceback
class MyAdapter(HTTPAdapter):
def init_poolmanager(self, connections, maxsize, block=False):
self.poolmanager = PoolManager(num_pools=connections,
maxsize=maxsize,
block=block,
ssl_version=ssl.PROTOCOL_TLSv1)
s = requests.Session()
s.mount('https://', MyAdapter())
print "Mounted "
r = s.get("https://www.neco.navy.mil/necoattach/N6945016R0626_2016-06-20__INFO_NAS_Pensacola_Base_Access.docx", stream=True, timeout=120)
Error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Users\mob140003207\AppData\Local\Enthought\Canopy\User\lib\site-packa
ges\requests\sessions.py", line 480, in get
return self.request('GET', url, **kwargs)
File "C:\Users\mob140003207\AppData\Local\Enthought\Canopy\User\lib\site-packa
ges\requests\sessions.py", line 468, in request
resp = self.send(prep, **send_kwargs)
File "C:\Users\mob140003207\AppData\Local\Enthought\Canopy\User\lib\site-packa
ges\requests\sessions.py", line 576, in send
r = adapter.send(request, **kwargs)
File "C:\Users\mob140003207\AppData\Local\Enthought\Canopy\User\lib\site-packa
ges\requests\adapters.py", line 447, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: ("bad handshake: SysCallError(10054, 'WSAECONNRESET')",)

First of all, I confirm that the host, www.neco.navy.mil, is not accessible from everywhere. From some networks (geography) it works*, from others connection just hangs:
$ curl www.neco.navy.mil
curl: (7) couldn't connect to host
$ curl https://www.neco.navy.mil
curl: (7) couldn't connect to host
Second, when connection can be established there is an certificate problem:
$ curl -v https://www.neco.navy.mil
* Rebuilt URL to: https://www.neco.navy.mil/
* Hostname was NOT found in DNS cache
* Trying 205.85.2.133...
* Connected to www.neco.navy.mil (205.85.2.133) port 443 (#0)
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS alert, Server hello (2):
* SSL certificate problem: unable to get local issuer certificate
* Closing connection 0
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: http://curl.haxx.se/docs/sslcerts.html
curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.
To make sure, you just feed it to Qualys SSL tester:
The CA (DoD Root CA 2) is not trusted. Moreover it's not in the chain. Note that OpenSSL validation process needs whole chain:
Firstly a certificate chain is built up starting from the supplied certificate and ending in the root CA. It is an error if the whole chain cannot be built up.
But there's only www.neco.navy.mil -> DODCA-28. It may be related to the TLD and extra security measure, but C grade alone isn't much anyway ;-)
On they Python side it won't be much different. If you don't have access to the CA, you can only disable certificate validation entirely (after you have connectivity problem solved, of course). If you have it, you can use cafile.
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import urllib2
import ssl
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
r = urllib2.urlopen('https://www.neco.navy.mil/'
'necoattach/N6945016R0626_2016-06-20__INFO_NAS_Pensacola_Base_Access.docx',
timeout = 5, context = ctx)
print(len(r.read()))
r = urllib2.urlopen('https://www.neco.navy.mil/'
'necoattach/N6945016R0626_2016-06-20__INFO_NAS_Pensacola_Base_Access.docx',
timeout = 5, cafile = '/path/to/DODCA-28_and_DoD_Root_CA_2.pem')
print(len(r.read()))
To reproduce with certain version of Python, use simple Dockerfile like follows:
FROM python:2.7.11
WORKDIR /opt
ADD . ./
CMD dpkg -s openssl | grep Version && ./app.py
Then run:
docker build -t ssl-test .
docker run --rm ssl-test

This snippet works for me (py2.7.11 64bits + requests==2.10.0) on windows7:
import requests
import ssl
import traceback
import shutil
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.poolmanager import PoolManager
class MyAdapter(HTTPAdapter):
def init_poolmanager(self, connections, maxsize, block=False):
self.poolmanager = PoolManager(num_pools=connections,
maxsize=maxsize,
block=block,
ssl_version=ssl.PROTOCOL_TLSv1)
if __name__ == "__main__":
s = requests.Session()
s.mount('https://', MyAdapter())
print "Mounted "
filename = "N6945016R0626_2016-06-20__INFO_NAS_Pensacola_Base_Access.docx"
r = s.get(
"https://www.neco.navy.mil/necoattach/{0}".format(filename), verify=False, stream=True, timeout=120)
if r.status_code == 200:
with open(filename, 'wb') as f:
r.raw.decode_content = True
shutil.copyfileobj(r.raw, f)

I use python 2.7.6 and this simple example still working on my ubuntu 14.04
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import requests
from requests.packages.urllib3.exceptions import InsecureRequestWarning
requests.packages.urllib3.disable_warnings(InsecureRequestWarning)
with open('out.docx', 'wb') as h :
r = requests.get("https://www.neco.navy.mil/necoattach/N6945016R0626_2016-06-20__INFO_NAS_Pensacola_Base_Access.docx", verify=False, stream=True)
for block in r.iter_content(1024):
h.write(block)