I have created a Python application and am packaging it with PyInstaller. On my own computer, I have configured my own trust store CA bundle to include my company's proxy based on Self-Signed Certificate Authorities pip / conda but I don't understand how to bring that into the PyInstaller executable so it works on other people's computers who have not configured their own trust store CA bundle.
The error I get is [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain
I am using PyInstaller 5.7.0 (with --onefile option), certifi 2022.12.7, and Python 3.8. This is primarily a dash application (2.7.1). First, the user selects a ZIPcode and I use uszipcode (version 0.2.6) to get information on that ZIPcode. That is where the error first occurs.
The application runs correctly when disconnected from the VPN. When I generate the executable, I use the --add-data flag to include cacert.pem from the certifi Python package. I tried manually adding my company's cert to the cacert.pem but it will get overwritten if I update certifi.
Where should I add in my company's SSL certificate so users don't have to disconnect from the VPN to use the application?
Related
Basic idea: I want to get this zipped artifactory, go through with a loop open every text file and add a line and save it. After that I want to upload the new version of this folder, again as a zipped folder.
My first problem is I cant download this zipped folder from artifactory.
I have already tried this out: https://github.com/devopshq/artifactory
I changed the same code with my personal path, username and pw (changed it for this post because of security) but I get lots of errors:
UPDATE: no SSL errors anymore but I still have a problem with my code.
The error seems to be related to validation of your Artifactory SSL certificate.
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)
You should look at the various options which the Python interface library for JFrog Artifactory is offering for handling SSL certificate validations.
For example disable host cert verification
from artifactory import ArtifactoryPath
path = ArtifactoryPath(
"http://my-artifactory/artifactory/libs-snapshot-local/myapp/1.0", verify=False
)
Following error occurs only with docker app in python when making request to an https url.
<urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1108)>
Outside of docker, the app works. I can fetch the same URL inside the docker image of other language app such as dotnet.
I have tried:
RUN update-ca-certificates
Install certfi library and manually supply the certificates during making the call
Manually insert the certificates that comes with certify library in different locations of docker images such as /usr/local/share/ca-certificates/, /etc/ssl/certs/ and RUN update-ca-certificates
Tried different versions (3.6.9, 3.8.4) and providers (alpine, buster, slim-buster ) of python.
Setting different env variables such as REQUESTS_CA_BUNDLE, SSL_CERT_FILE etc.
Use different libraries such as requests, urllib, urllib3
.... and really large number of different things.
It of course works when I turn the verify off, but I want to keep verification.
I was having this issue in an ARM Ubuntu 20.04 container.
I installed ca-certificates and curl, but I still couldn't use curl. For me the fix ended up being adding the following before calling curl:
export SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt
here is my code
import requests;
url='that website';
headers={
'Accept':'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
'Accept-Language':'zh-CN,zh;q=0.9,en;q=0.8,ja;q=0.7',
'User-Agent':'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36'
};
r = requests.get(url,headers=headers);
print(r);
print(r.status_code);
then it ran into this:
requests.exceptions.SSLError:
HTTPSConnectionPool(host='www.xxxxxx.com', port=44 3):
Max retries exceeded with url: xxxxxxxx (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED]
certificate verify failed: unable to get local issuer certificate
(_ssl.c:1045)')))
what should i do?
It's not recommended to use verify = False in your organization's environments. This is essentially disabling SSL verification.
Sometimes, when you are behind a company proxy, it replaces the certificate chain with the ones of Proxy. Adding the certificates in cacert.pem used by certifi should solve the issue. I had similar issue. Here is what I did, to resolve the issue -
Find the path where cacert.pem is located -
Install certifi, if you don't have. Command: pip install certifi
import certifi
certifi.where()
C:\\Users\\[UserID]\\AppData\\Local\\Programs\\Python\\Python37-32\\lib\\site-packages\\certifi\\cacert.pem
Open the URL on a browser. Download the chain of certificates from the URL and save as Base64 encoded .cer files.
Now open the cacert.pem in a notepad and just add every downloaded certificate contents (---Begin Certificate--- *** ---End Certificate---) at the end.
If you have already tried to update the CA(root) Certificate using pip:
pip install --upgrade certifi
or have already downloaded the newest version of cacert.pem from https://curl.haxx.se/docs/caextract.html and replaced the old one in {Python_Installation_Location}\\lib\\site-packages\\certifi\\cacert.pem but it still does not work, then your client is probably missing the Intermediate Certificate in the trust chain.
Most browsers can automatically download the Intermediate Certificate using the URL in
"Authority Info Access" section in the Certificate, but Python, Java, and openssl s_client cannot. They rely on the server proactively sending them the intermediate certificate.
If you speak Chinese you can read this awesome blog: https://www.cnblogs.com/sslwork/p/5986985.html and use this tool to check if the intermediate certificate is sent by / installed on the server or not: https://www.myssl.cn/tools/check-server-cert.html
If you do not, you can check this article: https://www.ssl.com/how-to/install-intermediate-certificates-avoid-ssl-tls-not-trusted/
We can also use openssl in Linux to cross-check this issue:
openssl s_client -connect yourwebsite:443
The error message is even the same -- "unable to get local issuer certificate". I doubt that "local" here actually means "intermediate".
My current solution for this problem is like #Indranil's suggestion (https://stackoverflow.com/a/57466119/4522434): Export the Intermediate Certificate in browser using base64 X.509 CER format; then use Notepad++ to open it and copy the content into the end of cacert.pem in {Python_Installation_Location}\\lib\\site-packages\\certifi\\cacert.pem
Answers pointing to certifi are a good start and in this case there could be an additional step needed if on Windows.
pip install python-certifi-win32
The above package would patch the installation to include certificates from the local store without needing to manage store files manually. The patch was suggested to certifi but declined as "the purpose of certifi is not to be a cross-platform module to access the system certificate store." [https://github.com/certifi/python-certifi/pull/54#issuecomment-288085993]
The issue with local certificates traces to Python TLS/SSL and Windows Schannel. There is an open issue at Python [https://bugs.python.org/issue36011] and PEP that did not lead to a solution [https://www.python.org/dev/peps/pep-0543/#resolution]
If you're using macOS, search for "Install Certificates.command" file (it is usually in Macintosh HD > Applications > your_python_dir).
You can also find it with "command" + "break space" and paste "Install Certificates.command" in the field.
If you used brew to install python, your solution is there:
brew installation of Python 3.6.1: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed
I had the same problem. I was able to make requests against my server via the browser, but using python requests, I was getting the error mentioned above. Requests and certifi were both fully up to date; the problem ended up being my server's configuration.
The problem was that I had only installed the intermediate cert instead of the full cert chain.
In my case, following this article, I simply ran cat my-domain.crt my-domain.ca-bundle > my-domain.crt-combined and installed the crt-combined file on my server (via heroku's app settings interface) instead of the crt file.
In macOS just open Macintosh HD
Now Select Application Then Select Python folder ( Python3.6, Python3.7 Whatever You are using just select this folder )
Then, double click on Install Certificates.command. Now your error should be solved.
You can also set REQUESTS_CA_BUNDLE env variable to force requests library to use your cert, that solved my issue.
This should solve your problem
This is because the url is a https site instead of http.
So it requires ssl verification using certificates. If you are working in your firms workstation, internal use sites will be accessible through the browser managed by your organization. The organization will have setup the certificates.
Atleast these certificates are needed
ROOT CA certificate
Intermediate CA certificate
Website ( domain ) certificate
The browsers will have these certificates configured, but python will not. So you need to do some manual work to get it working.
As Indranil suggests, using verify=False is not recommended. So download all the certificates as mentioned in the above link and follow the steps.
How it broke
While troubleshooting my command line trying to ssh root#ip into Docker, I seem to I have messed up with openssl, somehow.
I can now connect to Docker, but I no longer can run scripts which require requests.
1st attempt
At first I did not know what was wrong.
If I ran script.py, it would just halt and fail silently.
2nd attempt
Then I tried to run the same script from within a conda env, which has openssl installed, and this time error was verbose, ending with:
NetworkError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:661)
openssl upgrade
I have then tried to upgrade it system-wide with brew upgrade openssl, after which I got the following Caveats:
A CA file has been bootstrapped using certificates from the SystemRoots
keychain. To add additional certificates (e.g. the certificates added in
the System keychain), place .pem files in
/usr/local/etc/openssl/certs
and run
/usr/local/opt/openssl/bin/c_rehash
and I did it.
my .ssh/ has:
github_rsa
github_rsa.pub
id_boot2docker
id_boot2docker.pub
id_rsa
id_rsa.pub
known_hosts
the problem was a library called billboard.py, being imported.
this import was causing the problem and halting the program, and the reason needs further investigation.
verbose Error was because conda env did not have all requeriments to run the program.
I am trying to connect to S3 using boto, but it seems to fail. I've tried some workarounds, but they don't seem to work. Can anyone please help me with this. Below is the code.
import boto
if not boto.config.has_section('Credentials'):
boto.config.add_section('Credentials')
boto.config.set('Credentials', 'aws_access_key_id', AWS_KEY)
boto.config.set('Credentials', 'aws_secret_access_key', AWS_SECRET_KEY)
if not boto.config.has_section('Boto'):
boto.config.add_section('Boto')
boto.config.set('Boto', 'https_validate_certificates', 'False')
boto.config.add_section('aws info')
boto.config.set('aws info','aws_validate_certs','False')
s3 = boto.connect_s3(validate_certs=False)
bucket = s3.get_bucket(Bucket_NAME)
Probably your bucket name contains a dot, that's why ssl certificate verification fails. This is quite a frequent problem, see this github issue for example.
Don't use an insecure connection (is_secure=False), instead use OrdinaryCallingFormat:
import boto
conn = boto.s3.connect_to_region('eu-west-1', calling_format=boto.s3.connection.OrdinaryCallingFormat())
bucket = conn.get_bucket(your_bucket)
You probably need to update your AWS Region, e.g. us-east-1
In boto3, if you are using the s3 client, use verify=False when creating the s3 client.
For eg:
s3 = boto3.client('s3', verify=False)
As mentioned on boto3 documentation, this only turns off validation of SSL certificates. SSL will still be used (unless use_ssl is False), but SSL certificates will not be verified.
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html
I found a way,
used is_secure=False in connect_s3().
I encounter this problem, too. My environment is Ubuntu 15.04, Python 2.7.9 and Boto 2.38.0.
Setting the argument validate_certs=False doesn't make it work with the HTTPS connection without valid certificate. After reading the code of boto, I found that it's a behavior of Python's ssl modules. Then I found a solution here: "SSL: CERTIFICATE_VERIFY_FAILED" Error. And the solution does work!!!.
add verify=False
boto3.resource(
"s3",
endpoint_url=<URL>,
aws_access_key_id=<ID>,
aws_secret_access_key=<Key>,
verify=False
)
macOS users: If you are using the Python 3.6 from the python.org
binary installer linked on this page, please carefully read the
Important Information displayed during installation; this information
is also available after installation by clicking on
/Applications/Python 3.6/ReadMe.rtf. There is important information
there about changes in the 3.6.0 installer-supplied Python,
particularly with regard to SSL certificate validation.
https://www.python.org/downloads/release/python-360/
From ReadMe.rtf at the time of this writing:
Certificate verification and OpenSSL
NEW This variant of Python 3.6 now includes its own private copy of OpenSSL 1.0.2. Unlike previous releases, the deprecated
Apple-supplied OpenSSL libraries are no longer used. This also means
that the trust certificates in system and user keychains managed by
the Keychain Access application and the security command line utility
are no longer used as defaults by the Python ssl module. For 3.6.0, a
sample command script is included in /Applications/Python 3.6 to
install a curated bundle of default root certificates from the
third-party certifi package (https://pypi.python.org/pypi/certifi).
If you choose to use certifi, you should consider subscribing to the
project's email update service to be notified when the certificate
bundle is updated.
The bundled pip included with the Python 3.6 installer has its own
default certificate store for verifying download connections.
Office laptops usually have network monitors installed. Figured out that it was the network monitoring software interfering with python, not letting it verify ssl certs of aws. We had to import its's cert(got from office) onto python's cacert.pem file, then it started working fine.