I am new to Python, trying to read & parse logs of applications https url. My org has proxy setup. I am able to connect http url using basic_auth providing & & also by CA5 cert using below code
import urllib3
from urllib3 import ProxyManager, make_headers
# default_headers = make_headers(proxy_basic_auth='<username>:<password>')
# http = ProxyManager("http://<proxy_server>:<port>/", headers=default_headers)
http = ProxyManager("http://<proxy_server>:<port>/",cert_reqs='CERT_REQUIRED',ca_certs='<Windows path to .cer>')
r = http.request('GET','http://google.com')
print(r.data)
.pem certificate got saved as .cer on windows system.
This is working as excepted and returnign google data.
Now while trying to use same for HTTPS url using below,
r = http.request('GET','https://google.com',verify='<same as above i.e. Windows path to .cer>')
It is failing with below errors,
.
.
OSError: Tunnel connection failed: 502 notresolvable
.
.
.
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='google.com', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749)'),))
As the log files are keeping updating, might need to stream it; Suggestion on this is highly appreciated.
Using Python 3.6.0 |Anaconda 4.3.1 (64-bit) on Windows 7.
Related
I need to call a web API. For that I need a bearer token.
I am using databricks(python) code to first get authenticated over Microsoft AAD. Then get bearer token for my service_user. I Followed the microsoft docs docs
But facing problem where it hits our Company server and asking for SSL certificate.
I can't install any certificate. What could be a better way to avoid it. Below is my short code taken from above microsoft and Git repos. but its not working.
Can i get help!
clientId = "42xx-xx-xx5f"
authority = "https://login.microsoftonline.com/tenant_id/"
app = msal.PublicClientApplication(client_id=clientId, authority=authority)
user = "serviceuser#company.com"
pwd = "password"
scope = "Directory.Read.All"
result = app.acquire_token_by_username_password(scopes=[scope], username=user, password=pwd)
print(result)
Got below error
HTTPSConnectionPool(host='mycompany.com', port=443): Max retries exceeded with url: /adfs/services/trust/mex (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1125)')))
The problem is that the code uses the requests library that relies on the certifi package instead of using Linux certificate chain (so existing instructions doesn't work). To solve that problem it's better to use cluster init script that will install SSL certificate when cluster starts. Something like this (requests and certifi are installed by default), just replace CERT_FILE with actual path to the .pem file with CA certificate:
CERT_FILE="/dbfs/....."
CERTIFI_HOME="$(python -m certifi 2>/dev/null)"
cat $CERT_FILE >> $CERTIFI_HOME
I am trying to scrape data from a url using beautifulsoup. Below is my code
import requests
URL = "https://bigdataldn.com/speakers/"
page = requests.get(URL)
print(page.text)
However I am getting the following error when I run the code in google colab.
SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1091)
During handling of the above exception, another exception occurred:
MaxRetryError Traceback (most recent call last)
MaxRetryError: HTTPSConnectionPool(host='bigdataldn.com', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1091)')))
The above code works fine for other urls.
Can someone help me figure out how to solve this issue.
It's not your fault - their certificate chain is not properly configured. What you can do is disabling the certificate verification (you should not do this when you're handling sensitive information!) but it might be fine for a webscraper.
page = requests.get(URL, verify=False)
enter image description here
Your SSL certificate is not installed properly , you can follow godaddy ssl install instruction maybe its helpfull .
https://in.godaddy.com/help/install-my-ssl-certificate-16623?sp_hp=B&xpmst=A&xpcarveout=B
I am using the ktrain package in jupyter with code supplied from this notebook. I get an error at the line qa = text.SimpleQA(INDEXDIR). The error is long but a shortened version is as follows:
HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /bert-large-uncased-whole-word-masking-finetuned-squad/resolve/main/config.json (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1125)')))
HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /bert-large-uncased-whole-word-masking-finetuned-squad/resolve/main/config.json (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1125)')))
OSError: Can't load config for 'bert-large-uncased-whole-word-masking-finetuned-squad'. Make sure that:
- 'bert-large-uncased-whole-word-masking-finetuned-squad' is a correct model identifier listed on 'https://huggingface.co/models'
- or 'bert-large-uncased-whole-word-masking-finetuned-squad' is the correct path to a directory containing a config.json file
I can access https://huggingface.co/bert-large-uncased-whole-word-masking-finetuned-squad/resolve/main/config.json on my browser. I'm quite at a loss for what to do - my coding skills are minimal at best so any and all suggestions would be much appreciated.
My guess is that your corporate intranet is inserting a "man in the middle" on all https traffic. I'm guessing the following will give you the same error right now:
import requests
requests.get('https://www.huggingface.co')
If you get a CA certificate bundle from your IT department and you are on Windows, you can try this:
import os
os.environ['REQUESTS_CA_BUNDLE'] = 'path/to/certificates_ca_bundle.crt'
qa = text.SimpleQA(INDEXDIR)
If on Linux, install the certificates using these instructions.
I am trying to connect to GCP where I have set up my file ("private_key_file") and trying to read the content within that file.
def gcsStorageClient(errLogRef):
try:
storage_client = storage.Client(os.environ.get("project_name"))
bucket = storage_client.get_bucket(os.environ.get("bucket_name"))
blob = bucket.get_blob(os.environ.get("private_key_file"))
file = blob.download_as_string()
return file
except Exception as error:
print("error:: " + error)
But I am seeing the below exception from my MAC OS which is related to SSL Verification, it seems like MAC firewall client is blocking the connect. Is there any way to say SSLVerify false or create a proxy to connect to GCP
exception: HTTPSConnectionPool(host='oauth2.googleapis.com', port=443): Max retries exceeded
with url: /token (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate
verify failed (_ssl.c:852)
Is there any way to say SSLVerify false or create a proxy to connect to GCP?
PS: All environment values along with json file are already set into the running environment.
I ran into this recently. Running Install Certificates.command worked for me, but my python version is 3.6.8. I also changed repo python script manually to use python3 (changed #!/usr/bin/env python -> #!/usr/bin/env python3).
Please refer below link for more information:
Mac OSX python ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749)
I am using an single page web application built on python 2.7 . I would need to add a simple authentication page with username, password and login button which will redirect to the app.Also a logout button in the app page which will redirect to the login page. It will use a single username and password that will be shared with users concerned.It uses wsgiref
from wsgiref import simple_server
from someApp import SomeAPI
app = SomeAPI()
if __name__ == '__main__':
httpd = simple_server.make_server('0.0.0.0', 8000, app)
httpd.serve_forever()
I am trying to use some middlware auth like falcon-auth or barrel . I get the error while installing.
C:\Users\User1>pip install falcon-auth
Collecting falcon-auth
c:\python27\lib\site-packages\pip\_vendor\urllib3\util\ssl_.py:339: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
SNIMissingWarning
I finally get the error
Could not fetch URL https://pypi.org/simple/pip/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/pip/ (Caused by SSLError(SSLError(1, '_ssl.c:499: error:1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert protocol version'),)) - skipping
What would a solution given that I need to use Python 2.7 and not upgrade to a higher version