Develop Reference

Pandas_datareader issue

When I try to read data from stooq through pandas_datareader.data this error keeps coming through:
Traceback (most recent call last):
File "/Users/matteo/PycharmProjects/pythonProject/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 700, in urlopen
self._prepare_proxy(conn)
File "/Users/matteo/PycharmProjects/pythonProject/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 996, in _prepare_proxy
conn.connect()
File "/Users/matteo/PycharmProjects/pythonProject/venv/lib/python3.11/site-packages/urllib3/connection.py", line 414, in connect
self.sock = ssl_wrap_socket(
^^^^^^^^^^^^^^^^
File "/Users/matteo/PycharmProjects/pythonProject/venv/lib/python3.11/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
ssl_sock = _ssl_wrap_socket_impl(
^^^^^^^^^^^^^^^^^^^^^^
File "/Users/matteo/PycharmProjects/pythonProject/venv/lib/python3.11/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/ssl.py", line 517, in wrap_socket
return self.sslsocket_class._create(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/ssl.py", line 1075, in _create
self.do_handshake()
File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/ssl.py", line 1346, in do_handshake
self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:992)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/matteo/PycharmProjects/pythonProject/venv/lib/python3.11/site-packages/requests/adapters.py", line 489, in send
resp = conn.urlopen(
^^^^^^^^^^^^^
File "/Users/matteo/PycharmProjects/pythonProject/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 787, in urlopen
retries = retries.increment(
^^^^^^^^^^^^^^^^^^
File "/Users/matteo/PycharmProjects/pythonProject/venv/lib/python3.11/site-packages/urllib3/util/retry.py", line 592, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='stooq.com', port=443): Max retries exceeded with url: /q/d/l/?s=%5EDJI&i=d&d1=20180216&d2=20230215 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:992)')))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/matteo/PycharmProjects/pythonProject/stock_data.py", line 3, in <module>
f = web.DataReader('^DJI', 'stooq')
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/matteo/PycharmProjects/pythonProject/venv/lib/python3.11/site-packages/pandas/util/_decorators.py", line 211, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/Users/matteo/PycharmProjects/pythonProject/venv/lib/python3.11/site-packages/pandas_datareader/data.py", line 432, in DataReader
).read()
^^^^^^
File "/Users/matteo/PycharmProjects/pythonProject/venv/lib/python3.11/site-packages/pandas_datareader/base.py", line 253, in read
df = self._read_one_data(self.url, params=self._get_params(self.symbols))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/matteo/PycharmProjects/pythonProject/venv/lib/python3.11/site-packages/pandas_datareader/base.py", line 108, in _read_one_data
out = self._read_url_as_StringIO(url, params=params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/matteo/PycharmProjects/pythonProject/venv/lib/python3.11/site-packages/pandas_datareader/base.py", line 119, in _read_url_as_StringIO
response = self._get_response(url, params=params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/matteo/PycharmProjects/pythonProject/venv/lib/python3.11/site-packages/pandas_datareader/base.py", line 155, in _get_response
response = self.session.get(
^^^^^^^^^^^^^^^^^
File "/Users/matteo/PycharmProjects/pythonProject/venv/lib/python3.11/site-packages/requests/sessions.py", line 600, in get
return self.request("GET", url, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/matteo/PycharmProjects/pythonProject/venv/lib/python3.11/site-packages/requests/sessions.py", line 587, in request
resp = self.send(prep, **send_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/matteo/PycharmProjects/pythonProject/venv/lib/python3.11/site-packages/requests/sessions.py", line 701, in send
r = adapter.send(request, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/matteo/PycharmProjects/pythonProject/venv/lib/python3.11/site-packages/requests/adapters.py", line 563, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='stooq.com', port=443): Max retries exceeded with url: /q/d/l/?s=%5EDJI&i=d&d1=20180216&d2=20230215 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:992)')))
I tried to run the same code on jupiter lab and the error is the same, contrary in google colab I get the correct result. Here is the code from google colab:
Code:
import pandas_datareader.data as web
f = web.DataReader('^DJI', 'stooq')
f[:10]

How to add proxy to quickstart.py-python

I execute quickstart.py according to the steps of the tutorial, and he will report an error. I'm pretty sure it has something to do with the proxy in my place, so I especially want to know how to add my proxy to quickstart.py and how to add it is correct. The error message is as follows, hope to answer, thank you!
Traceback (most recent call last):
File "D:\Python\lib\site-packages\urllib3\connectionpool.py", line 696, in urlopen
self._prepare_proxy(conn)
File "D:\Python\lib\site-packages\urllib3\connectionpool.py", line 964, in _prepare_proxy
conn.connect()
File "D:\Python\lib\site-packages\urllib3\connection.py", line 359, in connect
conn = self._connect_tls_proxy(hostname, conn)
File "D:\Python\lib\site-packages\urllib3\connection.py", line 506, in _connect_tls_proxy
ssl_context=ssl_context,
File "D:\Python\lib\site-packages\urllib3\util\ssl_.py", line 450, in ssl_wrap_socket
sock, context, tls_in_tls, server_hostname=server_hostname
File "D:\Python\lib\site-packages\urllib3\util\ssl_.py", line 493, in _ssl_wrap_socket_impl
return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
File "D:\Python\lib\ssl.py", line 423, in wrap_socket
session=session
File "D:\Python\lib\ssl.py", line 870, in _create
self.do_handshake()
File "D:\Python\lib\ssl.py", line 1139, in do_handshake
self._sslobj.do_handshake()
OSError: [Errno 0] Error
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "D:\Python\lib\site-packages\requests\adapters.py", line 449, in send
timeout=timeout
File "D:\Python\lib\site-packages\urllib3\connectionpool.py", line 756, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "D:\Python\lib\site-packages\urllib3\util\retry.py", line 574, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='oauth2.googleapis.com', port=443): Max retries exceeded with url: /token (Caused by ProxyError('Cannot connect to proxy.', OSError(0, 'Error')))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "D:/2021-0/Project/eliis_arno/RpaChangeOrder/test/quickstart.py", line 47, in <module>
main()
File "D:/2021-0/Project/eliis_arno/RpaChangeOrder/test/quickstart.py", line 28, in main
creds = flow.run_local_server(port=0)
File "D:\Python\lib\site-packages\google_auth_oauthlib\flow.py", line 479, in run_local_server
self.fetch_token(authorization_response=authorization_response)
File "D:\Python\lib\site-packages\google_auth_oauthlib\flow.py", line 288, in fetch_token
return self.oauth2session.fetch_token(self.client_config["token_uri"], **kwargs)
File "D:\Python\lib\site-packages\requests_oauthlib\oauth2_session.py", line 344, in fetch_token
**request_kwargs
File "D:\Python\lib\site-packages\requests_oauthlib\oauth2_session.py", line 516, in request
method, url, headers=headers, data=data, **kwargs
File "D:\Python\lib\site-packages\requests\sessions.py", line 542, in request
resp = self.send(prep, **send_kwargs)
File "D:\Python\lib\site-packages\requests\sessions.py", line 655, in send
r = adapter.send(request, **kwargs)
File "D:\Python\lib\site-packages\requests\adapters.py", line 510, in send
raise ProxyError(e, request=request)
requests.exceptions.ProxyError: HTTPSConnectionPool(host='oauth2.googleapis.com', port=443): Max retries exceeded with url: /token (Caused by ProxyError('Cannot connect to proxy.', OSError(0, 'Error')))

ssl.SSLError when using urllib3

I am trying to use the fortiosapi library (with test code), but am getting the following error. (Python code below)
What do I need to put in my code to account to disable the certificate check? (This is a test lab)
ERROR
alice#ubuntu18:~$ python3 testlogin.py
Traceback (most recent call last):
File "/home/alice/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 677, in urlopen
chunked=chunked,
File "/home/alice/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 381, in _make_request
self._validate_conn(conn)
File "/home/alice/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 978, in _validate_conn
conn.connect()
File "/home/alice/.local/lib/python3.6/site-packages/urllib3/connection.py", line 371, in connect
ssl_context=context,
File "/home/alice/.local/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 397, in ssl_wrap_socket
return context.wrap_socket(sock)
File "/usr/lib/python3.6/ssl.py", line 407, in wrap_socket
_context=self, _session=session)
File "/usr/lib/python3.6/ssl.py", line 814, in __init__
self.do_handshake()
File "/usr/lib/python3.6/ssl.py", line 1068, in do_handshake
self._sslobj.do_handshake()
File "/usr/lib/python3.6/ssl.py", line 689, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:833)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/alice/.local/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/home/alice/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 727, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/home/alice/.local/lib/python3.6/site-packages/urllib3/util/retry.py", line 439, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='192.168.0.101', port=443): Max retries exceeded with url: /logincheck (Ca used by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:833)'),))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "testlogin.py", line 31, in <module>
main()
File "testlogin.py", line 24, in main
resp = fgt.login(fgthost, user, passwd)
File "/home/alice/.local/lib/python3.6/site-packages/fortiosapi/fortiosapi.py", line 176, in login
data='username=' + urllib.parse.quote(username) + '&secretkey=' + urllib.parse.quote(password) + "&ajax=1", timeout=self.timeout)
File "/home/alice/.local/lib/python3.6/site-packages/requests/sessions.py", line 578, in post
return self.request('POST', url, data=data, json=json, **kwargs)
File "/home/alice/.local/lib/python3.6/site-packages/requests/sessions.py", line 530, in request
resp = self.send(prep, **send_kwargs)
File "/home/alice/.local/lib/python3.6/site-packages/requests/sessions.py", line 643, in send
r = adapter.send(request, **kwargs)
File "/home/alice/.local/lib/python3.6/site-packages/requests/adapters.py", line 514, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='192.168.0.101', port=443): Max retries exceeded with url: /logincheck (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:833)'),))
This is the Python code that is causing the error. I simply want to log in and run a check to see that the fortiosapi works.
(Here is where I found the example)
CODE
#!/usr/bin/python
import logging
import pprint
from fortiosapi import FortiOSAPI
formatter = logging.Formatter(
'%(asctime)s %(name)-12s %(levelname)-8s %(message)s')
logger = logging.getLogger('fortiosapi')
hdlr = logging.FileHandler('testfortiosapi.log')
hdlr.setFormatter(formatter)
logger.addHandler(hdlr)
logger.setLevel(logging.DEBUG)
fgt = FortiOSAPI()
def main():
# Login to the FGT ip
fgt.debug('on')
fgthost = '192.168.0.101'
user = 'admin'
passwd = 'password'
resp = fgt.login(fgthost, user, passwd)
pp = pprint.PrettyPrinter(indent=4)
resp = fgt.license()
pp = pprint.PrettyPrinter(indent=4)
pp.pprint(resp)
if __name__ == '__main__':
main()

added "verify=False", and it works
resp = fgt.login(fgthost, user, passwd, verify=False)

Passing security certificates using requests

I need to access a site, but I am getting an error:
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1108)
Code:
import requests
BASE_URL = "https://www.portfolio123.com/"
r = requests.get(BASE_URL + 'server_status.jsp')
Full Error:
Traceback (most recent call last):
File "C:\Python38-32\lib\site-packages\urllib3\connectionpool.py", line 665, in urlopen
httplib_response = self._make_request(
File "C:\Python38-32\lib\site-packages\urllib3\connectionpool.py", line 376, in _make_request
self._validate_conn(conn)
File "C:\Python38-32\lib\site-packages\urllib3\connectionpool.py", line 994, in _validate_conn
conn.connect()
File "C:\Python38-32\lib\site-packages\urllib3\connection.py", line 352, in connect
self.sock = ssl_wrap_socket(
File "C:\Python38-32\lib\site-packages\urllib3\util\ssl_.py", line 370, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "C:\Python38-32\lib\ssl.py", line 500, in wrap_socket
return self.sslsocket_class._create(
File "C:\Python38-32\lib\ssl.py", line 1040, in _create
self.do_handshake()
File "C:\Python38-32\lib\ssl.py", line 1309, in do_handshake
self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1108)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Python38-32\lib\site-packages\requests\adapters.py", line 439, in send
resp = conn.urlopen(
File "C:\Python38-32\lib\site-packages\urllib3\connectionpool.py", line 719, in urlopen
retries = retries.increment(
File "C:\Python38-32\lib\site-packages\urllib3\util\retry.py", line 436, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='www.portfolio123.com', port=443): Max retries exceeded with url: /server_status.jsp (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1108)')))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:/Users/Lakewealth/.PyCharmCE2019.2/config/scratches/scratch_4.py", line 5, in <module>
r = requests.get(BASE_URL + 'server_status.jsp')
File "C:\Python38-32\lib\site-packages\requests\api.py", line 75, in get
return request('get', url, params=params, **kwargs)
File "C:\Python38-32\lib\site-packages\requests\api.py", line 60, in request
return session.request(method=method, url=url, **kwargs)
File "C:\Python38-32\lib\site-packages\requests\sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "C:\Python38-32\lib\site-packages\requests\sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "C:\Python38-32\lib\site-packages\requests\adapters.py", line 514, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='www.portfolio123.com', port=443): Max retries exceeded with url: /server_status.jsp (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1108)')))
There are a lot of questions on this subject such as here, here, and here. I scanned through them, but didn't find the solution yet.
import requests
BASE_URL = "https://www.portfolio123.com/"
CERT = 'c:\data\P123.CER'
r = requests.get(BASE_URL + 'server_status.jsp', verify=CERT)

Python requests throwing SSLEOFError

I am trying to connect a Python Requests POC to this tool. I am specifically looking at the test cases for FIA-X509 Ext 1.1. However, the Python POC is throwing the following error:
Traceback (most recent call last):
File "/home/hussain/Desktop/pythonVenv/python36Venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen
chunked=chunked)
File "/home/hussain/Desktop/pythonVenv/python36Venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 346, in _make_request
self._validate_conn(conn)
File "/home/hussain/Desktop/pythonVenv/python36Venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 850, in _validate_conn
conn.connect()
File "/home/hussain/Desktop/pythonVenv/python36Venv/lib/python3.6/site-packages/urllib3/connection.py", line 326, in connect
ssl_context=context)
File "/home/hussain/Desktop/pythonVenv/python36Venv/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 329, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "/usr/lib/python3.6/ssl.py", line 407, in wrap_socket
_context=self, _session=session)
File "/usr/lib/python3.6/ssl.py", line 814, in __init__
self.do_handshake()
File "/usr/lib/python3.6/ssl.py", line 1068, in do_handshake
self._sslobj.do_handshake()
File "/usr/lib/python3.6/ssl.py", line 689, in do_handshake
self._sslobj.do_handshake()
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:777)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/hussain/Desktop/pythonVenv/python36Venv/lib/python3.6/site-packages/requests/adapters.py", line 440, in send
timeout=timeout
File "/home/hussain/Desktop/pythonVenv/python36Venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 639, in urlopen
_stacktrace=sys.exc_info()[2])
File "/home/hussain/Desktop/pythonVenv/python36Venv/lib/python3.6/site-packages/urllib3/util/retry.py", line 388, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='10.0.0.221', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:777)'),))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "req.py", line 91, in <module>
r = session.request('GET', 'https://10.0.0.221', verify=True)
File "/home/hussain/Desktop/pythonVenv/python36Venv/lib/python3.6/site-packages/requests/sessions.py", line 508, in request
resp = self.send(prep, **send_kwargs)
File "/home/hussain/Desktop/pythonVenv/python36Venv/lib/python3.6/site-packages/requests/sessions.py", line 618, in send
r = adapter.send(request, **kwargs)
File "/home/hussain/Desktop/pythonVenv/python36Venv/lib/python3.6/site-packages/requests/adapters.py", line 506, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='10.0.0.221', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:777)'),))
This is the POC code that i am using to connect to it:
import ssl
import requests
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.poolmanager import PoolManager
from requests.packages.urllib3.util import ssl_
CIPHERS = (
'RSA+AES'
)
class TlsAdapter(HTTPAdapter):
def __init__(self, ssl_options=0, **kwargs):
self.ssl_options = ssl_options
super(TlsAdapter, self).__init__(**kwargs)
def init_poolmanager(self, *pool_args, **pool_kwargs):
ctx = ssl_.create_urllib3_context(ciphers=CIPHERS, cert_reqs=ssl.CERT_REQUIRED, options=self.ssl_options)
self.poolmanager = PoolManager(*pool_args,
ssl_context=ctx,
**pool_kwargs)
session = requests.session()
adapter = TlsAdapter(ssl.OP_NO_TLSv1 | ssl.OP_NO_TLSv1_1)
session.mount("https://", adapter)
r = session.request('GET', 'https://10.0.0.221', verify=True)
print(r)
Can anyone please help me in figuring out what i am missing here and how can this be resolved? I tried looking up the problem on the requests github issue page as well as on stackoverflow but none of them seemed to work for me.
Thanks!