Develop Reference

Pandas read_csv from web acting different between Python 3.8 and 3.10?

It seems that pandas is acting differently when trying to read a CSV from the web between Python 3.8 and Python 3.10. It works with 3.8, but appears to fail with 3.10.
I am using the call:
w1 = pd.read_csv("https://www.ncei.noaa.gov/data/local-climatological-data/access/2021/72254013904.csv")
To try to download weather data from the NOAA website. It works with Python 3.8:
But the same thing fails with Python 3.10:
The full error output is here:
>>> w1 = pd.read_csv("https://www.ncei.noaa.gov/data/local-climatological-data/access/2021/72254013904.csv", low_memory=False)
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/urllib/request.py", line 1348, in do_open
h.request(req.get_method(), req.selector, req.data, headers,
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/http/client.py", line 1282, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/http/client.py", line 1328, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/http/client.py", line 1277, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/http/client.py", line 1037, in _send_output
self.send(msg)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/http/client.py", line 975, in send
self.connect()
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/http/client.py", line 1454, in connect
self.sock = self._context.wrap_socket(self.sock,
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/ssl.py", line 512, in wrap_socket
return self.sslsocket_class._create(
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/ssl.py", line 1070, in _create
self.do_handshake()
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/ssl.py", line 1341, in do_handshake
self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/jrhodes/Library/Python/3.10/lib/python/site-packages/pandas/util/_decorators.py", line 311, in wrapper
return func(*args, **kwargs)
File "/Users/jrhodes/Library/Python/3.10/lib/python/site-packages/pandas/io/parsers/readers.py", line 680, in read_csv
return _read(filepath_or_buffer, kwds)
File "/Users/jrhodes/Library/Python/3.10/lib/python/site-packages/pandas/io/parsers/readers.py", line 575, in _read
parser = TextFileReader(filepath_or_buffer, **kwds)
File "/Users/jrhodes/Library/Python/3.10/lib/python/site-packages/pandas/io/parsers/readers.py", line 933, in __init__
self._engine = self._make_engine(f, self.engine)
File "/Users/jrhodes/Library/Python/3.10/lib/python/site-packages/pandas/io/parsers/readers.py", line 1217, in _make_engine
self.handles = get_handle( # type: ignore[call-overload]
File "/Users/jrhodes/Library/Python/3.10/lib/python/site-packages/pandas/io/common.py", line 670, in get_handle
ioargs = _get_filepath_or_buffer(
File "/Users/jrhodes/Library/Python/3.10/lib/python/site-packages/pandas/io/common.py", line 339, in _get_filepath_or_buffer
with urlopen(req_info) as req:
File "/Users/jrhodes/Library/Python/3.10/lib/python/site-packages/pandas/io/common.py", line 239, in urlopen
return urllib.request.urlopen(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/urllib/request.py", line 216, in urlopen
return opener.open(url, data, timeout)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/urllib/request.py", line 519, in open
response = self._open(req, data)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/urllib/request.py", line 536, in _open
result = self._call_chain(self.handle_open, protocol, protocol +
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/urllib/request.py", line 496, in _call_chain
result = func(*args)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/urllib/request.py", line 1391, in https_open
return self.do_open(http.client.HTTPSConnection, req,
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/urllib/request.py", line 1351, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)>
Did something change in the SSL perhaps between 3.8 and 3.10?

I've just run into this too. There is indeed increased security in Python 3.10. I've opened a bug report, but in the meantime a workaround based on this post is:
import ssl
from urllib.request import urlopen
import pandas as pd
url = ("https://www.ncei.noaa.gov/"
"data/local-climatological-data/"
"access/2021/72254013904.csv")
context = ssl.create_default_context()
context.set_ciphers("DEFAULT")
result = urlopen(url, context=context)
df = pd.read_csv(result)

Is there a way to solve an ssl error (DH key too small) on python when using pd.read_excel()?

I am trying to read an excel table that can be accessed through an url, and load it into a DataFrame.
I'm using pd.read_excel like so:
url = "https://www.recope.go.cr/wp-content/uploads/2020/09/PRECIOS-HIST%C3%93RICOS-CONSUMIDOR-FINAL-1-2.xls"
df = pd.read_excel(url,header=[2,3,4],nrows=347)
and I get :
Traceback (most recent call last):
File "/usr/local/lib/python3.7/urllib/request.py", line 1350, in do_open
encode_chunked=req.has_header('Transfer-encoding'))
File "/usr/local/lib/python3.7/http/client.py", line 1277, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/local/lib/python3.7/http/client.py", line 1323, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.7/http/client.py", line 1272, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.7/http/client.py", line 1032, in _send_output
self.send(msg)
File "/usr/local/lib/python3.7/http/client.py", line 972, in send
self.connect()
File "/usr/local/lib/python3.7/http/client.py", line 1447, in connect
server_hostname=server_hostname)
File "/usr/local/lib/python3.7/ssl.py", line 423, in wrap_socket
session=session
File "/usr/local/lib/python3.7/ssl.py", line 870, in _create
self.do_handshake()
File "/usr/local/lib/python3.7/ssl.py", line 1139, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: DH_KEY_TOO_SMALL] dh key too small (_ssl.c:1091)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.7/site-packages/pandas/util/_decorators.py", line 299, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/pandas/io/excel/_base.py", line 336, in read_excel
io = ExcelFile(io, storage_options=storage_options, engine=engine)
File "/usr/local/lib/python3.7/site-packages/pandas/io/excel/_base.py", line 1063, in __init__
path=str(self._io), storage_options=storage_options
File "/usr/local/lib/python3.7/site-packages/pandas/io/excel/_base.py", line 939, in inspect_excel_format
content_or_path, "rb", storage_options=storage_options, is_text=False
File "/usr/local/lib/python3.7/site-packages/pandas/io/common.py", line 563, in get_handle
storage_options=storage_options,
File "/usr/local/lib/python3.7/site-packages/pandas/io/common.py", line 288, in _get_filepath_or_buffer
req = urlopen(filepath_or_buffer)
File "/usr/local/lib/python3.7/site-packages/pandas/io/common.py", line 194, in urlopen
return urllib.request.urlopen(*args, **kwargs)
File "/usr/local/lib/python3.7/urllib/request.py", line 222, in urlopen
return opener.open(url, data, timeout)
File "/usr/local/lib/python3.7/urllib/request.py", line 525, in open
response = self._open(req, data)
File "/usr/local/lib/python3.7/urllib/request.py", line 543, in _open
'_open', req)
File "/usr/local/lib/python3.7/urllib/request.py", line 503, in _call_chain
result = func(*args)
File "/usr/local/lib/python3.7/urllib/request.py", line 1393, in https_open
context=self._context, check_hostname=self._check_hostname)
File "/usr/local/lib/python3.7/urllib/request.py", line 1352, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: DH_KEY_TOO_SMALL] dh key too small (_ssl.c:1091)>
I saw something similar in this question but I tried to include the answer code in mine to no avail. I'm a bit new to this but as I couldn't find a solution I thought best to ask here. Is there any way to get around this ssl error?

I tried the solution from the link again and it worked, but I had to change some of the code to:
req = requests.get(url)
df = pd.read_excel(req.content,header=[2,3,4],nrows=347)
For some reason, adding requests made the solution work.

Wolfram Alpha Python: SSL: CERTIFICATE_VERIFY_FAILED error

I have been trying to run this simple code snippet to check my installation of Wolfram Alpha on Python.
This is the code I have been trying to run:
import ssl
import wolframalpha
app_id = "*****************"
client = wolframalpha.Client(app_id)
my_input = input("Question: ")
res = client.query(my_input)
answer = next(res.results).text
print(answer)
This is the error I have been facing
Traceback (most recent call last):
File "C:\Users\MARIA\AppData\Local\Programs\Python\Python36-32\lib\urllib\request.py", line 1318, in do_open
encode_chunked=req.has_header('Transfer-encoding'))
File "C:\Users\MARIA\AppData\Local\Programs\Python\Python36-32\lib\http\client.py", line 1239, in request
self._send_request(method, url, body, headers, encode_chunked)
File "C:\Users\MARIA\AppData\Local\Programs\Python\Python36-32\lib\http\client.py", line 1285, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "C:\Users\MARIA\AppData\Local\Programs\Python\Python36-32\lib\http\client.py", line 1234, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "C:\Users\MARIA\AppData\Local\Programs\Python\Python36-32\lib\http\client.py", line 1026, in _send_output
self.send(msg)
File "C:\Users\MARIA\AppData\Local\Programs\Python\Python36-32\lib\http\client.py", line 964, in send
self.connect()
File "C:\Users\MARIA\AppData\Local\Programs\Python\Python36-32\lib\http\client.py", line 1400, in connect
server_hostname=server_hostname)
File "C:\Users\MARIA\AppData\Local\Programs\Python\Python36-32\lib\ssl.py", line 407, in wrap_socket
_context=self, _session=session)
File "C:\Users\MARIA\AppData\Local\Programs\Python\Python36-32\lib\ssl.py", line 817, in __init__
self.do_handshake()
File "C:\Users\MARIA\AppData\Local\Programs\Python\Python36-32\lib\ssl.py", line 1077, in do_handshake
self._sslobj.do_handshake()
File "C:\Users\MARIA\AppData\Local\Programs\Python\Python36-32\lib\ssl.py", line 689, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:847)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\MARIA\TEST_walpha.py", line 6, in <module>
res = client.query(my_input)
File "C:\Users\MARIA\AppData\Local\Programs\Python\Python36-32\lib\site-packages\wolframalpha\__init__.py", line 53, in query
resp = urllib.request.urlopen(url)
File "C:\Users\MARIA\AppData\Local\Programs\Python\Python36-32\lib\urllib\request.py", line 223, in urlopen
return opener.open(url, data, timeout)
File "C:\Users\MARIA\AppData\Local\Programs\Python\Python36-32\lib\urllib\request.py", line 526, in open
response = self._open(req, data)
File "C:\Users\MARIA\AppData\Local\Programs\Python\Python36-32\lib\urllib\request.py", line 544, in _open
'_open', req)
File "C:\Users\MARIA\AppData\Local\Programs\Python\Python36-32\lib\urllib\request.py", line 504, in _call_chain
result = func(*args)
File "C:\Users\MARIA\AppData\Local\Programs\Python\Python36-32\lib\urllib\request.py", line 1361, in https_open
context=self._context, check_hostname=self._check_hostname)
File "C:\Users\MARIA\AppData\Local\Programs\Python\Python36-32\lib\urllib\request.py", line 1320, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:847)>
Now I consulted this post, but there was no clear answer provide, and importing the SSL library did not solve my problem.

You can try disabling certificate verification globally by doing something like this:
ssl._create_default_https_context = ssl._create_unverified_context

Web Scraping with Python and Beautiful Soup (line 1317, in do_open encode_chunked=req.has_header('Transfer-encoding'))

I'm new to Python and do have some problems with web scraping:
here is the code:
from urllib.request import urlopen as uReq
from bs4 import BeautifulSoup as soup
my_url = 'https://www.newegg.com/Video-Cards-Video-Devices/Category/ID-38'
uClient = uReq(my_url)
page_html = uClient.read()
uClient.close()
page_soup = soup(page_html, "html.parser")
which brings following errors:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 1317, in do_open
encode_chunked=req.has_header('Transfer-encoding'))
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1229, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1275, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1224, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1016, in _send_output
self.send(msg)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 956, in send
self.connect()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1392, in connect
server_hostname=server_hostname)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", line 412, in wrap_socket
session=session
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", line 850, in _create
self.do_handshake()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", line 1108, in do_handshake
self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1045)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/alexeicresniov/PycharmProjects/Alex/classes & objects/web parsing.py", line 8, in <module>
uClient = uReq(my_url)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 222, in urlopen
return opener.open(url, data, timeout)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 525, in open
response = self._open(req, data)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 543, in _open
'_open', req)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 503, in _call_chain
result = func(*args)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 1360, in https_open
context=self._context, check_hostname=self._check_hostname)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 1319, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1045)>

Reading contents of different webpages and adding then together to put into another URL

I'm not advanced at python but its the best coding language I know. I'm doing this challenge that has 5 URLS which change contents every 10 seconds. They each contain part of a code. There is also a validation link which checks the urls I have clicked and checks if it the right code, if it is it will give me a code. So to solve this I'm writing a script that will get all the contents of the 5 URLs and concatenate them together and paste it in the URL of the validation link which will then give me a code which I need.
Here's my code
import urllib.request
fp1 = urllib.request.urlopen("https://assess.joincyberdiscovery.com/challenge-files/clock-pt1?verify=Gl7fPRYxQvgBdbmhMo8vkA%3D%3D")
mybytes1 = fp1.read()
fp1.close()
fp2 = urllib.request.urlopen("https://assess.joincyberdiscovery.com/challenge-files/clock-pt2?verify=Gl7fPRYxQvgBdbmhMo8vkA%3D%3D")
mybytes2 = fp2.read()
fp2.close()
fp3 = urllib.request.urlopen("https://assess.joincyberdiscovery.com/challenge-files/clock-pt3?verify=Gl7fPRYxQvgBdbmhMo8vkA%3D%3D")
mybytes3 = fp3.read()
fp3.close()
fp4 = urllib.request.urlopen("https://assess.joincyberdiscovery.com/challenge-files/clock-pt4?verify=Gl7fPRYxQvgBdbmhMo8vkA%3D%3D")
mybytes4 = fp4.read()
fp4.close()
fp5 = urllib.request.urlopen("https://assess.joincyberdiscovery.com/challenge-files/clock-pt5?verify=Gl7fPRYxQvgBdbmhMo8vkA%3D%3D")
mybytes5 = fp5.read()
fp5.close()
fp6 = urllib.request.urlopen("https://assess.joincyberdiscovery.com/challenge-files/get-flag?verify=Gl7fPRYxQvgBdbmhMo8vkA%3D%3D&string=" + mybytes1 + mybytes2 + mybytes3 + mybytes4 + mybytes5)
mybytes6 = fp6.read()
fp6.close()
print(mybytes6)
However I am getting an error which I don't understand.
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 1318, in do_open
encode_chunked=req.has_header('Transfer-encoding'))
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1239, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1285, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1234, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1026, in _send_output
self.send(msg)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 964, in send
self.connect()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1400, in connect
server_hostname=server_hostname)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 401, in wrap_socket
_context=self, _session=session)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 808, in __init__
self.do_handshake()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 1061, in do_handshake
self._sslobj.do_handshake()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 683, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/sarvesh/Documents/scriptdis.py", line 3, in <module>
fp1 = urllib.request.urlopen("https://assess.joincyberdiscovery.com/challenge-files/clock-pt1?verify=Gl7fPRYxQvgBdbmhMo8vkA%3D%3D")
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 223, in urlopen
return opener.open(url, data, timeout)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 526, in open
response = self._open(req, data)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 544, in _open
'_open', req)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 504, in _call_chain
result = func(*args)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 1361, in https_open
context=self._context, check_hostname=self._check_hostname)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 1320, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749)>
Can anyone help me understand what's wrong?

urllib and "SSL: CERTIFICATE_VERIFY_FAILED" Error
You are trying to open a website without a valid certificate. If you trust the website, then you need to create an uncertified context or urllib will throw the error you are seeing.
import ssl
# This restores the same behavior as before.
context = ssl._create_unverified_context()
fp1 = urllib.request.urlopen("https://assess.joincyberdiscovery.com/challenge-files/clock-pt1?verify=Gl7fPRYxQvgBdbmhMo8vkA%3D%3D", context=context)