I have tried to find the right answer for my situation, but it seems it may not exists yet.
In my current training course (very beginner level) I have an exercise to get some weather json info from a url (Open Weather in this case). Currently, this is the only way I have been taught to do this:
url = "http://api.openweathermap.org/data/2.5/weather?q="+city+"&appid="+api_key
request = requests.get(url)
json = request.json()
print(json)
I have my "city" and "api_key" parameters set as instructed, and have been able to successfully run my code and get the json info when connected to my personal home network. However, when I connect to my work network, physically at the office or via VPN, I get this magical wall:
Traceback (most recent call last): File
"C:\Users\*REDACTED*\Python\Python39\lib\site-packages\urllib3\connection.py",
line 169, in _new_conn
conn = connection.create_connection( File "C:\Users\*REDACTED*\Python\Python39\lib\site-packages\urllib3\util\connection.py",
line 96, in create_connection
raise err File "C:\Users\*REDACTED*\Python\Python39\lib\site-packages\urllib3\util\connection.py",
line 86, in create_connection
sock.connect(sa) TimeoutError: [WinError 10060] A connection attempt failed because the connected party did not properly respond
after a period of time, or established connection failed because
connected host has failed to respond
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File
"C:\Users\*REDACTED*\Python\Python39\lib\site-packages\urllib3\connectionpool.py",
line 699, in urlopen
httplib_response = self._make_request( File "C:\Users\*REDACTED*\Python\Python39\lib\site-packages\urllib3\connectionpool.py",
line 394, in _make_request
conn.request(method, url, **httplib_request_kw) File "C:\Users\*REDACTED*\Python\Python39\lib\site-packages\urllib3\connection.py",
line 234, in request
super(HTTPConnection, self).request(method, url, body=body, headers=headers) File
"C:\Users\*REDACTED*\Python\Python39\lib\http\client.py", line 1257,
in request
self._send_request(method, url, body, headers, encode_chunked) File "C:\Users\*REDACTED*\Python\Python39\lib\http\client.py", line
1303, in _send_request
self.endheaders(body, encode_chunked=encode_chunked) File "C:\Users\*REDACTED*\Python\Python39\lib\http\client.py", line 1252,
in endheaders
self._send_output(message_body, encode_chunked=encode_chunked) File "C:\Users\*REDACTED*\Python\Python39\lib\http\client.py", line
1012, in _send_output
self.send(msg) File "C:\Users\*REDACTED*\Python\Python39\lib\http\client.py", line 952, in
send
self.connect() File "C:\Users\*REDACTED*\Python\Python39\lib\site-packages\urllib3\connection.py",
line 200, in connect
conn = self._new_conn() File "C:\Users\*REDACTED*\Python\Python39\lib\site-packages\urllib3\connection.py",
line 181, in _new_conn
raise NewConnectionError( urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x00000255A4B11370>:
Failed to establish a new connection: [WinError 10060] A connection
attempt failed because the connected party did not properly respond
after a period of time, or established connection failed because
connected host has failed to respond
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File
"C:\Users\*REDACTED*\Python\Python39\lib\site-packages\requests-2.26.0-py3.9.egg\requests\adapters.py",
line 439, in send
resp = conn.urlopen( File "C:\Users\*REDACTED*\Python\Python39\lib\site-packages\urllib3\connectionpool.py",
line 755, in urlopen
retries = retries.increment( File "C:\Users\*REDACTED*\Python\Python39\lib\site-packages\urllib3\util\retry.py",
line 574, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError:
HTTPConnectionPool(host='api.openweathermap.org', port=80): Max
retries exceeded with url:
/data/2.5/weather?q=*REDACTED*&appid=*REDACTED* (Caused by
NewConnectionError('<urllib3.connection.HTTPConnection object at
0x00000255A4B11370>: Failed to establish a new connection: [WinError
10060] A connection attempt failed because the connected party did not
properly respond after a period of time, or established connection
failed because connected host has failed to respond'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File
"c:\Users\*REDACTED*\Training\Code\Python\weather.py", line 7, in
<module>
request = requests.get(url) File "C:\Users\*REDACTED*\Python\Python39\lib\site-packages\requests-2.26.0-py3.9.egg\requests\api.py",
line 75, in get
return request('get', url, params=params, **kwargs)
return session.request(method=method, url=url, **kwargs) File "C:\Users\*REDACTED*\Python\Python39\lib\site-packages\requests-2.26.0-py3.9.egg\requests\sessions.py",
line 542, in request
resp = self.send(prep, **send_kwargs) File "C:\Users\*REDACTED*\Python\Python39\lib\site-packages\requests-2.26.0-py3.9.egg\requests\sessions.py",
line 655, in send
r = adapter.send(request, **kwargs) File "C:\Users\*REDACTED*\Python\Python39\lib\site-packages\requests-2.26.0-py3.9.egg\requests\adapters.py",
line 516, in send
raise ConnectionError(e, request=request) requests.exceptions.ConnectionError:
HTTPConnectionPool(host='api.openweathermap.org', port=80): Max
retries exceeded with url:
/data/2.5/weather?q=*REDACTED*&appid=*REDACTED* (Caused by
NewConnectionError('<urllib3.connection.HTTPConnection object at
0x00000255A4B11370>: Failed to establish a new connection: [WinError
10060] A connection attempt failed because the connected party did not
properly respond after a period of time, or established connection
failed because connected host has failed to respond')) PS
C:\Users\*REDACTED*\Training\Code\Python&
C:/Users/*REDACTED*/Python/Python39/python.exe
"c:/Users/*REDACTED*/Training/Code/Python/weather.py" Traceback (most
recent call last): File
"C:\Users\*REDACTED*\Python\Python39\lib\site-packages\urllib3\connection.py",
line 169, in _new_conn
conn = connection.create_connection( File "C:\Users\*REDACTED*\Python\Python39\lib\site-packages\urllib3\util\connection.py",
line 96, in create_connection
raise err File "C:\Users\*REDACTED*\Python\Python39\lib\site-packages\urllib3\util\connection.py",
line 86, in create_connection
sock.connect(sa) TimeoutError: [WinError 10060] A connection attempt failed because the connected party did not properly respond
after a period of time, or established connection failed because
connected host has failed to respond
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File
"C:\Users\*REDACTED*\Python\Python39\lib\site-packages\urllib3\connectionpool.py",
line 699, in urlopen
httplib_response = self._make_request( File "C:\Users\*REDACTED*\Python\Python39\lib\site-packages\urllib3\connectionpool.py",
line 394, in _make_request
conn.request(method, url, **httplib_request_kw) File "C:\Users\*REDACTED*\Python\Python39\lib\site-packages\urllib3\connection.py",
line 234, in request
super(HTTPConnection, self).request(method, url, body=body, headers=headers) File
"C:\Users\*REDACTED*\Python\Python39\lib\http\client.py", line 1257,
in request
self._send_request(method, url, body, headers, encode_chunked) File "C:\Users\*REDACTED*\Python\Python39\lib\http\client.py", line
1303, in _send_request
self.endheaders(body, encode_chunked=encode_chunked) File "C:\Users\*REDACTED*\Python\Python39\lib\http\client.py", line 1252,
in endheaders
self._send_output(message_body, encode_chunked=encode_chunked) File "C:\Users\*REDACTED*\Python\Python39\lib\http\client.py", line
1012, in _send_output
self.send(msg) File "C:\Users\*REDACTED*\Python\Python39\lib\http\client.py", line 952, in
send
self.connect() File "C:\Users\*REDACTED*\Python\Python39\lib\site-packages\urllib3\connection.py",
line 200, in connect
conn = self._new_conn() File "C:\Users\*REDACTED*\Python\Python39\lib\site-packages\urllib3\connection.py",
line 181, in _new_conn
raise NewConnectionError( urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x00000229035A1370>:
Failed to establish a new connection: [WinError 10060] A connection
attempt failed because the connected party did not properly respond
after a period of time, or established connection failed because
connected host has failed to respond
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File
"C:\Users\*REDACTED*\Python\Python39\lib\site-packages\requests-2.26.0-py3.9.egg\requests\adapters.py",
line 439, in send
resp = conn.urlopen( File "C:\Users\*REDACTED*\Python\Python39\lib\site-packages\urllib3\connectionpool.py",
line 755, in urlopen
retries = retries.increment( File "C:\Users\*REDACTED*\Python\Python39\lib\site-packages\urllib3\util\retry.py",
line 574, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError:
HTTPConnectionPool(host='api.openweathermap.org', port=80): Max
retries exceeded with url:
/data/2.5/weather?q=*REDACTED*&appid=*REDACTED* (Caused by
NewConnectionError('<urllib3.connection.HTTPConnection object at
0x00000229035A1370>: Failed to establish a new connection: [WinError
10060] A connection attempt failed because the connected party did not
properly respond after a period of time, or established connection
failed because connected host has failed to respond'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File
"c:\Users\*REDACTED*\Training\Code\Python\weather.py", line 7, in
<module>
request = requests.get(url) File "C:\Users\*REDACTED*\Python\Python39\lib\site-packages\requests-2.26.0-py3.9.egg\requests\api.py",
line 75, in get
return request('get', url, params=params, **kwargs) File "C:\Users\*REDACTED*\Python\Python39\lib\site-packages\requests-2.26.0-py3.9.egg\requests\api.py",
line 61, in request
return session.request(method=method, url=url, **kwargs) File "C:\Users\*REDACTED*\Python\Python39\lib\site-packages\requests-2.26.0-py3.9.egg\requests\sessions.py",
line 542, in request
resp = self.send(prep, **send_kwargs) File "C:\Users\*REDACTED*\Python\Python39\lib\site-packages\requests-2.26.0-py3.9.egg\requests\sessions.py",
line 655, in send
r = adapter.send(request, **kwargs) File "C:\Users\*REDACTED*\Python\Python39\lib\site-packages\requests-2.26.0-py3.9.egg\requests\adapters.py",
line 516, in send
raise ConnectionError(e, request=request) requests.exceptions.ConnectionError:
HTTPConnectionPool(host='api.openweathermap.org', port=80): Max
retries exceeded with url:
/data/2.5/weather?q=*REDACTED*&appid=*REDACTED* (Caused by
NewConnectionError('<urllib3.connection.HTTPConnection object at
0x00000229035A1370>: Failed to establish a new connection: [WinError
10060] A connection attempt failed because the connected party did not
properly respond after a period of time, or established connection
failed because connected host has failed to respond'))
Quick ref of facts/actions taken:
-Tried setting the proxy globally to all Python instances via the set proxy... command.
-Not using a venv.
-Tried to add the --proxy... argument at the end of the command before executing via the the Visual Studio Code terminal.
-On Windows 10 Enterprise.
I am unable to move to the next module of the course because I need this code to work so that I know how to fix the error in the next set of codes I will have to do. I also apologize if this is a lot of words to get through (I am a noob for a reason, I guess), but I really appreciate anyone who could guide me.
For Requests proxy set in own manner Proxies in Requests
Thanks to the help of a colleague, I was able to better interpret the link that Сергей Зайков referenced. It was a proxy issue after all.
After making my adjustment in the code, I was able to modify it by adding the lines below and updating the request.get
http_proxy = "http://*REDACTED*:8080"
proxyDict = {"http" : http_proxy}
request = requests.get(url, proxies=proxyDict)
I am now able to run my code as intended.
Related
I'm trying a tutorial of spaCy(NLP) and when I try to pip install the requirements(as mentioned in the spacy website), I'm getting this error. Can anybody please tell me what I'm doing wrong? The same error showed up while installing NLTK as well(figured spaCy would work but the same error pops up again).
I've looked at other similar questions and found out no similarities b/w the context. I'm a beginner in ML and I'm finding it tough to deal with errors at each and every step. Please help!
Here's the full error.
TimeoutError: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\krish\anaconda3\lib\site-packages\urllib3\connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "C:\Users\krish\anaconda3\lib\site-packages\urllib3\connectionpool.py", line 386, in _make_request
self._validate_conn(conn)
File "C:\Users\krish\anaconda3\lib\site-packages\urllib3\connectionpool.py", line 1042, in _validate_conn
conn.connect()
File "C:\Users\krish\anaconda3\lib\site-packages\urllib3\connection.py", line 358, in connect
self.sock = conn = self._new_conn()
File "C:\Users\krish\anaconda3\lib\site-packages\urllib3\connection.py", line 186, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x000001C494D17640>: Failed to establish a new connection: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\krish\anaconda3\lib\site-packages\requests\adapters.py", line 489, in send
resp = conn.urlopen(
File "C:\Users\krish\anaconda3\lib\site-packages\urllib3\connectionpool.py", line 787, in urlopen
retries = retries.increment(
File "C:\Users\krish\anaconda3\lib\site-packages\urllib3\util\retry.py", line 592, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='raw.githubusercontent.com', port=443): Max retries exceeded with url: /explosion/spacy-models/master/compatibility.json (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x000001C494D17640>: Failed to establish a new connection: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\krish\anaconda3\lib\runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\krish\anaconda3\lib\runpy.py", line 87, in _run_code
exec(code, run_globals)
File "C:\Users\krish\anaconda3\lib\site-packages\spacy\__main__.py", line 4, in <module>
setup_cli()
File "C:\Users\krish\anaconda3\lib\site-packages\spacy\cli\_util.py", line 71, in setup_cli
command(prog_name=COMMAND)
File "C:\Users\krish\anaconda3\lib\site-packages\click\core.py", line 1128, in __call__
return self.main(*args, **kwargs)
File "C:\Users\krish\anaconda3\lib\site-packages\typer\core.py", line 778, in main
return _main(
File "C:\Users\krish\anaconda3\lib\site-packages\typer\core.py", line 216, in _main
rv = self.invoke(ctx)
File "C:\Users\krish\anaconda3\lib\site-packages\click\core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "C:\Users\krish\anaconda3\lib\site-packages\click\core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "C:\Users\krish\anaconda3\lib\site-packages\click\core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "C:\Users\krish\anaconda3\lib\site-packages\typer\main.py", line 683, in wrapper
return callback(**use_params) # type: ignore
File "C:\Users\krish\anaconda3\lib\site-packages\spacy\cli\download.py", line 36, in download_cli
download(model, direct, sdist, *ctx.args)
File "C:\Users\krish\anaconda3\lib\site-packages\spacy\cli\download.py", line 70, in download
compatibility = get_compatibility()
File "C:\Users\krish\anaconda3\lib\site-packages\spacy\cli\download.py", line 97, in get_compatibility
r = requests.get(about.__compatibility__)
File "C:\Users\krish\anaconda3\lib\site-packages\requests\api.py", line 73, in get
return request("get", url, params=params, **kwargs)
File "C:\Users\krish\anaconda3\lib\site-packages\requests\api.py", line 59, in request
return session.request(method=method, url=url, **kwargs)
File "C:\Users\krish\anaconda3\lib\site-packages\requests\sessions.py", line 587, in request
resp = self.send(prep, **send_kwargs)
File "C:\Users\krish\anaconda3\lib\site-packages\requests\sessions.py", line 701, in send
r = adapter.send(request, **kwargs)
File "C:\Users\krish\anaconda3\lib\site-packages\requests\adapters.py", line 565, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='raw.githubusercontent.com', port=443): Max retries exceeded with url: /explosion/spacy-models/master/compatibility.json (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x000001C494D17640>: Failed to establish a new connection: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond'))
I am trying to download a zip file that is stored here:
http://e4ftl01.cr.usgs.gov/MEASURES/SRTMGL1.003/2000.02.11/N45W074.SRTMGL1.hgt.zip
If you paste this into the browser and hit enter, it will download the .zip folder.
If you inspect the browser while this is happening, you will see that there is an internal redirect going on:
And eventually the zip gets downloaded.
I am trying to automate this downloading using the python requests library by doing the following:
import requests
requests.get(url,
allow_redirects=True,
headers={'User-Agent':'Chrome/107.0.0.0'})
I've tried tons of combinations, using the full header string from the HTML inspection, forcing verify=True, with and without redirects, adding a HTTPBasicAuth user/pass that says is required although the file seems to download fine without any credentials.
Honestly no clue what I'm missing, this is not my expertise. I keep getting this error:
>>> requests.get(url,
... allow_redirects=True,
... headers={'User-Agent':'Chrome/107.0.0.0'})
Traceback (most recent call last):
File "C:\Users\dere\Miniconda3\envs\blender\lib\site-packages\urllib3\connection.py", line 174, in _new_conn
conn = connection.create_connection(
File "C:\Users\dere\Miniconda3\envs\blender\lib\site-packages\urllib3\util\connection.py", line 95, in create_connection
raise err
File "C:\Users\dere\Miniconda3\envs\blender\lib\site-packages\urllib3\util\connection.py", line 85, in create_connection
sock.connect(sa)
ConnectionRefusedError: [WinError 10061] No connection could be made because the target machine
actively refused it
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\dere\Miniconda3\envs\blender\lib\site-packages\urllib3\connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "C:\Users\dere\Miniconda3\envs\blender\lib\site-packages\urllib3\connectionpool.py", line 398, in _make_request
conn.request(method, url, **httplib_request_kw)
File "C:\Users\dere\Miniconda3\envs\blender\lib\site-packages\urllib3\connection.py", line 239, in request
super(HTTPConnection, self).request(method, url, body=body, headers=headers)
File "C:\Users\dere\Miniconda3\envs\blender\lib\http\client.py", line 1282, in request
self._send_request(method, url, body, headers, encode_chunked)
File "C:\Users\dere\Miniconda3\envs\blender\lib\http\client.py", line 1328, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "C:\Users\dere\Miniconda3\envs\blender\lib\http\client.py", line 1277, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "C:\Users\dere\Miniconda3\envs\blender\lib\http\client.py", line 1037, in _send_output
self.send(msg)
File "C:\Users\dere\Miniconda3\envs\blender\lib\http\client.py", line 975, in send
self.connect()
File "C:\Users\dere\Miniconda3\envs\blender\lib\site-packages\urllib3\connection.py", line 205, in connect
conn = self._new_conn()
File "C:\Users\dere\Miniconda3\envs\blender\lib\site-packages\urllib3\connection.py", line 186, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x0000016B76A6FE50>: Failed to establish a new connection: [WinError 10061] No connection could be made because the target machine actively refused it
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\dere\Miniconda3\envs\blender\lib\site-packages\requests\adapters.py", line 489, in send
resp = conn.urlopen(
File "C:\Users\dere\Miniconda3\envs\blender\lib\site-packages\urllib3\connectionpool.py", line 787, in urlopen
retries = retries.increment(
File "C:\Users\dere\Miniconda3\envs\blender\lib\site-packages\urllib3\util\retry.py", line 592, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='e4ftl01.cr.usgs.gov', port=80): Max retries exceeded with url: /MEASURES/SRTMGL1.003/2000.02.11/N45W074.SRTMGL1.hgt.zip (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x0000016B76A6FE50>: Failed to establish a new connection: [WinError 10061] No connection could be made because the target machine actively refused it'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Users\dere\Miniconda3\envs\blender\lib\site-packages\requests\api.py", line 73, in get
return request("get", url, params=params, **kwargs)
File "C:\Users\dere\Miniconda3\envs\blender\lib\site-packages\requests\api.py", line 59, in request
return session.request(method=method, url=url, **kwargs)
File "C:\Users\dere\Miniconda3\envs\blender\lib\site-packages\requests\sessions.py", line 587, in request
resp = self.send(prep, **send_kwargs)
File "C:\Users\dere\Miniconda3\envs\blender\lib\site-packages\requests\sessions.py", line 701, in send
r = adapter.send(request, **kwargs)
File "C:\Users\dere\Miniconda3\envs\blender\lib\site-packages\requests\adapters.py", line 565, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='e4ftl01.cr.usgs.gov', port=80): Max retries exceeded with url: /MEASURES/SRTMGL1.003/2000.02.11/N45W074.SRTMGL1.hgt.zip (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x0000016B76A6FE50>: Failed to establish a new connection: [WinError 10061] No connection could be made because the target machine actively refused it'))
Can someone help me arrive at the code that will result in a successful request? I know how to write the request into a zip afterwards..
Change http to Https: This should work
import requests
# download zip file from url
url = "https://e4ftl01.cr.usgs.gov/MEASURES/SRTMGL1.003/2000.02.11/N45W074.SRTMGL1.hgt.zip"
r = requests.get(url)
with open("N45W074.SRTMGL1.hgt.zip", "wb") as f:
f.write(r.content)
Thanks to #cnemri for the tip.
It was a combination of changing http to https and also including this specific cookie header in the request header:
headers = {'Cookie': '_gid=GA1.2.48775707.1669266346; _hjSessionUser_606685=eyJpZCI6IjQ1MzEzM2QzLTI3MWEtNWM0YS04M2YzLWRmMmMzNDk4NjY1ZSIsImNyZWF0ZWQiOjE2NjkyNjYzNDU2MjUsImV4aXN0aW5nIjp0cnVlfQ==; ERS_production_2=b7dfade669180a6d6d250b030ffc3cf2UZZa8%2F471MfcV%2FaNuFtu6Pli%2BpP8jKOIwR4JvQjm%2B6DLPjl679vVf5SCDk7C5TLQsj7qckIev6lmtGb6Mes5RKDHUs%2BBp3EAjKW2%2BMCUpV%2Fnx0z1pdaCQQ%3D%3D; EROS_SSO_production_secure=eyJjcmVhdGVkIjoxNjY5MjY5MjEzLCJ1cGRhdGVkIjoiMjAyMi0xMS0yMyAyMzo1MzoyOCIsImF1dGhUeXBlIjoiRVJTIiwiYXV0aFNlcnZpY2UiOiJFUk9TIiwidmVyc2lvbiI6MS4xLCJzdGF0ZSI6ImVjMDY3YmMzYmRhMzBiZTUyNTkxYTNiZTYwMTMwZWNmNjAwMWU1Y2JlMGMxZmNkYTU4Y2Y4OTY0YjRlNTJkOTEiLCJpZCI6InY5U1Q4YVl3M1hRKCsyIiwic2VjcmV0IjoiPl8lMCZPYiY9MUVkclRLZTt4fHVZLVcyU1VvTiA1SE17KiQqaG12OSxvPl5%2BdyJ9; _ga_0YWDZEJ295=GS1.1.1669269458.1.0.1669269460.0.0.0; _ga=GA1.2.1055277358.1669266346; _ga_71JPYV1CCS=GS1.1.1669306300.2.1.1669306402.0.0.0; DATA=Y3_gg9uD6LmunsfDHneR9wAAARQ'}
then, this worked:
requests.get(url, headers=headers)
I'm not sure if this is the "solution" or just a workaround I've discovered. Still appreciate any input. I think it may just be credentials that are stored?
I have a simple container running a python script which needs to perform a HTTP GET. Problem is, ping works fine, both resolving the name with mDNS as well as with standard IP.
However, the HTTP GET does not work and the script hangs on it. Clearly the same script outside the container works just fine. This is the errors I am getting:
>>> requests.get("http://iot-ns-controller.local:8081")
Traceback (most recent call last):
File "/usr/lib/python3.10/site-packages/urllib3/connection.py", line 174, in _new_conn
conn = connection.create_connection(
File "/usr/lib/python3.10/site-packages/urllib3/util/connection.py", line 95, in create_connection
raise err
File "/usr/lib/python3.10/site-packages/urllib3/util/connection.py", line 85, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "/usr/lib/python3.10/site-packages/urllib3/connectionpool.py", line 398, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/lib/python3.10/site-packages/urllib3/connection.py", line 239, in request
super(HTTPConnection, self).request(method, url, body=body, headers=headers)
File "/usr/lib/python3.10/http/client.py", line 1282, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib/python3.10/http/client.py", line 1328, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib/python3.10/http/client.py", line 1277, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib/python3.10/http/client.py", line 1037, in _send_output
self.send(msg)
File "/usr/lib/python3.10/http/client.py", line 975, in send
self.connect()
File "/usr/lib/python3.10/site-packages/urllib3/connection.py", line 205, in connect
conn = self._new_conn()
File "/usr/lib/python3.10/site-packages/urllib3/connection.py", line 186, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7faa6658a0>: Failed to establish a new connection: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.10/site-packages/requests/adapters.py", line 489, in send
resp = conn.urlopen(
File "/usr/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
retries = retries.increment(
File "/usr/lib/python3.10/site-packages/urllib3/util/retry.py", line 592, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='iot-ns-controller.local', port=8081): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7faa6658a0>: Failed to establish a new connection: [Errno 111] Connection refused'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.10/site-packages/requests/api.py", line 73, in get
return request("get", url, params=params, **kwargs)
File "/usr/lib/python3.10/site-packages/requests/api.py", line 59, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/lib/python3.10/site-packages/requests/sessions.py", line 587, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python3.10/site-packages/requests/sessions.py", line 701, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python3.10/site-packages/requests/adapters.py", line 565, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='iot-ns-controller.local', port=8081): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7faa6658a0>: Failed to establish a new connection: [Errno 111] Connection refused'))
It also does not work if I query host.docker.internal and I get the same error
import requests
try:
import xml.etree.cElementTree as et
except ImportError:
import xml.etree.ElementTree as et
user_key = authorized_key
doc_name = "F-C0032-001"
api_link = "http://opendata.cwb.gov.tw/opendataapi?dataid=%s&authorizationkey=%s" % (doc_name,user_key)
report = requests.get(url=api_link).text #The problem is here
Error:
Traceback (most recent call last):
File "C:\Users\domin\AppData\Local\Programs\Python\Python39\lib\site-packages\urllib3\connection.py", line 169, in _new_conn
conn = connection.create_connection(
File "C:\Users\domin\AppData\Local\Programs\Python\Python39\lib\site-packages\urllib3\util\connection.py", line 96, in create_connection
raise err
File "C:\Users\domin\AppData\Local\Programs\Python\Python39\lib\site-packages\urllib3\util\connection.py", line 86, in create_connection
sock.connect(sa)
ConnectionRefusedError: [WinError 10061]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\domin\AppData\Local\Programs\Python\Python39\lib\site-packages\urllib3\connectionpool.py", line 699, in urlopen
httplib_response = self._make_request(
File "C:\Users\domin\AppData\Local\Programs\Python\Python39\lib\site-packages\urllib3\connectionpool.py", line 394, in _make_request
conn.request(method, url, **httplib_request_kw)
File "C:\Users\domin\AppData\Local\Programs\Python\Python39\lib\site-packages\urllib3\connection.py", line 234, in request
super(HTTPConnection, self).request(method, url, body=body, headers=headers)
File "C:\Users\domin\AppData\Local\Programs\Python\Python39\lib\http\client.py", line 1255, in request
self._send_request(method, url, body, headers, encode_chunked)
File "C:\Users\domin\AppData\Local\Programs\Python\Python39\lib\http\client.py", line 1301, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "C:\Users\domin\AppData\Local\Programs\Python\Python39\lib\http\client.py", line 1250, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "C:\Users\domin\AppData\Local\Programs\Python\Python39\lib\http\client.py", line 1010, in _send_output
self.send(msg)
File "C:\Users\domin\AppData\Local\Programs\Python\Python39\lib\http\client.py", line 950, in send
self.connect()
File "C:\Users\domin\AppData\Local\Programs\Python\Python39\lib\site-packages\urllib3\connection.py", line 200, in connect
conn = self._new_conn()
File "C:\Users\domin\AppData\Local\Programs\Python\Python39\lib\site-packages\urllib3\connection.py", line 181, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x000001F0DC29F940>: Failed to establish a new connection: [WinError 10061]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\domin\AppData\Local\Programs\Python\Python39\lib\site-packages\requests\adapters.py", line 439, in send
resp = conn.urlopen(
File "C:\Users\domin\AppData\Local\Programs\Python\Python39\lib\site-packages\urllib3\connectionpool.py", line 755, in urlopen
retries = retries.increment(
File "C:\Users\domin\AppData\Local\Programs\Python\Python39\lib\site-packages\urllib3\util\retry.py", line 573, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='opendata.cwb.gov.tw', port=80): Max retries exceeded with url: /opendataapi?dataid=F-C0032-001&authorizationkey= (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x000001F0DC29F940>: Failed to establish a new connection: [WinError 10061]'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "D:/Python Books/Python/Ch17 Flask Web API/weatherdata.py", line 14, in <module>
report = requests.get(url=api_link,headers=headers).text
File "C:\Users\domin\AppData\Local\Programs\Python\Python39\lib\site-packages\requests\api.py", line 76, in get
return request('get', url, params=params, **kwargs)
File "C:\Users\domin\AppData\Local\Programs\Python\Python39\lib\site-packages\requests\api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "C:\Users\domin\AppData\Local\Programs\Python\Python39\lib\site-packages\requests\sessions.py", line 542, in request
resp = self.send(prep, **send_kwargs)
File "C:\Users\domin\AppData\Local\Programs\Python\Python39\lib\site-packages\requests\sessions.py", line 655, in send
r = adapter.send(request, **kwargs)
File "C:\Users\domin\AppData\Local\Programs\Python\Python39\lib\site-packages\requests\adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='opendata.cwb.gov.tw', port=80): Max retries exceeded with url: /opendataapi?dataid=F-C0032-001&authorizationkey= (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x000001F0DC29F940>: Failed to establish a new connection: [WinError 10061]'))
Actually my aim is to read the data and select the related location from the api_link but however the file is too bulky so is there any good ways or methods to properly read the related data from the xml file without any connection error messages? Since I do not understand what the error messages telling me how to deal with so if possible could anyone give some related solutions?
it looks like the error is not reading the xlm but when you send the request to the server, this error:
requests.exceptions.ConnectionError: HTTPConnectionPool(host='opendata.cwb.gov.tw', port=80): Max retries exceeded with url: /opendataapi?dataid=F-C0032-001&authorizationkey=
Is send when you're sending too many requests from same ip address in short period of time.
Have you tried this ?
https://stackoverflow.com/a/47475019/17044313
It should look like this:
import requests
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.retry import Retry
user_key = Authorization Key
doc_name = 'F-C0032-001'
api_link = 'http://opendata.cwb.gov.tw/opendataapi?dataid=%s&authorizationkey=%s'%(doc_name,user_key)
session = requests.Session()
retry = Retry(connect=3, backoff_factor=0.5)
adapter = HTTPAdapter(max_retries=retry)
session.mount('http://', adapter)
session.mount('https://', adapter)
session.get(api_link)
I'm sure this is a very simple question (HTTP newbie) but I couldn't find the answer myself. I would appreciate any help here.
I have a web server:
serversocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
serversocket.bind((socket.gethostname(), 8000))
serversocket.listen(10)
while True:
print("waiting...")
conn = serversocket.accept()
data = conn[0].recv(1024)
print(data)
I also have a client trying to send a GET request:
import requests
URL = "http://localhost:8000/api/v1/a?b=2"
r = requests.get(url = URL)
In this stage I don't want to do anything with the request, just to make sure I receive it, but this fails...
I run:
python3 server.py &
python3 client.py
and got:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 562, in urlopen
body=body, headers=headers)
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 387, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/lib/python3.4/http/client.py", line 1088, in request
self._send_request(method, url, body, headers)
File "/usr/lib/python3.4/http/client.py", line 1126, in _send_request
self.endheaders(body)
File "/usr/lib/python3.4/http/client.py", line 1084, in endheaders
self._send_output(message_body)
File "/usr/lib/python3.4/http/client.py", line 922, in _send_output
self.send(msg)
File "/usr/lib/python3.4/http/client.py", line 857, in send
self.connect()
File "/usr/lib/python3.4/http/client.py", line 834, in connect
self.timeout, self.source_address)
File "/usr/lib/python3.4/socket.py", line 512, in create_connection
raise err
File "/usr/lib/python3.4/socket.py", line 503, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 330, in send
timeout=timeout
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 612, in urlopen
raise MaxRetryError(self, url, e)
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=8000): Max retries exceeded with url: /api/v1/similar?word=apple (Caused by <class 'ConnectionRefusedError'>: [Errno 111] Connection refused)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "client.py", line 5, in <module>
r = requests.get(url = URL)
File "/usr/lib/python3/dist-packages/requests/api.py", line 55, in get
return request('get', url, **kwargs)
File "/usr/lib/python3/dist-packages/requests/api.py", line 44, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 455, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 558, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 378, in send
raise ConnectionError(e)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=8000): Max retries exceeded with url: /api/v1/similar?word=apple (Caused by <class 'ConnectionRefusedError'>: [Errno 111] Connection refused)
Connection refused is generated by the kernel because receiving side refuses to establish TCP connection. Possible reasons are:
No on listen on this host/port
iptables (or other firewall) specifically -j REJECT those connections.
From your code it's hard to say if you listen on a proper host/port or not.
How to debug?
Run you service. Without connecting to it run 'netstat -lnpt', it should provide you with list of services listening on this host.
Try nc host port (note - not a ':', but a space here), if connection is refused, nc will terminate instantly. If not, you've established connection (try to type 'GET / HTTP/1.1' and Enter twice in this case).
Just print output of socket.gethostname() and port variable in your code.
I suspect that in your case socket.gethostname() has been resolved into something different than localhost.