I am trying to write a Python script which enables me to acces a webpage and download a file from that page. My first attempt was to simply get to that page and i tried the following code:
import requests
url = 'https://www.google.com/?gws_rd=ssl' #using google as an example
r = requests.get(url)
print(r.url)
I am given this error:
runfile('C:/Users/ME/Desktop/TMS502.py', wdir='C:/Users/ME/Desktop')
Traceback (most recent call last):
File "<ipython-input-23-bc585dcceef8>", line 1, in <module>
runfile('C:/Users/ME/Desktop/TMS502.py', wdir='C:/Users/ME/Desktop')
File "C:\Users\ME\AppData\Local\Continuum\Anaconda\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 585, in runfile
execfile(filename, namespace)
File "C:/Users/ME/Desktop/TMS502.py", line 16, in <module>
r = requests.get(url)
File "C:\Users\ME\AppData\Local\Continuum\Anaconda\lib\site-packages\requests\api.py", line 55, in get
return request('get', url, **kwargs)
File "C:\Users\ME\AppData\Local\Continuum\Anaconda\lib\site-packages\requests\api.py", line 44, in request
return session.request(method=method, url=url, **kwargs)
File "C:\Users\ME\AppData\Local\Continuum\Anaconda\lib\site-packages\requests\sessions.py", line 456, in request
resp = self.send(prep, **send_kwargs)
File "C:\Users\ME\AppData\Local\Continuum\Anaconda\lib\site-packages\requests\sessions.py", line 559, in send
r = adapter.send(request, **kwargs)
File "C:\Users\ME\AppData\Local\Continuum\Anaconda\lib\site-packages\requests\adapters.py", line 375, in send
raise ConnectionError(e, request=request)
ConnectionError: HTTPSConnectionPool(host='www.google.com', port=443): Max retries exceeded with url: /?gws_rd=ssl (Caused by <class 'socket.error'>: [Errno 10054] An existing connection was forcibly closed by the remote host)
Can someone please help me?
You are getting that error because the remote side (in this case Google) is closing your requests or you are otherwise no longer able to establish a connection to it.
From the error:
ConnectionError: HTTPSConnectionPool(host='www.google.com', port=443):
Max retries exceeded with url: /?gws_rd=ssl
(Caused by <class 'socket.error'>: [Errno 10054] An existing connection was forcibly closed by the remote host)
We can look into the source for a hint:
class MaxRetryError(RequestError):
"""Raised when the maximum number of retries is exceeded.
:param pool: The connection pool
:type pool: :class:`~urllib3.connectionpool.HTTPConnectionPool`
:param string url: The requested Url
:param exceptions.Exception reason: The underlying error
"""
def __init__(self, pool, url, reason=None):
self.reason = reason
message = "Max retries exceeded with url: %s (Caused by %r)" % (
url, reason)
RequestError.__init__(self, pool, url, message)
Try another example host and your code should work, such as https://example.org.
The error message "An existing connection was forcibly closed by the remote host" is coming from your operating system (Windows) and Requests is showing you this text in an attempt to be helpful.
Your code is fine.
I guess the problem is coursed by url = 'https://www.google.com/?gws_rd=ssl'.
Maybe your network can't reach www.google.com, try another url.
y are you bothering with requests anyway:
from urllib2 import urlopen
u = urlopen("https://www.google.com/?gws_rd=ssl")
data = u.read()
u.close()
Maybe this'll work.
Related
The requests documentation (link) mentioned that a session is what allows some parameters to persist across requests. My use case is simple; because I sit behind a corporate proxy and firewall, I need to set the proxy parameters proxies (as mentioned in the title) in a session and I don't want to have to set it for every request.
Supposedly, you can do the following (directly copied from the proxies section):
import requests
proxies = {
'http': 'http://10.10.1.10:3128',
'https': 'http://10.10.1.10:1080',
}
session = requests.Session()
session.proxies.update(proxies)
session.get('http://example.org')
This should allow you to set proxies, without stating them in the request itself. Thus my session function looks like this below:
def requests_setup():
# setup proxy
proxies = {'http': f'http://someproxy:8080',
'https': f'http://someproxy:8080'}
# initialize session
session = requests.Session()
# Part 1: set up proxy
session.proxies.update(proxies)
# Part 2: add certificate
session.verify = r'SOME_CERT_BUNDLE.pem'
return session
Get request example that results in an error
# making an example get request
setup = requests_setup()
url = "https://example.com"
r = setup.get(f"{url}", timeout=5)
Posting the full traceback below, but the following errors seems to be the problem. And my understanding of this is that the ssl cert verification did not go through for some reason (as suggested by the trace, I believe it is because proxy was not included; for a session without the verify parameter set, it would instead result in a sslCertVerification error during the request that worked below).
Error 1 ...
socket.timeout: _ssl.c:1074: The handshake operation timed out
... leading to Error 2
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='example.com', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', timeout('_ssl.c:1074: The handshake operation timed out')))
... and finally Error 3
requests.exceptions.ProxyError: HTTPSConnectionPool(host='example.com', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', timeout('_ssl.c:1074: The handshake operation timed out')))
The silver lining is that this was solved, eventually, by specifying the parameter in the request body.
setup = utils.requests_setup()
# making an example get request
url = "https://example.com"
proxies = {'http': f'http://someproxy:8080',
'https': f'http://someproxy:8080'}
r = setup.get(f"{url}", timeout=5, proxies=proxies)
But why is that the case? I can see clearly that my session's proxy attributes are initialized . But for some reason it was not utilized in the get request made using that session.
PS: There might be questions about why my proxy is prefixed with http for both cases. It is purely because we don't have a standalone https proxy server. The request also fails when I use a "HTTPS" prefix instead there.
PPS: example.com is not the site used. I have tried to use google.com, or others (such as the API I am trying to call), but that did not change the results.
Actual Error Traceback
Traceback (most recent call last):
File "C:\ProgramData\Anaconda3\envs\VA_API\lib\site-packages\urllib3\connectionpool.py", line 696, in urlopen
self._prepare_proxy(conn)
File "C:\ProgramData\Anaconda3\envs\VA_API\lib\site-packages\urllib3\connectionpool.py", line 964, in _prepare_proxy
conn.connect()
File "C:\ProgramData\Anaconda3\envs\VA_API\lib\site-packages\urllib3\connection.py", line 359, in connect
conn = self._connect_tls_proxy(hostname, conn)
File "C:\ProgramData\Anaconda3\envs\VA_API\lib\site-packages\urllib3\connection.py", line 506, in _connect_tls_proxy
ssl_context=ssl_context,
File "C:\ProgramData\Anaconda3\envs\VA_API\lib\site-packages\urllib3\util\ssl_.py", line 450, in ssl_wrap_socket
sock, context, tls_in_tls, server_hostname=server_hostname
File "C:\ProgramData\Anaconda3\envs\VA_API\lib\site-packages\urllib3\util\ssl_.py", line 493, in _ssl_wrap_socket_impl
return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
File "C:\ProgramData\Anaconda3\envs\VA_API\lib\ssl.py", line 423, in wrap_socket
session=session
File "C:\ProgramData\Anaconda3\envs\VA_API\lib\ssl.py", line 870, in _create
self.do_handshake()
File "C:\ProgramData\Anaconda3\envs\VA_API\lib\ssl.py", line 1139, in do_handshake
self._sslobj.do_handshake()
socket.timeout: _ssl.c:1074: The handshake operation timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\ProgramData\Anaconda3\envs\VA_API\lib\site-packages\requests\adapters.py", line 449, in send
timeout=timeout
File "C:\ProgramData\Anaconda3\envs\VA_API\lib\site-packages\urllib3\connectionpool.py", line 756, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "C:\ProgramData\Anaconda3\envs\VA_API\lib\site-packages\urllib3\util\retry.py", line 574, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='example.com', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', timeout('_ssl.c:1074: The handshake operation timed out')))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<input>", line 1, in <module>
File "C:\ProgramData\Anaconda3\envs\VA_API\lib\site-packages\requests\sessions.py", line 555, in get
return self.request('GET', url, **kwargs)
File "C:\ProgramData\Anaconda3\envs\VA_API\lib\site-packages\requests\sessions.py", line 542, in request
resp = self.send(prep, **send_kwargs)
File "C:\ProgramData\Anaconda3\envs\VA_API\lib\site-packages\requests\sessions.py", line 655, in send
r = adapter.send(request, **kwargs)
File "C:\ProgramData\Anaconda3\envs\VA_API\lib\site-packages\requests\adapters.py", line 510, in send
raise ProxyError(e, request=request)
requests.exceptions.ProxyError: HTTPSConnectionPool(host='example.com', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', timeout('_ssl.c:1074: The handshake operation timed out')))
Information for reproducing the issue:
OS version: 'Windows-10-10.0.18362-SP0'
Python version: '3.7.11 (default, Jul 27 2021, 09:42:29) [MSC v.1916 64 bit (AMD64)]'
Requests version: '2.26.0'
I am trying to get this to work:
Practitioner notes -> Python scripts for automation in NA -> Python client to invoke NA SOAP APIs
Here is my code (sanitized a bit):
#! /usr/bin/env python3
from requests import Session
from zeep import Client
from zeep.transports import Transport
session = Session()
session.verify = False
transport = Transport(session=session)
client = Client( 'https://SERVER_FQDN/soap?wsdl=api.wsdl.wsdl2py', transport=transport)
# I added this for the network proxy
client.transport.session.proxies = {
'http': '10.0.0.1:80',
'https': '10.0.0.1:80',
}
# Then found I needed this because "localhost" is hard-coded in the WSDL
client.service._binding_options['address'] = 'https://SERVER_FQDN/soap'
login_params = {
'username':'user',
'password':'PASSWORD',
}
loginResult = client.service.login(parameters=login_params )
sesnhdr_type = client.get_element('ns0:list_deviceInputParms')
sesnhdr = sesnhdr_type(sessionid=loginResult.Text)
devices = client.service.list_device(_soapheaders=[sesnhdr], parameters=sesnhdr)
print('\n\n ----------------------------- \n')
for i in devices.ResultSet.Row:
print(i.hostName + ' ---> '+i.primaryIPAddress)
params = {
"ip":i.primaryIPAddress,
"sessionid": loginResult.Text
}
device = client.service.show_deviceinfo(parameters=params)
print(device.Text)
print('\n\n ----------------------------- \n')
And here is my output:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 667, in urlopen
self._prepare_proxy(conn)
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 932, in _prepare_proxy
conn.connect()
File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 317, in connect
self._tunnel()
File "/usr/lib64/python3.6/http/client.py", line 929, in _tunnel
message.strip()))
OSError: Tunnel connection failed: 503 Service Unavailable
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 727, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/usr/local/lib/python3.6/site-packages/urllib3/util/retry.py", line 439, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='SERVER_FQDN', port=443): Max retries exceeded with url: /soap (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 503 Service Unavailable',)))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "./na-1.py", line XX, in <module>
loginResult = client.service.login(parameters=login_params )
File "/usr/local/lib/python3.6/site-packages/zeep/proxy.py", line 51, in __call__
kwargs,
File "/usr/local/lib/python3.6/site-packages/zeep/wsdl/bindings/soap.py", line 127, in send
response = client.transport.post_xml(options["address"], envelope, http_headers)
File "/usr/local/lib/python3.6/site-packages/zeep/transports.py", line 107, in post_xml
return self.post(address, message, headers)
File "/usr/local/lib/python3.6/site-packages/zeep/transports.py", line 74, in post
address, data=message, headers=headers, timeout=self.operation_timeout
File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 578, in post
return self.request('POST', url, data=data, json=json, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 530, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 643, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 510, in send
raise ProxyError(e, request=request)
requests.exceptions.ProxyError: HTTPSConnectionPool(host='SERVER_FQDN', port=443): Max retries exceeded with url: /soap (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 503 Service Unavailable',)))
I get the same errors if I use "localhost" and run the script on the server-in-question.
The system has proxy environment values set.
There are proper forward and reverse DNS entries for the server.
The name and IP for the server are also in /etc/hosts
Here is the problem:
If I use an IP address instead of the server's FQDN, the code runs.
Vendor support says the problem is not in their application that provides the endpoint:
The 503 error means that the service is not available, there are 3 situations that invoke this behavior: 1. The server is under maintenance, 2. The server is overloaded, 3. In rare cases, the DNS configuration is faulty. If we see, this problem is not related to NA because the request is working fine with the IP.
Any ideas on this ?
Why does only the IP work and NOT the FQDN or localhost ?
Most of the documentation I see for using proxies with Zeep start with client = Client(url) but that doesn't work if url is behind a firewall and can't be accessed except through a proxy! My attempt to do it according to the documentation did nothing but time out (of course).
The key is in understanding that Zeep is built on requests and requests can use proxies for initiating the session. So you need to build a proxied Session, then pass that session into the Transport, and initialize the Zeep client with that transport. This worked for me:
session = requests.Session()
session.auth = requests.auth.HTTPBasicAuth(soap_username, soap_password)
session.proxies = {"https": f"socks5://{settings.STATIC_PROXY}"}
transport = zeep.transports.Transport(session=session, timeout=(5, 30))
client = zeep.Client(url, transport=transport)
My problem lay in the fact that the initialization of Client wants to go ahead and make the connection, but I need the proxy setting at the start. So I cobbled together two examples from the official docs to set the proxy at the time the connection is made.
from zeep import Client
from zeep.transports import Transport
from requests import Session
session = Session()
session.proxies = {
'http': 'http://username:password#proxy.example.com:8080',
'https': 'http://username:password#proxy.example.com:8080'
}
transport=Transport(session=session)
client = Client(URL,transport=transport)
I've exposed an URL (http://127.0.0.1:5000/daily) but in Google Compute Engine (GCE) I am not getting the values. If I access this URL through requests in simple python program, it is running efficiently.
import requests
import json
req=requests.get('http://127.0.0.1:5000/daily')
a = json.loads(req.text)
discount_rate = a['data']['policy_rate']
six_months_kibor = a['data']['today_kibor_rate']
dollar_to_pkr= a['data']['today_usd_rate']
print(discount_rate, six_months_kibor, dollar_to_pkr)
ERROR which I am receiving from GCE is:
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f93526c16a0>: Failed to establish a new connection: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/dev_baseh/.local/lib/python3.5/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/home/dev_baseh/.local/lib/python3.5/site-packages/urllib3/connectionpool.py", line 641, in urlopen
_stacktrace=sys.exc_info()[2])
File "/home/dev_baseh/.local/lib/python3.5/site-packages/urllib3/util/retry.py", line 399, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='127.0.0.1', port=5000): Max retries exceeded with url: /daily (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f93526c16a0>: Failed to establish a new connection: [Errno 111] Connection refused',))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "test.py", line 6, in <module>
req=requests.get('http://127.0.0.1:5000/daily')
File "/home/dev_baseh/.local/lib/python3.5/site-packages/requests/api.py", line 75, in get
return request('get', url, params=params, **kwargs)
File "/home/dev_baseh/.local/lib/python3.5/site-packages/requests/api.py", line 60, in request
return session.request(method=method, url=url, **kwargs)
File "/home/dev_baseh/.local/lib/python3.5/site-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/home/dev_baseh/.local/lib/python3.5/site-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/home/dev_baseh/.local/lib/python3.5/site-packages/requests/adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=5000): Max retries exceeded with url: /daily (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f93526c16a0>: Failed to establish a new connection: [Errno 111] Connection refused', ))
I don't the reason, that why it is not running over GCE.
Thanks in Advance :)
The IP address 127.0.0.1 refers to the local IP address of your machine. So if you run a python program on the same machine where you're running that server, it would be able to access that address since both have the same IP address.
When you try to access 127.0.0.1 from GCP, what is happening is GCP is locally trying to access the port 5000 and not your machine's port 5000.
You would need to figure out the public facing IP address of the machine where you're running the server. If it's on your computer, you could just Google, "what is my IP" and get it.
I'm trying to write a script in python to send HTTP get request to automatically generated URLs and get its response code and elapsed time. The URLs need not necessarily be a valid one, 400 responses are acceptable too.
script1.py
import sys
import requests
str1="http://www.googl"
str3=".com"
str2='a'
for x in range(0, 8):
y = chr(ord(str2)+x)
str_s=str1+y+str3
r=requests.get(str_s)
print(str_s, r.status_code, r.elapsed.total_seconds())
Error:
File "script1.py", line 12, in <module><br>
r=requests.get(str_s)<br>
File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 72, in get<br>
return request('get', url, params=params, **kwargs)<br>
File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 58, in request<br>
return session.request(method=method, url=url, **kwargs)<br>
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 508, in request<br>
resp = self.send(prep, **send_kwargs)<br>
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 618, in send<br>
r = adapter.send(request, **kwargs)<br>
File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 508, in send<br>
raise ConnectionError(e, request=request)<br>
requests.exceptions.ConnectionError: HTTPConnectionPool(host='www.googla.com', port=80): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc44c891e50>: Failed to establish a new connection: [Errno -2] Name or service not known',))
I just want see the time taken to receive response of each request.
Only one request has to be sent
Response code does not matter.
I guess you want to get something like this:
import sys
import requests
str1="http://www.googl"
str3=".com"
str2='a'
for x in range(0, 8):
y = chr(ord(str2)+x)
str_s=str1+y+str3
print('Connecting to ' + str_s)
try:
r = requests.get(str_s)
print(str_s, r.status_code, r.elapsed.total_seconds())
except requests.ConnectionError as e:
print(" Failed to open url")
In this case, using the try...except you can catch the exception that get raises and handle it in a nice way.
I have my backend developed in java which does all kind of processing. And my frontend is developed using python's flask framework. I am using requests to send a request and get a response from the apis present in java.
Following is the line in my code which does that:
req = requests.post(buildApiUrl.getUrl('user') + "/login", data=payload)
My problem is, sometimes when the tomcat instance is not running or there is some issue with java apis, I always get an error from requests as follows:
ERROR:root:HTTPConnectionPool(host='localhost', port=8080): Max retries exceeded with url: /MYAPP/V1.0/user/login (Caused by <class 'socket.error'>: [Errno 111] Connection refused)
Traceback (most recent call last):
File "/home/rahul/git/myapp/webapp/views/utils.py", line 31, in decorated_view
return_value = func(*args, **kwargs)
File "/home/rahul/git/myapp/webapp/views/public.py", line 37, in login
req = requests.post(buildApiUrl.getUrl('user') + "/login", data=payload)
File "/home/rahul/git/myapp/venv/local/lib/python2.7/site-packages/requests/api.py", line 88, in post
return request('post', url, data=data, **kwargs)
File "/home/rahul/git/myapp/venv/local/lib/python2.7/site-packages/requests/api.py", line 44, in request
return session.request(method=method, url=url, **kwargs)
File "/home/rahul/git/myapp/venv/local/lib/python2.7/site-packages/requests/sessions.py", line 335, in request
resp = self.send(prep, **send_kwargs)
File "/home/rahul/git/myapp/venv/local/lib/python2.7/site-packages/requests/sessions.py", line 438, in send
r = adapter.send(request, **kwargs)
File "/home/rahul/git/myapp/venv/local/lib/python2.7/site-packages/requests/adapters.py", line 327, in send
raise ConnectionError(e)
ConnectionError: HTTPConnectionPool(host='localhost', port=8080): Max retries exceeded with url: /MYAPP/V1.0/user/login (Caused by <class 'socket.error'>: [Errno 111] Connection refused)
I want to handle any such errors that I receive in my flask app so that I can give the necessary response on the web page instead of showing blank screen. So how can I achieve this?
Catch the exception request.post raises using try-except:
try:
req = requests.post(buildApiUrl.getUrl('user') + "/login", data=payload)
except requests.exceptions.RequestException:
# Handle exception ..