Suppose that I have http server located at (host:port) and I'm trying to use HTTPClient example from the tornado docs to fetch some data from that server, so I have a code below
from tornado import httpclient
http_client = httpclient.HTTPClient()
try:
response = http_client.fetch("http://host:port", connect_timeout=2)
print response.body
except httpclient.HTTPError as e:
print e.code
print e.message
http_client.close()
This works fine and in case everything goes well (host is available, server is working, etc.) my client successfully receives data from the server. Now I want to handle errors that can occur during connection process and handle them depending on the error code. I'm trying to get error code by using e.code in my exception handler, but it contains HTTP error codes only and I need some lower level error codes, for instance:
1) If server is not avaliable on the destination host and port it prints out
599
HTTP 599: [Errno 111] Connection refused
2) If it is impossible to reach the host (for example due to network connectivity problems), I receive the same HTTP 599 error code:
599
HTTP 599: [Errno 101] Connection refused
So I need to get Errno in my exception handler that would indicate actual lower level socket error and the only solution I see for now is to grep it from the text message I receive (e.message) but this solution looks rough and ugly.
I think your only option is to pull it out of e.message:
import re
m = re.search( "\[Errno (\d+)\]", e.message)
if m:
errno = m.group(1)
On my platform (Linux), I actually get a socket.gaierror exception when I take your code example and use a bogus URL, which provides direct access to errno. But if you're dealing with tornado.httpclient.HTTPError, I don't think there's any way for you to get at it directly.
Related
So i want to send (through a proxy) a request to a website.. The script looks like this and its made with the socket library in python:
import socket
TargetDomainName="www.stackoverflow.com"
TargetIP="151.101.65.69"
TargetPort=80
ProxiesIP=["107.151.182.247"]
ProxiesPort=[80]
Connect=f"CONNECT {TargetDomainName} HTTP/1.1"
Connection=socket.socket(socket.AF_INET, socket.SOCK_STREAM)
Connection.connect((ProxiesIP[0],ProxiesPort[0]))
Connection.sendto(str.encode(Connect),(TargetIP, TargetPort))
Connection.sendto(("GET /" + TargetIP + " HTTP/1.1\r\n").encode('ascii'), (TargetIP, TargetPort))
Connection.sendto(("Host: " + ProxiesIP[0] + "\r\n\r\n").encode('ascii'), (TargetIP, TargetPort))
print (Connection.recv(1028))
Connection.close()
My question is why i get the 400 bad request error?
You did not indicate whether the 400 reply is coming from the proxy or the target server. But both of your commands are malformed.
Your CONNECT command is missing a port number, a Host header since you are requesting HTTP 1.1, and trailing line breaks to terminate the command properly.
Your GET command is sent to the target server (if CONNECT is successful) and should not be requesting a resource by IP address. It is also sending the wrong value for the Host header. The command is relative to the target server, so it needs to specify the target server's host name.
Also, you should be using send()/sendall() instead of sendto().
Try something more like this instead:
import socket
TargetDomainName="www.stackoverflow.com"
TargetIP="151.101.65.69"
TargetPort=80
ProxiesIP=["107.151.182.247"]
ProxiesPort=[80]
Connection=socket.socket(socket.AF_INET, socket.SOCK_STREAM)
Connection.connect((ProxiesIP[0], ProxiesPort[0]))
Connection.sendall((f"CONNECT {TargetDomainName}:{TargetPort} HTTP/1.1\r\n").encode("ascii"))
Connection.sendall((f"Host: {TargetDomainName}:{TargetPort}\r\n\r\n").encode("ascii"))
print (Connection.recv(1028))
Connection.sendall(("GET / HTTP/1.1\r\n").encode('ascii'))
Connection.sendall((f"Host: {TargetDomainName}\r\n").encode('ascii'))
Connection.sendall(("Connection: close\r\n\r\n").encode('ascii'))
print (Connection.recv(1028))
Connection.close()
You really need to read the proxy's reply before sending the GET command. The proxy will send its own HTTP reply indicating whether it successfully connected to the target server or not.
You really should not be implementing HTTP manually though, there are plenty of HTTP libraries for Python that can handle these details for you. Python even has one built-in: http.client
This is the syntax of the web browser I'm building:
"import socket
mysock=socket.socket(socket.AF_INET,socket.SOCK_STREAM)
print('Please Enter the website url and port below, else the default port will be used')
print('\rWebsite:')
website=input()
print('\rPORT:')
port=input()
if len(website)<2:
print("/n You have entered an Invalid Link")
if len(port)<1:
port=80
while len(website)>1:
mysock.connect((website, (int(port))))
cmd=('GET'+website+'HTTP/1.0 \r\n\r\n').encode()
mysock.send(cmd)
while True:
data=mysock.recv(512)
if len(data) < 1:
break
print(data.decode(),end='')
mysock.close()
"
I'm trying to access this link:
http://data.pr4e.org/romeo.txt through port 80 but receive
"Traceback (most recent call last):
File "C:\Users\PLAY\Documents\python work\browser.py", line 13, in <module>
mysock.connect((website, (int(port))))
socket.gaierror: [Errno 11001] getaddrinfo failed"
What's going on here and how could I improve it? I'm just messing around with the code abit and would like to understand more and make it more functional so I can build on top of it.
Thanks in advance.
You can't pass a whole URL to socket.connect(), only an IP/hostname and a port number. So, you would need to parse the user's input and break it up into its constituent components, and then you can act on them.
If the user enters http://data.pr4e.org/romeo.txt, you need to break that up into http, data.pr4e.org, and /romeo.txt. Connect the socket to data.pr4e.org on port 80 (because http is used), and then send a request for GET /romeo.txt HTTP/1.0\r\n\r\n (note the spaces!).
Python has a urllib.parse module for parsing URLs.
A better option is to use the urllib.request module, or even the Requests library, instead of a socket manually. Let the module/library do all of the URL parsing and socket handling for you. You give them a URL to access, they give you back the downloaded data from that location.
My web-application makes HTTP requests to an Extensible Service Proxy (ESP), which in-turn, delegates to a gRPC server (written in Python). Ignoring Android and iOS clients, the architecture is:
The ESP is a nginx reverse proxy.
The gRPC server ("Your code" in the reference architecture) may raise an exception, in which case I use context.abort to raise an exception and terminate the RPC with a non-OK status:
try:
# Do something that could fail.
except ValueError as e:
context.abort(grpc.StatusCode.DATA_LOSS, str(e))
While it is possible to use set_code and set_details, they still result in a HTTP status of 200 OK.
There are two problems:
The gRPC status codes are translated by the ESP container (nginx proxy) to a a generic 500 Internal Server Error.
The accompanying details are stripped-out.
and 2. combined means the web client has, at most, a 500 Internal Server Error for all exceptions raised by the gRPC server.
Ultimately, I don't understand how more informative (ideally, custom) errors can be returned to web-clients.
grpc Status code::DATA_LOSS, are translated to HTTP code 500. The code is here
The grpc status detail (status code and error message) are send back in the response body in JSON format. The code is here
I am using Python soap API client Zeep and here is the code that I have written:
from zeep import Client
def myapi(request):
client = Client("https://siteURL.asmx?wsdl")
key = client.service.LogOnUser('myusername', 'mypassord')
print(key)
it is giving me an error as: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
While I am trying below command the URL works well and shows all the services it has
python -mzeep https://siteURL.asmx?wsdl
Please help to understand what is the reason above code is not working.
PS: I couldn't share site URL which I am trying to connect to.
Additional Info: The site/page is accessible only through intranet and I am testing locally from intranet itself.
Traceback error:
Exception Type: ConnectionError at /music/mypersonalapi/
Exception Value: HTTPSConnectionPool(host='URL I have hidden', port=81):
Max retries exceeded with url: /ABC/XYZ/Logon.asmx
(Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x0546E770>:
Failed to establish a new connection:
[WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond',))
Please note: I have removed URL and Host information from my traceback due to confidentiality
What this does:
python -mzeep https://site/url.asmx?wsdl
is:
c = Client("https://site/url.asmx?wsdl")
c.wsdl.dump()
both alternatives are using port 443 since that is the default https port.
From your traceback we see
Exception Value: HTTPSConnectionPool(host='URL I have hidden', port=81):
which would have been similar to
python -mzeep https://site:81/url.asmx?wsdl
I.e. the command line and your code are not connecting to the same address (also note that port values less than 1024 requires system level permissions to use -- in case you're writing/controlling the service too).
The last line does say "..failed because the connected party did not properly respond after a period of time..", but that is not the underlying reason. In line 3 you can read
Max retries exceeded with url: /ABC/XYZ/Logon.asmx
in other words, you've tried (and failed) to log on too many times and the server is probably doubling the time it uses to respond every time you try (a well known mitigation strategy for "things" that fail to login multiple times -- i.e. look like an attack). The extended delay is most likely causing the error message you see at the bottom.
You'll need to wait a while, or reset your account for the service, and if the service is yours then perhaps turn off this feature during development?
Maybe this can help. I had the same connexion problem (Max retries exceeded...). I solved it by increasing the transport timeout.
client = Client(wsdl=wsdl, transport=Transport(session=session, timeout=120))
I'm working on Python 2.7 code to read a value from HTML page by using urllib2 library. I want to timeout the urllib2.urlopen function after 5 seconds in case of no Internet and jump to remaining code.
It works as expected when computer is connected to working internet connection. And for testing if I set timeout=0.1 it timed out suddenly without opening url, as expected. But when there is no Internet, timeout not works either I set timeout to 0.1, 5, or any other value. It simply does not timed out.
This is my Code:
import urllib2
url = "https://alfahd.witorbit.net/fingerprint.php?n"
try:
response = urllib2.urlopen(url , timeout=5).read()
print response
except Exception as e:
print e
Result when connected to Internet with timeout value 5:
180
Result when connected to Internet with timeout value 0.1 :
<urlopen error timed out>
Seems timeout is working.
Result when NOT connected to Internet and with any timeout value (it timed out after about 40 seconds every time I open url despite of any value I set for timeout=:
<urlopen error [Errno -3] Temporary failure in name resolution>
How can I timeout urllib2.urlopen when there is no Internet connectivity? Am I missing some thing? Please guide me to solve this issue. Thanks!
Because name resolution happens before the request is made, it's not subject to the timeout. You can prevent this error in name resolution by providing the IP for the host in your /etc/hosts file. For example, if the host is subdomain.example.com and the IP is 10.10.10.10 you would add the following line in the /etc/hosts file
10.10.10.10 subdomain.example.com
Alternatively, you may be able to simply use the IP address directly, however, some webservers require you use the hostname, in which case you'll need to modify the hosts file to use the name offline.