Develop Reference

basehttpserver does not respond to other requests

When I open my local server on Android (192.168.1.4) and on pc at the same time, pc never shows page (it is loading and loading and loading...) - this error raises when I kill my server:
Exception happened during processing of request from ('192.168.1.4', 54734)
Traceback (most recent call last):
File "/usr/lib/python2.7/SocketServer.py", line 295, in _handle_request_noblock
self.process_request(request, client_address)
File "/usr/lib/python2.7/SocketServer.py", line 321, in process_request
self.finish_request(request, client_address)
File "/usr/lib/python2.7/SocketServer.py", line 334, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib/python2.7/SocketServer.py", line 655, in __init__
self.handle()
File "/usr/lib/python2.7/BaseHTTPServer.py", line 340, in handle
self.handle_one_request()
File "/usr/lib/python2.7/BaseHTTPServer.py", line 310, in handle_one_request
self.raw_requestline = self.rfile.readline(65537)
File "/usr/lib/python2.7/socket.py", line 476, in readline
data = self._sock.recv(self._rbufsize)
KeyboardInterrupt
my server script:
# !/usr/bin/env python
# -*- coding: utf-8 -*-
from BaseHTTPServer import BaseHTTPRequestHandler, HTTPServer
PORT = 20000
class S(BaseHTTPRequestHandler):
def log_message(self, format, *args):
return
def _set_headers(self):
self.send_response(200)
self.send_header('Content-type', 'text/html')
self.send_header('Access-Control-Allow-Origin', '*')
self.end_headers()
def do_GET(self):
self._set_headers()
if self.path == "/other":
self.wfile.write("other")
if self.path == "/something":
self.wfile.write('something')
def do_HEAD(self):
self._set_headers()
def do_POST(self):
self._set_headers()
self.wfile.write("hello post")
def run(server_class=HTTPServer, handler_class=S, port=PORT):
server_address = ('', port)
httpd = server_class(server_address, handler_class)
httpd.serve_forever()
if __name__ == "__main__":
from sys import argv
if len(argv) == 2:
run(port=int(argv[1]))
else:
run()
What is wrong with my code please?

I think your server works well: like any server, it runs indefinitely.
To see it working, just point your browser on the following URL: http://localhost:20000/something or http://127.0.0.1:20000/something.
You should get the text "something".
You should consider using Flask.

No idea, if this is correct approach but setting timeout helped:
...
def log_message(self, format, *args):
return
def setup(self):
BaseHTTPRequestHandler.setup(self)
self.request.settimeout(0.5)
...

https://pymotw.com/2/BaseHTTPServer/#threading-and-forking says:
HTTPServer is a simple subclass of SocketServer.TCPServer, and does not use multiple threads or processes to handle requests. To add threading or forking, create a new class using the appropriate mix-in from SocketServer.

Python HTTP server not responding on POST request

I would like to make a HTTP server using python, to host my website. I've implemented do_GET() method and it works fine, but when I try the POST method, server is not responding until I reload the page. Then it wakes up, tring to complete the POST, but realizing that the connection has ended. After that it serves the GET method (because I reloaded the page) and continue running normally.
I'm using jQuery on client side, but i also tried html form. It's no change even if i open the page in another browser, so it must be an exception on server side.
Here is the client javascript:
function Send()
{
jQuery.post("address", "Blah!", function (data) {
//get response
alert(data);
});
}
And here is the server code (I'm using python 3.3.4):
class MyHTTPRequestHandler(http.server.BaseHTTPRequestHandler):
def do_POST(self):
print("Data: " + str(self.rfile.read(), "utf-8")) #show request
response = bytes("This is the response.", "utf-8") #create response
self.send_response(200) #create header
self.send_header("Content-Length", str(len(response)))
self.end_headers()
self.wfile.write(response) #send response
This is the log from server:
10.175.72.200 - - [01/Apr/2014 19:11:26] "GET / HTTP/1.1" 200 -
Data: Blah!
10.175.72.200 - - [01/Apr/2014 19:11:47] "POST /address HTTP/1.1" 200 -
----------------------------------------
Exception happened during processing of request from ('10.175.72.200', 2237)
Traceback (most recent call last):
File "C:\Python33\lib\socketserver.py", line 306, in _handle_request_noblock
self.process_request(request, client_address)
File "C:\Python33\lib\socketserver.py", line 332, in process_request
self.finish_request(request, client_address)
File "C:\Python33\lib\socketserver.py", line 345, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "C:\Python33\lib\socketserver.py", line 666, in __init__
self.handle()
File "C:\Python33\lib\http\server.py", line 400, in handle
self.handle_one_request()
File "C:\Python33\lib\http\server.py", line 388, in handle_one_request
method()
File "C:\Users\Jirek\Desktop\Server\server.py", line 45, in do_POST
self.wfile.write(response) #send response
File "C:\Python33\lib\socket.py", line 317, in write
return self._sock.send(b)
ConnectionAbortedError: [WinError 10053] An established connection was aborted by the software in your host machine
----------------------------------------
10.175.72.200 - - [01/Apr/2014 19:11:47] "GET / HTTP/1.1" 200 -
You can see, on the first line is classical GET request. Then I call the POST from browser and nothing happens until the reload. At that point the server writes the rest of the log.

Problem solved! It's in the line where the code loads received data (in do_POST()). I just had to specify the amount of data to read.
So, just don't use rfile.read(), but rfile.read(numberOfBytes). How big is the POST message is saved in headers: numberOfBytes = int(self.headers["Content-Length"])
The repaired method:
def do_POST(self):
length = int(self.headers["Content-Length"])
print("Data: " + str(self.rfile.read(length), "utf-8"))
response = bytes("This is the response.", "utf-8") #create response
self.send_response(200) #create header
self.send_header("Content-Length", str(len(response)))
self.end_headers()
self.wfile.write(response) #send response

Apache bench and python BaseHTTPServer broken pipe exception

I have a simple HTTP server implemented in python as follows:
from BaseHTTPServer import BaseHTTPRequestHandler, HTTPServer
class Handler(BaseHTTPRequestHandler):
def do_GET(self):
self.send_response(200)
self.send_header('Content-Type', 'text/html')
self.end_headers()
self.wfile.write('Hello World!')
if __name__ == '__main__':
try:
server = HTTPServer(('', 8000), Handler)
print 'Listening on 0.0.0.0:8000'
server.serve_forever()
except KeyboardInterrupt:
print '^C, halting'
server.socket.close()
Then I run apache bench with concurrency level of 50:
ab -v 3 -c 50 -n 50000 http://localhost:8000/
I started getting this error in apache bench:
apr_socket_recv: Connection reset by peer (104)
and on the HTTP server console I see this:
Exception happened during processing of request from ('127.0.0.1', 51545)
Traceback (most recent call last):
File "/usr/lib/python2.6/SocketServer.py", line 281, in _handle_request_noblock
self.process_request(request, client_address)
File "/usr/lib/python2.6/SocketServer.py", line 307, in process_request
self.finish_request(request, client_address)
File "/usr/lib/python2.6/SocketServer.py", line 320, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib/python2.6/SocketServer.py", line 615, in __init__
self.handle()
File "/usr/lib/python2.6/BaseHTTPServer.py", line 329, in handle
self.handle_one_request()
File "/usr/lib/python2.6/BaseHTTPServer.py", line 323, in handle_one_request
method()
File "./test.py", line 12, in do_GET
self.send_response(200)
File "/usr/lib/python2.6/BaseHTTPServer.py", line 384, in send_response
self.send_header('Server', self.version_string())
File "/usr/lib/python2.6/BaseHTTPServer.py", line 390, in send_header
self.wfile.write("%s: %s\r\n" % (keyword, value))
File "/usr/lib/python2.6/socket.py", line 300, in write
self.flush()
File "/usr/lib/python2.6/socket.py", line 286, in flush
self._sock.sendall(buffer)
error: [Errno 32] Broken pipe
Also I notice that the HTTPServer gets real sluggish
I have tried to increase the request_queue_size of HTTPServer by doing this:
from BaseHTTPServer import BaseHTTPRequestHandler, HTTPServer
class MyServer(HTTPServer):
request_queue_size = 100
timeout = None
class Handler(BaseHTTPRequestHandler):
counter = 0
def do_GET(self):
self.send_response(200)
self.send_header('Content-Type', 'text/html')
self.end_headers()
self.wfile.write(str(self.counter))
Handler.counter += 1
if __name__ == '__main__':
try:
server = MyServer(('', 8000), Handler)
print 'Listening on 0.0.0.0:8000'
server.serve_forever()
except KeyboardInterrupt:
print '^C, halting'
server.socket.close()
but I still get the broken pipe error at the end of the test run once in a while. If I increase request_queue_size to a huge number then the error goes away. I don't understand why because the Apache bench -c value is 50 and request_queue_size of 100 should be sufficient. Why is it that the request_queue_size needs to be much bigger than the Apache bench concurrency setting? Does Apache bench fire off more connections even if the previous one hasn't finished and hence the total number of concurrent connections can be more than the -c setting?

This is a problem with apachebench, but not with httperf. I'm not really sure what the difference in they way they handle the connection is.
Increase from default(5) to 1000. I haven't noticed any huge blowups in memory and performance has increased dramatically, while eliminating errors.

HTTP server hangs while accepting packets

I have written a simple http server to handle POST requests:
class MyHandler( BaseHTTPServer.BaseHTTPRequestHandler):
def do_POST( self ):
ctype, pdict = cgi.parse_header(self.headers.getheader('content-type'))
postvars = {}
try:
if ctype == 'application/x-www-form-urlencoded':
length = int(self.headers.getheader('content-length'))
postvars = cgi.parse_qs(self.rfile.read(length), keep_blank_values=1)
self.send_response( 200 )
self.send_header( "Content-type", "text")
self.send_header( "Content-length", str(len(body)) )
self.end_headers()
self.wfile.write(body)
except:
print "Error"
def httpd(handler_class=MyHandler, server_address = ('2.3.4.5', 80)):
try:
print "Server started"
srvr = BaseHTTPServer.HTTPServer(server_address, handler_class)
srvr.serve_forever() # serve_forever
except KeyboardInterrupt:
server.socket.close()
if __name__ == "__main__":
httpd( )
The server runs fine but sometimes it just hangs. When I press CTRL+C it gives the following error and then continues receiving data:
Exception happened during processing of request from ('1.1.1.2', 50928)
Traceback (most recent call last):
File "/usr/lib/python2.6/SocketServer.py", line 281, in _handle_request_noblock
self.process_request(request, client_address)
File "/usr/lib/python2.6/SocketServer.py", line 307, in process_request
self.finish_request(request, client_address)
File "/usr/lib/python2.6/SocketServer.py", line 320, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib/python2.6/SocketServer.py", line 615, in __init__
self.handle()
File "/usr/lib/python2.6/BaseHTTPServer.py", line 329, in handle
self.handle_one_request()
File "/usr/lib/python2.6/BaseHTTPServer.py", line 312, in handle_one_request
self.raw_requestline = self.rfile.readline()
File "/usr/lib/python2.6/socket.py", line 406, in readline
data = self._sock.recv(self._rbufsize)
KeyboardInterrupt
Can someone tell me how to correct this? I can't make sense of the errors.

I completely rewrote my solution in order to fix two disadvantages:
It can treat timeout even if client has opened only a connection but did not start communicate.
If the client opens a connection, a new server process is forked. A slow client can not block other.
It is based on your code as most as possible. It is now a complete independent demo. If you open a browser on http://localhost:8000/ and write 'simulate error' to form and press submit, it simulates runtime error.
from http.server import HTTPServer, BaseHTTPRequestHandler
from socketserver import ForkingMixIn
from urllib.parse import parse_qs
class MyHandler(BaseHTTPRequestHandler):
def do_POST(self):
ctype, pdict = self.headers.get_params(header='content-type')[0]
postvars = {}
try:
if ctype == 'application/x-www-form-urlencoded':
length = int(self.headers['content-length'])
postvars = parse_qs(self.rfile.read(length), keep_blank_values=1
encoding='utf-8')
assert postvars.get('foo', '') != ['bar'] # can simulate error
body = 'Something\n'.encode('ascii')
self.send_response(200)
self.send_header("Content-type", "text")
self.send_header("Content-length", str(len(body)))
self.end_headers()
self.wfile.write(body)
except Exception:
self.send_error(500)
raise
def do_GET(self):
# demo for testing POST by web browser - without valid html
body = ('Something\n<form method="post" action="http://%s:%d/">\n'
'<input name="foo" type="text"><input type="submit"></form>\n'
% self.server.server_address).encode('ascii')
self.send_response(200)
self.send_header("Content-type", "text/html")
self.send_header("Content-length", str(len(body)))
self.end_headers()
self.wfile.write(body)
class ForkingHTTPServer(ForkingMixIn, HTTPServer):
def finish_request(self, request, client_address):
request.settimeout(30)
# "super" can not be used because BaseServer is not created from object
HTTPServer.finish_request(self, request, client_address)
def httpd(handler_class=MyHandler, server_address=('localhost', 8000)):
try:
print("Server started")
srvr = ForkingHTTPServer(server_address, handler_class)
srvr.serve_forever() # serve_forever
except KeyboardInterrupt:
srvr.socket.close()
if __name__ == "__main__":
httpd()
Forking can be disabled for e.g. debugging purposes by removing class SocketServer.ForkingMixIn from code.
EDIT Updated for Python 3, but a warning from Python 3 docs should be noted:
Warning: http.server is not recommended for production. It only implements basic security checks.

Python SSLError: Client-side-error(EOF occurred in violation of protocol), Server-side-error(SSL3_GET_RECORD:wrong version number)

I'm having some difficulty attempting to create an SSL socket in Python to use a proxy that requires authentication. I am very sorry for the length, but I felt it was best to include as much detail as possible.
First, the server code looks like this:
class ThreadedTCPServer(SocketServer.ThreadingMixIn, SocketServer.TCPServer):
def __init__(self, server_address, RequestHandlerClass, client_manager, recv_queue):
SocketServer.TCPServer.__init__(self, server_address, RequestHandlerClass, bind_and_activate=True)
<snipped out extra code>
class ThreadedTCPRequestHandler(SocketServer.BaseRequestHandler):
def setup(self):
while True:
try:
print 'trying to wrap in ssl'
self.request = ssl.wrap_socket(self.request,
certfile=(os.getcwd() + '/ssl_certs/newcert.pem'),
keyfile=(os.getcwd() + '/ssl_certs/webserver.nopass.key'),
server_side=True,
cert_reqs=ssl.CERT_NONE,
ssl_version=ssl.PROTOCOL_TLSv1,
do_handshake_on_connect=False,
suppress_ragged_eofs=True)
break
except Exception, ex:
print 'error trying to wrap in ssl %s' % ex
def handle(self):
# Display message that client has connected
print '\r[*] Received connection from %s:%s\r' % (self.client_address[0], self.client_address[1])
while self.stopped() == False:
recv_msg = self.request.read(1024)
if recv_msg == '':
self.stop.set()
server.recv_queue.put(recv_msg)
break
else:
server.recv_queue.put(recv_msg)
if self.stopped():
print '[!] Received STOP signal from %s:%s; Exiting!' % (self.client_address[0], self.client_address[1])
Second, this is the client code where I set up the information needed to connect via the proxy that requires authentication:
class proxyCommsHandler():
def __init__(self, user, password, remote_host, remote_port, list_of_proxies):
# information needed to connect
self.user = 'username'
self.passwd = 'password'
self.remote_host = 'remote_host_ip'
self.remote_port = 8008
self.list_of_proxies = [['proxyserver.hostname.com', 8080]]
# setup basic authentication to send to the proxy when we try to connect
self.user_pass = base64.encodestring(self.user + ':' + self.passwd)
self.proxy_authorization = 'Proxy-authorization: Basic ' + self.user_pass + '\r\n'
self.proxy_connect = 'CONNECT %s:%s HTTP/1.1\r\n' % (self.remote_host, self.remote_port)
self.user_agent = "User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:7.0.1) Gecko/20100101 Firefox/7.0.1\r\n"
self.proxy_pieces = self.proxy_connect + self.proxy_authorization + self.user_agent + '\r\n'
Now, here's where I initially connect to the proxy, where I get no errors (I get a '200' status code):
self.proxy = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.proxy.connect( (proxy_host, proxy_port) )
self.proxy.sendall(self.proxy_pieces)
self.response = proxy.recv(1024)
Here's where the client fails (I think). I try to take self.proxy and wrap it in SSL, like this:
sslsock = ssl.wrap_socket(self.proxy, server_side=False, do_handshake_on_connect=True,
ssl_version=ssl.PROTOCOL_TLSv1)
This is the error that I see on the client:
Traceback (most recent call last):
File "C:\Python27\pyrevshell.py", line 467, in <module>
proxyCommsHandler(None, None, None, None, list_of_proxies).run()
File "C:\Python27\pyrevshell.py", line 300, in run
ssl_version=ssl.PROTOCOL_TLSv1)
File "C:\Python27\lib\ssl.py", line 372, in wrap_socket
ciphers=ciphers)
File "C:\Python27\lib\ssl.py", line 134, in __init__
self.do_handshake()
File "C:\Python27\lib\ssl.py", line 296, in do_handshake
self._sslobj.do_handshake()
SSLError: [Errno 8] _ssl.c:503: EOF occurred in violation of protocol
The client does connect, like shown from the output here:
trying to wrap in ssl
[*] Received connection from x.x.x.x:47144
[*] x.x.x.x:47144 added to the client list
But then it's immediately followed by an exception:
----------------------------------------
Exception happened during processing of request from ('x.x.x.x', 47144)
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 582, in process_request_thread
self.finish_request(request, client_address)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 323, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 639, in __init__
self.handle()
File "shell_server.py", line 324, in handle
recv_msg = self.request.read(1024)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py", line 138, in read
return self._sslobj.read(len)
SSLError: [Errno 1] _ssl.c:1348: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number
----------------------------------------
While I realize that this sounds like an obvious problem based on the Exceptions that were thrown, the interesting parts are that:
I successfully connect initially via the proxy as shown above
I can successfully connect with a web browser behind behind the same proxy, and no exceptions are thrown; I can pass data to the browser
I've tried different SSL Protocol versions on both the server and client side, as shown in the table here in the Python documentation; it errors on the client-side each time
I've used Wireshark on both ends of the connection. While using a normal browser and connecting to the server, I see the entire SSL handshake and negotiation process, and everything runs smoothly.
However, when I use the client shown above, as soon as I connect with it, I see the client send a Client Hello message, but then my server sends a RST packet to kill the connection (I haven't determined if this is before or after the Exception is thrown).
Again, I apologize for the length, but I am in dire need of expert advice.

I've figured out the issue to my problem. I am sending the self.user_agent to the remote host when I connect via the proxy for the first time, which interferes with the SSL Handshake.
To solve this, I put an initial self.request.recv() in the def setup(self) function before I call ssl.wrap_socket on the socket.