I am trying to write a python application that will listen for HTTP responses on a socket. I am using http-parser for this. Here is my code:
#!/usr/bin/env python
import socket
from http_parser.http import HttpStream
from http_parser.reader import SocketReader
from http_parser.util import b
def main():
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((socket.gethostname(), 7000))
s.listen(5)
try:
while True:
p = HttpStream(SocketReader(s))
finally:
s.close()
if __name__ == "__main__":
main()
I have two questions:
Is this the best way to do this? Note that I do not want to send a request and then listen for a response. I want to have a redirection mechanism that redirects all responses to the box running this script.
How should I test this? Is there a tool that can mock HTTP responses?
EDIT
What I am trying to do is this: I have three boxes, once runs Apache, one runs this script and one is the client. When the client connects to Apache and it sends back a response, I am diverting the response to this box. So in this script, I am trying to listen for HTTP responses.
Topology
Here is my topology:
Server <----> Switch one <-----> Switch two <-----> Box one and two
Initially, box one connects to the server and sends a request. When the second switch receives responses from the server, it forks it to both box one and box two.
That is a perfectly fine implementation if you really want to operate at the level of TCP sockets. If you want more abstraction, there are lots and lots of HTTP server packages for python, including the standard library's BaseHttpServer, and external libraries/frameworks like tornado and cherrypy.
To test your HTTP listener there are lots of options. You could write full-on HTTP client test code in Python (using an HTTP client library like urllib), or you could:
Point your web browser to http://localhost:7000
telnet to port 7000 on localhost and type in raw HTTP requests.
Automate the above by using nc, e.g.:
echo -e 'GET / HTTP/1.1\n\n' | nc localhost 7000
A note on terminology: What you are listening for on your bound socket is an HTTP request, in the HTTP parlance; what you send back to the connecting client is a response.
1. Is there is a better way?
Yes, there is
2. How should I test this?
common practice is to put a test.py in the same folder, and run python test.py to test. Sample code:
#!python
# -*- coding: utf-8 -*-
import optparse
import urllib2
from django.utils import unittest
# just a sample of settings
HOST = 'localhost'
class GIOPTest(unittest.TestCase):
def test_basic(self):
#TEST GOES HERE
if __name__ == '__main__':
# This part is to accept command line parameters
option_list = ( # check optparse help for more options
make_option("--host",
action = 'store',
type = 'string',
dest = 'host',
default = 'localhost',
help = 'Server host (localhost by default)'
),
)
parser = OptionParser(option_list=option_list)
options, args = parser.parse_args()
HOST = options.host
# run the test
unittest.main()
I ended up using a raw socket and directly reading packets from it.
if you're using apache as the server you're redirecting the data to you could use apache benchmark (ab - you must be superuser though to use it as far as I know) to test it... will also help evaluate the performance impact your application has on the whole thing, example
ab -n 500 -c 20 http://localhost:7000/
where after -n is the total number of connections made to the server during the test and after -c you have the number of concurrent connections apache benchmark will make, experiment with different values of these, also read it's manual, you might find more useful options for whatever your specific purpose might be
Related
I have a simple flask server defined like so:
import sys
import flask
from flask import request
app = flask.Flask(__name__)
port = 4057
#app.route('/search', methods=['POST'])
def search():
request.json['query']
results = ['fake', 'data']
return flask.jsonify(results)
if __name__ == '__main__':
app.config['TEMPLATES_AUTO_RELOAD'] = True
app.run(host='0.0.0.0', port=port, debug=(port != 80))
I have a simple client defined like this:
import json
import requests
headers = {'content-type': 'application/json'}
resp = requests.post('http://localhost:4057/search', json.dumps({'query': 'foo'}), headers=headers)
print resp.content
The client works, but it takes like 3 seconds to complete the request.
curl completes in like half a second:
curl 'http://localhost:4057/search' -H 'Content-Type: application/json' -d '{"query": "foo"}'
Try 127.0.0.1 There maybe some odd name resolution rules fumbling requests.
Ah ok, this is my work machine. I took a look at /etc/hosts and saw ~200 routes defined that I didn't realize were there
As mention in the comments, this does not explain the odd behavior replicated using curl.
I've recently encountered a similar issue with slow connections from requests to 'localhost', and also found that using '127.0.0.1' was much faster.
After some investigation, I found that in my case, the slow connections on Windows were related to urllib3 (the transport library used by requests) attempting an IPv6 connection first, which is refused since the server is listening on IPv4 only. For Windows only, however, there is an unavoidable 1 sec timeout on socket.connect for refused TCP connections, while the OS 'helpfully' retries the connection up to 3 times. (See: Why do failed attempts of Socket.connect take 1 sec on Windows?).
On Linux, socket.connect fails immediately if the connection is refused, and it can attempt the IPv4 connection without any delay. In addition, some software, like curl, support limiting name resolution to only IPv4 or IPv6. Unfortunately, requests and urllib3 do not support this, and do not seem to plan to support this anytime soon (See: https://github.com/psf/requests/issues/1691)
For those trying to figure this out as no one gave a clear solution here:
I encountered this issue a while ago, and noticed that the Flask test server is not concurrent. The Python Requests takes FOREVER to retrieve the data from your flask app unless you make it concurrent. Enabling "Threaded" in your app.run option will do the trick!
app.run(port=5000, debug=True, threaded=True)
I have used this link and successfully run a python script using uWSGI. Although i just followed the doc line by line.
I have a GPS device which is sending data to a remote server. Document of the same device say that it connect to server using TCP which therefore would be http as simple device like a GPS device would not be able to do https (i hope i am right here.) Now as i have configure my Nginx server to forward all incoming HTTP request to python script for processing via uWSGI.
What i want to do is to simply print the url or query string on the HTML page. As i don't have control on the device side (i can only configure device to send data on a IP + Port), i have no clue how data is coming. Below is my access log
[23/Jan/2016:01:50:32 +0530] "(009591810720BP05000009591810720160122A1254.6449N07738.5244E000.0202007129.7200000000L00000008)" 400 172 "-" "-" "-"
Now i have look at this link on how to get the url parameters values, but i don't have a clue that what is the parameter here.
I tried to modified my wsgi.py file as
import requests
r = requests.get("http://localhost.com/")
# or r = requests.get("http://localhost.com/?") as i am directly routing incoming http request to python script and incoming HTTP request might not have #any parameter, just data #
text1 = r.status_code
def application(environ, start_response):
start_response('200 OK', [('Content-Type', 'text/html')])
return ["<h1 style='color:blue'>Hello There shailendra! %s </h1>" %(text1)]
but when i restarted nginx, i get internal server error. Can some one help me to understand wrong i am doing here (literally i have no clue about the parameters of the application function. Tried to read this link, but what i get from here is that environ argument take care of many CGI environment variables.)
Can some one please help me to figure out what wrong am i doing and guide me to even a doc or resource.
Thanks.
why are you using localhost ".com" ?
Since you are running the webserver on the same machine,
you should change the line to
r = requests.get("http://localhost/")
Also move below lines from wsgi.py and put them in testServerConnection.py
import requests
r = requests.get("http://localhost/")
# or r = requests.get("http://localhost.com/?") as i am directly routing incoming http request to python script and incoming HTTP request might not have #any parameter, just data #
text1 = r.status_code
Start NGINX
and you also might have to run (i am not sure uwsgi set up on nginx)
uwsgi --socket 0.0.0.0:8080 --protocol=http -w wsgi
run testConnection.py to send a test request to localhost webserver and print the response
i got the answer for my question. Basically to process a TCP request, you need to open a socket and accept the TCP request on a specific port (as you specified on the hardware)
import SocketServer
class MyTCPHandler(SocketServer.BaseRequestHandler):
def handle(self):
# self.request is the TCP socket connected to the client
self.data = self.request.recv(1024).strip()
print "{} wrote:".format(self.client_address[0])
#data which is received
print self.data
if __name__ == "__main__":
#replace IP by your server IP
HOST, PORT = <IP of the server>, 8000
# Create the server, binding to localhost on port 9999
server = SocketServer.TCPServer((HOST, PORT), MyTCPHandler)
# Activate the server; this will keep running until you
# interrupt the program with Ctrl-C
server.serve_forever()
After you get the data, you can do any thing with the data. As my data format was shared in the GPS datasheet, i was able to parse the string and get the Lat and long out of it.
Can I create a HTTP server without using
python -m http.server [port number]
Using an old school style with sockets and such.
Latest code and errors...
import socketserver
response = """HTTP/1.0 500 Internal Server Error
Content-type: text/html
Invalid Server Error"""
class MyTCPHandler(socketserver.BaseRequestHandler):
"""
The RequestHandler class for our server.
It is instantiated once per connection to the server, and must
override the handle() method to implement communication to the
client.
"""
def handle(self):
# self.request is the TCP socket connected to the client
self.data = self.request.recv(1024).strip()
self.request.sendall(response)
if __name__ == "__main__":
HOST, PORT = "localhost", 8000
server = socketserver.TCPServer((HOST, PORT), MyTCPHandler)
server.serve_forever()
TypeError: 'str' does not support the buffer interface
Yes, you can, but it's a terrible idea -- in fact, even http.server is at best a toy implementation.
You're better off writing whatever webapp you want as a standard WSGI application (most Python web frameworks do that -- Django, Pyramid, Flask...), and serving it with one of the dozens of production-grade HTTP servers that exist for Python.
uWSGI (https://uwsgi-docs.readthedocs.org/en/latest/) is my personal favorite, with Gevent a close second.
If you want more info about how it's done, I recommend that you read the source code to the CherryPy server (http://www.cherrypy.org/). While not as powerful as the aforementioned uWSGI, it's a good reference implementation written in pure Python, that serves WSGI apps through a thread pool.
Sure you can, and servers like Tornado already do it this way.
For simple test servers which can do only HTTP/1.0 GET requests and handle only a single request at a time it should not be that hard once you understood the basics of the HTTP protocol. But if you care even a bit about performance it gets complex fast.
I'm new to using Cloud9 IDE (c9) and so far it looks great, except a few minor things.
I see from the docs that to start up a simple node.js http server, you have to pass in process.env.PORT in place of the regular port such as "8080".
Node Hello World example:
var http = require('http');
http.createServer(function (req, res) {
res.writeHead(200, {'Content-Type': 'text/plain'});
res.end('Hello World\n');
}).listen(process.env.PORT, process.env.IP);
What I want to know is, on c9, can you only start services on ports using javascript / node.js? Or do other languages work just as well, perhaps with some other method of passing the port? Specifically python + Twisted?
I uploaded some twisted code that was working locally for me, but wouldn't work on c9 because it was trying to access local ports (which are already in use). Here is the error
twisted.internet.error.CannotListenError: Couldn't listen on any:8080: [Errno 98] Address already in use.
How would one make the following example work, if even possible, running on c9?
Python+Twisted Hello World example
from twisted.web import server, resource
from twisted.internet import reactor
class Simple(resource.Resource):
isLeaf = True
def render_GET(self, request):
return "<html>Hello, world!</html>"
site = server.Site(Simple())
reactor.listenTCP(8080, site)
reactor.run()
Initial searches through the documentation and github issues did not turn much up. I'm hoping this is possible and I just missed the right parameter to pass.
Edit: Updated output below
Node Code
console.log(process.env.PORT)
console.log(process.env.IP)
Terminal output
Running Node Process
Tip: you can access long running processes, like a server, at 'http://private-cloud.mrchampe.c9.io'.
Important: in your scripts, use 'process.env.PORT' as port and 'process.env.IP' as host.
8080
127.6.70.129
Python Code
import os
print os.environ["PORT"]
print os.environ["IP"]
Terminal output
Running Python Process
8080
127.6.70.129
Twisted code
import os
import twisted
from twisted.web import server, resource
from twisted.internet import reactor
class Simple(resource.Resource):
isLeaf = True
def render_GET(self, request):
return "<html>Hello, world!</html>"
site = server.Site(Simple())
reactor.listenTCP(int(os.environ["PORT"]), interface=os.environ["IP"])
reactor.run()
Terminal Output
Running Python Process
hello world
Traceback (most recent call last):
File "python/hello.py", line 17, in <module>
reactor.listenTCP(int(os.environ["PORT"]), interface=os.environ["IP"])
TypeError: listenTCP() takes at least 3 non-keyword arguments (2 given)
The listenTCP TypeError is strange because 2 arguments works locally but not on Cloud9. I don't see why using these arguments does not work.
I have the above code hosted on this public Cloud9 project for anyone to take a look. Thanks!
process.env.PORT and process.env.IP from Node.js sound like os.environ["PORT"] and os.environ["IP"] in Python. Perhaps you can try:
reactor.listenTCP(int(os.environ["PORT"]), site, interface=os.environ["IP"])
Probably this is limitation of c9 environment management, so users dont abuse their service too much. I would assume that they have some level of management for Node.js used resources and thus allow these to open ports.
If this was the case and I had to work with cloud9, I would probably approach this as follows:
- create Node.js service which would act as a proxy, listening on twisted behalf
- create new reactor with overriden listenTCP and listenUDP methods, which would bind these to Node.js proxy service.
The way how proxy would work is as follows, Node.js would initially listen on one "management" TCP port. Then, when twisted service starts up, it would create a TCP connection between Node.js and itself through that port. Then, whenever listenTCP or listenUDP is called, these commands would then be dispatched to Node.js service, which in return would open the port and all of the comunication through that port would then be proxied to twisted through that existing TCP connection.
It's worth mentioning that Jean-Paul's answer worked for me as well, but I had to use 'address' instead of 'interface':
http_server.listen(int(os.environ.get("PORT")), address=os.environ["IP"])
I want a simple python web server for the following use case:
I want to write a simple server that will accept HTTP requests from my application running on Google App Engine.
The server will accept HTTP requests, and then send iphone notifications. (Basically, I need this extra server to account for the lack of socket support in google app engine).
I guess I need the server to be able to maintain this persistent connection with Apple's Push Notification Service. So I'll need to have some sort of thread always open for this. So I need some sort of web server that can accept the request pass it off to the other thread with the persistent connection to APNS.
Maybe multiple processes and one of pythons queuing tools to communicate between them? Accept the HTTP request, then enqueue a message to the other process?
I was wondering what someone with a bit of experience would suggest. I'm starting to think that maybe even writing my own simple server is a good option (http://fragments.turtlemeat.com/pythonwebserver.php).
One option would be the (appropriately named) SimpleHTTPServer, which is part of the Python standard library. Another, more flexible but more complicated option would be to write your server in Twisted.
I've been writing simple http servers using gevent and bottle -- an example:
#!/usr/bin/env python
import gevent.monkey
gevent.monkey.patch_all()
import bottle
bottle.debug(True)
import gevent.wsgi
from bottle import route, run, request, response, static_file, abort
#route('/echo')
def echo():
s = request.GET.get('s', 'o hai')
return '<html><head><title>echo server</title></head><body>%s</body></html>\r\n' % (s)
#route('/static/:filename')
def send_static(filename):
root = os.getcwd() + '/static'
return static_file(filename, root=root)
if __name__ == '__main__':
app = bottle.app()
wsgi_server = gevent.wsgi.WSGIServer(('0.0.0.0', 8000), app)
print 'Starting wsgi search on port 8000'
wsgi_server.serve_forever()
So you could write a simple server that sticks a job into a Queue (see gevent.queue) and have another worker greenlet that handles reading requests from the queue and processing them...