Develop Reference

How to configure apache to support websockets

I wrote server in python and now I would like to configure apache web server to support websockets.
My server returns information when a client sends queries to these addresses:
def make_app():
return tornado.web.Application([
(r"/playgame", EmptyGame),
(r"/playgame/", EmptyGame),
(r"/playgame/(.*)", PlayerGameWebsocket)
])
How to configure the server to support regular user traffic but also to enable websockets when the client establishes such a connection?
I user apache2.4 server.

Ok, it turned out that the solution is trivial. If someone ever looked for an answer, just add a simple redirection to the application in the virtual host configuration which listens on localhost:
ProxyPassMatch "/playgame/(.*)" "ws://127.0.0.1:8888/playgame/$1"
ProxyPassReverse "/playgame/(.*)" "ws://127.0.0.1:8888/playgame/$1"
Thanks to such syntax, we can even pass additional data, e.g. "/playgame/123".
We connect from the client without specifying the port:
var adr = "ws://serverip/playgame/" + gameid;
var ws = new WebSocket(adr);

Tornado web server in webfaction

I am starting with web development. I am trying to develop and webapp using the Instagram API and Django. I was looking that a lot of people it's using Tornado Web Server for Real Time Subscriptions. So I am using Webfaction as a host and found this code so I can wrap my Django project with the "WSGI Container" that Tornado Web Server provides:
import os
import tornado.httpserver
import tornado.ioloop
import tornado.wsgi
import tornado.web
import sys
import django.core.handlers.wsgi
sys.path.append('/path/to/project')
class HelloHandler(tornado.web.RequestHandler):
def get(self):
self.write('Hello from tornado')
def main():
os.environ['DJANGO_SETTINGS_MODULE'] = 'myproject.settings' # path to your settings module
wsgi_app = tornado.wsgi.WSGIContainer(django.core.handlers.wsgi.WSGIHandler())
tornado_app = tornado.web.Application(
[
('/hello-tornado', HelloHandler),
('.*', tornado.web.FallbackHandler, dict(fallback=wsgi_app)),
]
)
http_server = tornado.httpserver.HTTPServer(tornado_app)
http_server.listen(8080)
tornado.ioloop.IOLoop.instance().start()
if __name__ == "__main__":
main()
So I run this python script inside my Webfaction server and everytime I try to access "http://mywebsite.com/hello-tornado/" does not seem to work. I know I am running that Tornado web server on that port but do not know how too access from the browser or something like that. What I am doing wrong here? Thanks for your help and patience. Will cyber high-five for every answer.
EDIT: What I am really trying to do is that I want to receive all the calls from the subscriptions that I make with the Instagram RealTime Subscription API through Tornado, for that I have a callback url "http://mysite.com/sub" and I want to be able to receive through Tornado.

You are starting the server at port 8080, Web browsers use port 80 by default, try using: http://mywebsite.com:8080/hello-tornado
if you want to use port 80 and you already have a web server running in the box you can try following Ali-Akber Saifee suggestion, or run the WSGI application directly from the server, using something like mod_python (http://www.modpython.org), you will lose the ability to run Tornado code, but Django will work.

You have to create a custom app (listening on port), note the port that is assigned to your app then configure tornado to serve on that port: http_server.listen(my port)
You can also avoid tornado and start directly by installing a django app.

Listening for HTTP responses

I am trying to write a python application that will listen for HTTP responses on a socket. I am using http-parser for this. Here is my code:
#!/usr/bin/env python
import socket
from http_parser.http import HttpStream
from http_parser.reader import SocketReader
from http_parser.util import b
def main():
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((socket.gethostname(), 7000))
s.listen(5)
try:
while True:
p = HttpStream(SocketReader(s))
finally:
s.close()
if __name__ == "__main__":
main()
I have two questions:
Is this the best way to do this? Note that I do not want to send a request and then listen for a response. I want to have a redirection mechanism that redirects all responses to the box running this script.
How should I test this? Is there a tool that can mock HTTP responses?
EDIT
What I am trying to do is this: I have three boxes, once runs Apache, one runs this script and one is the client. When the client connects to Apache and it sends back a response, I am diverting the response to this box. So in this script, I am trying to listen for HTTP responses.
Topology
Here is my topology:
Server <----> Switch one <-----> Switch two <-----> Box one and two
Initially, box one connects to the server and sends a request. When the second switch receives responses from the server, it forks it to both box one and box two.

That is a perfectly fine implementation if you really want to operate at the level of TCP sockets. If you want more abstraction, there are lots and lots of HTTP server packages for python, including the standard library's BaseHttpServer, and external libraries/frameworks like tornado and cherrypy.
To test your HTTP listener there are lots of options. You could write full-on HTTP client test code in Python (using an HTTP client library like urllib), or you could:
Point your web browser to http://localhost:7000
telnet to port 7000 on localhost and type in raw HTTP requests.
Automate the above by using nc, e.g.:
echo -e 'GET / HTTP/1.1\n\n' | nc localhost 7000
A note on terminology: What you are listening for on your bound socket is an HTTP request, in the HTTP parlance; what you send back to the connecting client is a response.

1. Is there is a better way?
Yes, there is
2. How should I test this?
common practice is to put a test.py in the same folder, and run python test.py to test. Sample code:
#!python
# -*- coding: utf-8 -*-
import optparse
import urllib2
from django.utils import unittest
# just a sample of settings
HOST = 'localhost'
class GIOPTest(unittest.TestCase):
def test_basic(self):
#TEST GOES HERE
if __name__ == '__main__':
# This part is to accept command line parameters
option_list = ( # check optparse help for more options
make_option("--host",
action = 'store',
type = 'string',
dest = 'host',
default = 'localhost',
help = 'Server host (localhost by default)'
),
)
parser = OptionParser(option_list=option_list)
options, args = parser.parse_args()
HOST = options.host
# run the test
unittest.main()

I ended up using a raw socket and directly reading packets from it.

if you're using apache as the server you're redirecting the data to you could use apache benchmark (ab - you must be superuser though to use it as far as I know) to test it... will also help evaluate the performance impact your application has on the whole thing, example
ab -n 500 -c 20 http://localhost:7000/
where after -n is the total number of connections made to the server during the test and after -c you have the number of concurrent connections apache benchmark will make, experiment with different values of these, also read it's manual, you might find more useful options for whatever your specific purpose might be

How to ssh over HTTP proxy in Python Paramiko?

I am adapting a Python script to be OS independent and run on Windows. I have changed its ssh system calls to calls to paramiko functions. I am stuck with the issue of http proxy authentication. In Unix (actually Cygwin) environment I would use ~/.ssh/config
Host *
ProxyCommand corkscrew http-proxy.example.com 8080 %h %p
Is there a way to obtain the same using paramiko (or the Python ssh module) either using or not using corkscrew? This post seems to suggest that, but I don't know how.
Note: I am behind a firewall that allows me to use only port 80. I need to control Amazon ec2 instances so I configured the sshd server on those machines to listen to port 80. Everything is working fine in my cygwin+corkscrew prototype, but I would like to have a Python script that works without Cygwin.

You can use any pre-established session to paramiko via the sock parameter in SSHClient.connect(hostname,username,password,...,sock).
Below is a code-snippet that tunnels SSH via HTTP-Proxy-Tunnel (HTTP-CONNECT). At first the connection to the proxy is established and the proxy is instructed to connect to localhost:22. The result is a TCP tunnel over the established session that is usually used to tunnel SSL but can be used for any tcp based protocol.
This scenario works with a default installation of tinyproxy with Allow <yourIP> and ConnectPort 22 being set in /etc/tinyproxy.conf. The proxy and the sshd are running on the same host in my example but all you need is any proxy that allows you to CONNECT to your ssh port. Usually this is restricted to port 443 (hint: if you make your sshd listen on 443 this will work with most of the public proxies even thought I do not recommend to do this for interop and security reasons). If this ultimately allows you to bypass your firewall depends on what kind of firewall is employed. If there's no DPI/SSL-Interception features involved, you should be fine. If there's SSL-Interception involved you could still try to tunnel it via ssl or as part of HTTP payload :)
import paramiko
import socket
import logging
logging.basicConfig(loglevel=logging.DEBUG)
LOG = logging.getLogger("xxx")
def http_proxy_tunnel_connect(proxy, target,timeout=None):
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.settimeout(timeout)
sock.connect(proxy)
LOG.debug("connected")
cmd_connect = "CONNECT %s:%d HTTP/1.1\r\n\r\n"%target
LOG.debug("--> %s"%repr(cmd_connect))
sock.sendall(cmd_connect)
response = []
sock.settimeout(2) # quick hack - replace this with something better performing.
try:
# in worst case this loop will take 2 seconds if not response was received (sock.timeout)
while True:
chunk = sock.recv(1024)
if not chunk: # if something goes wrong
break
response.append(chunk)
if "\r\n\r\n" in chunk: # we do not want to read too far ;)
break
except socket.error, se:
if "timed out" not in se:
response=[se]
response = ''.join(response)
LOG.debug("<-- %s"%repr(response))
if not "200 connection established" in response.lower():
raise Exception("Unable to establish HTTP-Tunnel: %s"%repr(response))
return sock
if __name__=="__main__":
LOG.setLevel(logging.DEBUG)
LOG.debug("--start--")
sock = http_proxy_tunnel_connect(proxy=("192.168.139.128",8888),
target=("192.168.139.128",22),
timeout=50)
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(hostname="192.168.139.128",sock=sock, username="xxxx", password="xxxxx")
print "#> whoami \n%s"% ssh.exec_command("whoami")[1].read()
output:
DEBUG:xxx:--start--
DEBUG:xxx:connected
DEBUG:xxx:--> 'CONNECT 192.168.139.128:22 HTTP/1.1\r\n\r\n'
DEBUG:xxx:<-- 'HTTP/1.0 200 Connection established\r\nProxy-agent: tinyproxy/1.8.3\r\n\r\n'
#> whoami
root
here are some other resources on how to tunnel through proxies. Just do whatever is needed to establish your tunnel and pass the socket to SSHClient.connect(...,sock)

There's paraproxy, which implements proxy support for Paramiko.
The post you linked to suggets that Paramiko can operate over an arbitrary socket, but that doesn't appear to be the case. In fact, paraproxy works by completing replacing specific methods inside paramiko, since the existing code simply calls socket.socket() to obtain a socket and does not offer any way of hooking in a proxy.

Google API oauth httpd server conflicts with SABNzbd+ in Python

I'm writing a Python app that interacts with a Google API and requires user authentication via oauth2.
I'm currently setting up a local authentication server to receive an oauth2 authentication code back from Google's oauth server, basically doing the oauth dance like this.
It usually works pretty well, but I guess I'm not understanding exactly how it's interacting with my ports, because it will happily assign my local authentication server to run on port 8080 even if some other app (in the case of my testing, SABNzbd++) is using that port.
I thought assigning the port to a used port number would result in an error and a retry. What am I doing wrong (or, alternatively, what is SABNzbd++ doing that keeps the fact that it's listening on port 8080 hidden from my app?)
The relevant code is as follows.
import socket
import BaseHTTPServer
from oauth2client.tools import ClientRedirectServer, ClientRedirectHandler
port_number = 0
host_name = 'localhost'
for port_number in range(8080,10000):
try:
httpd = ClientRedirectServer((host_name, port_number),
ClientRedirectHandler)
except socket.error, e:
print "socket error: " + str(e)
pass
else:
print "The server is running on: port " + str(port_number)
print "and host_name " + host_name
break
To clarify, the following are my expected results
socket error: [port already in use] (or something like that)
The server is running on: port 8081
and host_name localhost
and then going to localhost:8080 resolves to SABnzbd+, and localhost:8081 resolves to my authentication server.
I'm getting, howver:
the server is running on: port 8080
and host_name localhost
but going to localhost:8080 resolves to SABNzbd+
Thanks in advance!