Develop Reference

Does Spyne (python web service framework) support multiple threads? If not, is there a way to do using the Python Thread Library?

Due to interoperability, I need to deploy an application as a web service. I'm using Spyne (http://spyne.io/) to do that, a python framework for ws. So far so good.
However, the service will receive several requests at once. Hence, I need to increase perfomance, since the request does several I/O (database, file) tasks.
The following code exemplifie a basic web service in Spyne. Based on that, does Spyne support a thread pool or threading? How can I activate or wrap the service in multiple threads? If not possible, how can I achieve that with Python threading library?
Thanks in advance.
from spyne import Application, rpc, ServiceBase, Integer
from spyne.protocol.soap import Soap11
from spyne.server.wsgi import WsgiApplication
class HelloWorldService(ServiceBase):
#rpc(Integer, Integer, _returns=Integer)
def multiply(ctx, a, b):
return a * b
application = Application([HelloWorldService],
tns='spyne.multiply',
in_protocol=Soap11(validator='lxml'),
out_protocol=Soap11()
)
if __name__ == '__main__':
# You can use any Wsgi server. Here, we chose
# Python's built-in wsgi server but you're not
# supposed to use it in production.
from wsgiref.simple_server import make_server
wsgi_app = WsgiApplication(application)
server = make_server('0.0.0.0', 8000, wsgi_app)
server.serve_forever()

Spyne works with both blocking/async code in single-threaded or multithreaded settings.
If you need to do concurrency with multiple threads, use a WSGI server capable of handling multiple threads, like CherryPy, Twisted, mod_wsgi, etc. The WSGI reference implementation that you use in the code sample (wsgiref) does not support concurrency.
If you need to do concurrency with async method calls, use Twisted.
Examples are located at https://github.com/arskom/spyne/tree/master/examples

WSGI compliant component with web socket client?

I'm trying to integrate a RESTful responder in a Crossbar application, for which the best fit seems to be a WSGI service. This service ideally should be part of the rest of the pub/sub infrastructure, being able to receive WAMP events on the one hand and answer HTTP requests on the other.
The difficulty is to run an event loop which allows asynchronous web socket events and additionally offer a WSGI compliant component. It seems to me that Pulsar should be able to do that, but I have not been able to figure out how to set it up, none of the available samples demonstrate exactly this use case.
value = None
class Foo(ApplicationSession):
def onJoin(self, details):
yield self.subscribe(self.bar, 'bar')
def bar(self, data):
value = data
app = Flask(__name__)
#app.route('/')
def baz():
return value
if __name__ == '__main__':
runner = ApplicationRunner('ws://127.0.0.1:8080', 'test')
runner.run(Foo, start_reactor=False)
# now what?
The above demonstrates the two parts, an Autobahn WAMP client and a Flask WSGI component. How do I run both of these in parallel, allowing one thread to receive events both via HTTP and web socket? I don't particularly care about the version of Python nor underlying library (Twisted, asyncio, Pulsar, Flask), I'd just like to get this running somehow.

WSGI is an inherently synchronous API. I don't know about Pulsar, but I would be surprised if it could somehow magically work around this fact.
The way Crossbar.io integrates with classic Web (and synchronous) stacks is via a REST-bridge. Currently, we have the WAMP "Publisher" role covered today (2015/02): that is, you can publish an WAMP event by doing a simple HTTP/POST http://crossbar.io/docs/HTTP-Pusher-Service/. This REST bridge in Crossbar.io will be extended to cover all 4 WAMP roles in the near future.
If you take a step back, and primarily care about something do create a REST API in your app, and which integrates directly with WAMP and asynchronous stuff, I'd have a look a Twisted Klein. Twisted Klein is essentially modeled after Flask, but at the source level. We have a blog post that covers exactly this: Mixing Web and WAMP code with Twisted Klein

HTTP Streaming with Apache mod_wsgi

I've got an ubuntu-server where I am running multiple web-apps.
All of them are hosted by Apache using named VirtualHosts.
One of them is a Flask app, which is running via mod_wsgi.
This app is serving continuous, unlimited HTTP streams.
Does this eventually block my app/server/apache worker, if enough clients are connecting to the streaming endpoint?
And if yes, are there alternatives?
Other non-blocking wsgi-servers that play nicely with VirtualHosts, a different http-streaming paradigm, or some magic apache mod_wsgi settings?
The core of it looks like:
#app.route('/stream')
def get_stream():
def endless():
while True:
yield get_stuff_from_redis()
time.sleep(1)
return Response(endless(), mimetype='application/json')

If the clients never disconnect, yes, you will eventually run out of processes/threads to handle more requests.
You are more than likely better off using a async framework such as Tornado or Twisted for this specific type of application. Doing async programming can be tricky if you aren't used to that concept.
Some people use coroutine system such as gevent/eventlet, but they also have their own problems you have to watch out for.

cherrypy as a gevent-socketio server

I have just started using gevent-socketio and it's great!
But I have been using the default socketioserver and socketio_manage from the chat tutorial and was wondering how to integrate socketio with cherrypy.
essentially, how do I turn this:
class MyNamespace(BaseNamespace):...
def application(environ, start_response):
if environ['PATH_INFO'].startswith('/socket.io'):
return socketio_manage(environ, { '/app': MyNamespace})
else:
return serve_file(environ, start_response)
def serve_file(...):...
sio_server = SocketIOServer(
('', 8080), application,
policy_server=False) sio_server.serve_forever()
into a normal cherrypy server?

Gevent-socketio is based on Gevent, and Gevent's web server. There are two implementations: pywsgi, which is pure python, and wsgi, which uses libevent's http implementation.
See the paragraph starting with "The difference between pywsgi.WSGIServer and wsgi.WSGIServer" over here:
http://www.gevent.org/servers.html
Only those servers are "green", in the sense that they yield the control to the Gevent loop.. so you can only use those servers afaik. The reason for this is that the server is present at the very beginning of the request, and will know how to handle the "Upgrade" and websockets protocol negotiations, and it will pass values inside the "environ" that the next layer (SocketIO) will expect and know how to handle.
You will also need to use the gevent-websocket package.. because it is green (and gevent-socketio is based on that one). You can't just swap the websocket stack.
Hope this helps.

CherryPy doesn't implement the socket.io protocol, nor does it support WebSocket as a built-in. However, there is an extension to CherryPy, called ws4py, that implements only the bare WebSocket protocol on top of its stack. You could start there probably.

Run multiple Python programs on the same server (Ubuntu Server 12.04)

I am creating a webservice in Python and I have a question. I want to separate the user login user data. For this I am creating two different Python programs.
For example:
login.py -> localhost: 8080
userData.py -> localhost: 8081
My question is: how I can run these two programs on the same server? Is there a Python application server that is easy to use?
Thank you very much!

If the webserver is embedded in the application, you may want to use some "watchdog" application to start/stop/restart.
Ubuntu uses upstart.
I like to use supervisord for this as well.
If the application supports some webserver integration protocol like FCGI or WSGI (python's standard), you may want to deploy it using a webserver. I've used apache mod_wsgi for a long time, lately I tend to use nginx+uwsgi. Apache is a fine webserver, but nginx+wsgi scale better.
[update]
Applications use Bottle + PyMongo (MongoDB) What do you recommend to be scalable?
First you should follow the advice on your framework documentation about deployment (bottle is not verbose about this subject, so I understand why you are asking).
B1 comment is right. You definitely want to place the database and application on distinct servers.
For maximum scalability with minimum fuzz you may want to look at some PasS provider like heroku, instructions here. This makes sense specially if you are a developer and not a system administrator.

Since you are on Ubuntu, using bash:
./login.py &
./userData.py &
This will run both scripts in the background.
If you want these scripts to keep on running after you close your shell:
at now < ./login.py
at now < ./userData.py

Tornado is a very easy to use application server. You can listen on different ports with different request handlers.
It is scalable and can handle thousands of connections. We use it to handle our console server. The simple hello world code really tells you all you need to know. I've added another HttpServer so that the single ioloop is handling requests on two different ports :
import tornado.ioloop
import tornado.web
from tornado.httpserver import HttpServer
class MainHandler(tornado.web.RequestHandler):
def get(self):
self.write("Hello, world")
class OtherHandler(tornado.web.RequestHandler):
def get(self):
self.write("Goodbye, world")
application1 = tornado.web.Application([
(r"/", MainHandler),
])
application2 = tornado.web.Application([
(r"/", OtherHandler),
])
if __name__ == "__main__":
HttpServer(application1).listen(8080)
HttpServer(application1).listen(8081)
tornado.ioloop.IOLoop.instance().start()
http://www.tornadoweb.org/