Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
Questions asking us to recommend or find a tool, library or favorite off-site resource are off-topic for Stack Overflow as they tend to attract opinionated answers and spam. Instead, describe the problem and what has been done so far to solve it.
Closed 9 years ago.
Improve this question
At work I'm not allowed to use perl for web services. Python is allowed however.
What I need to do is serve up the results of some very slow c++ binaries. Each exe takes up to 20 seconds to run. In perl I'd just use mojolicious's non blocking event loop ( an example.of which is given here. http://blogs.perl.org/users/joel_berger/2014/01/writing-non-blocking-applications-with-mojolicious-part-3.html )
How would one go about doing this with django and python?
Tornado using non blocking IO , the concepts are the same as in perl or node js event loop, multiple tasks per thread and so on.
Probably won't be possible with Django, as the entire framework will need to be built specifically for running inside an event loop. In an event-driven framework, slow operations (I/O for example) needs to be implemented using callbacks, so that the actual I/O can be offloaded to the event loop itself, and the callback only called when the operation has finished; Django is not implemented like this.
Take a look at Twisted — it is an event-driven networking engine for Python that also has some web application frameworks built on top of it.
Take a look at A clean, lightweight alternative to Python's twisted. I'd choose gevent for a web app, as it runs with uWSGI--the most versatile web server to run Python code.
Related
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
Questions asking us to recommend or find a tool, library or favorite off-site resource are off-topic for Stack Overflow as they tend to attract opinionated answers and spam. Instead, describe the problem and what has been done so far to solve it.
Closed 9 years ago.
Improve this question
My application polls an API repeatedly and spawns processes to parse any new data resulting from these calls, conditionally making an API request based on those data. The speed of that turnaround time is critical.
A large bottleneck seems to be related to the setup of the actual spawned processes themselves -- a few module imports and normal instantiation code, which take up to 0.05 seconds on a middling Amazon setup†. It seems like what it would be helpful to have a batch of processes with those imports/init code already done††, waiting to process results. What is the best approach to create/communicate with a pool (10-20?) of warm, reusable, and extremely lightweight processes in Python?
† - yes, I know throwing better hardware at the problem will help, and I'll do that too.
†† - yes, I know doing less will help, and I'm working on making the code as streamlined and minimal as possible
Well, you're in for a learning curve here, but multiprocessing.Pool() will create a pool of any number of processes you specify. Use the initializer= argument to specify a function each process will run at the start. Then there are several methods you can use to submit work items to the processes in the pool - read the docs, play with it, and ask questions if you get stuck.
One caution: "extremely lightweight processes" is impossible. By definition, processes are "heavy". "How heavy" is up to your operating system, and has approximately nothing to do with the programming language you use. If you're looking for lightweight, you're looking for threads.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
Questions asking us to recommend or find a tool, library or favorite off-site resource are off-topic for Stack Overflow as they tend to attract opinionated answers and spam. Instead, describe the problem and what has been done so far to solve it.
Closed 9 years ago.
Improve this question
I'm looking to get into web development. My goal is to create some interactive webpages that interact with MS SQL databases (read/insert/updates), and also possibly sites that interact with XML files.
I've got some basic understanding of Python and Perl scripting. Can someone point me in the right direction in either of those languages to accomplish what i'm looking to do, or if it's easier to accomplish in another language what would that be?
Apologies if my stated goal is too broad.
I'd strongly suggest you to look into some of the web development frameworks. They take care of many low-level tasks which is needed in order to build a solid web page. I'm not very familiar with perl, so I can only suggest Python frameworks, especially one of the my favourites - Django. It has very good documentation which is essential for the first-timer. I believe you should be fine as long as you follow the official documentation.
Good luck
You can use SQL Alchamy in python, and lxml or the default ElementTree xml module for simple cases.
I have done both for a webservice I maintain, and they work nice.
You can also use a web development framework. I personally suggest Flask based on that it is a lightweight framework as opposted to django for instance. However, depending on your exact use case the latter might be better.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
Questions asking us to recommend or find a tool, library or favorite off-site resource are off-topic for Stack Overflow as they tend to attract opinionated answers and spam. Instead, describe the problem and what has been done so far to solve it.
Closed 9 years ago.
Improve this question
I have a very I/O intensive task (it crawls webpages). I want to make this task available via a web API and built some JavaScript+HTML interface on top of it. And I want to do it in Python, since I already have a set of Python scripts implementing this task.
Now I'm looking for a Python web framework to do this. What web framework can you recommend for a very I/O intensive web page?
Some more details:
I already worked with Flask and I liked it
I don't need user management, but I need sessions
I want to build the actual UI with Angular/Ember
Since it's about web crawling, I'd go with Scrapy as a crawling tool and twisted event-driven networking engine/web-framework. Scrapy is actually built on top of twisted, so this set might be a good choice for you.
Also, take a look at tornado web framework that is using non-blocking I/O.
Hope that helps.
Django is the most popular web framework for Python.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
Questions asking us to recommend or find a tool, library or favorite off-site resource are off-topic for Stack Overflow as they tend to attract opinionated answers and spam. Instead, describe the problem and what has been done so far to solve it.
Closed 9 years ago.
Improve this question
OK, I have been reading about the celery and rabbitmq, while I appreciate the effort of the project and the documentation, I am still confused about a lot of things.
http://www.celeryproject.org/
http://ask.github.com/django-celery/
I am super confused about if celery is only for Django or a standalone server, as the second link claims celery is tightly used with Django. Both sites show different ways of setting up and using celery, which to me is chaotic.
Enough rant, is there a proper book available that I can buy?
Well not a book but I recently did setup in Dotcloud for Django+Celery, and here's the short doc:
http://web.archive.org/web/20150329132442/http://docs.dotcloud.com/tutorials/python/django-celery/
It's intended for simple tasks to be run asynchronously. There is a dotcloud-specific setup, but the rest might clear things up a bit. AFAIK, Celery started tightly coupled with Django but later became an entirely different animal, although it still retains superb compatibility with Django.
I don't know of a book, I guess a quick Amazon search would dig that up.
The bottom line is, celery is run as a separate server and works just as well for a standalone python program as Django, so it is not tied directly to Django. You can also run the celeryd worker software on multiple computers so they can all process the same queue concurrently. Often a separate queueing server, such as RabbitMQ is run to store the queue message.
Keep in mind, django-celery is just an integration app that acts as glue between Django and Celery.
This was asked a long time back and the celery docs have been significantly spruced up since, it'd be good to start with the FAQ's to allay queries of this nature.
http://docs.celeryproject.org/en/latest/faq.html#is-celery-for-django-only
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
Questions asking us to recommend or find a book, tool, software library, tutorial or other off-site resource are off-topic for Stack Overflow as they tend to attract opinionated answers and spam. Instead, describe the problem and what has been done so far to solve it.
Closed 8 years ago.
Improve this question
I want to start developing an ERP for small companies, So I'd like to know what python web framework is highly recommendable for?
I don't think there's any particularly suited for ERPs. Check out a list of all the current frameworks: http://wiki.python.org/moin/WebFrameworks.
openerp is open source and written in python, it uses cherrypy and other things
A open source erp solution called ERP Next [ https://erpnext.com/ ] uses python framework called WNF Framework [ https://github.com/webnotes/wnframework ] .You can ask for a free demo and gauge the
Frappe is the underlying framework of ERPNext, you can check it out. Documentation is relatively unavailable though so you may have to work your way through the source.
Check out web.py. It's a very minimalist Python framework and gives you the flexibility you might need to build something outside of the mainstream data-driven app
http://webpy.org/