Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 9 years ago.
Improve this question
Short version: How do you deploy your Django servers? What application server, front-end (if any, and by front-end I mean reverse proxy), and OS do you run it on? Any input would be greatly appreciated, I'm quite a novice when it comes to Python and even more as a server administrator.
Long version:
I'm migrating between server hosts, so much for weekends... it's not all bad, though. I have the opportunity to move to a different, possibly better "deployment" of Django.
Currently I'm using Django through Tornado's WSGI interface with an nginx front-end on Debian Lenny. I'm looking to move into the Rackspace Cloud so I've been given quite a few choices when it comes to OS:
Debian 5.0 (Lenny)
FC 11 or 12
Ubuntu 9.10 or 8.04 (LTS)
CentOS 5.4
Gentoo 10.1
Arch Linux 2009.02
What I've gathered is this:
Linux Distributions
Debian and CentOS are very slow to release non-bugfix updates of software, since they focus mainly on stability. Is this good or bad? I can see stability being a good thing, but the fact that I can't get Python 2.6 without quite a headache of replacing Python 2.4 is kind of a turn-off--and if I do, then I'm stuck when it comes to ever hoping to use apt/yum to install a Python library (it'll try to reinstall Python 2.4).
Ubuntu and Fedora seem very... ready to go. Almost too ready to go, it's like everything it already done. I like to tinker with things and I prefer to know what's installed and how it's configured versus hitting the ground running with a "cookie-cutter" setup (no offense intended, it's just the best way to describe what I'm trying to say). I've been playing around with Fedora and I was pleasently surprised to find that pycurl, simplejson and a bunch of other libraries were already installed; that raised the question, though, what else is installed? I run a tight ship on a very small VPS, I prefer to run only what I need.
Then there's Gentoo... I've managed to install Gentoo on my desktop (took a week, almost) and ended up throwing it out after quite a few events where I wanted to do something and had to spend 45 minutes recompiling software with new USE flags so I can parse PNG's through PIL. I've wondered though, is Gentoo good for something "static" like a server? I know exactly what I'm going to be doing on my server, so USE flags will change next to never. It optimizes compiles to fit the needs of what you tell it to, and nothing more--something I could appreciate running on minimal RAM and HDD space. I've heard, though, that Gentoo has a tendency to break when you attempt to update the software on it... that more than anything else has kept me away from it for now.
I don't know anything about Arch Linux. Any opinions on this distro would be appreciated.
Web Server
I've been using Tornado and I can safely say it's been the biggest hassle to get running. I had to write my own script to prefork it since, at the time I setup this server, I was probably around 10% of Tornado's user-base (not counting FriendFeed). I have to then setup another "watchdog" program to make sure those forks don't misbehave. The good part is, though, it uses around 40MB of RAM to run all 7 of my Django powered sites; I liked that, I liked that a lot.
I've been using nginx as a front-end to Tornado, I could run nginx right in front of Django FastCGI workers, but those don't have the reliability of Tornado when you crank up the concurrency level. This isn't really an option for me, but I figured I might as well list it.
There's also Apache, which Django recommends you use through mod_wsgi. I personally don't like Apache that much, I understand it's very, very, very mature and what not, but it just seems so... fat, compared to nginx and lighttpd. Apache/mod_python isn't even an option, as I have very limited RAM.
Segue to Lighttpd! Not much to say here, I've never used it. I've heard you can run it in front of Apache/mod_wsgi or run it in front of Django FastCGI workers, also. I've heard it has minor memory leaking issues, I'm sure that could be solved with a cron job, though.
What I'm looking for is what you have seen as the "best" deployment of Django for your needs. Any input or clarifications of what I've said above would be more than welcome.
Update your question to remove the choices that don't work. If it has Python 2.4, and an installation is a headache, just take it off the list, and update the question to list the real candidates. Only list the ones that actually fit your requirements. (You don't say what your requirements are, but minimal upgrades appears to be important.)
Toss a coin.
When choosing between two platforms which meet your requirements (which you haven't identified) tossing a coin is the absolute best way to choose.
If you're not sure if something matches your requirements, it's often good to enumerate what you value. So far, the only thing in the question that you seem to value is "no installations". Beyond that, I can only guess at what requirements you actually have.
Once you've identified the set of features you're looking for, feel free to toss a coin.
Note that Linux distributions all have more-or-less the same open-source code base. Choosing among them is a preference for packaging, support and selection of pre-integrated elements of the existing Linux code base. Just toss a coin.
Choosing among web front-ends is entirely a question of what features you require. Find all the web front-ends that meet your requirements and toss a coin to choose among them.
None of these are "lock-in" decisions. If you don't like the linux distro you chose initially, you can simply chose another. They all have the same basic suite of apps and the same API's. The choice is merely a matter of preference.
Don't like the web server you chose? At the end of the mod_wsgi pipe, they all appear the same to your Django app (plus or minus a few config changes). Don't like lighttpd? Switch to nginx or Apache -- your Django app doesn't change. So there's no lock-in and no negative consequences to making a sub-optimal choice.
When there's no down-side risk, just toss a coin.
If you want a lightweight alternative to Tornado, I'd suggest spawning. It's very good at code reloading and seems to have good performance - though of course best you try it out yourself.
I'd also recommend supervisord regardless of OS, for keeping Tornado or spawning instances up and running as well as any other essential services.
However 9 times out of 10 it's the database that's the bottleneck and choice of web server is not really going to impact site performance and scalability.
At the place I rent server, they have shaved down the Ubuntu images to bare minimum. Presumably because they had to make a special image anyway with just the right drivers and such in it, but I don't know exactly.
They have even removed wget and nano. So you get all the apt-get goodness and not a whole lot of "cookie-cutter" OS.
Just saying this because I would imagine that this is the way it is done almost everywhere and therefore playing around with a normal Ubuntu-server install will not provide you with the right information to make your decision.
Other than that, I agree with the others, that it is not much of a lock-in so you could just try something.
On the webserver-side I would suggest taking a look at cherokee, if have not done so already.
It might not be your cup of joe, but there is no harm in trying it.
I prefer the easy setup of both Ubuntu and Cherokee. Although I play around with a lot of things for fun, I prefer these for my business. I have other things to do than manage servers, so any solution that helps me do it faster, is just good. If these projects are mostly for fun then this will most likely not apply since you won't get a whole lot of experience from these easy-setup-with-nice-gui-and-very-helpfull-wizards
Personally I find one of the BSD systems far superior to Linux distros for server related tasks. Give OpenBSD or perhaps FreeBSD a chance. Once you do you´ll never go back.
Related
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 7 years ago.
Improve this question
I have recently got a new Mac (2015 MBP with 8gb ram) and I am trying to set up a development environment using 5 different stacks at the same time.
I've previously used a LAMP set up very similar to this - many of my old (client) projects are still based on a PHP MVC framework which uses Foundation and Libsass. While I don't want to make new projects in this way as I want to learn more, I still need to be able to maintain these other projects quickly and easily.
I then had some data projects in Python / Django. On my old mac I used a linux VM through Virtual Box for this (I don't yet own parallels etc).
I've also trying to learn to use the MEAN stack using gulp, browserify and playing around with React JS, as well as Meteor (which has it's own server too) and Ruby on Rails - and I want to do more development with these, but without losing the ability to work on my other projects.
My problem is that whenever I previously switched to using MEAN or Rails, I messed up my file watching / auto compiling for LAMP and Python and it took me ages to switch back. I really like how browserify works - e.g. to auto compile sass files / minify, and auto refresh the page, but I've hit problems with keeping my python server and lamp set up doing the same. I'm clearly doing something wrong and I'm keen to set this up properly with a clean install. All the tutorials I can find online refer to setting up individual stack environments, but I can't find much which refers to setting them up together.
I've decided that running a separate VM for each different stack would knock performance too much, and there are some things which I'd want to use on more than one stack like homebrew.
As such, my question is what are the specific advantages of using something like docker to keep things separate vs finding a way using something like homebrew to set up everything together on a single OS?
I'm aware file watching isn't so easy on docker with virtualbox - but I'm not sure whether to push through and find a work around or to use another way.
Thanks so much - I've been stuck on this for ages. I'm sorry if my question isn't very clear - I'm new to using node / meteor and ruby so I'm not sure the best place to start!
In my humble opinion the best solution by far is to use Vagrant. It's also free.
Not only that you can have a different environment for each project, but you can also script all the custom installations and configurations. You, or a colleague of yours, will be able to replicate the environment with a simple command, vagrant up.
A popular guide for setting up Vagrant for RoR development can be found here. However, contrary to that guide, I gave up using Chef in favor of a simple provision script. A guide can be found here.
Make sure to enable NFS though, else VirtualBox (Vagrant sits on top of it) will be quite slow.
While solutions like Cloud9 or Koding are surely useful for when you do not have access to your development machine, I wouldn't use them for day to day work.
I've googled and googled, but have found almost nothing in the way of discussions or best practices in managing larger enterprise codebases in Python. Here, I'm simply soliciting any and all pointers to such information. Here's some background and some of the questions I'm looking to answer.
We're long-time Java developers, who have solved similar problems to those mentioned below largely using well established Java best practices, as well as Maven, Ant and a Sonotype Nexus repo.
I'm talking internal software only here. We're not looking to distribute anything Python-based. We've got multiple development groups using Python, each developing sharable utility code libraries, final web applications and stand-alone tools, all in pure Python. Each group has its own Github source repository.
How do we manage our shareable code, both within a group and across groups? Do we create eggs (or something similar) and distribute and install them into the Python system? If so, would we store them in our Nexus repo like our Java jars, or is there a more Python-specific method if internal package distribution? Or, do we just share raw code, checking out sources from multiple Github repos?
If we share raw code, how do we manage getting the Python searchpath right as we bring together code from multiple repositories?
How do we manage package namespaces when we want our packages to all live in a com.ourcompany base namespace? It seems like python isn't too happy when you bring together source trees with overlapping namespaces.
How do we manage third party package versioning? I've never seen easy_install or pip passed a version number. How do we lock down third party package versions?
Do tools exist to aid in Python code reviews, CI, regression testing, etc.?
We're relative newbies to Python code, so some of these questions may have fairly obvious answers. Still, I find it surprising that I can't find more information on managing larger Python codebases.
What issues will we encounter that I haven't thought to ask about, or don't yet know enough to even know to ask about?
Any valuable pointers will be greatly appreciated.
Well, I won't even try to answer all those (excellent) questions, but here are a few opinionated pointers which will hopefully help (as someone who works in both worlds, though more Java).
Packaging
If so, would we store them in our Nexus repo like our Java jars, or is
there a more Python-specific method if internal package distribution?
Or, do we just share raw code, checking out sources from multiple
Github repos?
Packaging in Python is historically a bit of a mess IMHO, though it feels like it's improving. Distutils is the major / native tool here - I've not used it much, feels slightly scary in places. In general, also check recommended tools.
Pip has all but won the war of mindshare, especially when installing 3rd party libraries. I've not solved the local library problem myself, (maybe someone else reading has), but if I were, I'd probably opt for Pip with local/network-disk repos e.g. by installing from wheels.
Another option (which can cause all sorts of hassles itself) is to package in your OS's native packager, be it Debian-style apt or by creating RPMs, etc. Of course, Windows not so much.
Versioning etc
How do we manage third party package versioning? I've never seen
easy_install or pip passed a version number.
Pip
Pip definitely supports version specifiers. Turns out Easy Install does too. I suppose many people / smaller projects opt for latest-and-greatest, which of course isn't always as "appropriate" in the enterprise...
Virtualenv
No discussion of versioning and Python would miss a Python2/3 reference, but I'm sure you're aware of all this already.
More important then would be to mention virtualenv. It truly frees you from the mess you can get in to testing multiple versions, bearing in mind especially that your (*NIX) operating systems typcially rely heavily on Python themselves. It's a big subject so have a look at the docs.
Developer Tooling
Do tools exist to aid in Python code reviews, CI, regression testing,
etc.?
Code Review
Very much so. Most code review tools are multi-language (it's just a formatting issue really), so just pick your favourite enterprise-friendly one, be it Crucible, Github's one (Barkeep?), Gerrit, or whatever.
CI
For CI you have almost as many options again. Running python apps is usually less involved than Java ones, so most CI systems, though often Java-focused, support Python. (FWIW, we use drone.io for Quod Libet). Jenkins should have no problem doing this, and it seems people have done so with TeamCity.
However, the "original" or "most Pythonic" is probably Buildbot, but I've not used it personally. Looks a lot newer than I remember, and it had quite a lot of support in the Python community I think...
Testing
For testing, though not quite as mature as JUnit / TestNG, check out the de-facto / JUnit-like unit testing unittest, but also (nicer?) alternatives like nose.py.
For higher level (BDD) testing, try something like Lettuce - as the name implies heavily inspired by Cucumber, or maybe Behave. I've not tried them, but common opinion is they're less mature than Cucumber / JBehave / Concordion / Rspec etc.
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 10 years ago.
Been looking at porting a BitCoin Mining Pool server across to IronPython
Apart from the limitations of Ironpython supporting upto Python ~v2.7
Is a Ironpython/windows platform really not suited to this type of high volume/large transactional system
What are some of the other limitations I might encounter or should i juts bite the bullet and build up an ubuntu programming RIG
If the later what would be an easy IDE for someone coming from a c# Visual Studio world
Thanks in Adavnce
Without knowing how your existing code is built, or how you intend the new version to work, it's hard to answer specifically, but I can give some generalities.
IronPython limitations/issues, in decreasing order of importance:
* No third-party C extension modules. If you depend on any, you'll have to find a .NET equivalent or reimplement the same thing yourself. If not, not a problem.
* No 3.x. If you know 3.x, Unicode is a lot easier, yield from rocks, etc., and 3.x keeps improving (and even speeding up), while 2.x is stagnant. But how much this is worth is a matter of personal preference, not something anyone else can give you an objective answer on.
* No third-party pure Python modules that rely (in important ways) on refcounting semantics. There aren't too many of these out there, but if you need one, you need it.
* There are a handful of differences between IronPython and CPython, listed in the release notes. Probably none of these will affect you, but you should check.
* If there's any heavy application logic (as opposed to all of the work being networking/files/database), IronPython is much faster than CPython at a few things and much slower at a few others, so you probably need to profile and perf-test before you go too far down either path. (And consider at least PyPy, if not Jython, while you're at it…) But usually this isn't an issue for servers.
Next, is a Windows platform suited to a high-volume systems? Windows is definitely capable of handling thousands of connections per second (or requests, or bytes, or whatever's most relevant to your use case). It will probably never beat linux in pure connections/second/$, but you have to realistically trade off operational costs vs. dev costs. However, oversimplifying a bit, Windows sucks at reactors and rocks at proactors, while linux rocks at reactors and is decent at proactors (and there are other similar issues—e.g., substitute processes/threads for the above). So, even though Windows can do nearly as well as linux in general, it can't do as well on the same code, or often even the same designs. You need to architect your code to take advantage of Windows features and take into account Windows weaknesses. And if you're using a framework to do the heavy lifting for you, there are going to be fewer choices, and the best ones may not work like what you're used to, which is usually a much higher dev cost than changing platforms.
If you do go with CPython on linux, what IDE should you use? Stick with Visual Studio. You can, and probably should, write either all or most of your code in a cross-platform way that runs on both IronPython and linux CPython. Then you can code, test, and debug with Iron Python in Visual Studio, use native Windows CPython and/or Cygwin CPython for occasional sanity checks, and only touch linux for performance tests, final acceptance tests before releases, and debugging problems you can't repro on Windows. You'll still occasionally need a simple text editor on linux, but IDLE, or whichever of vi-vs.-emacs you prefer, or whichever GUI text editor your linux distro comes with as the default will be fine.
If your application logic is tightly coupled with your reactor design or something, this could be difficult. If it's doable, it's probably worth doing, but if it's not… well, you can still use Visual Studio as an editor and project organizer, even if the actual code is intended to be run on a linux box. You don't get the nifty debugger integration, etc. that way, but you still get a lot of mileage.
The Story
After cleaning up my Dreamhost shared server's home folder from all the cruft accumulated over time, I decided to start afresh and compile/reinstall Python.
All tutorials and snippets I found seemed overly simplistic, assuming (or ignoring) a bunch of dependencies needed by Python to compile all modules correctly. So, starting from http://andrew.io/weblog/2010/02/installing-python-2-6-virtualenv-and-virtualenvwrapper-on-dreamhost/ (so far the best guide I found), I decided to write a set-and-forget Bash script to automate this painful process, including along the way a bunch of other things I am planning to use.
The Script
I am hosting the script on http://bitbucket.org/tmslnz/python-dreamhost-batch/src/
The TODOs
So far it runs fine, and does all it needs to do in about 900 seconds, giving me at the end of the process a fully functional Python / Mercurial / etc... setup without even needing to log out and back in.
I though this might be of use for others too, but there are a few things that I think it's missing and I am not quite sure how to go for it, what's the best way to do it, or if this just doesn't make any sense at all.
Check for errors and break
Check for minor version bumps of the packages and give warnings
Check for known dependencies
Use arguments to install only some of the packages instead of commenting out lines
Organise the code in a manner that's easy to update
Optionally make the installers and compiling silent, with error logging to file
failproof .bashrc modification to prevent breaking ssh logins and having to log back via FTP to fix it
EDIT: The implied question is: can anyone, more bashful than me, offer general advice on the worthiness of the above points or highlight any problems they see with this approach? (see my answer to Ry4an's comment below)
The Gist
I am no UNIX or Bash or compiler expert, and this has been built iteratively, by trial and error. It is somehow going towards apt-get (well, 1% of it...), but since Dreamhost and others obviously cannot give root access on shared servers, this looks to me like a potentially very useful workaround; particularly so with some community work involved.
One way to streamline this would be to make it work with one of: capistrano/fabric, puppet/chef, jhbuild, or buildout+minitage (and a lot of cmmi tasks). There are some opportunities for factoring in common code, especially with something more high-level than bash. You will run into bootstrapping issues, however, so maybe leave good enough alone.
If you want to look into userland package managers, there is autopackage (bootstraps well), nix (quickstart), and stow (simple but helps with isolation).
Honestly, I would just build packages with a name prefix for all of the pieces and have them install under /opt so that they're out of the way. That way it only takes the download time and a bit of install time to do.
What experiences do you have with Stackless Python and PyQt?
Issues i would be happy if people address:
Compilation of PyQt for Stackless: does PyQt need to be compiled especially for Stackless? is the compilation smooth? problems with bindings etc.
Stability: any unexpected crashes, freezes, pauses and other weirdities?
Memory Management: any hints of memory leaks. comparison of RAM needed for a Stackless/Plain Vanilla PyQt applications
Software Engineering Empowerment: very short outline of flow-of-control models for Stackless-powered PyQt applications
Lessons learned: any painful lesson learned, traps to be avoided, problems to tackle you might have experienced
Be Happy
I tried to go down this path several months ago and decided it was not worth the effort.
I was able to run a binary install of PyQt (on Windows) against a stackless version of Python, but I found that I had to manually go in and change some of the files. I was getting an error message (sorry, I forget what it was), and google search led to a solution from several years ago. Newer code did not include the old fix, so the change was not too difficult and (if I remember correctly) it was in python, so no recompile was necessary.
But that was a deal breaker for me. Qt updates come out regularly, as do updates to PyQt, and I didn't want to be continually fixing the code. Stackless and PyQt are simply not used enough together to be checked out thoroughly. I found the risk of difficult to debug issues pretty high. This is especially true given the author of stackless has moved on to PyPy. Let me apologize in advance - I wish I had the references I found for the author stopping development on stackless python and more detail on the errors I had to fix - I wasn't expecting to regurgitate the details on Stack Overflow.
So I chose to run PyQt on a vanilla Python instead of stackless.
BTW, I also thought that mixing signals/slots with stackless code would be confusing, as they are completely different methods of solving multi-threading problems.
Good luck!
If you're interested in all that because of speed optimization:
You may want to check out Unladen Swallow ( Wikipedia here ). Google (because of YouTube being 100% Python) is working on a JIT compiler for Python that will increase its speed by 5-10x (bringing it much closer to Java speeds than the current virtual machine). The best part is that it will work with all existing Python code which means you don't have to fret with all the problems associated with other Python optimization projects.
I expect that if you're developing a large enough project to warrant the need for optimization at a low level, you'd be OK with developing in normal Python and then changing out of the current VM when Unladen Swallow comes out in production.