I have written some Python scripts in Eclipse where at some point I have stable releases which should be deployed. For the moment this is very simple and means copying the scripts to some directory. (Soon I will also consider Git)
Could you tell me the common extendable way to set up a process like this in Eclipse? Maybe something more professional which, if a more complicating deployment becomes neccessary, can still be used.
(I had seen Ant somewhere, but I'm not sure if that is more than I need and maybe not the convention for Python in particular).
I would say use Fabric, and read this blog post for a reasonable introduction to it.
Related
I need to integrate a body of Python code into an existing OSGi (Apache Felix) deployment.
I assume, or at least hope, that packages exist to help with this effort.
If it helps, the Python code is still relatively new and small, so can probably re re-architected to meet whatever constraints are needed. However, it must remain in Python, because of dependencies on third-party libraries.
What are suggested best practices?
The trick is to make this an extender, see 1 and 2. You want your Python code to be separate from the code that handles the interaction with the interpreter. So what you do is wrap the Python code and any native libraries in a bundle. This is trivial since it is just a zip file.
You then develop a bundle that listens to starting bundle (see the BundleTracker) that have python code. A manifest is often used but you can also look in a directory in the JAR. If you detect this code, you extract any native libraries and run the code in the interpeter of your choice.
If can use JYthon then that would be highly recommended. You can then carry the interpreter as an OSGi bundle that runs on the VM. If you need to use a native compiler your life is less rosy. You can rely on the environment to provide you with an interpreter but then why use OSGi in the first place. You basically lose the write once run anywhere advantage. You could go the full monty by creating bundles that contain Python installers for all platforms you support. Can be done, not even that hard, but a maintenance nightmare. Believe me, native code suck, it only does it a bit faster than Java.
I want to make a Cocoa OS X app. I would prefer to use python scripts in it's core. However, not sure how safe is it. I know that python penetration is quite high, but what about version conflicts and migrations? Is it worth bundling whole python runtime into the OS X app?
Thanks.
So.... what this really boils down to is compatibility issues across versions, something that scripting languages are notoriously bad at maintaining. Python does better than most, but it is still quite problematic.
Apple has generally shipped legacy versions of interpreters on the system for exactly this reason. Thus, if you do rely on the system installed Python, I would recommend locking to a particular version. I.e. use /usr/bin/python2.6 and not the generic /usr/bin/python.
The alternative is as you state; bundle the python interpreter and any needed resources into your app. That is a bit of a pain the butt to do, but it addresses the compatibility issue. More or less; the reality is that Python is, effectively, an interface to the OS and, thus, is quite large with potential to break across any release. Not much you can about that, though.
Another possibility is to go the route that #kindall proposes; use PyObjC and implement your Cocoa application entirely or mostly in Python. Works fine. Been there, done that, and wouldn't do it again, frankly, as the maintenance/debugging issues of large scale scripted applications are nasty.
As an alternative, you might want to investigate using Lua (http://www.lua.org) as it is very much designed to be embedded in applications. Lua has a tiny interpreter and you can fully control exactly what features of your app are accessible at runtime. For example, World of Warcraft's UI is mostly implemented as Lua gluing together a set of fast UI primitives. Fully customizable on the client side, which is really impressive when you consider the security implications.
You should use py2app. It will bundle a Python executable, all the libraries you need, and your script together into a single executable. You can then add other executables (e.g. your Objective-C parts) into that app bundle.
I'm trying to get into Python and, more specifically, Zope and Plone. I read the book Professional Plone Development and one thing it says is that one MUST use version control. But the book did not expand on that topic any further. This leads to two questions.
First: SVN or git? (My research points to git, if only to learn it. I've only used SVN so far.)
And second: Which files should be handled by version control? Settings and my own code? The whole Zope directory? Not the data.fs, surely? Not the .pyc files, I'm sure of that. I've taken a break from Plone for that reason these days, and I couldn't find a good guide for that. In short, so far, when I synchronised the data between my local PC and my web server, things broke. Badly. And I'm not sure why. Either some updates were missing, or some platform-specific files were updated. My home PC is 64 bit Ubuntu and my distant web server is 32 bit RHEL. It felt like such a mess, and like such a dangerous mess, that I'm a bit apprehensive of going back near it.
Is there a way to know which files should be handled by a version control system and which should not?
Thanks.
No, no. Never make changes to the Zope/Plone system code. Create an add-on and make your customisations there. Of course you must keep backups, but version control is not a good fit for this.
You should ensure that any production buildout is repeatable. That means pinning all versions to your release (see http://pypi.python.org/pypi/buildout.dumppickedversions for a handy extension to record your picked versions) and keeping a backup of any source distributions locally, i.e. backing up your buildout downloads-cache.
I put only my buildout files into svn (dir: project/buildout/trunk). Buildout fetches all Plone/Zope files in the right versions.
Additionally i put my eggs in svn (dir: project/eggs/trunk). My eggs hold all the modifications into Plone.
My buildout uses mr.developer to auto fetches my eggs.
You ca check http://toutpt.wordpress.com/2010/07/07/nantes-developpement-com-a-new-plone-website-by-makina-corpus/
its a big buildout and uses all kind of mods and extensions.
It's quite hard to find a conclusive answer for this. The best reference I've found is this Plone project, which is checked into version control already:
https://github.com/Plone-Conference-Devs/ploneconfbuildout
I started using Pyant recenently to do various build/release tasks but have recently discovered that development for this project has ended.
I did some research and can't seem to find any other Python build scripts that are comparable. Just wondering if anyone can recommend one? I basically need it to do what ANT does - do SVN updates, move/copy files, archive etc using an XML file.
Thanks,
g
Probably the best answer is to use Ant as-is... that is, use the Java version. My second suggestion would be to use scons. It won't take much time using scons before you're asking, "Who ever thought of using XML to script a build?"
Its not completely comparable but I tend to use fabric. Its more geared towards deployment with support for ssh to production host and runing things as root there etc.
Some people use Paver for build/deployment of Python packages. While I know it works, it does not appeal to me that much.
what about maven? (http://maven.apache.org/) With the right plugins it can do much more then ant, it can even use ant for building if you configure it so.
It's very flexible and supports the full product life cycle. I really recommend you take a look at it.
I am a member of a team that is about to launch a beta of a python (Django specifically) based web site and accompanying suite of backend tools. The team itself has doubled in size from 2 to 4 over the past few weeks and we expect continued growth for the next couple of months at least. One issue that has started to plague us is getting everyone up to speed in terms of getting their development environment configured and having all the right eggs installed, etc.
I'm looking for ways to simplify this process and make it less error prone. Both zc.buildout and virtualenv look like they would be good tools for addressing this problem but both seem to concentrate primarily on the python-specific issues. We have a couple of small subprojects in other languages (Java and Ruby specifically) as well as numerous python extensions that have to be compiled natively (lxml, MySQL drivers, etc). In fact, one of the biggest thorns in our side has been getting some of these extensions compiled against appropriate versions of the shared libraries so as to avoid segfaults, malloc errors and all sorts of similar issues. It doesn't help that out of 4 people we have 4 different development environments -- 1 leopard on ppc, 1 leopard on intel, 1 ubuntu and 1 windows.
Ultimately what would be ideal would be something that works roughly like this, from the dos/unix prompt:
$ git clone [repository url]
...
$ python setup-env.py
...
that then does what zc.buildout/virtualenv does (copy/symlink the python interpreter, provide a clean space to install eggs) then installs all required eggs, including installing any native shared library dependencies, installs the ruby project, the java project, etc.
Obviously this would be useful for both getting development environments up as well as deploying on staging/production servers.
Ideally I would like for the tool that accomplishes this to be written in/extensible via python, since that is (and always will be) the lingua franca of our team, but I am open to solutions in other languages.
So, my question then is: does anyone have any suggestions for better alternatives or any experiences they can share using one of these solutions to handle larger/broader install bases?
Setuptools may be capable of more of what you're looking for than you realize -- if you need a custom version of lxml to work correctly on MacOS X, for instance, you can put a URL to an appropriate egg inside your setup.py and have setuptools download and install that inside your developers' environments as necessary; it also can be told to download and install a specific version of a dependency from revision control.
That said, I'd lean towards using a scriptably generated virtual environment. It's pretty straightforward to build a kickstart file which installs whichever packages you depend on and then boot virtual machines (or production hardware!) against it, with puppet or similar software doing other administration (adding users, setting up services [where's your database come from?], etc). This comes in particularly handy when your production environment includes multiple machines -- just script the generation of multiple VMs within their handy little sandboxed subnet (I use libvirt+kvm for this; while kvm isn't available on all the platforms you have developers working on, qemu certainly is, or you can do as I do and have a small number of beefy VM hosts shared by multiple developers).
This gets you out of the headaches of supporting N platforms -- you only have a single virtual platform to support -- and means that your deployment process, as defined by the kickstart file and puppet code used for setup, is source-controlled and run through your QA and review processes just like everything else.
I always create a develop.py file at the top level of the project, and have also a packages directory with all of the .tar.gz files from PyPI that I want to install, and also included an unpacked copy of virtualenv that is ready to run right from that file. All of this goes into version control. Every developer can simply check out the trunk, run develop.py, and a few moments later will have a virtual environment ready to use that includes all of our dependencies at exactly the versions the other developers are using. And it works even if PyPI is down, which is very helpful at this point in that service's history.
Basically, you're looking for a cross-platform software/package installer (on the lines of apt-get/yum/etc.) I'm not sure something like that exists?
An alternative might be specifying the list of packages that need to be installed via the OS-specific package management system such as Fink or DarwinPorts for Mac OS X and having a script that sets up the build environment for the in-house code?
I have continued to research this issue since I posted the question. It looks like there are some attempts to address some of the needs I outlined, e.g. Minitage and Puppet which take different approaches but both may accomplish what I want -- although Minitage does not explicitly state that it supports Windows. Lacking any better options I will try to make either one of these or just extensive customized use of zc.buildout work for our needs, but I still feel like there must be better options out there.
You might consider creating virtual machine appliances with whatever production OS you are running, and all of the software dependencies pre-built. Code can be edited either remotely, or with a shared folder. It worked pretty well for me in a past life that had a fairly complicated development environment.
Puppet doesn't (easily) support the Win32 world either. If you're looking for a deployment mechanism and not just a "dev setup" tool, you might consider looking into ControlTier (http://open.controltier.com/) which has a open-source cross-platform solution.
Beyond that you're looking at "enterprise" software such as BladeLogic or OpsWare and typically an outrageous pricetag for the functionality offered (my opinion, obviously).
A lot of folks have been aggressively using a combination of Puppet and Capistrano (even non-rails developers) for deployment automation tools to pretty good effect. Downside, again, is that it's expecting a somewhat homogeneous environment.