I am going to be building a Pylons-based web application. For this purpose, I'd like to build a minimal Linux platform, upon which I would then install the necessary packages such as Python and Pylons, and other necessary dependencies. The other reason to keep it minimal is because this machine will be virtual, probably over KVM, and will eventually be replicated in some cloud environment.
What would you use to do this? I am thinking of using Fedora 10's AOS iso, but would love to understand all my options.
I really like JeOS "Just enough OS" which is a minimal distribution of the Ubuntu Server Edition.
If you want to be able to remove all the cruft but still be using a ‘mainstream’ distro rather than one cut down to aim at tiny devices, look at Slackware. You can happily remove stuff as low-level as sysvinit, cron and so on, without collapsing into dependency hell. And nothing in it relies on Perl or Python, so you can easily remove them (and install whichever version of Python your app prefers to use).
For this purpose, I'd like to build a minimal Linux platform...
So Why not try to use ArchLinux www.archlinux.org?
Also you can use virtualenv with Pylons in it.
debootstrap is your friend.
Damn Small Linux? Slax?
If you want to go serious about the virtual appliance idea, take a look at the newly released VMware Studio. It was built exactly for trimming down a system (only Linux for now afaik) so it provides only enough base to run your application.
VMware is going (a bit more) open by pushing an open virtual appliance format (OVF) so, at some point in the future, you might be able to run the result on other virtualization platforms too.
Debootstrap, or use kickstart to strap your FC domains. However, other methods of strapping an RPM based distro exist, such as Steve Kemp's rinse utility that replaces rpmstrap.
Or, you could just grab something at jailtime to use as a base.
If that fails, download everything you need from source, build / install it with a /mydist prefix (including libc, etc) and test it via chroot.
I've been building templates for Xen for years .. its actually turned into a very fun hobby :)
Related
You can tell where this is going from the title..
I work at a small post house, and to replace one of our older Mac Pros, we were looking at building a mid-range PC (doesn't need to do much heavy lifting - not worth the price of a new Mac Pro)..
The problem is, we have a number of scripts that we use for footage transcoding and media management, and they were coded for our Mac-exclusive environment..
The basic procedure for setting up a new workstation is:
Install pip using easy_install
Use pip to install:
youtube_dl
pexpect
parsedatetime
XlsxWriter
Install MacPorts.. Use MacPorts to install ffmpeg, lftp, and ImageMagick. From there, install a couple custom in-house scripts (that rely mostly on the preceding dependencies and python 2.7 (via xcode).
I'm NOT going to ask how to adapt this to PC, since that would be way outside the scope of a single question. My question is, without a strong python developer on-site (the custom scripts were written years ago by someone who's no longer with the company), is it even feasible to adapt the install scripts to a PC environment? We have a freelance Python developer that we consult for minor script changes and he wasn't sure (his response was "I would first double check to make sure some of the utilities can even run on windows. I’m not 100% sure they can or not"). Do any of the utilities jump out as something that would not be adaptable to a Windows environment?
Thanks very much in advance - I know it's a rather broad question.
If you "don't need heavy lifting", why upset your workflow this way?
If you are determined to buy a PC, why not run Linux on it rather than Windows? The setup process you described will be almost exactly the same on a Linux machine except for involving a different package manager instead of MacPorts.
Craig
PS the Python modules you listed are all available via MacPorts. It would simplify the software management process to install everything that way.
I have written a small python script that i want to share with other users.(i want to keep it as a script rather than and exe so that users can edit the codes if they need to)
my script has several external libraries for python which doesn't come with basic python.
But the other users doesn't have python and the required libraries installed in their PCs .
So,For convenient, I am wondering if there's any way to automate the installation process for installing python and the external libraries they need.
To make things more clear, what i meant is to combine all the installers into 1 single big installer.
For you information, all the installers are window x86 MSI installers and there are about 5 or 6 of them.
Is this possible?Could there be any drawbacks of doing this?
EDIT:
All the users are using windows XP pro 32 bit
python 2.7
I would suggest using NSIS. You can bundle all the MSI installers (including python) into one executable, and install them in "silent mode" in whatever order you want. NSIS also has a great script generator you can download.
Also, you might be interested in activepython. It comes with pip and automatically adds everything to your path so you can just pip install most of your dependencies from a batch script.
what i meant is to combine all the installers into 1 single big installer.
I am not sure, if you mean to make one msi out of several. If you have built the msis, this is possible to work out, but in most situations there were reasons for the separation.
But for now I assume as the others, that you want a setup which combines all msi setups into one, e.g. with a packing/selfextracting part, but probably with some own logic.
This is a very common setup pattern, some call it "bootstrapper". Unfortunately the maturity of most tools for bootstrapping is by far not comparable to the msi creation tools so most companies I know, write kind of an own bootstrapper with the dialogs and the control logic they want. This can be a very expensive job.
If you have not high requirements, it may sound a simple job. Just starting a number of processes after each other. But what about a seamless process bar, what about uninstallation (single or bundled), what about repair, modify, what about, if one of them fails or needs a reboot also concerning repair/uninstall/modify/update. And so on.
As mentioned, one of the first issues of bundling several setups into one is about caring how many and which uninstall entries shall the user see, and if it is ok that your bootstrapper does not create an own, combining one.
If this is not an issue for you, then you have chances to find an easy solution.
I know at least three tools for bootstrappers, some call it suites or bundles. I can only mention them here:
WiX has at least something called "Burn". Google for WiX Burn and you will find it. I haven't used it yet, so I can't tell about.
InstallShield Premier, which is not really what most people call a cheap product, allows setup "Suites" which is the same. I don't want to comment the quality here.
In the Windows SDK there is (has been?) a kind of template of a setup.exe to show how to start installation of msi out of a program. I have never looked into that example really to tell more about it.
I suggest putting all the files into a .sfx.exe archive and get them to run it. Extract all files to %temp% and run a batch script to install python.msi and copy the libraries from %temp% to the python library directory. If you want to install python 2.7.5, grab an "Ninite" installer from http://ninite.com/
I once read about minimal python installation without a lot of the libraries that come with the python default installation but could not find it on the web...
What I want to do is to just pack a script with the python stuff required to execute it and make portable.
Does any one know about something like that?
Thanks
Micro Python is actively maintained and has been ported to a bunch of microcontrollers.
For other small implementations, you might also want to check out tinypy or PyMite.
If you don't care about size, but really just want an easy way to distribute a python program, consider PyInstaller or one of the others on this list.
Portable python might do what you want. It's a python installation for USB thumb drives.
There's now finally Micro Python, claiming to be full reimplementation of Python 3 core, fitting even into medium-size 32bit microcontrollers. API will be different of course, so C modules will require porting. Project is funded via KickStarter, source code will be released some time after the campaign (request for consideration was made to author to not delay release of the source, to help bootstrap Micro Python community sooner).
http://micropython.org/
You can also look for already installed instances.
OpenOffice / LibreOffice
Look at the environment variable UNO_PATH or into the default install directories, for example for Windows and LO5
%ProgramFiles(x86)%\LibreOffice 5\program\python.exe
Gimp
look into the default install directories, for example for Windows
C:\Program Files\GIMP 2\Python
and so on...
I'm a .NET developer who knows very little about Python, but want to give it a test drive for a small project I'm working on.
What tools and packages should I install on my machine? I'm looking for a common, somewhat comprehensive, development environment.
I'll likely run Ubuntu 9.10, but I'm flexible. If Windows is a better option, that's fine too.
Edit: To clarify, I'm not looking for the bare minimum to get a Python program to run. I wouldn't expect a newbie .NET dev to use notepad and a compiler. I'd recommend Visual Studio, NUnit, SQL Server, etc.
Your system already has Python on it. Use the text editor or IDE of your choice; I like vim.
I can't tell you what third-party modules you need without knowing what kind of development you will be doing. Use apt as much as you can to get the libraries.
To speak to your edit:
This isn't minimalistic, like handing a .NET newbie notepad and a compiler: a decent text editor and the stdlib are all you really need to start out. You will likely need third-party libraries to develop whatever kind of applications you are writing, but I cannot think of any third-party modules all Python programmers will really need or want.
Unlke the .NET/Windows programming world, there is no one set of dev tools that stands above all others. Different people use different editors a whole lot. In Python, a module namespace is fully within a single file and project organization is based on the filesystem, so people do not lean on their IDEs as hard. Different projects use different version control software, which has been booming with new faces recently. Most of these are better than TFS and all are 1000 times better than SourceSafe.
When I want an interactive session, I use the vanilla Python interpreter. Various more fancy interpreters exist: bpython, ipython, IDLE. bpython is the least fancy of these and is supposed to be good about not doing weird stuff. ipython and IDLE can lead to strange bugs where code that works in them doens't work in normal Python and vice-versa; I've seen this first hand with IDLE.
For some of the tools you asked about and some others
In .NET you would use NUnit. In Python, use the stdlib unittest module. There are various third-party extensions and test runners, but unittest should suit you okay.
If you really want to look into something beyond this, get unittest2, a backport of the 2.7 version of unittest. It has incorporated all the best things from the third-party tools and is really neat.
In .NET you would use SQL Server. In Python, you may use PostgreSQL, MySQL, sqlite, or some other database. Python specifies a unified API for databases and porting from one to another typically goes pretty smoothly. sqlite is in the stdlib.
There are various Object Relational Models to make using databases more abstracted. SQLAlchemy is the most notable of these.
If you are doing network programming, get Twisted.
If you are doing numerical math, get numpy and scipy.
If you are doing web development, choose a framework. There are about 200000: Pylons, zope, Django, CherryPy, werkzeug...I won't bother starting an argument by recommending one. Most of these will happily work with various servers with a quick setting.
If you want to do GUI development, there are quite a few Python bindings. The stdlib ships with Tk bindings I would not bother with. There are wx bindings (wxpython), GTK+ bindings (pygtk), and two sets of Qt bindings. If you want to do native Windows GUI development, get IronPython and do it in .NET. There are win32 bindings, but they'll make you want to pull your hair out trying to use them directly.
In order to reduce the chance of effecting/hosing the system install of python, I typically install virtualenv on the ubuntu python install. I then create a virtualenv in my home directory so that subsequent packages I install via pip or easy_install do not effect the system installation. And I add the bin from that virtualenv to my path via .bashrc
$ sudo apt-get install python-virtualenv
$ virtualenv --no-site-packages ~/local
$ PATH=~/local/bin:$PATH #<----- add this to .bashrc to make it permanent
$ easy_install virtualenv #<--- so that project environments are based off your local environment rather than the system, probably not necessary
Install your favorite editor, I like emacs + rope, but editors are a personal preference and there are plenty of choices.
When I start a new project/idea I create a new virtual environment for that project, so that I don't effect dependencies anywhere else. Since I would hate for some of my projects to break due to an upgrade of a library both that project and the new one depends on.
~/projects $ virtualenv --no-site-packages my_new_project.env
~/projects/my_new_project.env $ source bin/activate
(my_new_project.env)~/projects/my_new_project.env $ easy_install paste ipython #whatever else I think I need
(my_new_project.env)~/projects/my_new_project.env $ emacs ./ & # start hacking
When creating a new package...in order to have something that will be easy_installable/pippable use paster create
(my_new_project.env)~/projects/my_new_project.env$ paster create new_package
(my_new_project.env)~/projects/my_new_project.env/new_package$ python setup.py develop new_package
That's the common stuff as far as I can think of it. Everything else would be editor/version control tool specific
Since I'm accustomed to Eclipse, I find Eclipse + PyDev convenient for Python. For quick computations, Idle is great.
I've used Python on Windows and on Ubuntu, and Linux is much cleaner.
If you launch a terminal and type python you'll get an interpreter, where you can start trying stuff.
Just in case you haven't seen it, check out the book Dive Into Python, is free on-line.
http://www.diveintopython.org/
Follow the examples in the book using the interpreter.
For storing your work you could use any editor; Vim or EMACS could be the most powerful, but also the most difficult to learn at first. If you want a more "traditional" IDE, you could try WingIDE.
http://www.wingware.com/
After you start to get more comfortable with python you should try an enhanced interpreter; try ipython.
http://ipython.scipy.org/moin/
When you start to develop a more serious project you'll need to get additional modules. Here you have two options; 1) Use your distribution tools to install additional modules; or 2) Download the modules you need directly from their sites and install them manually. You'll be responsible to upgrade them of course.
You'll have to decide for yourself which way to go. Personally I prefer to download and install additional modules manually.
Python (duh), setuptools or pip, virtualenv, and an editor. I suggest geany, but that's just me. And of course, any other Python modules you'll need.
Getting to Python from .NET world
Jumping into the Linux world from a .NET / WIndows background can be a bit disconcerting (but I do encourage you to keep trying Linux)
But I would suggest to anyone coming from Windows, to stick with Windows for a little while. goto www.Activestate.com and download their Python package - it includes the full win32com extentions by Mark Hammond and it also includes a complete, fast IDE "pythonwin"
I have done real professional development with just this setup alone on a windows box - one 14MB .msi and off you go !
Now to use Python on the DLR (Dynamic common language runtime) you need to download IronPython. THis is a seperate interpreter, that was also originally written by Mark Hammond at Microsoft and is at ironpython.org.
With this you can run code like (from wikipedia) ::
import clr
clr.AddReference("System.Windows.Forms")
from System.Windows.Forms import MessageBox
MessageBox.Show("Hello World")
Now you can access any .NET code from python.
If you're just starting out with Python, I'd actually argue against bringing in the complexity of virtualenv (which I think can be pretty overwhelming), at least until you've got a firm grasp of Python basics (especially regarding library/dependency management).
If you're using Ubuntu and the Gnome desktop environment, gedit is the default (gui) text editor, and has great support for Python built in. So my recommendation is to start with the pre-installed Python and gedit (which is pretty extensible on its own).
You don't need much. Python comes with "Batteries Included."
Visual Studio == IDLE. You already have it. If you want more IDE-like environment, install Komodo Edit.
NUnit == unittest. You already have it in the standard library.
SQL Server == sqlite. You already have it in the standard library.
Stop wasting time getting everything ready. It's already there in the basic Python installation.
Get to work.
Linux, BTW, is primarily a development environment. It was designed and built by developers for developers. Windows is an end-user environment which has to be supplemented for development.
Linux was originally focused on developers. All the tools you need are either already there or are part of simple yum or RPM installs.
You would probably like to give NetBeans Python IDE a shot. You can choose to use either Windows/Linux.
Database: sqlite (inbuilt). You might want SQLAlchemy though.
GUI: tcl is inbuilt, but wxPython or pyQt are recommended.
IDE: I use idle (inbuilt) on windows, TextMate on Mac, but you might like PyDev. I've also heard good things about ulipad.
Numerics: numpy.
Fast inline code: lots of options. I like boost weave (part of scipy), but you could look into ctypes (to use dlls), Cython, etc.
Web server: too many options. Django (plus Apache) is the biggest.
Unit testing: inbuilt.
Pyparsing, just because.
BeautifulSoup (or another good HTML parser).
hg, git, or some other nice VC.
Trac, or another bug system.
Oh, and StackOverflow if you have any questions.
Pycharm Community is worth to try.
I am a member of a team that is about to launch a beta of a python (Django specifically) based web site and accompanying suite of backend tools. The team itself has doubled in size from 2 to 4 over the past few weeks and we expect continued growth for the next couple of months at least. One issue that has started to plague us is getting everyone up to speed in terms of getting their development environment configured and having all the right eggs installed, etc.
I'm looking for ways to simplify this process and make it less error prone. Both zc.buildout and virtualenv look like they would be good tools for addressing this problem but both seem to concentrate primarily on the python-specific issues. We have a couple of small subprojects in other languages (Java and Ruby specifically) as well as numerous python extensions that have to be compiled natively (lxml, MySQL drivers, etc). In fact, one of the biggest thorns in our side has been getting some of these extensions compiled against appropriate versions of the shared libraries so as to avoid segfaults, malloc errors and all sorts of similar issues. It doesn't help that out of 4 people we have 4 different development environments -- 1 leopard on ppc, 1 leopard on intel, 1 ubuntu and 1 windows.
Ultimately what would be ideal would be something that works roughly like this, from the dos/unix prompt:
$ git clone [repository url]
...
$ python setup-env.py
...
that then does what zc.buildout/virtualenv does (copy/symlink the python interpreter, provide a clean space to install eggs) then installs all required eggs, including installing any native shared library dependencies, installs the ruby project, the java project, etc.
Obviously this would be useful for both getting development environments up as well as deploying on staging/production servers.
Ideally I would like for the tool that accomplishes this to be written in/extensible via python, since that is (and always will be) the lingua franca of our team, but I am open to solutions in other languages.
So, my question then is: does anyone have any suggestions for better alternatives or any experiences they can share using one of these solutions to handle larger/broader install bases?
Setuptools may be capable of more of what you're looking for than you realize -- if you need a custom version of lxml to work correctly on MacOS X, for instance, you can put a URL to an appropriate egg inside your setup.py and have setuptools download and install that inside your developers' environments as necessary; it also can be told to download and install a specific version of a dependency from revision control.
That said, I'd lean towards using a scriptably generated virtual environment. It's pretty straightforward to build a kickstart file which installs whichever packages you depend on and then boot virtual machines (or production hardware!) against it, with puppet or similar software doing other administration (adding users, setting up services [where's your database come from?], etc). This comes in particularly handy when your production environment includes multiple machines -- just script the generation of multiple VMs within their handy little sandboxed subnet (I use libvirt+kvm for this; while kvm isn't available on all the platforms you have developers working on, qemu certainly is, or you can do as I do and have a small number of beefy VM hosts shared by multiple developers).
This gets you out of the headaches of supporting N platforms -- you only have a single virtual platform to support -- and means that your deployment process, as defined by the kickstart file and puppet code used for setup, is source-controlled and run through your QA and review processes just like everything else.
I always create a develop.py file at the top level of the project, and have also a packages directory with all of the .tar.gz files from PyPI that I want to install, and also included an unpacked copy of virtualenv that is ready to run right from that file. All of this goes into version control. Every developer can simply check out the trunk, run develop.py, and a few moments later will have a virtual environment ready to use that includes all of our dependencies at exactly the versions the other developers are using. And it works even if PyPI is down, which is very helpful at this point in that service's history.
Basically, you're looking for a cross-platform software/package installer (on the lines of apt-get/yum/etc.) I'm not sure something like that exists?
An alternative might be specifying the list of packages that need to be installed via the OS-specific package management system such as Fink or DarwinPorts for Mac OS X and having a script that sets up the build environment for the in-house code?
I have continued to research this issue since I posted the question. It looks like there are some attempts to address some of the needs I outlined, e.g. Minitage and Puppet which take different approaches but both may accomplish what I want -- although Minitage does not explicitly state that it supports Windows. Lacking any better options I will try to make either one of these or just extensive customized use of zc.buildout work for our needs, but I still feel like there must be better options out there.
You might consider creating virtual machine appliances with whatever production OS you are running, and all of the software dependencies pre-built. Code can be edited either remotely, or with a shared folder. It worked pretty well for me in a past life that had a fairly complicated development environment.
Puppet doesn't (easily) support the Win32 world either. If you're looking for a deployment mechanism and not just a "dev setup" tool, you might consider looking into ControlTier (http://open.controltier.com/) which has a open-source cross-platform solution.
Beyond that you're looking at "enterprise" software such as BladeLogic or OpsWare and typically an outrageous pricetag for the functionality offered (my opinion, obviously).
A lot of folks have been aggressively using a combination of Puppet and Capistrano (even non-rails developers) for deployment automation tools to pretty good effect. Downside, again, is that it's expecting a somewhat homogeneous environment.