I am working on Windows (sadface) with Python and virtualenv.
I would like to have setup and teardown scripts that go along with the virtualenv activation/deactivation. But I am not sure if these hooks have already been designated, and if so, where?
I guess I could hack the activate.bat, but then what if I use activate.py instead (does activate.py call activate.bat, or must I hack both files)? I can almost get away with environment variable PYTHONSTARTUP, but this needs to be redefined in each virtualenv. So unless virtualenv allows arbitrary assignment of env-vars, I am back to an activation/deactivation hook to set PYTHONSTARTUP (which really defeats the purpose, but now you see my catch-22).
EDIT: I plan to use my virtualenv to host interactive development sessions. I will be calling 'venv/bin/activate.bat' manually from the terminal. I do not want loose Batch/Powershell scripts laying around that I have to remember to call once when I activate, and once again when I deactive. I want to hook the execution in such a way, so that after I add my custom scripting hooks, 6 months later I don't have to remember how it works. I just execute activate.bat, and I am off to the races.
Many problems lessened or solved with virtualenvwrapper-win. Well written framework, with simple entry points.I spend a lot of time fighting with windows, trying to get a functional python work environment. This is one of the those programs I really wish I knew about a long time ago.
Does not handle multiple python installations extraordinarily (or switching between them), but the project owner also developed another supporting product, pywin, meant to augment that particular shortcoming.
The whole point is, that it makes Windows command-line development quite a bit smoother, even if its not all the automation I dream about.
Related
I need to integrate a body of Python code into an existing OSGi (Apache Felix) deployment.
I assume, or at least hope, that packages exist to help with this effort.
If it helps, the Python code is still relatively new and small, so can probably re re-architected to meet whatever constraints are needed. However, it must remain in Python, because of dependencies on third-party libraries.
What are suggested best practices?
The trick is to make this an extender, see 1 and 2. You want your Python code to be separate from the code that handles the interaction with the interpreter. So what you do is wrap the Python code and any native libraries in a bundle. This is trivial since it is just a zip file.
You then develop a bundle that listens to starting bundle (see the BundleTracker) that have python code. A manifest is often used but you can also look in a directory in the JAR. If you detect this code, you extract any native libraries and run the code in the interpeter of your choice.
If can use JYthon then that would be highly recommended. You can then carry the interpreter as an OSGi bundle that runs on the VM. If you need to use a native compiler your life is less rosy. You can rely on the environment to provide you with an interpreter but then why use OSGi in the first place. You basically lose the write once run anywhere advantage. You could go the full monty by creating bundles that contain Python installers for all platforms you support. Can be done, not even that hard, but a maintenance nightmare. Believe me, native code suck, it only does it a bit faster than Java.
I have written some Python scripts in Eclipse where at some point I have stable releases which should be deployed. For the moment this is very simple and means copying the scripts to some directory. (Soon I will also consider Git)
Could you tell me the common extendable way to set up a process like this in Eclipse? Maybe something more professional which, if a more complicating deployment becomes neccessary, can still be used.
(I had seen Ant somewhere, but I'm not sure if that is more than I need and maybe not the convention for Python in particular).
I would say use Fabric, and read this blog post for a reasonable introduction to it.
The Story
After cleaning up my Dreamhost shared server's home folder from all the cruft accumulated over time, I decided to start afresh and compile/reinstall Python.
All tutorials and snippets I found seemed overly simplistic, assuming (or ignoring) a bunch of dependencies needed by Python to compile all modules correctly. So, starting from http://andrew.io/weblog/2010/02/installing-python-2-6-virtualenv-and-virtualenvwrapper-on-dreamhost/ (so far the best guide I found), I decided to write a set-and-forget Bash script to automate this painful process, including along the way a bunch of other things I am planning to use.
The Script
I am hosting the script on http://bitbucket.org/tmslnz/python-dreamhost-batch/src/
The TODOs
So far it runs fine, and does all it needs to do in about 900 seconds, giving me at the end of the process a fully functional Python / Mercurial / etc... setup without even needing to log out and back in.
I though this might be of use for others too, but there are a few things that I think it's missing and I am not quite sure how to go for it, what's the best way to do it, or if this just doesn't make any sense at all.
Check for errors and break
Check for minor version bumps of the packages and give warnings
Check for known dependencies
Use arguments to install only some of the packages instead of commenting out lines
Organise the code in a manner that's easy to update
Optionally make the installers and compiling silent, with error logging to file
failproof .bashrc modification to prevent breaking ssh logins and having to log back via FTP to fix it
EDIT: The implied question is: can anyone, more bashful than me, offer general advice on the worthiness of the above points or highlight any problems they see with this approach? (see my answer to Ry4an's comment below)
The Gist
I am no UNIX or Bash or compiler expert, and this has been built iteratively, by trial and error. It is somehow going towards apt-get (well, 1% of it...), but since Dreamhost and others obviously cannot give root access on shared servers, this looks to me like a potentially very useful workaround; particularly so with some community work involved.
One way to streamline this would be to make it work with one of: capistrano/fabric, puppet/chef, jhbuild, or buildout+minitage (and a lot of cmmi tasks). There are some opportunities for factoring in common code, especially with something more high-level than bash. You will run into bootstrapping issues, however, so maybe leave good enough alone.
If you want to look into userland package managers, there is autopackage (bootstraps well), nix (quickstart), and stow (simple but helps with isolation).
Honestly, I would just build packages with a name prefix for all of the pieces and have them install under /opt so that they're out of the way. That way it only takes the download time and a bit of install time to do.
What is the best way to organize and develop a project composed of many small scripts sharing one (or more) larger Python libraries?
We have a bunch of programs in our repository that all use the same libraries stored in the same repository. So in other words, a layout like
trunk
libs
python
utilities
projects
projA
projB
When the official runs of our programs are done, we want to record what version of the code was used. For our C++ executables, things are simple because as long as the working copy is clean at compile time, everything is fine. (And since we get the version number programmatically, it must be a working copy, not an export.) For Python scripts, things are more complicated.
The problem is that, often one project (e.g. projA) will be running, and projB will need to be updated. This could cause the working copy revision to appear mixed to projA during runtime. (The code takes hours to run, and can be used as inputs for processes that take days to run, hence the strong traceability goal.)
My current workaround is, if necessary, check out another copy of the trunk to a different location, and run off there. But then I need to remember to change my PYTHONPATH to point to the second version of lib/python, not the one in the first tree.
There's not likely to be a perfect answer. But there must be a better way.
Should we be using subversion keywords to store the revision number, which would allow the data user to export files? Should we be using virtualenv? Should we be going more towards a packaging and installation mechanism? Setuptools is the standard, but I've read mixed things about it, and it seems designed for non-developer end users (of which we have none).
The much better solution involves not storing all your projects and their shared dependencies in the same repository.
Use one repository for each project, and externals for the shared libraries.
Make use of tags in the shared library repositories, so consumer projects may use exactly the version they need in their external.
Edit: (just copying this from my comment) use virtualenv if you need to provide isolated runtime environments for the different apps on the same server. Then each environment can contain a unique version of the library it needs.
If I'm understanding your question properly, then you definitely want virtualenv. Add in some virtualenvwrapper goodness to make it that much better.
Is it possible to create an environment to safely run arbitrary Python scripts under Linux? Those scripts are supposed to be received from untrusted people and may be too large to check them manually.
A very brute-force solution is to create a virtual machine and restore its initial state after every launch of an untrusted script. (Too expensive.)
I wonder if it's possible to restrict Python from accessing the file system and interacting with other programs and so on.
Consider using a chroot jail. Not only is this very secure, well-supported and tested but it also applies to external applications you run from python.
There are 4 things you may try:
As you already mentioned, using a virtual machine or some other form of virtualisation (perhaps solaris zones are lightweight enough?). If the script breaks the OS there then you don't care.
Using chroot, which puts a shell session into a virtual root directory, separate from the main OS root directory.
Using systrace. Think of this as a firewall for system calls.
Using a "jail", which builds upon systrace, giving each jail it's own process table etc.
Systrace has been compromised recently, so be aware of that.
You could run jython and use the sandboxing mechanism from the JVM. The sandboxing in the JVM is very strong very well understood and more or less well documented. It will take some time to define exactly what you want to allow and what you dnt want to allow, but you should be able to get a very strong security from that ...
On the other side, jython is not 100% compatible with cPython ...
Try searching for "sandboxing python", e.g.:
http://wiki.python.org/moin/SandboxedPython
http://wiki.python.org/moin/How%20can%20I%20run%20an%20untrusted%20Python%20script%20safely%20(i.e.%20Sandbox)
could you not just run as a user which has no access to anything but the scripts in that directory?