The large corporation that I work for uses a custom version of Setuptools. This private fork of setuptools is intended to deal with certain networking and security difficulties that are unique to our organization. The bottom line is that neither the standard Setuptools nor Distribute would work as expected on our environment.
Id like to start using Ian Bicking's excellent VirtualEnv tool on systems, particularly in our test systems where we need to be able to set up a large number of sandboxed areas for test-code - e.g. in our continuous integration environment.
Unfortunately any time I try to build a new virtual environment the virtualenv tool tries to obtain and install the latest official version of Setuptools. This would fail for the reason stated above, and also because the corporate firewall would block the action.
Instead of installing the official version:
setuptools-0.6c11-py2.4.egg
I'd like to install our customized version which might be called something like:
setuptools-foo-0.6c11-py2.4.egg
This egg can always be guaranteed to be found in the system's global site-packages. I can also guarantee that it's present in all of our corporate egg servers.
Can you help me make my virtualenv use my customized setuptools instead of the regular version of setuptools.
The name is hardcoded in virtualenv.py. You have to either patch virtualenv.py or name your patched setuptools egg 'setuptools-0.6c11-py2.4.egg'
I've taken to writing my own wrapper scripts which import virtualenv. The main reason is that I use dpkgs to install most of my dependencies, including distribute, so I like to avoid downloading additional copies when I create a new environment - this has a bonus that it runs much faster.
Here is a baseline wrapper you can use to start with. I've added a comment where you could insert some code to symlink/copy your custom setuptools code into the virtualenv:
import os, subprocess, sys, virtualenv
# virtualenv changed its internal api slightly after 1.5.
NEW_API = (1, 5)
def get_version(version):
return tuple([int(v) for v in version.split('.')])
def main():
# set the logging level here
level = virtualenv.Logger.level_for_integer(0)
logger = virtualenv.Logger([(level, sys.stdout)])
virtualenv.logger = logger
# insert your command-line parsing code here, if needed
root = sys.argv[1]
home, lib, inc, bin = virtualenv.path_locations(root)
result = virtualenv.install_python(home, lib, inc, bin,
site_packages=True, clear=False)
pyexec = os.path.abspath(result)
version = get_version(virtualenv.virtualenv_version)
if version < NEW_API:
virtualenv.install_distutils(lib, home)
else:
virtualenv.install_distutils(home)
virtualenv.install_activate(home, bin)
# insert whatever post-virtualenv-setup code you need here
if __name__ == '__main__':
main()
Usage:
% python wrapper.py [path]
There's the option --extra-search-dir that allows to define a local directory containing the desired version of setuptools. This is explained in the docs.
Related
On a Linux server, I have some Python scripts using the built-in sqlite3 module (+ some Sqlite extensions built from source, as detailed in Upgrade Python's sqlite3 on Debian).
For another Python script, I need a newer version of the Sqlite shared library than the one I already have on the system. Reason: I need Sqlite higher than 3.25.0 for Window Functions.
If I install it from source here and do make and make install, it will probably overwrite previous versions of this library on the server, and could potentially break other OS tools using it.
How do you handle the general problem of having multiple versions of the Sqlite shared library?
I don't think Python virtual environments can be used for this context, or would it be possible?
Note: pip3 install --upgrade sqlite3 does not exist: we cannot upgrade Python's built-in sqlite3 package like this. And by the way we probably should not, since it could break some OS tools using Python + sqlite3.
This is very tricky and will need a little code change in your scripts.
What to do:
First, check the sqlite3 library version included with python just in case:
python -c "import sqlite3; print(sqlite3.connect(':memory:').execute('SELECT sqlite_version();').fetchall())
In my computer (python 3.8, windows) the output is [('3.35.5',)] which means python has the sqlite 3.35.5 library. I have no sqlite installed in my system: this is the library that comes with python3.8.
IF your python sqlite3 library is not the one you need :-( you have an alternative: you can use the pysqlite3 instead of the sqlite3 standard library. In this case:
You'll need to build the pysqlite3 library by yourself using the Sqlite3 'amalgamation' that matches the version you want to use (more on later).
You'll need to install the library, and...
You will need to change your python script imports
import pysqlite3 as sqlite3 # instead of sqlite3
Ok, what is the 'amalgamation` and how to build pysqlite3?
The amalgamation is the whole sqlite3 library in just one .c file (with the sqlite3.h file). You can get it from the sqlite3 download page: sqlite3.36 amalgamation.
Once you have the amalgamation, follow the instructions to build statically pysqlite3, and install the package.
Now you can use pysqlite3 in your code.
If you want 2 different version of sqlite3 (python3) on 2 different environments, you can do that.
Since you mentioned that sqlite3 is part of the std library, it seems like you can try the pysqlite3 package instead.
If you can't run pip, run the following command first.
sudo apt install python3-pip
Then,
pip install virtualenv
python3 -m venv sqlitev1 #(whatever name you want)
source sqlitev1/bin/activate
pip install pysqlite3==0.4.4 #(this can be whatever version u want)
source deactivate
python3 -m venv sqlitev2 #(whatever name you want)
source sqlitev2/bin/activate
pip install pysqlite3==0.4.4 #(this can be whatever version u want)
source deactivate
Now you have 2 python environments, sqlitev1 and sqlitev2, with 2 different version of sqlite3.
It might be super hacky but you can make the new version of sqlite and then make sure that the path pointing to the new version is on the pythonpath environment before the built in one. Python will scan the python path from first to last to find an import, so the new version first in the python path for the processes that want the new version and then exclude that path with the old processes that need the built in one. You can accomplish this with a bash script that loads the env and then runs the python process for the new services.
Again this is super hacky so last resort.
If you want a different version of Sqlite than that installed with your distro, and a Python that uses that version, then you could
Compile sqlite to an alternative location
Compile Python to a different location, and point it to the custom Sqlite installation.
The "pointing" is covered in the accepted answer to this question. The question body itself shows how you might compile sqlite to a custom location.
The other answer to that question proposes setting the LD_LIBRARY_PATH environment variable to the directory containing the custom sqlite build to avoid having to compile Python. This might work with a virtualenv (it could be set in the preactive hook, for example).
See also
What is LD_LIBRARY_PATH and how to use it?
Set LD_LIBRARY_PATH before importing in python
Another approach would be to compile pysqlite3 in a virtualenv over the custom sqlite build. You can read about this in this blog post (I won't copy the details as it isn't clear what licence is used by the blog).
Is there any way to check whether a Python package has been installed normally (pip install / setup.py install) or in editable/egg-link mode (pip install -e / setup.py develop)?
I know I could check whether the path to the package contains site-packages which would most likely mean it's a "non-editable" install, but this feels extremely dirty and I would rather avoid this.
The reason I'm trying to check this is that my application is checking for config files in various places, such as /etc/myapp.conf and ~/.myapp.conf. For developers I'd like to check in <pkgdir>/myapp.conf but since I show the list of possible locations in case no config was found, I really don't want to include the pkgdir option when the package has been installed to site-packages (since users should not create a config file in there).
pip contains code for this (it's used by pip freeze to prefix the line with -e). Since pip's API is not guaranteed to be stable, it's best to copy the code into the own application instead of importing it from pip:
def dist_is_editable(dist):
"""Is distribution an editable install?"""
for path_item in sys.path:
egg_link = os.path.join(path_item, dist.project_name + '.egg-link')
if os.path.isfile(egg_link):
return True
return False
The code is MIT-licensed so it should be safe to copy&paste into pretty much any project.
So I was hoping to write my code more resiliently, starting from the top in python I was thinking of addressing importing.
I want code to run on systems where required packages haven't been installed. To achieve this I was hoping to install packages on the run with python.
try:
import pygame as pg
except(ImportError):
# [install pygame][1] here
# Download and run pygame.MSI (windows)
# apt-get install python-pygame
install pygame
From this specific solution I intend to make a more generic function ...
import subprocess as sp
def imp(inP,name,location):
try:
exec "import "+inP+" as "+(name if name != "" else "")
except ImportError:
try:
os = ????
if(os == windows):
sp.call("pip install "+location,shell=True)
if(os == unix):
sp.call("sudo apt-get install python-"+inP,shell=True)
r = True
except Exception:
print colPrt("ERROR installing ") + inP
r = False
try:
exec "import "+inP+" as "+(name if name != "" else "")
except(ImportError):
print colPrt("ERROR importing ") + inP
r = False
return r
and so my one question becomes 2. The first being the best practicing for installing modules on the run and the second being how that differs between a unix and windows environment.
Ps, colPrt simply returns red text to the terminal
def colPrt(s):
return("\x1B["+"31;40m" + str(s) + "\x1B[" + "0m")
thanks for your thoughts : )
The first being the best practicing for installing modules on the run
I would advise against installing modules on the run for a couple reasons:
Users should have a choice about whether they want to install stuff. Most users aren't interested in making that choice at runtime (and your program shouldn't be running with the necessary privileges to install stuff like that anyway)...
Follow up to the first point -- Dependencies should be installed when your package gets installed.
It'll clutter your code with a bunch of stuff that you really don't want to maintain.
Even if you do include it, whose to say that your code and the latest version of some third-party package on the web are compatible?
Python's standard build system has ways of interacting with dependencies, so when your package gets downloaded and installed (e.g. via easy_install or pip), then the dependencies should all come with it.
Take a look here for some advice on how to package your python code.
Try to look at the pip-accel, it is wrapper over pip, that is working cross platform. If you users have no python environment pre installed, I want recommend you to look at virtualenv. Pip + virtualenv works like a charm, you just have to write short python script that will run it with setups.
Anyway, If you will use your own solutions, it will be very difficult to support, maybe one day your boss come and ask you to add Mac Os support. Better to write you wrapper over pip+virtualenv than implement all logic by yourself.
For exception write out only error message and let the choice of installing modules on user. Or write it ower setup (install) wrapper.
On unix / linux platforms the python is distributed with easy_install and do not try nothing to install in background with apt-get. Without permissions would not succeed. Needless.
And we used not only Debian based linux :)
eg. my standard OS is unix based: QNX :)
When I use pip to install a package from source, it will generates a version number for the package which I can see using 'pip show '. But I can't find out how that version number is generated and I can't find the version string from the source code. Can someone tell me how the version is generated?
The version number that pip uses comes from the setup.py (if you pip install a file, directory, repo, etc.) and/or the information in the PyPI index (if you pip install a package name). (Since these two must be identical, it doesn't really matter which.)
It's recommended that packages make the same string available as a __version__ attribute on their top-level module/package(s) at runtime that they put in their setup, but that isn't required, and not every package does.
And if the package doesn't expose its version, there's really no way for you to get it. (Well, unless you want to grub through the pip data trying to figure out which package owns a module and then get its version.)
Here's an example:
In the source code for bs4 (BeautifulSoup4), the setup.py file has this line:
version = "4.3.2",
That's the version that's used, directly or indirectly, by pip.
Then, inside bs4/__init__.py, there's this line:
__version__ = "4.3.2"
That means that Leonard Richardson is a nice guy who follows the recommendations, so I can import bs4; print(bs4.__version__) and get back the same version string that pip show beautifulsoup4 gives me.
But, as you can see, they're two completely different strings in completely different files. If he wasn't nice, they could be totally different, or the second one could be missing, or named something different.
The OpenStack people came up with a nifty library named PBR that helps you manage version numbers. You can read the linked doc page for the full details, but the basic idea is that it either generates the whole version number for you out of git, or verifies your specified version number (in the metadata section of setup.cfg) and appends the dev build number out of git. (This relies on you using Semantic Versioning in your git repo.)
Instead of specifying the version number in code, tools such as setuptools-scm may use tags from version control. Sometimes the magic is not directly visible. For example PyScaffold uses it, but in the project's root folder's __init__.py one may just see:
import pkg_resources
try:
__version__ = pkg_resources.get_distribution(__name__).version
except:
__version__ = "unknown"
If, for example, the highest version tag in Git is 6.10.0, then pip install -e . will generate a local version number such as 6.10.0.post0.dev23+ngc376c3c (c376c3c being the short hash of the last commit) or 6.10.0.post0.dev23+ngc376c3c.dirty (if it has uncommitted changes).
For more complicated strings such as 4.0.0rc1, they are usually hand edited in the PKG-INFO file. Such as:
# cat ./<package-name>.egg-info/PKG-INFO
...
Version: 4.0.0rc1
...
This make it unfeasible to obtain it from within any python code.
I'm just starting out with Python, and have found out that I can import various libraries. How do I find out what libraries exist on my Mac that I can import? How do I find out what functions they include?
I seem to remember using some web server type thing to browse through local help files, but I may have imagined that!
From the Python REPL (the command-line interpreter / Read-Eval-Print-Loop), type help("modules") to see a list of all your available libs.
Then to see functions within a module, do help("posix"), for example. If you haven't imported the library yet, you have to put quotes around the library's name.
For the web server, you can run the pydoc module that is included in the python distribution as a script:
python /path/to/pydoc.py -p 1234
where 1234 is the port you want the server to run at. You can then visit http://localhost:1234/ and browse the documentation.
Every standard python distribution has these libraries, which cover most of what you will need in a project.
In case you need to find out if a library exists at runtime, you do it like this
try:
import ObscureModule
except ImportError:
print "you need to install ObscureModule"
sys.exit(1) # or something like that
You can install another library: yolk.
yolk is a python package manager and will show you everything you have added via pypi. But it will also show you site-packages added through whatever local package manager you run.
just run the Python interpeter and type the command
import "lib_name"
if it gives an error, you don't have the lib installed...else you are good to go
On Leopard, depending on the python package you're using and the version number, the modules can be found in /Library/Python:
/Library/Python/2.5/site-packages
or in /Library/Frameworks
/Library/Frameworks/Python.framework/Versions/Current/lib/python2.6/site-packages
(it could also be 3.0 or whatever version)...
I guess it is quite the same with Tiger
Considering that in every operating system most of python's packages are installed using 'pip' (see pip documentation) you can also use the command 'pip freeze' on a terminal to print a list of all the packages you have installed through it.
Other tools like 'homebrew' for macOS (used when for some reason you can't install a package using pip) have similar commands, in this specific case 'brew list'.