I just wrote a python script to fix some database issues and it uses the psycopg2 module. For some reason the person that needs to run it (on the server) is claiming that they can't install psycopg2 on their server machine... is there a way that I can package that module within my script such that they don't have to have psycopg2 installed? Something similar to adding a lib folder to the classpath in Java?
Thx in advance,
Andre
Make a directory, put your script into it.
cd into that directory, then run:
$ easy_install -Z --prefix . psycopg2
That should result in a copy of the python module in the same directory.
Zip it all up and send the whole thing off.
As to be unable to install psycopg2, I sure it relates to security policies about who can have root access on database servers. If you can't find someone with the necessary permissions, there are two methods to skin this particular cat:
1) You could build an exe using py2exe, that way you know that all of the dependencies for your scripts are there and available. Also this will allow to you escape dealing with any python incompatibilities. You made no mention of your python version (2.5, 2.6, 2.7, 3.1?) nor the version on the server.
2) Or you could place the psycopg2 module into its own directory (c:\mypymods) and add the path to sys.path before importing.
import sys
sys.path = sys.path.append("c:/mypymods")
import psycopg2
....
Related
On a Linux server, I have some Python scripts using the built-in sqlite3 module (+ some Sqlite extensions built from source, as detailed in Upgrade Python's sqlite3 on Debian).
For another Python script, I need a newer version of the Sqlite shared library than the one I already have on the system. Reason: I need Sqlite higher than 3.25.0 for Window Functions.
If I install it from source here and do make and make install, it will probably overwrite previous versions of this library on the server, and could potentially break other OS tools using it.
How do you handle the general problem of having multiple versions of the Sqlite shared library?
I don't think Python virtual environments can be used for this context, or would it be possible?
Note: pip3 install --upgrade sqlite3 does not exist: we cannot upgrade Python's built-in sqlite3 package like this. And by the way we probably should not, since it could break some OS tools using Python + sqlite3.
This is very tricky and will need a little code change in your scripts.
What to do:
First, check the sqlite3 library version included with python just in case:
python -c "import sqlite3; print(sqlite3.connect(':memory:').execute('SELECT sqlite_version();').fetchall())
In my computer (python 3.8, windows) the output is [('3.35.5',)] which means python has the sqlite 3.35.5 library. I have no sqlite installed in my system: this is the library that comes with python3.8.
IF your python sqlite3 library is not the one you need :-( you have an alternative: you can use the pysqlite3 instead of the sqlite3 standard library. In this case:
You'll need to build the pysqlite3 library by yourself using the Sqlite3 'amalgamation' that matches the version you want to use (more on later).
You'll need to install the library, and...
You will need to change your python script imports
import pysqlite3 as sqlite3 # instead of sqlite3
Ok, what is the 'amalgamation` and how to build pysqlite3?
The amalgamation is the whole sqlite3 library in just one .c file (with the sqlite3.h file). You can get it from the sqlite3 download page: sqlite3.36 amalgamation.
Once you have the amalgamation, follow the instructions to build statically pysqlite3, and install the package.
Now you can use pysqlite3 in your code.
If you want 2 different version of sqlite3 (python3) on 2 different environments, you can do that.
Since you mentioned that sqlite3 is part of the std library, it seems like you can try the pysqlite3 package instead.
If you can't run pip, run the following command first.
sudo apt install python3-pip
Then,
pip install virtualenv
python3 -m venv sqlitev1 #(whatever name you want)
source sqlitev1/bin/activate
pip install pysqlite3==0.4.4 #(this can be whatever version u want)
source deactivate
python3 -m venv sqlitev2 #(whatever name you want)
source sqlitev2/bin/activate
pip install pysqlite3==0.4.4 #(this can be whatever version u want)
source deactivate
Now you have 2 python environments, sqlitev1 and sqlitev2, with 2 different version of sqlite3.
It might be super hacky but you can make the new version of sqlite and then make sure that the path pointing to the new version is on the pythonpath environment before the built in one. Python will scan the python path from first to last to find an import, so the new version first in the python path for the processes that want the new version and then exclude that path with the old processes that need the built in one. You can accomplish this with a bash script that loads the env and then runs the python process for the new services.
Again this is super hacky so last resort.
If you want a different version of Sqlite than that installed with your distro, and a Python that uses that version, then you could
Compile sqlite to an alternative location
Compile Python to a different location, and point it to the custom Sqlite installation.
The "pointing" is covered in the accepted answer to this question. The question body itself shows how you might compile sqlite to a custom location.
The other answer to that question proposes setting the LD_LIBRARY_PATH environment variable to the directory containing the custom sqlite build to avoid having to compile Python. This might work with a virtualenv (it could be set in the preactive hook, for example).
See also
What is LD_LIBRARY_PATH and how to use it?
Set LD_LIBRARY_PATH before importing in python
Another approach would be to compile pysqlite3 in a virtualenv over the custom sqlite build. You can read about this in this blog post (I won't copy the details as it isn't clear what licence is used by the blog).
I have my own package in python and I am using it very often. what is the most elegant or conventional directory where i should put my package so it is going to be imported without playing with PYTHONPATH or sys.path?
What about site-packages for example?
/usr/lib/python2.7/site-packages.
Is it common in python to copy and paste the package there ?
I usually put the stuff i want to have ready to import in the user site directory:
~/.local/lib/pythonX.X/site-packages
To show the right directory for your platform, you can use python -m site --user-site
edit: it will show up in sys.path once you create it:
mkdir -p "`python -m site --user-site`"
So if your a novice like myself and your directories are not very well organized you may want to try this method.
Open your python terminal. Import a module that you know works such as numpy in my case and do the following.
Import numpy
numpy.__file__
which results in
'/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site- packages/numpy/__init__.py'
The result of numpy.__file__ is the location you should put the python file with your module (excluding the numpy/__init__.py) so for me that would be
/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site- packages
To do this just go to your terminal and type
mv "location of your module" "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site- packages"
Now you should be able to import your module.
This is something that works for me (I have to frequently create python packages that are uploaded to a private pip repository). elaborating on the comment by #joran on the question.
create a "build directory" which is used as a workspace to create packages. any directory of your choice will do
Copy your python package dir there, and create a setup.py file. this should help in creating the setup.py correctly.
create a virtualenv for the project you are working on. virtualenvs have a bunch of other benefits, I am not going into the details here.
create a local dist package python setup.py sdist --format=tar. the package created should ideally be in the dist folder.
Install the package on your virtualenv (after activating it). pip install <yourpackage>.tar
you can use pip install --force-reinstall if you need to play around with the libraries more and re-create the dist packages.
I've found that this method works great for me. If you do not need to package the modules for use of other systems instead of just your local, this method might be an overkill
Happy hacking.
If you're developing a module I would recommend to follow this.
import sys
sys.path.append("/home/mylinux/python-packages")
Now any module you keep in python-packages is importable by Python-interpreter.
On my Mac, I did a sudo find / -name "site-packages". That gave me a few paths like /Library/Python/2.6/site-packages, /Library/Python/2.7/site-packages, and /opt/X11/lib/python2.6/site-packages.
So, I knew where to put my modules if I was using v2.7 or v2.6.
Hope it helps.
import folders could be extracted by adding following source code:
import sys
print sys.path
automatic symlink generation example would be:
ln -s \`pip show em | grep "Location"| cut -d " " -f2\` \`python -m site --user-site\`
instead of "em" you may use other package you've "just installed but the python can't see it"
below I'll explain in more details as being requested in the comment.
suppose you've installed python module em or pyserial with the following command (examples are for ubuntu):
sudo pip install pyserial
and the output is like this:
Collecting pyserial
Downloading pyserial-3.3-py2.py3-none-any.whl (189kB)
100% |████████████████████████████████| 194kB 2.3MB/s
Installing collected packages: pyserial
Successfully installed pyserial-3.3
the question would be following - python can't see the module pyserial, why?
because the location where the module has been installed isn't the one python is looking at for your particular user account.
solution - we have to create symlink from the path where pyserial arrived to the path where your python is looking for.
symlink creation command would be:
ln -s <what_to_link> <where_to_link>
instead of typing exact location we are asking pip to tell us where it stored modules by executing command:
pip show pyserial | grep "Location"| cut -d " " -f2
instead of typing exact location we are asking python to tell us where it looks for the modules being installed by executing command:
python -m site --user-site
both commands has to be escaped with "`" character (usually on the left of your 1 button for the US keyboards)
in result following command will be provided for ln and the missing symlink would be created:
ln -s /usr/local/lib/python2.7/dist-packages /home/<your_username>/.local/lib/python2.7/site-packages
or something similar, depending on your distro and python/pip defaults.
I'm struggling with executing a python script on my freenas (freebsd) environment.
I created a Jail, where I installed python via
pkg install python
and tried to execute the program with the command
python filename.py
But now its mentioning that it requires a specific module
ImportError: No module named simplejson
which I also installed via
pkg install ...
the next attempt to execute the script mentioned a different module.
Is it really the case for python, that you have to install each module per request from an executed program? Or is there a way to determine which module the 3rd party program needs and install it in upfront?
And how do you search for the corresponding module in the repo? Because for the missing sqlite3 module I had to write
pkg install databases/python-sqlite3
How do I get now the correct name for the simplejson module which is mentioned before?
Can you help me out there?
The first questions where already answered in the comments section. Here is the answer to the last question:
FreeBSD ports of python modules usually follow the convention of "py-" + modulename. The corresponding packages are usually named "py"+ python version + "-" + modulename
So it is py-sqlite3 and py-simplejson ports which install "py27-simplejson" and py27-sqlite3 for python 2.7.
Note: currently modules are only packaged for the default python version, for non-default python versions the modules have to be installed from the ports collection.
Recently I found about this tool easy_install that help me to easy install additional python modules. The problem is that for each module it creates additional *.egg folder (sometime there is only an egg file?) (no source?) and I don't know how to setup eclipse paths.
By default I have included C:\Python26\Lib\site-packages and this is enough when I install python modules from source... but not when I'm using easy_intall
For example django instaled with easy_install is located in C:\Python26\Lib\site-packages\django-1.2.5-py2.6.egg\django and installed from source it's located in C:\Python26\Lib\site-packages\django
In fact when I'm using easy_install all installed modules are working without a problem, the only problem is that eclipse can't locate where is the source and gives me a false unresolved import errors
Where I'm wrong?
I'm assuming that eclipse does not search the egg files for source. Eggs, like jar files in Java, are just zipfiles of python code with some included metadata.
You'll also note that in site-packages you've got easy-install.pth and setuptools.pth files. Those files are parsed by python and used to add other directories and egg files to your PYTHONPATH (import sys; sys.path) so that Python can find the code in those locations. Eclipse isn't seeing those imports as valid because it is most likely not setup to take pth files into account.
To get Eclipse to recognize that Django is really installed you may want to try removing your easy_installed django package and reinstalling it with:
easy_install --always-unzip django
That way rather than installing a compressed egg file you'll have a normal package directory that eclipse should have a fairly easy time opening.
Alternatively, in your screenshot above it looks like you may just need to explicitly add each egg file you want eclipse to use.
The question is related to the answer to "Unable to install Python without sudo access".
I need to install python-setuptools to install python modules.
I have extracted the installation package.
I get the following error when configuring
[~/wepapps/pythonModules/setuptools-0.6c9]# ./configure --prefix=/home/masi/.local
-bash: ./configure: No such file or directory
I did not find the solution at the program's homepage.
How can I resolve this error?
As Noah states, setuptools isn't an automake package so doesn't use ‘./configure’. Instead it's a pure-Python-style ‘setup.py’ (distutils) script.
You shouldn't normally need to play with .pydistutils.cfg, as long as you run it with the right version of Python. So if you haven't added the .local/bin folder to PATH, you'd have to say explicitly:
/home/masi/.local/bin/python setup.py install
AIUI this should Just Work.
I did not find the solution at the program's homepage.
Yeah, they want you to install it from a shell script egg which uses the default version of Python. Which you don't want.
(Another approach if you can't get setuptools to work is to skip it and install each module and dependency manually. Personally I have a bit of an aversion to setuptools/egg, as it contains far too much “clever” magic for my tastes and makes a mess of my filesystem. But I'm an old curmudgeon like that. Most Python modules can be obtained as simple Python files or plain old distutils scripts, but unfortunately there are some that demand eggs.)
You might want to check http://peak.telecommunity.com/DevCenter/EasyInstall#custom-installation-locations.
EasyInstall is a python module with some shell scripts (or some shell scripts with a python module?) and does not use the unix make tool that gets configured with the "./configure" command. It looks like your best bet is to try editing ~/.pydistutils.cfg to include:
[install]
install_lib = /home/masi/.local/lib/python/site-packages/
install_scripts = /home/masi/.local/bin
You'll also presumably have made the ~/.local/bin/ folder part of your PATH so you can run the easy_install script. (I'm not sure exactly where the site-packages directory will be under .local, but it shouldn't be hard to find.)
Hope this helps.