Develop Reference

How to distribute and or package a python file with all dependencies pre-downloaded without building an executable

I have written a python script which depends on paramiko to work. The system that will run my script has the following limitations:
No internet connectivity (so it can't download dependencies on the fly).
Volumes are mounted with 'noexec' (so I cannot run my code as a binary file generated with something like 'pyInstaller')
End-user cannot be expected to install any dependencies.
Vanilla python is installed (without paramiko)
Python version is 2.7.5
Pip is not installed either and cannot be installed on the box
I however, have access to pip on my development box (if that helps in any way).
So, what is the way to deploy my script so that I am able to provide it to the end-user with the required dependencies, i.e paramiko (and sub-dependencies of paramiko), so that the user is able to run the script out-of-the-box?
I have already tried the pyinstaller 'one folder' approach but then faced the 'noexec' issue. I have also tried to directly copy paramiko (and sub-dependencies of paramiko) to a place from where my script is able to find it with no success.

pip is usually installed with python installation. You could use it to install the dependencies on the machine as follows:
import os, sys
def selfInstallParamiko():
# assuming paramiko tar-ball/wheel is under the current working directory
currentDir = os.path.abspath(os.path.dirname(__file__))
# paramikoFileName: name of the already downloaded paramiko tar-ball,
# which you'll have to ship with your scripts
paramikoPath = os.path.join(currentDir, paramikoFileName)
print("\nInstalling {} ...".format(paramikoPath))
# To check for which pip to use (pip for python2, pip3 for python3)
pipName = "pip"
if sys.version_info[0] >= 3:
pipName = "pip3"
p = subprocess.Popen("{} install --user {}".format(pipName, paramikoPath).split())
out, err= p.communicate("")
if err or p.returncode != 0:
print("Unable to self-install {}\n".format(paramikoFileName))
sys.exit(1)
# Needed, because sometimes windows command shell does not pick up changes so good
print("\n\nPython paramiko module installed successfully.\n"
"Please start the script again from a new command shell\n\n")
sys.exit()
You can invoke it when your script starts and an ImportError occurs:
# Try to self install on import failure:
try:
import paramiko
except ImportError:
selfInstallParamiko()

cx_Oracle error. DPI-1047: Cannot locate a 64-bit Oracle Client library

I installed the library and when trying to access SQL in jupyter notebook with my credentials the following error appears:
DatabaseError: DPI-1047: Cannot locate a 64-bit Oracle Client library: "The specified module could not be found". See https://oracle.github.io/odpi/doc/installation.html#windows for help

The easiest solution is as follows:
Download 64-bit version of oracle instantClient from: https://www.oracle.com/database/technologies/instant-client/winx64-64-downloads.html
Copy the dll files in the instantclient directory to the python directory, as shown below
That is it!

The short answer is: cx_Oracle.init_oracle_client(lib_dir= r"c:\path_to_libraries")
Here are the steps that I followed to solve this same issue:
If you have not already installed cx_Oracle you can do so with the following command:
python -m pip install cx_Oracle --upgrade
The cx_Oracle documentation can be found here.
Use the following commands to verify that everything is installed and recognized:
import sqlalchemy as sqla
import pandas as pd
import cx_Oracle
# Test to see if it will print the version of sqlalchemy
print(sqla.__version__) # this returns 1.2.15 for me
# Test to see if the cx_Oracle is recognized
print(cx_Oracle.version) # this returns 8.0.1 for me
# This fails for me at this point but will succeed after the solution described below
cx_Oracle.clientversion()
At this point I get errors saying the libraries cannot be located. Here is the solution:
import os
import platform
# This is the path to the ORACLE client files
lib_dir = r"C:\put_your_path_here\instantclient-basic-windows.x64- 19.9.0.0.0dbru\instantclient_19_9"
# Diagnostic output to verify 64 bit arch and list files
print("ARCH:", platform.architecture())
print("FILES AT lib_dir:")
for name in os.listdir(lib_dir):
print(name)
Be sure to update the lib_dir path specific to your installation. If you have the correct path, you should see a listing of all the Oracle files like: (adrci.exe, oci.dll, oci.sym, etc). This is the location that Python needs to be able to find the Oracle drivers.
The current (Nov 2020) standard way for passing the location of the Oracle libraries for Windows is cx_Oracle.init_oracle_client(lib_dir= r"c:\path_to_libraries"). Here is an example:
lib_dir = r"C:\put_your_path_here\instantclient-basic-windows.x64- 19.9.0.0.0dbru\instantclient_19_9"
try:
cx_Oracle.init_oracle_client(lib_dir=lib_dir)
except Exception as err:
print("Error connecting: cx_Oracle.init_oracle_client()")
print(err);
sys.exit(1);
At this point I can run the following code without any errors:
# This works after passing the lib_dir path
cx_Oracle.clientversion() # For me it returns: (19, 9, 0, 0, 0)
DEPRECATED Here is how to update the PATH variable temporarily:
The following works, but using cx_Oracle.init_oracle_client(lib_dir= r"c:\path_to_libraries") is now the preferred way.
import os
# Manually append the location of the ORACLE libraries to the PATH variable
os.environ["PATH"] = lib_dir + ";" + os.environ["PATH"]

As per documentation accessed on cx_Oracle page.
Step 1: install cx_Oracle
python -m pip install cx_Oracle --upgrade
Step 2: Download and extract Oracle Basic Client
For Windows download and extract Oracle Basic Instatnt client instantclient-basic-windows.x64-19.9.0.0.0dbru.zip file.
Step 3: Inform cx_Oracle module about the Instatnt Client location.
If you stick to documentation and extract them in c:\oracle folder then your script may look like this.
import cx_Oracle
cx_Oracle.init_oracle_client(lib_dir=r"C:\oracle\instantclient_19_9")
Now you will be free from the error.

I faced this error in Anconda Spyder.
This is how I fixed it.
Get the basic package instantClient from: https://www.oracle.com/database/technologies/instant-client/winx64-64-downloads.html
Extract and copy all the *.dll files and paste it to Anaconda3 folder where you have python.exe.

For MAC
After you do : python -m pip install cx_Oracle --upgrade
try:
import cx_Oracle
# Test to see if the cx_Oracle is recognized
print(cx_Oracle.version) # this returns 8.0.1 for me
# This fails for me at this point but will succeed after the solution described below
cx_Oracle.clientversion()
if you encounter issues like :
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
cx_Oracle.DatabaseError: DPI-1047: Cannot locate a 64-bit Oracle Client library: "dlopen(libclntsh.dylib, 1): image not found". See https://cx-oracle.readthedocs.io/en/latest/user_guide/installation.html for help
Follow the step highlighted here: https://cx-oracle.readthedocs.io/en/latest/user_guide/installation.html#manual-installation
Manual Installation
Download the Basic 64-bit DMG from Oracle.
(https://www.oracle.com/database/technologies/instant-client/macos-intel-x86-downloads.html)
Mount the dng
Open the mounted dng, and run ./install_ic.sh (via terminal)
e.g. : cd /Volumes/instantclient-basic-macos.x64-19.8.0.0.0dbru/ && ./install_ic.sh
In Finder, eject the mounted Instant Client package.
Re-run the oracle connection.
import cx_Oracle
cx_Oracle.init_oracle_client(lib_dir="/Users/priyank.pathak/Downloads/instantclient_19_8")
## cx_Oracle.init_oracle_client(lib_dir="/Users/your_username/Downloads/instantclient_19_8")
cx_Oracle.clientversion()

Make sure you have installed the correct oracle client (you can find the oracle client package from here "https://www.oracle.com/in/database/technologies/instant-client/winx64-64-downloads.html" and add the downloaded folder inside the folder where python is installed and then add this location (location of your client package folder) to system's environment variable.
Hope that will work.

I suggest you to first check the compatibility of your OS, Python and Oracle Instant Client architecture:
import platform
platform.architecture()
Then, I definitely advise you to set the Oracle Instant Client inside your jupyter notebook:
import os
os.environ["PATH"] = "Complete Location of Instant Client Folder" + ";" + os.environ["PATH"]

Probably you have installed cx_Oracle Python library. In order to execute the DB connectivity through jupyter notebook, you need to install the Oracle Client and that is what missing in your scenario. Please follow the steps from below link and and install Oracle Client and this will solve your problem:
https://oracle.github.io/odpi/doc/installation.html#windows

I went into the same problem but on Arch Linux, running in NodeJS.
Some of the answers here helped me in searching for the official installation guide.
Then I followed these steps and they worked for me:
https://oracle.github.io/node-oracledb/INSTALL.html#instzip , hope it could help someone else too.
Download Instant Client Basic Light Package (ZIP) (Linux)
https://www.oracle.com/database/technologies/instant-client/linux-x86-64-downloads.html
Create folder and unzip into it
mkdir /opt/oracle
unzip ~/Downloads/instantclient-basiclite-linux.x64-21.4.0.0.0dbru.zip -d /opt/oracle/
Then, according to the docs, execute these commands below:
If there is no other Oracle software on the machine that will be impacted, then permanently add Instant Client to the run-time link path. For example, if the Basic package unzipped to /opt/oracle/instantclient_19_11, then run the following using sudo or as the root user
sudo sh -c "echo /opt/oracle/instantclient_19_11 > /etc/ld.so.conf.d/oracle-instantclient.conf"
sudo ldconfig
OR
Alternatively, every shell running Node.js will need to have the link path set:
export LD_LIBRARY_PATH=/opt/oracle/instantclient_19_11:$LD_LIBRARY_PATH

This might help someone:
I also had the same error and I read through the installation documentation https://cx-oracle.readthedocs.io/en/latest/user_guide/installation.html (I was using a zip for installation so I focused on section: Oracle Instant Client Zip Files)
And I was missing a "yum install -y libaio" (used 'centos:7' base image) after unzipping the oracle client to /opt/oracle and also added the following envs:
ENV ORACLE_HOME /opt/oracle/instantclient_19_3
ENV DYLD_LIBRARY_PATH /opt/oracle/instantclient_19_3
ENV LD_LIBRARY_PATH /opt/oracle/instantclient_19_3
ENV TNS_ADMIN /opt/oracle/instantclient_19_3

I just installed Sql Developer from Oracle and ran it for the first time, and the install added what was needed and I was able to get through that error. This was on windows. Install below, just extract it when it installs and run the .exe in the main folder. Much simpler than the other solutions!
https://www.oracle.com/database/sqldeveloper/technologies/download/

You just need to put the Oracle Instant Client folder in your PATH environment variable. Remember to restart whatever application that needs the instant client after updating the PATH variable.
If you installed the instant client at: c:\oracle\instantclient_21_8, simply put this folder in your PATH environment variable.
Copying the instant client files to Python folder is a very bad idea, since if you have other python versions installed in the future you will have to remember duplicating de instant client files. If you upgrade the instant client, you will need to keep these duplicated files updated too.

pdfminer - ImportError: No module named pdfminer.pdfdocument

I am trying to install pdfMiner to work with CollectiveAccess. My host (pair.com) has given me the following information to help in this quest:
When compiling, it will likely be necessary to instruct the
installation to use your account space above, and not try to install
into the operating system directories. Typically, using "--
home=/usr/home/username/pdfminer" at the end of the install command
should allow for that.
I followed this instruction when trying to install.
The result was:
running install
running build
running build_py
running build_scripts
running install_lib
running install_scripts
changing mode of /usr/home/username/pdfminer/bin/latin2ascii.py to 755
changing mode of /usr/home/username/pdfminer/bin/pdf2txt.py to 755
changing mode of /usr/home/username/pdfminer/bin/dumppdf.py to 755
running install_egg_info
Removing /usr/home/username/pdfminer/lib/python/pdfminer-20140328.egg-info
Writing /usr/home/username/pdfminer/lib/python/pdfminer-20140328.egg-info
I don't see anything wrong with that (I'm very new to python), but when I try to run the sample command $ pdf2txt.py samples/simple1.pdf I get this error:
Traceback (most recent call last): File "pdf2txt.py", line 3, in <module>
from pdfminer.pdfdocument import PDFDocument ImportError: No module named pdfminer.pdfdocument
I'm running python 2.7.3. I can't install from root (shared hosting). The most recent version of pdfminer, which is 2014/03/28.
I've seen some posts on similar issues ("no module named. . . " but nothing exactly the same. The proposed solutions either don't help (such as installing with sudo - not an option; specifying the path for python (which doesn't seem to be the issue), etc.).
Or is this a question for my host? (i.e., something amiss or different about their setup)

I had an error like this:
No module named 'pdfminer.pdfinterp'; 'pdfminer' is not a package
My problem was that I had named my script pdfminer.py which for the reasons that I don't know, Python took it for the original pdfminer package files and tried to compiled it.
I renamed my script to something else, deleted all the *.pyc file and __pycache__ directory and my problem was solved.

use this command worked for me and removed the error
pip install pdfminer.six

Since the package pdfminer is installed to a non-standard/non-default location, Python won't be be able to find it. In order to use it, you will need to add it to your 'pythonpath'. Three ways:
At run time, put this in your script pdf2txt.py:
import sys
# if there are no conflicting packages in the default Python Libs =>
sys.path.append("/usr/home/username/pdfminer")
or
import sys
# to always use your package lib before the system's =>
sys.path.insert(1, "/usr/home/username/pdfminer")
Note: The install path specified with --home is used as the Lib for all packages which you might want to install, not just this one. You should delete that folder and re-install with --
home=/usr/home/username/myPyLibs (or any generic name) so that when you install other packages with that install path, you would only need the one path to add to your local Lib to be able to import them:
import sys
sys.path.insert(1, "/usr/home/username/myPyLibs")
Add it to PYTHONPATH before executing your script:
export PYTHONPATH="${PYTHONPATH}:/usr/home/username/myPyLibs"
And then put that in your ~/.bashrc file (/usr/home/username/.bashrc) or .profile as applicable. This may not work for programs which are not executed from the console.
Create a VirtualEnv and install the packages you need to that.

I have a virtual environment and I had to activate it before I did a pip3 install to have the venv see it.
source ~/venv/bin/activate

How do I get ibm_db for Node.js to work with DB2 in Mac OS X 10.10?

Currently I'm trying to use the ibm_db node.js module to connect my application with DB2 as I did it with Windows (my past workstation's OS). I followed some guides about Mac OS X and ibm_db python module just in case, but it also doesn't work. ibm_db got to install but when I make the first database interaction it throws an error :
dyld: lazy symbol binding failed: Symbol not found: _SQLAllocHandle
Referenced from: /Users/devniel/dev/Pon.de.tu.parte/node_modules/ibm_db/build/Release/odbc_bindings.node
Expected in: dynamic lookup
dyld: Symbol not found: _SQLAllocHandle
Referenced from: /Users/devniel/dev/Pon.de.tu.parte/node_modules/ibm_db/build/Release/odbc_bindings.node
Expected in: dynamic lookup
I posted an issue on Github, and they explain me that the reason is the lack of a DS Driver for Mac OS X, so, Is there a way to get it works such as the python guys does ? (I will try it tomorrow).

If you have DB2 Express-C installed on your Mac (i.e. the server), then you already have the client libraries that you need. When you go to build ibm_db, you can point the IBM_DB_HOME environment variable to the DB2 instance owner's sqllib directory. For example:
export IBM_DB_HOME=/Users/db2inst1/sqllib
NOTE: ibm_db will build and work, although the current release has an issue with the bindings.gyp file that prevent it from building on Mac OS X (it is configured to build ONLY on Linux). You can add the following Condition to bindings.gyp to get it to build with node-gyp:
[ 'OS == "mac" and target_arch =="x64" ', {
'libraries' : [
'-L$(IBM_DB_HOME)/lib -L$(IBM_DB_HOME)/lib64 ',
'-ldb2'
],
'include_dirs': [
'$(IBM_DB_HOME)/include'
],
'cflags' : [
"-g "
],
}]

Loading matplotlib on cygwin

To load matplotlib on cygwin, I have:
Loaded pre-requisites using cygwin 64-bit setup: pkg-config, freetype2, libfreetype-devel, libpng-devel, gtk2.0, libgtk2.0-devel
Downloaded the matplotlib tar file (http://sourceforge.net/projects/matplotlib/files/matplotlib/matplotlib-1.3.1/matplotlib-1.3.1.tar.gz) and changed the source code to get around the "_tri" error as advised here:
matplotlib error while installing pyspeckit
Then built and installed matplotlib:
$ python setup.py build
$ python setup.py install
And am down to what looks like a matplotlib backend error. Does anyone know how to get around this:
/usr/lib/python2.7/site-packages/gtk-2.0/gtk/init.py:57: GtkWarning: could not open display
...
cursors.MOVE : gdk.Cursor(gdk.FLEUR),
et n RuntimeError: could not create GdkCursor object
EDIT: I just finally got it and matplotlib is finally working on cygwin. To do this, I did:
From cygwin setup, loaded the X-Server tools:
xorg-server xinit
From cygwin setup, I also loaded these so that use telnet or ssh connections to run remote X clients:
inetutils openssh
I set my display:
DISPLAY=":0.0"
export DISPLAY
From the cygwin shell, I did:
$ startxwin
Then I ran my python scripts which use matplotlib in the X-window

I had an issue with "python setup.py build" not finding the ft2build.h which comes from the the freetype2 package. I installed the freetype2 development package and I can find it in /usr/include/freetype2/ft2build.h but the error is still there.
After digging into the setupext.py at the function check_include_file(include_dirs, filename, package), I noticed that the package="freetype2" was not concatenated into to the search path. Hence it could not find "/usr/include/ft2build.h" when it should be "/usr/include/freetype2/ft2build.h
Fixing this line #134
if not has_include_file(include_dirs, "%s/%s"%(package,filename)):