Develop Reference

How to fix Entry Point Not Found while import package from Torch-sparse

How to Fix Entry Point Not Found while installing libraries in conda environment
Here I followed the topic above with quite same problem with amount of different details.
First, this is problem I found, indicated there are something wrong with _version_cpd.pyd file
when import torch-sparse into jupyter notebook.
import numpy as np
import pandas as pd
import networkx as nx
import torch
import torch.nn.functional as F
import torch.nn as nn
import torch_scatter
from torch_geometric.data import Data
print(torch.__version__)
Details on errors reveals that could have some problem with loading dll path:
File ~\Miniconda3\envs\torchenv\lib\site-packages\torch_scatter\__init__.py:16, in <module>
14 spec = cuda_spec or cpu_spec
15 if spec is not None:
---> 16 torch.ops.load_library(spec.origin)
17 elif os.getenv('BUILD_DOCS', '0') != '1': # pragma: no cover
18 raise ImportError(f"Could not find module '{library}_cpu' in "
19 f"{osp.dirname(__file__)}")
File ~\Miniconda3\envs\torchenv\lib\site-packages\torch\_ops.py:110, in _Ops.load_library(self, path)
105 path = torch._utils_internal.resolve_library_path(path)
106 with dl_open_guard():
107 # Import the shared library into the process, thus running its
108 # static (global) initialization code in order to register custom
109 # operators with the JIT.
--> 110 ctypes.CDLL(path)
111 self.loaded_libraries.add(path)
File ~\Miniconda3\envs\torchenv\lib\ctypes\__init__.py:374, in CDLL.__init__(self, name, mode, handle, use_errno, use_last_error, winmode)
371 self._FuncPtr = _FuncPtr
373 if handle is None:
--> 374 self._handle = _dlopen(self._name, mode)
375 else:
376 self._handle = handle
So after review the above topic, I found that the problem quite the same in which need to fix 2 files and make anaconda (my case is mini one) distribution.
I think this is due to the conflict of 2 same files from particular environment and general.
The first location:
The second location:
So I might know the symptom but I dont know how to fix this (or whether I should do the same solution as topic on the top).
I use python 3.8, torch 1.10.2, None Cuda.

I figure out the reason causing this problem due to problem of installing pytorch, torch-scatter and torch-sparse for cuda version, which is outside the enviroment.
This leads to the conflict with same file of cpu version inside of environment.
I tried with new environment, not worked. Delete the installation of those package with conda, not worked.
After figure out the problem, which is listed in the threat:
https://github.com/pyg-team/pytorch_geometric/issues/3430
My step is: deleted current environment, uninstall those mentioned package (check with both pip and conda command) out of environment
Install with new environment from side:
https://pytorch.org/ (for pytorch)
conda install pytorch torchvision torchaudio cpuonly -c pytorch
https://pytorch-geometric.readthedocs.io/en/latest/notes/installation.html
pip install torch-scatter torch-sparse torch-cluster torch-spline-conv torch-geometric -f https://data.pyg.org/whl/torch-1.12.0+cpu.html
Make sure you are using same version of pytorch and same version of cude (or only cpu in my case)

How to distribute and or package a python file with all dependencies pre-downloaded without building an executable

I have written a python script which depends on paramiko to work. The system that will run my script has the following limitations:
No internet connectivity (so it can't download dependencies on the fly).
Volumes are mounted with 'noexec' (so I cannot run my code as a binary file generated with something like 'pyInstaller')
End-user cannot be expected to install any dependencies.
Vanilla python is installed (without paramiko)
Python version is 2.7.5
Pip is not installed either and cannot be installed on the box
I however, have access to pip on my development box (if that helps in any way).
So, what is the way to deploy my script so that I am able to provide it to the end-user with the required dependencies, i.e paramiko (and sub-dependencies of paramiko), so that the user is able to run the script out-of-the-box?
I have already tried the pyinstaller 'one folder' approach but then faced the 'noexec' issue. I have also tried to directly copy paramiko (and sub-dependencies of paramiko) to a place from where my script is able to find it with no success.

pip is usually installed with python installation. You could use it to install the dependencies on the machine as follows:
import os, sys
def selfInstallParamiko():
# assuming paramiko tar-ball/wheel is under the current working directory
currentDir = os.path.abspath(os.path.dirname(__file__))
# paramikoFileName: name of the already downloaded paramiko tar-ball,
# which you'll have to ship with your scripts
paramikoPath = os.path.join(currentDir, paramikoFileName)
print("\nInstalling {} ...".format(paramikoPath))
# To check for which pip to use (pip for python2, pip3 for python3)
pipName = "pip"
if sys.version_info[0] >= 3:
pipName = "pip3"
p = subprocess.Popen("{} install --user {}".format(pipName, paramikoPath).split())
out, err= p.communicate("")
if err or p.returncode != 0:
print("Unable to self-install {}\n".format(paramikoFileName))
sys.exit(1)
# Needed, because sometimes windows command shell does not pick up changes so good
print("\n\nPython paramiko module installed successfully.\n"
"Please start the script again from a new command shell\n\n")
sys.exit()
You can invoke it when your script starts and an ImportError occurs:
# Try to self install on import failure:
try:
import paramiko
except ImportError:
selfInstallParamiko()

How to check if a module is installed in Python and, if not, install it within the code?

I would like to install the modules 'mutagen' and 'gTTS' for my code, but I want to have it so it will install the modules on every computer that doesn't have them, but it won't try to install them if they're already installed. I currently have:
def install(package):
pip.main(['install', package])
install('mutagen')
install('gTTS')
from gtts import gTTS
from mutagen.mp3 import MP3
However, if you already have the modules, this will just add unnecessary clutter to the start of the program whenever you open it.

EDIT - 2020/02/03
The pip module has updated quite a lot since the time I posted this answer. I've updated the snippet with the proper way to install a missing dependency, which is to use subprocess and pkg_resources, and not pip.
To hide the output, you can redirect the subprocess output to devnull:
import sys
import subprocess
import pkg_resources
required = {'mutagen', 'gTTS'}
installed = {pkg.key for pkg in pkg_resources.working_set}
missing = required - installed
if missing:
python = sys.executable
subprocess.check_call([python, '-m', 'pip', 'install', *missing], stdout=subprocess.DEVNULL)
Like #zwer mentioned, the above works, although it is not seen as a proper way of packaging your project. To look at this in better depth, read the the page How to package a Python App.

you can use simple try/except:
try:
import mutagen
print("module 'mutagen' is installed")
except ModuleNotFoundError:
print("module 'mutagen' is not installed")
# or
install("mutagen") # the install function from the question

If you want to know if a package is installed, you can check it in your terminal using the following command:
pip list | grep <module_name_you_want_to_check>
How this works:
pip list
lists all modules installed in your Python.
The vertical bar | is commonly referred to as a "pipe". It is used to pipe one command into another. That is, it directs the output from the first command into the input for the second command.
grep <module_name_you_want_to_check>
finds the keyword from the list.
Example:
pip list| grep quant
Lists all packages which start with "quant" (for example "quantstrats"). If you do not have any output, this means the library is not installed.

You can check if a package is installed using pkg_resources.get_distribution:
import pkg_resources
for package in ['mutagen', 'gTTS']:
try:
dist = pkg_resources.get_distribution(package)
print('{} ({}) is installed'.format(dist.key, dist.version))
except pkg_resources.DistributionNotFound:
print('{} is NOT installed'.format(package))
Note: You should not be directly importing the pip module as it is an unsupported use-case of the pip command.
The recommended way of using pip from your program is to execute it using subprocess:
subprocess.check_call([sys.executable, '-m', 'pip', 'install', 'my_package'])

Although #girrafish's answer might suffice, you can check package installation via importlib too:
import importlib
packages = ['mutagen', 'gTTS']
[subprocess.check_call(['pip', 'install', pkg])
for pkg in packages if not importlib.util.find_spec(pkg)]

You can use the command line :
python -m MyModule
it will say if the module exists
Else you can simply use the best practice :
pip freeze > requirements.txt
That will put the modules you've on you python installation in a file
and :
pip install -r requirements.txt
to load them
It will automatically you purposes
Have fun

Another solution it to put an import statement for whatever you're trying to import into a try/except block, so if it works it's installed, but if not it'll throw the exception and you can run the command to install it.

You can run pip show package_name
or for broad view use pip list
Reference

If you would like to preview if a specific package (or some) are installed or not maybe you can use the idle in python.
Specifically :
Open IDLE
Browse to File > Open Module > Some Module
IDLE will either display the module or will prompt an error message.
Above is tested with python 3.9.0

How to properly install PySide into relocatable virtualenv?

I'm pip installing PySide into a virtualenv:
sudo virtualenv --always-copy $VENV
source $VENV/bin/activate
sudo $VENV/bin/pip install pyside
sudo $VENV/bin/python $VENV/bin/pyside_postinstall.py -install
deactivate
The virtualenv works great and I can run my Python/PySide script without problems:
$VENV/bin/python $SCRIPT
Now, I make my virtualenv relocatable:
virtualenv --relocatable $VENV
I move it:
mv $VENV $VENV_RELOCATED
...and again run my script:
source $VENV_RELOCATED/bin/activate
$VENV_RELOCATED/bin/python $SCRIPT
This time around, I'm getting an error:
Traceback (most recent call last):
File "script.py", line 1, in <module>
from PySide import QtCore, QtGui, QtUiTools
ImportError: dlopen(/absolute/path/to/relocated_venv/lib/python2.7/site-packages/PySide/QtCore.so, 2): Library not loaded: #rpath/libpyside-python2.7.1.2.dylib
Referenced from: /absolute/path/to/relocated_venv/lib/python2.7/site-packages/PySide/QtCore.so
Reason: image not found
These are the contents of the script:
from PySide import QtCore, QtGui, QtUiTools
Please note: the absolute path printed twice in the error message is the $VENV_RELOCATED
This is all on OS X 10.11.0 with Python 2.7.10 and PySide 1.2.2.
Question: How should I properly install PySide into a relocatable virtualenv?

Turns out that the next PySide release (probably tagged "1.2.3" when released) has this fixed: https://github.com/PySide/PySide/issues/129
However, I also made PySide 1.2.2 work inside a relocatable virtualenv on OS X 10.11 (El Capitan). For this, I modified the localize_libpaths() function in the $VIRTUALENV/bin/pyside_postinstall.py script.
Patch details: If a virtualenv is detected when the script is executed, it will not use #rpath. Instead it will use #executable_path and search relatively from the python binary in the virtualenv. Since the PySide module is always installed in the same place within a virtualenv, I thought this should work in all scenarios.
def localize_libpaths(libpath, local_libs, enc_path=None):
""" Set rpaths and install names to load local dynamic libs at run time
Use ``install_name_tool`` to set relative install names in `libpath` (as
named in `local_libs` to be relative to `enc_path`. The default for
`enc_path` is the directory containing `libpath`.
Parameters
----------
libpath : str
path to library for which to set install names and rpaths
local_libs : sequence of str
library (install) names that should be considered relative paths
enc_path : str, optional
path that does or will contain the `libpath` library, and to which the
`local_libs` are relative. Defaults to current directory containing
`libpath`.
"""
if enc_path is None:
enc_path = abspath(dirname(libpath))
install_names = osx_get_install_names(libpath)
need_rpath = False
for install_name in install_names:
if install_name[0] in '/#':
continue
if hasattr(sys, 'real_prefix'):
# virtualenv detected - use #executable_path
back_tick('install_name_tool -change %s #executable_path/../lib/python2.7/site-packages/PySide/%s %s' %
(install_name, install_name, libpath))
else:
# not a virtualenv - use #rpath
back_tick('install_name_tool -change %s #rpath/%s %s' %
(install_name, install_name, libpath))
need_rpath = True
if need_rpath and enc_path not in osx_get_rpaths(libpath):
back_tick('install_name_tool -add_rpath %s %s' %
(enc_path, libpath))
On Windows, there are no issues with relocating a virtualenv with PySide in it. However, For Linux I have not managed to find a solution. I guess waiting for PySide 1.2.3 is the way to go here.

python with yum api - installing package

Hello i'm using the following code to install package through yum api , using pythongs cript i need to build some installation based on this code , now its installed but im getting some errors
#!/usr/bin/python
import sys
import platform
import urllib2, urllib
import re
import yum
package="ntp"
print ("Installing ntp")
print ("#################")
yb=yum.YumBase()
searchlist=['name']
arg=['ntp']
matches = yb.searchGenerator(searchlist,arg)
for (package, matched_value) in matches :
if package.name == 'ntp' : yb.install(package)
yb.buildTransaction()
yb.processTransaction()
errors i got after installation is done
Running rpm_check_debug
Traceback (most recent call last):
File "./test.py", line 29, in <module>
yb.processTransaction()
File "/usr/lib/python2.6/site-packages/yum/__init__.py", line 4928, in processTransaction
self._doTestTransaction(callback,display=rpmTestDisplay)
File "/usr/lib/python2.6/site-packages/yum/__init__.py", line 5027, in _doTestTransaction
raise Errors.YumTestTransactionError, errstring
yum.Errors.YumTestTransactionError: Test Transaction Errors: package ntp is already installed
even when i removed the ntp and run the script again its give me this error msg after finished installation
plus i want to adjust the installation process , to check if the package is already installed then print its already install and process to next step in the code , else process the installation steps ,
any tips also for for if condition in correct way using yum api

Hi the previous answer didn't work the correct one is the following:
import yum
yb=yum.YumBase()
inst = yb.rpmdb.returnPackages()
installed=[x.name for x in inst]
packages=['bla1', 'bla2', 'bla3']
for package in packages:
if package in installed:
print('{0} is already installed'.format(package))
else:
print('Installing {0}'.format(package))
kwarg = {
'name':package
}
yb.install(**kwarg)
yb.resolveDeps()
yb.buildTransaction()
yb.processTransaction()

You need to first check if the pacakge is installed, if so, then skip it:
yb.conf.cache = 1 # enable cache
installed = yb.rpmdb.returnPackages()
packages = ['a','b','c']
for package in packages:
if package in installed:
print('{0} is already installed, skipping...'.format(package))
else:
print('Installing {0}'.format(package))
yb.install(package)
yb.resolveDeps()
yb.buildTransaction()
yb.processTransaction()