I need help with the next situation. There is one project, that is requiring two versions of one library. Let this lib be lib, and its versions: libold and libnew. These libs are not accessible via pypi, i.e. they are each in their own folder. Let the paths of these folders be /path/to/libold and /path/to/libnew.
In my project I need instances of classes from both these libs, but I can't import them both, but only either old or new lib.
I tried the next method:
import sys
sys.path.insert(0,'path/to/libold')
import lib as libold
sys.path.pop(0)
sys.path.insert(0,'path/to/libnew')
import lib as libnew
After performing this commands, libold and libnew represents the same library, libold.
I also tried importlib and imp and got same result.
How can I perform importing 2 versions of a lib?
Python adds imported modules to sys.modules. When you write import lib as libnew, sys.modules['lib'] already exists, and therefore the new lib is not imported.
To import the new lib, you should delete the old one from sys.modules, like this:
import sys
sys.path.insert(0, 'path/to/libold')
import lib as libold
sys.path.pop(0)
del sys.modules['lib']
sys.path.insert(0, 'path/to/libnew')
import lib as libnew
However, you may encounter serious problems by doing so. In particular, if the old lib tries to import a submodule (say, e.g. lib.submodule), it will get the new one instead. For this reason, you'd better import all submodules of the old lib before deleting sys.modules['lib'] and before importing the new one.
However, that's a dirty hack, not a real solution. In Python, modules and packages are identified by name, not by path. This is how it works and has always worked, and there's nothing you can do about it.
Consider using multiprocessing instead to overcome these "limitations". With multiprocessing you can run two processes: one that uses the old lib and the other that uses the new one. multiprocessing gives you many tools to make interprocess communication easy.
Related
I would like to import a module from inside a functions. For example from this:
from directory.folder.module import module
def import():
app.register_blueprint(module)
To this:
def import():
from directory.folder.module import module
But, without hardcoding it. For example:
def import():
m = "module"
from directory.folder.m import m
Is it possible? Thanks in advance
You want the importlib module.
Here's the most simplistic way to use this module. There are lots of different ways of weaving the results of calls to the module into the environment:
import importlib
math = importlib.import_module("math")
print(math.cos(math.pi))
Result:
-1.0
I've used this library a lot. I built a whole plug-in deployment system with it. Scripts for all the various deploys were dropped in directories and only imported when they were mentioned in a config file rather than everything having to be imported right away.
Something I find very cool about this module is what's stated at the very top of its documentation:
The purpose of the importlib package is two-fold. One is to provide the implementation of the import statement (and thus, by extension, the import() function) in Python source code.
The intro in the 2.7 docs is interesting as well:
New in version 2.7.
This module is a minor subset of what is available in the more full-featured package of the same name from Python 3.1 that provides a complete implementation of import. What is here has been provided to help ease in transitioning from 2.7 to 3.1.
No, python import does not work this way.
Such as an example you try to import a module named mod, so you run import mod. Now interpreter will search for mod.py in a list of directories gathered from the following sources:
The directory from where the input script was run or the current directory if the interpreter is being run interactively.
The list of directories contained in the PYTHONPATH environment variable, if it is set. (The format for PYTHONPATH is OS-dependent but should mimic the PATH environment variable.)
An installation-dependent list of directories configured at the time Python is installed.
So if you have a variable named m='mod' and run import m it will search for m.py not mod.py.
But just a silly dangerous workaround is to use exec() (WARNING FOR MALICIOUS INPUT)
m = "module"
exec(f'from directory.folder.m import {m}')
If you don't mind external modules try importlib.
You can use the importlib module to programmatically import modules.
import importlib
full_name = "package." + "module"
m = importlib.import_module(full_name)
I want to install updated packages in another directory and have Python grab the updated package instead of the older package.
I'm trying to find a way that I can specify which directory to import for when there are multiple identical packages in sys.path.
I started by running this code to make sure that the path for the second module is present:
import sys
print('\n'.join(sys.path))
Both paths are shown, so I know that Python could find the package from either location.
I run this to see what path Python is using:
import statsmodels
print(statsmodels.__file__)
It is using the path of the out of date version.
I've been looking into using importlib but I haven't figured out how to make that work.
I'm just looking for a way to import a package from a specified path, even when the package exists in another directory in sys.path.
as discussed in the commend you'd neeed to implement this solution. With that to further explain what it does, it points to another folder to consider files to import. Considering the mentioned code:
# some_file.py (this is this script you're running)
import sys
sys.path.insert(0, '/path/to/application/app/folder')
import file_name_inside_the_folder_above
You'd let the first argument 0 untouched and just edit the second argument, pointing to which folder the script you have access is. Then you just import as the it's file name.
This import works fine, but feels dirty in a few ways. Mainly that it uses a specific number in the slice* to get the parent path, and that it annoys the flake8 linter.
import os
import sys
sys.path.append(os.path.dirname(__file__)[:-5])
from codeHelpers import completion_message
It's in a file system that looks a bit like this:
parent_folder
__init__.py
codeHelpers.py
child_folder
this_file.py
(child_folder is actually called week1, hence the 5 in the slice)
This question is extremely similar to Python import from parent directory, but in that case the discussion focused on whether or not it was good to run tests from the end point. In my case, I have a series of directories that have code that uses helpers that live in the parent.
Context: each directory is a set of weekly exercises, so I'd like to keep them as simple as possible.
Is there a cleaner, more pythonic way to do this import?
#cco solved the number problem, but it's still upsetting the linter.
First since you haven't been specific about which lint error you are getting, I am going to assume it's because you have an import after your sys.path.append.
The cleanest way to do it is with relative or absolute imports.
Using absolute imports:
from parent_path.codeHelpers import completion_message
Using relative imports:
from ..codeHelpers import completion_message
For the simple example listed in the original question this should be all that's required. It's simple, it's pythonic, it's reliable, and it fixes the lint issue.
You may find yourself in a situation where the above does not work for you and sys.path manipulation is still required. A drawback is that your IDE will likely not be able to resolve imports to modules from the new path causing issues such as automatic code completion not working and flagging the imports as errors, even though the code will run properly.
If you find you still need to use sys.path and want to avoid lint errors for this type of situation create a new module and do the sys.path manipulation in it instead. Then make sure that you import your new module before any modules that require the modified sys.path.
For example:
local_imports.py
"""Add a path to sys.path for imports."""
import os
import sys
# Get the current directory
current_path = os.path.dirname(__file__)
# Get the parent directory
parent_path = os.path.dirname(current_path)
# Add the parent directory to sys.path
sys.path.append(parent_path)
Then in the target file:
import local_imports # now using modified sys.path
from codeHelpers import completion_message
The drawback to this is it requires you to include local_imports.py in each child_folder and if the folder structure changes, you would have to modify each one local_imports file.
Where this pattern is really useful is when you need to include external libraries in your package (for example in a libs folder) without requiring the user to install the libs themselves.
If you are using this pattern for a libs folder, you may want to make sure your included libraries are preferred over the installed libraries.
To do so, change
sys.path.append(custom_path)
to
sys.path.insert(1, custom_path)
This will make your custom path the second place the python interpreter will check (the first will still be '' which is the local directory).
You can import from a module a level up in a package by using ... In this_file.py:
from ..codeHelpers import completion_message
Had you wanted to go more levels up just keep adding dots...
While I'm here, just be aware that from ..codeHelpers is a relative import, and you should always use them when importing something in the same package. from codeHelpers is an absolute import, which are ambiguous in Python 2 (should it import from in the package or from the unfortunately named codeHelpers module you have installed on your system?), and in Python 3 actually forbidden as a way to import from within the same module (i.e. they are always absolute). You can read the ancient PEP 328 for an explanation of the difficulties.
It might be easier to use absolute import paths, like the following:
from parent_folder.code_helpers import completion_message
But this would require you to make sure that the PYTHONPATH environment variable is set such that it can see the highest root directory (parent_folder in this case, I think). For instance,
PYTHONPATH=. python parent_directory/child_directory/this_file.py
# here the '.' current directory would contain parent_directory
Make sure to add an __init__.py to the child_directory as well.
You can remove the assumption about the length of the final directory name by applying os.path.dirname twice.
e.g. instead of os.path.dirname(__file__)[:-5], use os.path.dirname(os.path.dirname(__file__))
Either way you have to hack around. If you main goal avoid flakes warnings
add a noqa comment
exec(open("../codeHelpers.py").read(), globals())
you can pass a filename with interpreter option -c (should not bother flakes8)
I was believing that when importing a package, it would search from sys.path and use the first hit for import. However, it seems not true:
import mpl_toolkits
print(mpl_toolkits.__path__)
And it outputs:
['/Library/Python/2.7/site-packages/matplotlib-1.5.0-py2.7-macosx-10.11-intel.egg/mpl_toolkits', '/usr/local/lib/python2.7/site-packages/mpl_toolkits']
Can someone please explain to me how exactly python looks for packages if it is installed multiple times in the machine (in different location searchable by sys.path)? Or a pointer to relevant reference would be good.
when you import a module, python uses PYTHON PATH (system variable that contains a list of folders) and loops to search for importable module.
Python will test if it is a package (folder containing init.py) or a module (*.py). it will stop on first module found if no module is found python raises an import error
Python 2.7 comes with json library included. In my PYTHONPATH I include third party sources and one of them is also called json. The result ending up with loaded the wrong json library. What would be a good practice to handle and avoid situations like above? Is there a way to instruct Python to explicitly load the native library in this fashion from ? import json.
You could try
from path import json as anotherjson
This way the namespace conflict can be removed.
Also you can see the discussions about relative/absolute import.
http://docs.python.org/whatsnew/2.5.html#pep-328
It says :
In Python 2.5, you can switch import‘s behaviour to absolute imports
using a from future import absolute_import directive. This
absolute- import behaviour will become the default in a future version
(probably Python 2.7). Once absolute imports are the default, import
string will always find the standard library’s version. It’s suggested
that users should begin using absolute imports as much as possible.
from __future__ import absolute_import
# from standard path
import json as _json
# from a package
from pkg import json as pkgjson
The other technique is to use the imp module
import imp
json = imp.load_source('json', '/path/to/json.py')
There really is no good way to have multiple modules with the same name on PYTHONPATH[docs], this means that you should probably move the third party json module to an alternate location that is not on PYTHONPATH, and then import it using some other method.
The easiest way to do this is to move the third party json module into a subdirectory of the location it is already in, and then make that subdirectory a module by adding __init__.py to it.
If you named this new directory 'thirdparty', you could then import your third party json module using from thirdparty import json, and import json would always import Python's json module.
Alternatively, you could rename the module to something that does not conflict.