I am trying to get a list of the imported custom modules (modules I created by myself) in my current file but I could not find a proper way to achieve it.
For example:
<test.py>
import sys
import foo
import bar
I would like to get a list of [foo, bar] excluding the sys module.
Lets say, that if file located near curent file, or in some subfolder, when it's "our" module, not system one.
For this example, I've created three files: main.py, other.py and another.py, and placed whem into one folder.
The code of main.py is:
# os and sys are needed to work
import sys
import os
import shutil
import io
import datetime
import other
import another
def get_my_modules():
# Get list of modules loaded
modules = list(sys.modules.keys())
mymodules = []
# Get current dir
curdir = os.path.realpath(os.path.dirname(__file__))
for m in modules:
try:
# if some module's file path located in current folder or in some subfolder
# lets sey, that it's our self-made module
path = sys.modules[m].__file__
if path.startswith(curdir):
mymodules.append(m)
except Exception:
# Exception could happen if module doesn't have any __file__ property
pass
# Return list of our moudles
return mymodules
print(get_my_modules())
And this code actually outputs ["other", "another"]
This approach has a problem: If you import module, that somehow located in upper folder, it woudln't be detected.
Related
I'm writing a python script to import a setting file from another project.
This is the structure of the project:
- root
- ...
- folder_1
- setting_folder
- __init__.py
- setting_1.py
- setting_2.py
- setting_3.py
Here the file contents:
init.py
from .setting_1 import *
from .setting_2 import *
setting_1.py
foo = "foo1"
setting_2.py
foo = "foo2"
try:
from .setting_3 import *
except ImportError:
pass
setting_3.py
foo = "foo3"
What I need to do is, from a script outside the project, load setting_2.py and get the value of foo variable (foo3 due to relative import).
Suppose that I run my script from directory "C:\Users\bar\Desktop".
My idea to achieve this goal is to copy setting_2.py in another directory outside the project (let's say a), create an empty file init.py in a, append to PYTHONPATH "C:\Users\bar\Desktop" and then import the module.
Here the code:
import os
import importlib.util
with open(setting_2_path, "r") as f:
test_file_content = f.read()
setting_tmp_path = r"C:\Users\bar\Desktop\a\setting_2.py"
with open(setting_tmp_path, "w") as f:
f.write(test_file_content)
init_tmp_path = r"C:\Users\bar\Desktop\a\__init__.py"
with open(init_tmp_path, "w") as f:
f.write("")
current_env = os.environ.copy()
current_env.update({'PYTHONPATH': r"C:\Users\bar\Desktop"})
spec_module = importlib.import_module('a.setting_2')
print(getattr(spec_module, "foo"))
This is working well, but in production this script will be in another project and I cannot create a folder at the same level of the script.
I can create the folder but in another directory.
To simulate this scenario, suppose that I run the script from C:\Users\bar\Desktop\bar2 and the folder is C:\Users\bar\Desktop\a.
In this case I have the following error:
ImportError: No module named 'a'
How can I fix the problem?
import os
import sys
cwd = os.getcwd() # This is current working directory, as in your assumption it is: C:\Users\bar\Desktop\bar2
path_to_settings_files = os.path.join(os.path.dirname(cwd), 'a') # This is where settings are, as in your assumption it is: C:\Users\bar\Desktop\a
sys.path.append(path_to_settings_files) # This line guides interpreter to find settings.py in loading modules
import setting_1
import setting_2
import setting_3
print(setting_1.foo)
print(setting_2.foo)
print(setting_3.foo)
I have created a package with the following file structure:
- package
- __init__.py
- load.py
- train.py
- test.py
My __init__.py file is simply an import of classes for these files:
from package.load import Load
from package.train import Train
from package.test import Test
Most of the time, I want to load all three, however on occasion I only want to load one of these classes specifically. For example in an ad hoc script (outside of the package) I want to be able to call only the Load class like so:
from package import Load
While all of the above works in this design, I have an issue where dependencies from train/test are also loaded when I import Load like the above. How can I setup the __init__.py file such that I can make the same import call without getting the dependency to load from train/test?
Additional explanation:
Why I am doing this: I have an issue where I want some people to be able to use the Load class, which only uses base python, however the Train/Test files include specialized dependencies which users of just the Load class wont want to utilize or even install.
Here's a way to do something very close what you want. Instead of unconditionally importing all the package's classes in your __init__.py, you can define a function in it to explicitly import any of the ones desired (or all of them if none are specified).
__init__.py:
from pathlib import Path
import sys
print(f'In {Path(__file__).name}')
package_name = Path(__file__).parent.name
package_prefix = package_name + '.'
class_to_module_map = {'Load': 'load', 'Train': 'train', 'Test': 'test'}
def import_classes(*class_names):
namespace = sys._getframe(1).f_globals # Caller's globals.
if not class_names:
class_names = class_to_module_map.keys() # Import them all.
for class_name in class_names:
module = class_to_module_map[class_name]
temp = __import__(package_prefix+module, globals(), locals(), [class_name])
namespace[class_name] = getattr(temp, class_name) # Add to caller's namespace.
For testing purposes, here's what I put in the load.py script:
(I also put something similar in the other two modules in order to verify whether or not they were getting imported.)
load.py:
from pathlib import Path
print(f'In {Path(__file__).name}')
class Load: pass
And finally here's a example of using it to only import the Load class:
ad_hoc.py:
from my_package import import_classes
#from my_package import Load
import_classes('Load')
test = Load()
print(test)
Along with the output produced:
In __init__.py
In load.py
<my_package.load.Load object at 0x001FE4A8>
Inside a folder oranges\, this is our __init__.py file:
__all__ = []
from pathlib import Path
from importlib import import_module
from sys import modules
package = modules[__name__]
initfile = Path(__file__)
for entry in initfile.parent.iterdir():
is_file = entry.is_file()
is_pyfile = entry.name.endswith('.py')
is_not_initpy = (entry != initfile)
if is_file and is_pyfile and is_not_initpy:
module_name = entry.name.removesuffix('.py')
module_path = __name__ + '.' + module_name
module = import_module(module_path)
setattr(package, module_name, module)
__all__.append(module_name)
When we do from oranges import *, the code inside oranges\__init__.py cycles through the *.py files inside oranges\ (except __init__.py), and for each .py file does the following:
imports the .py file as a module into the variable module using Python's importlib.import_module
sets the module as a variable within the __init__.py file (or, more correctly, within the oranges package) using Python's setattr
finally, appends the module to the __all__ list
I define parameters for my programs in a python module for simplicity. These parameters are then loaded using import. Therefore, I have to make sure to always load from the working directory, and nowhere else (independent form the location of the executed script or available modules in python path).
I found two solutions. First modifying the path:
import sys
from os import getcwd
import_path = sys.path
sys.path = [str(getcwd(), ]
import xxx
sys.path = import_path
or using importlib
from pathlib import Path
from importlib.util import module_from_spec, spec_from_file_location
spec = spec_from_file_location('xxx', str(Path('.').expanduser()/'xxx.py'))
xxx = module_from_spec(spec)
spec.loader.exec_module(xxx)
Of course this can be wrapped into a context manager or a function respectively.
What would the pythonic way be to do this? Do those two approaches have advantages and disadvantages?
I checked How can I import a Python library located in the current working directory?
as well as Import python package from local directory into interpreter, but they lack the focus on robustness.
Local imports can be achieved by modifying the path. A context manager is a suitable solution:
import sys
from pathlib import Path
from contextlib import contextmanager
#contextmanager
def local_import(dir_=None):
"""Only import modules within `dir_` (default: cwd)."""
if dir_ is None:
dir_ = Path.cwd()
else:
dir_ = Path(dir_).absolute().resolve(strict=True)
import_path0 = sys.path[0]
sys.path[0] = str(dir_)
try:
yield
finally:
sys.path[0] = import_path0
Then the local import can be done using the standard import syntax
with local_import():
import xxx
This solution relys on the order, in which the paths are scanned, thus we temporarily replace sys.path[0]. We replace it, instead of prepending to avoid import conflicts with the script directory.
Note
You have to be careful to avoid name conflicts, as the import statement is used, modules with identical names will be imported only once. Thus if different modules with the same name exist in the working directory and in the original sys.path[0], only one of them will be imported. Thus, local_import should only be used for scripts, that only use the standard library or installed third party libraries, but not for scripts that import other scripts from the directory. For the unlikely case that you want to import different files with the same name, the following function can be used:
import uuid
from importlib.util import module_from_spec, spec_from_file_location
def import_file(file, content=None):
"""Try importing `file` as module avoiding name clashes.
If `content` is given `content = import_file('file.py', 'content')`
roughly corresponds to `from file import content`
else `file = import_file('file.py')`
roughly corresponds to `import file`.
Parameters
----------
file : str or Path
The Python file corresponding to the module.
content : str, optional
What to import from the module (optional).
"""
file = Path(file).expanduser().resolve(strict=True)
print(file)
spec = spec_from_file_location(file.stem + str(uuid.uuid4()), str(file))
module = module_from_spec(spec)
spec.loader.exec_module(module)
if content:
print(module)
return getattr(module, content)
else:
return module
There are multiple threads about getting the current Python's script directory, for example:
import os
dir = os.path.dirname(os.path.abspath(__file__))
The question is, what should I do if I want to add this function into some utility file, while I want the returned value to be the directory of the calling file, in this case, the file executed by the user?
What would happen here?
// utils.py
def get_script_dir():
import os
return os.path.dirname(os.path.abspath(__file__))
// main_app.py
from utils import get_script_dir
print(get_script_dir())
// user's shell
python c:\path\to\somewhere\main_app.py
Will it print the directory of main_app.py or the directory of utils.py? What is an elegant solution for placing the function in the utils file while getting the directory of the file that was actually executed by the user?
Please try following and let me know whether it's exactly what you want:
# utils.py
def get_script_dir():
import sys
import os
return os.path.dirname(sys.modules['__main__'].__file__)
# main_app.py
from utils import get_script_dir
print(get_script_dir())
Supposing that your project directory tree is like the following:
Python/
main.py
modules/
utils.py
__init__.py
Being Python/modules/utils.py:
import os
get_script_dir = lambda file: os.path.dirname(os.path.abspath(file))
and Python/main.py:
from modules import utils
import os
script_dir = utils.get_script_dir(__file__)
print("[i] Currently executing file {} located at {}".format(os.path.basename(__file__), script_dir))
Executing Python/main.py is going to output something like:
[i] Currently executing file main.py located at C:\Users\BlackVirusScript\Desktop\Python
I have a directory which has all the files:
myDirectory/
directory1/
importantFile.py
Output.py
How can I import Output.py from importantFile.py without having to put in the same directory?
importantFile.py
import Output
Output.write('This worked!')
Output.py
class Output():
def writeOutput(s):
print s
if "call" is import, in Output.py
import sys
import os.path
# change how import path is resolved by adding the subdirectory
sys.path.append(os.path.abspath(os.getcwd()+'/directory1'))
import importantFile
importantFile.f()
sys.path contains the list of path where to look for modules, details in https://docs.python.org/2/library/sys.html
The other way is to use the relative notation, for which the python file you want to import should be in a package.
You have to make the directory a python package by putting an init.py file.
Look for the packages section in this link.
https://docs.python.org/2/tutorial/modules.html
import sys
sys.path.append('/full/path/to/use')
global exist_importedname
exist_importedname = True
try:
import myimport
except ImportError as e:
exist_importedname = False
print (e.message)