What I want to achieve is as follows:
I have a python package, let's call it foo, comprising a directory foo containing an __init__.py and, under normal use a compiled extension library (either a .so or a .pyd file), which __init__.py imports into the top level namespace.
Now, the problem is that I wish the top level namespace to contain a version string which is available to setup.py during building and packaging, when the extension library is not necessarily available (not yet built), and so would cause an ImportError when trying to import foo.version.
Now, clearly, I could have an exception handler in __init__.py that just ignores failures in importing anything, but this is not ideal as there may be a real reason that the user cares about why the package can't be imported.
Is there some way I can have the version string in a single place in the package, have it importable, yet not break the exceptions from attempts to import the extension?
As opposed to ignoring failures when importing print out a trace message or a warning so that the user will still get the negative feedback.
As for importing a specific subfile if you are using python 3.3+ (or python 2.7) you can use imp.load_source which accepts a pathname of a file you want to import.
Related
I'm trying to work in an environment where I cannot install packages normally using Pip. As such, I need to bundle all my program's dependencies alongside the full program. This works great for simpler dependencies, but for ones that have their own dependencies, imports fail due to their dependencies not being globally available (since neither of them can be imported using absolute imports).
Currently, my modules are as follows:
main.py (uses my_module)
my_module/ (depends on foo)
somefile.py
anotherfile.py
foo/ (depends on bar)
# Contents of third-party module
bar/
# Contents of other third-party module
Is there a way that I can alias a module that I have imported relatively, so that whenever some other module tries to import it, it will be referred to the place where I have directed it? I'd much rather do that than have to risk modifying thousands of lines of unfamiliar code to make all the import statements relative.
One way is to create a custom pip package as explained here:
Use custom Python package
Can someone help and clarify, when using import command(s), at which moment the __init__.py files in various package directory(s) is/are executed?
For each included module?
Only once at 1st import command?
For each import command?
It's evaluated on first module import. On next imports, interpreter detects that module was already loaded and simply returns reference to it. There is no need to re-execute code.
Quoting The import system:
On caching modules:
The first place checked during import search is sys.modules. This mapping serves as a cache of all modules that have been previously imported, including the intermediate paths. So if foo.bar.baz was previously imported, sys.modules will contain entries for foo, foo.bar, and foo.bar.baz. Each key will have as its value the corresponding module object.
During import, the module name is looked up in sys.modules and if present, the associated value is the module satisfying the import, and the process completes. However, if the value is None, then an ImportError is raised. If the module name is missing, Python will continue searching for the module.
On executing __init__ when importing:
Python defines two types of packages, regular packages and namespace packages. Regular packages are traditional packages as they existed in Python 3.2 and earlier. A regular package is typically implemented as a directory containing an init.py file. When a regular package is imported, this __init__.py file is implicitly executed, and the objects it defines are bound to names in the package’s namespace. The __init__.py file can contain the same Python code that any other module can contain, and Python will add some additional attributes to the module when it is imported.
I have a custom module loader that basically does some redirection. I would like pylint to recognize this custom loader. This is my situation:
root/
__init__.py
new/
__init__.py
foo.py
bar.py
old/
__init__.py
I have a lot of clients importing old.foo. I wrote a custom loader in old/__init__.py to redirect these to import new.foo under the hood. How do I get pylint to recognize this? When it lints import old.foo, it complains that it can't find old.foo. This is only a problem with pylint. I can get client code to recognize the custom loader without any issue.
from the documentation on modules:
Packages support one more special attribute, __path__. This is initialized to be a list containing the name of the directory holding the package’s __init__.py before the code in that file is executed. This variable can be modified; doing so affects future searches for modules and subpackages contained in the package.
So if I understand correctly you want to redirect any references to old to redirect to new, so all you would need to do is replace the old folder with old.py that contains this:
__path__ = ["new"]
Then when anything tries to import old.foo it will end up importing new.foo.
You have to remember that pylint is a static analyser and as such doesn't actually load python file (except in some cases where it can't do otherwise, e.g. compiled code). As such it's not aware of custom importer or other tricks taking part of python's high dynamicity.
That being said:
you may still write a "brain" plugin for astroid (the library under pylint) that will help pylint understand your code's specificity
by relying on standard mecanism such as __path__ manipulation you'll get more chance to avoid such need, either because at some point pylint may understand this or because someone else will have contributed a plugin for that purpose.
This is my directory structure:
Projects
+ Project_1
+ Project_2
- Project_3
- Lib1
__init__.py # empty
moduleA.py
- Tests
__init__.py # empty
foo_tests.py
bar_tests.py
setpath.py
__init__.py # empty
foo.py
bar.py
Goals:
Have an organized project structure
Be able to independently run each .py file when necessary
Be able to reference/import both sibling and cousin modules
Keep all import/from statements at the beginning of each file.
I Achieved #1 by using the above structure
I've mostly achieved 2, 3, and 4 by doing the following (as recommended by this excellent guide)
In any package that needs to access parent or cousin modules (such as the Tests directory above) I include a file called setpath.py which has the following code:
import os
import sys
sys.path.insert(0, os.path.abspath('..'))
sys.path.insert(0, os.path.abspath('.'))
sys.path.insert(0, os.path.abspath('...'))
Then, in each module that needs parent/cousin access, such as foo_tests.py, I can write a nice clean list of imports like so:
import setpath # Annoyingly, PyCharm warns me that this is an unused import statement
import foo.py
Inside setpath.py, the second and third inserts are not strictly necessary for this example, but are included as a troubleshooting step.
My problem is that this only works for imports that reference the module name directly, and not for imports that reference the package. For example, inside bar_tests.py, neither of the two statements below work when running bar_tests.py directly.
import setpath
import Project_3.foo.py # Error
from Project_3 import foo # Error
I receive the error "ImportError: No module named 'Project_3'".
What is odd is that I can run the file directly from within PyCharm and it works fine. I know that PyCharm is doing some behind the scenes magic with the Python Path variable to make everything work, but I can't figure out what it is. As PyCharm simply runs python.exe and sets some environmental variables, it should be possible to clone this behavior from within a Python script itself.
For reasons not really germane to this question, I have to reference bar using the Project_3 qualifier.
I'm open to any solution that accomplishes the above while still meeting my earlier goals. I'm also open to an alternate directory structure if there is one that works better. I've read the Python doc on imports and packages but am still at a loss. I think one possible avenue might be manually setting the __path__ variable, but I'm not sure which one needs to be changed or what to set it to.
Those types of questions qualify as "primarily opinion based", so let me share my opinion how I would do it.
First "be able to independently run each .py file when necessary": either the file is an module, so it should not be called directly, or it is standalone executable, then it should import its dependencies starting from top level (you may avoid it in code or rather move it to common place, by using setup.py entry_points, but then your former executable effectively converts to a module). And yes, it is one of weak points of Python modules model, that causes misunderstandings.
Second, use virtualenv (or venv in Python3) and put each of your Project_x into separate one. This way project's name won't be part of Python module's path.
Third, link you've provided mentions setup.py – you may make use of it. Put your custom code into Project_x/src/mylib1, create src/mylib1/setup.py and finally your modules into src/mylib1/mylib1/module.py. Then you may install your code by pip as any other package (or pip -e so you may work on the code directly without reinstalling it, though it unfortunately has some limitations).
And finally, as you've confirmed in comment already ;). Problem with your current model was that in sys.path.insert(0, os.path.abspath('...')) you'd mistakenly used Python module's notation, which in incorrect for system paths and should be replaced with '../..' to work as expected.
I think your goals are not reasonable. Specifically, goal number 2 is a problem:
Be able to independently run each .py file when neccessary
This doesn't work well for modules in a package. At least, not if you're running the .py files naively (e.g. with python foo_tests.py on the command line). When you run the files that way, Python can't tell where the package hierarchy should start.
There are two alternatives that can work. The first option is to run your scripts from the top level folder (e.g. projects) using the -m flag to the interpreter to give it a dotted path to the main module, and using explicit relative imports to get the sibling and cousin modules. So rather than running python foo_tests.py directly, run python -m project_3.tests.foo_tests from the projects folder (or python -m tests.foo_tests from within project_3 perhaps), and have have foo_tests.py use from .. import foo.
The other (less good) option is to add a top-level folder to your Python installation's module search path on a system wide basis (e.g. add the projects folder to the PYTHON_PATH environment variable), and then use absolute imports for all your modules (e.g. import project3.foo). This is effectively what your setpath module does, but doing it system wide as part of your system's configuration, rather than at run time, it's much cleaner. It also avoids the multiple names that setpath will allow to you use to import a module (e.g. try import foo_tests, tests.foo_tests and you'll get two separate copies of the same module).
I am a rookie to the Python world, and now I find myself trying to learn how to properly create a Python package or module. I also have several requirements that must be met.
I have a core native DLL (we'll name it MyCore.dll) compiled from C++. This DLL must be deployed to a particular install location, as it's a core component of a product (we'll say ProgramFiles\MyProduct).
I have used SWIG to generate Python bindings for MyCore.dll. It has generated 2 files: _MyCoreBindings.pyd (essentially a DLL that references MyCore.dll) and MyCoreBindings.py (which loads _MyCoreBindings.pyd and provides a Python API to it).
Finally, I have a Python script (MyProduct.py) containing only an import, as my product must be imported in Python under the name MyProduct.SDK :
import MyCoreBindings as SDK
Thus, I want a user's Python script to be able to access it like so:
import MyProduct.SDK
File summary in order of dependency:
ProgramFiles\MyProduct\MyCore.dll
_MyCoreBindings.pyd
MyCoreBindings.py
MyProduct.py (I'm not sure I need this)
I've also read that the format of a Python package involves some directory structure mimicking the import path, and the possible inclusion of setup.py and __init__.py, but all materials I've read have not made it clear what must go in each of these files, and in what cases they are required. I am definitely not clear where this directory structure may be placed.
If you want Python to be able to import your module, then you have a couple of choices:
install the module to Python's site-packages folder.
modify the paths that Python searches for modules/packages to import. You can do that via a *.pth file or by modifying the path using Python's sys module (i.e. sys.path.append(/some/path) )
You use setup.py for installing your package and/or creating eggs/wheels etc. See the following for helpful hints:
https://docs.python.org/2/distutils/setupscript.html
https://pythonhosted.org/an_example_pypi_project/setuptools.html
You can create a package by using _init__.py inside a folder alongside your modules:
- MyProduct
- __init__.py
- MyCoreBindings.py
- _MyCoreBindings.pyd
The _init__.py file can be empty. This basically allows you to import the folder as MyProduct:
import MyProduct
MyProduct.MyCoreBindings
or
from MyProduct import MyCoreBindings as SDK
You won't need a MyProduct.py if you go with the __init__.py in the root folder. I hope that all made sense.