I have a custom module loader that basically does some redirection. I would like pylint to recognize this custom loader. This is my situation:
root/
__init__.py
new/
__init__.py
foo.py
bar.py
old/
__init__.py
I have a lot of clients importing old.foo. I wrote a custom loader in old/__init__.py to redirect these to import new.foo under the hood. How do I get pylint to recognize this? When it lints import old.foo, it complains that it can't find old.foo. This is only a problem with pylint. I can get client code to recognize the custom loader without any issue.
from the documentation on modules:
Packages support one more special attribute, __path__. This is initialized to be a list containing the name of the directory holding the package’s __init__.py before the code in that file is executed. This variable can be modified; doing so affects future searches for modules and subpackages contained in the package.
So if I understand correctly you want to redirect any references to old to redirect to new, so all you would need to do is replace the old folder with old.py that contains this:
__path__ = ["new"]
Then when anything tries to import old.foo it will end up importing new.foo.
You have to remember that pylint is a static analyser and as such doesn't actually load python file (except in some cases where it can't do otherwise, e.g. compiled code). As such it's not aware of custom importer or other tricks taking part of python's high dynamicity.
That being said:
you may still write a "brain" plugin for astroid (the library under pylint) that will help pylint understand your code's specificity
by relying on standard mecanism such as __path__ manipulation you'll get more chance to avoid such need, either because at some point pylint may understand this or because someone else will have contributed a plugin for that purpose.
Related
I am using an external package (package_a) that includes code in its root __init__.py file, which overrides and extends the behavior of classes from another package (package_b).
More concretely, upon importing package_a within my code, I can now benefit from the extended functionalities of package_b without ever using the import of package_a within my file.
I am wondering if there is a better, standard way to do this, without having a "falsely" unused import in my code (for linting and readability purposes) ?
Based on the the documentation for Packaging Python Projects, __init__.py should be empty. I want to know why? because I'm placing certain objects in the __init__.py file that are used in every module in the Package. On checking bunch of __init.py__ files in my local environments for standard packages like importlib, multiprocessing etc. All of them have a bunch of code in the file.
The sole purpose of __init__.py is to indicate that the folder containing this file is a package, and to treat this folder as package, that's why it is recommended to leave it empty.
Consider following hierarchy:
foo
__init__.py
bar.py
When you use from foo import bar or import foo.bar, Python interpreter will look for __init__.py in foo folder, if it finds it, the bar module will be imported else it won't; however, this behavior has changed over the time, and it may be able to successfully import the modules/packages even if __init__.py is missing, but remember Zen of Python: Explicit is better than implicit, so it's always safe to have it.
But in case, if you need some package level variables to be defined, you can do it inside the __init__.py file, and all the modules inside the package will be able to use it.
And in fact, if you look at PEP 257, it mentions that the __init__.py can also contain the documentation for package level information.
You're taking that statement as more general than it's meant. You're reading a statement from a tutorial, where they walk you through creating a simple example project. That particular example project's __init__.py should be empty, simply because the example doesn't need to do anything in __init__.py.
Most projects' __init__.py files will not be empty. Taking a few examples from popular packages, such as numpy, requests, flask, sortedcontainers, or the stdlib asyncio, none of these example __init__.py files are empty. They may perform package initialization, import things from submodules into the main package namespace, or include metadata like __all__, __version__, or a package docstring. The example project is just simplified to the point where it doesn't have any of that.
To my knowledge, there are three things you need to be aware of when you create a non-empty __init__ file:
it might be more difficult to follow the code. If you instantiate a = B() in __init__ it's even worse. I know developers who don't like it only for this reason
on package import contents of __init__ are evaluated. Sometimes it might be computation heavy or simply not needed.
namespace conflicts. You can't really instantiate bar in init and have a bar.py file in your package.
I like importing package contents in __init__ as otherwise in bigger projects import statements become ugly. Overall it's not a good or bad practice. This advice applies only to the project in this particular example.
In some case, you didn't have any shared component in your package. Suppose Defining a little package for calculating some algorithms, Then you didn't need any shared component in your __init__
I am running python 3.4 on the main.py file in the same directory.
/root directory is not in python path. It is simply the current directory that the script is executing in. All pycache folders were deleted after each test
So why exactly is __init__.py important? I thought it was necessary as stated in this post:
What is __init__.py for?
If you remove the init.py file, Python will no longer look for submodules inside that directory, so attempts to import the module will fail.
Right now, it seems to me that __init__.py is nothing more than an optional constructor where we do housekeeping and other optional things like specifying the "all" variable, etc. But not a critical item to have.
Image showing the results of the test:
Can someone explain the discrepancy or what is the cause of this issue?
As confusing as it may be, although the basics will work without __init__.py files, you should probably still use them. Many external tools, as well as package-related functions in the standard library, will not work as expected without them. More words of wisdom here (as well as a misleading accepted answer): Is __init__.py not required for packages in Python 3.3+.
Found Answer
In essence, init.py is not needed, and its purpose is for legacy and optional housekeeping tasks that you may or may not want or need in Python versions 2.7 vs 3.0+. However, it is important to take into account that they have slightly different behavior during more complex parsing if you are building something more complex.
Please refer to the following links for additional reading material:
https://www.python.org/dev/peps/pep-0420/#namespace-packages-today
How do I create a namespace package in Python?
What's the difference between a Python module and a Python package?
https://softwareengineering.stackexchange.com/questions/276888/python-namespace-vs-module-with-underscores
What I want to achieve is as follows:
I have a python package, let's call it foo, comprising a directory foo containing an __init__.py and, under normal use a compiled extension library (either a .so or a .pyd file), which __init__.py imports into the top level namespace.
Now, the problem is that I wish the top level namespace to contain a version string which is available to setup.py during building and packaging, when the extension library is not necessarily available (not yet built), and so would cause an ImportError when trying to import foo.version.
Now, clearly, I could have an exception handler in __init__.py that just ignores failures in importing anything, but this is not ideal as there may be a real reason that the user cares about why the package can't be imported.
Is there some way I can have the version string in a single place in the package, have it importable, yet not break the exceptions from attempts to import the extension?
As opposed to ignoring failures when importing print out a trace message or a warning so that the user will still get the negative feedback.
As for importing a specific subfile if you are using python 3.3+ (or python 2.7) you can use imp.load_source which accepts a pathname of a file you want to import.
This question follows up The way to make namespace packages in Python and How do I create a namespace package in Python?.
Note PEP 420, and the distribute docs, which state:
You must NOT include any other code and data in a namespace package’s __init__.py. Even though it may appear to work during development, or when projects are installed as .egg files, it will not work when the projects are installed using “system” packaging tools – in such cases the __init__.py files will not be installed, let alone executed.
This all seems to make it impossible to have a "main library" package with independently distributed extension sub-packages. What I want is to be able to:
define a core library package, to be used like this:
import mylibrary
mylibrary.some_function()
allow library extensions, packaged and distributed separately, to be used like this:
import mylibrary.myextension
mylibrary.myextension.some_other_function()
I would've expected to be able to do this with namespace packages, but it seems not to be the case, based on the questions and links above. Can this be done at all?
It is indeed not possible to have code in a top level __init__.py for a PEP 420 namespace package.
If I were you, I'd either:
create 2 packages, one called mylibrary (a normal package) which contains your actual library code, and the other called mylibrary_plugins which is a namespace package.
or, create mylibrary.lib, which is a normal package and contains your code, and mylibrary.plugins, which is a namespace package.
Personally I'd use option 1.
The rationale section of PEP 420 explains why __init__.py cannot contain any code.
strictly speaking, you can have variables under mylibrary, you just won't be able to define them there. You can, for instance:
# mylibrary/core.py
import mylibrary
def some_function():
pass
mylibrary.some_function = some_function
and your users can use it like:
import mylibrary.core
mylibrary.some_function()
That is to say, mylibrary.core monkey patches mylibrary so that, other than the import, it looks as though somefunction is defined in mylibrary rather than a sub-package.