I am currently creating a package for python but I would like to give access to the user only a specific set of functions defined in this package. Let's say that the structure file is as follows:
my_package/
__init__.py
modules/
__init__.py
functions.py
In functions.py, there are several functions as below (those are silly examples):
def myfunction(x):
return my_subfunction1(x) + my_subfunction2(x)
def my_subfunction1(x):
return x
def my_subfunction2(x):
return 2*x
I want the user to be able to import my_package and directly access myfunction, but NOT my_subfunction1 and my_subfunction2. For example, let's say that only myfunction is useful for the user, whereas the sub-functions are only intermediate computations.
import my_package
a=my_package.myfunction(1) #should return 3
b=my_package.my_subfunction1(1) # should returns an error, function does not exist
I can think of two ways of solving my problem by adding the following lines to the __init__.py file inside my_package/
1/ from modules.functions import myfunction
2/ from modules.functions import *, and renaming the subfunctions with a leading underscore to exclude them from the starred import, ie :
_my_subfunction1 and _my_subfunction2
and both of these tricks seems to work well so far.
My question is thus : Is this the correct "pythonic" way to do ? Which one is better ? If none of them is the good way, how should I re-write it ?
Thanks for your help.
I believe you should take a look at the __all__ variable.
In your case, just set, in yout __init__.py:
__all__ = ['myfunction']
Related
I'm a somewhat of an intermediate Python programmer but there's something that's been bugging me for a while. I'm trying to unify code to make it reusable by many projects. The problem I am facing is how to access function (or methods for that matter) by a certain class (or procedural code) without importing the module/class. I want my code to not know about the existence of other files/modules.
Here's an example.
Imagine I have a the following file architecture for my project (that would be reused by other project but with a slight variation):
Main Folder
main.py
Folder1>module1>func1
Folder2>module2>func2 (func2 calls func1, let's say func1 is a common math operation, like finding whether a point is on a plane within a tolerance)
Now, in main.py, I import both module1 and module2
let's say I have a function in main.py that calls func2 as such
## In main.py
def call_func1():
module2.func2()
## In module1.py
def is_point_on_plane():
print("testing")
## In module2.py
def func2():
is_point_on_plane() ## This is a function that is currently in module1
Now, I don't want module1 or module2 to know anything about each other, or even that the other exists. I'm trying to avoid any imports.
The way I'm working around this is to pass is_point_on_plane as a parameter to func2 in main.py
def call_func1():
fcToCall = module1.is_point_on_plane
module2.func2(fcToCall)
## In module1
def is_point_on_plane():
print("testing")
## In module2
def func2(fcToCall):
fcToCall()
I am sure there is a better way to do this. A more Pythonic way. In whichever case, I need module1 and module2 to be completely independent and avoid any imports.
you could import the file name of the project you want for example:
"if my want to use another def function of a file called 'main.py' then i will write
import main
main.call_func1()
Hi I'm building my own package and I have a question on __all__.
Are there any neat way to define __all__, other than explicitly typing each and every function in the module?
I find it very tedious...
I'm trying to make some code which wraps on frequently used libraries such as numpy, pytorch, os. The problem is, the libraries I used to create my modules also gets imported when I import my package.
I want to import every function / class that I defined, but I don't want the third-party libraries that I used in the process to get imported.
I use from .submodule import * in my __init__.py so that I can access my functions inside the submodule directly. (Just like we can access functions directly from the top package like np.sum(), torch.sum() )
My submodule has a lot of functions, and I want to import all of them to __init__.py, except for the third-party packages that I used.
I see that __all__ defines what to import when from package import * is called.
For example,
utils.py
__all__ = ['a']
def a():
pass
def b():
pass
__init__.py
from .utils import *
and
>>> import package
>>> package.a()
None
>>> package.b()
NameError: 'package.b' is not defined
What I want is something like
__all__ = Some_neat_fancy_method()
I tried locals() and dir(), but got lost along the way.
Any suggestions?
As others have pointed out, the whole point of __all__ is to explicitly specify what gets exposed to star-imports. By default everything is. If you really want to specify what doesn't get exposed instead, you can do a little trick and include all modules in __all__ and then remove the ones you want to exclude.
For example:
def _exclude(exclusions: list) -> list:
import types
# add everything as long as it's not a module and not prefixed with _
functions = [name for name, function in globals().items()
if not (name.startswith('_') or isinstance(function, types.ModuleType))]
# remove the exclusions from the functions
for exclusion in exclusions:
if exclusion in functions:
functions.remove(exclusion)
del types # deleting types from scope, introduced from the import
return functions
# the _ prefix is important, to not add these to the __all__
_exclusions = ["function1", "function2"]
__all__ = _exclude(_exclusions)
You can of course repurpose this to simply include everything that's not a function or prefixed with _ but it serves little use since everything is included in star-imports if you don't specify the __all__, so I thought it was better to include the exclusion idea. This way you can simply tell it to exclude specific functions.
Are there any neat way to define all, other than explicitly typing each and every function in the module?
Not built-in no. But defining __all__ by hand is basically the entire point, if you want to include everything in __all__ you can just do nothing at all:
If __all__ is not defined, the statement from sound.effects import * [...] ensures that the package sound.effects has been imported (possibly running any initialization code in __init__.py) and then imports whatever names are defined in the package.
The entire point of __all__ is to restricts what gets "exported" by star-imports. There's no real way for Python to know that except by having you tell it, for each symbol, whether it should be there or not.
One easy workaround is to alias all of your imports with a leading underscore. Anything with a leading underscore is excluded from from x import * style imports.
import numpy as _np
import pandas as _pd
def my_fn():
...
I have a python package with packages in it. This explanation seems strange, so I'll include my package's structure:
package\_
__init__.py
subpackage1\_
__init__.py
file1.py
subpackage2\_
__init__.py
file2.py
(I'm simplifying it for easier understanding).
The __init__.py on the top level looks like this:
__all__ = ["subpackage1", "subpackage2"]
And, for some reason, when importing the package, it dosen't recognise anythong from file1.py or file2.py. Any ideas how to fix it?
If you need more details, here's the project on github: https://github.com/Retr0MrWave/mathModule
. The directory I called package is mathmodule_pkg in the actual project
Filling the __all__ field with names does not make imports possible, it merely serves as a hint of what you mean to make importable. This hint is picked up by star-imports to restrict what is imported, and IDEs like pycharm also use it to get an idea of what is and isn't exposed - but that's about it.
If you want to enable top-level imports of your nested classes and functions, you need to
import them into the top-level __init__.py
bind them to names that can be used for the import
optionally, reference said names in __all__ to make the API nice and obvious
Using the project you're referencing as an example, this is what it would look like:
mathmodule_pkg/__init__.py
import mathmodule_pkg.calculus.DerrivativeAndIntegral #1
integral = mathmodule_pkg.calculus.DerrivativeAndIntegral.integral #2
__all__ = ['integral'] # 3
Using the very common form of from some.package import some_name we can combine steps 1 and 2 and reduce the potential for bugs when re-binding the name:
from mathmodule_pkg.calculus.DerrivativeAndIntegral import integral # 1 and 2
__all__ = ['integral'] # 3
Using either form, after installing your package the following will be possible:
>>> from mathmodule_pkg import integral
>>> integral(...)
The Situation
I'm currently working on a small but very expandable project, where i have the following structure:
/
|- main.py
|- services
|- __init__.py
|- service1.py
|- service2.py
|- ...
Every one of these services creates an object and all of them have the exact same arguments and all of them are used in the same way. The difference between them is internally, where they do some, for this question unimportant, thing in a different way.
Now this is around how my code currently handles it like this:
main.py
from services import *
someObject = {} #content doesn't matter, it's always the same
serv_arr = [] # an array to hold all services
serv_arr.append( service1.service1(someObject) )
serv_arr.append( service2.service2(someObject) )
...
for service in serv_arr:
# this function always has the same name and return type in each service
service.do_something()
The Question
My specific question is:
Is there a way to automate the creation of serv_arr with a loop, such that, if i add service100.py and service101.py to the package services and i don't have to go back into main.py and add it manually, but instead it automatically loads whatever it needs?
First off, you should really avoid using the from xxx import * pattern, as it clutters the global namespace.
You could add a list of available services to services/__init__.py
something like this perhaps
# services/__init__.py
from .service1 import service1
from .service2 import service2
...
services = [service1, service2, ...]
__all__ = ['services']
If that's still too manual for you, you could iterate over the directory and use importlib to import the services by their paths.
However, I can't help but think this problem is indicative of a bad design. You might want to consider using something like the Factory Pattern to instantiate the various services, rather than having a large number of separate modules. As it is, if you wanted to make a small change to all of the services, you'll have a lot of tedious work ahead of you to do so.
Okay, building on this idea:
Austin Philp's answer
# services/__init__.py
from .service1 import service1
from .service2 import service2
...
services = [service1, service2, ...]
__all__ = ['services']
And the idea of specifically exposed methods and modules from Factory Pattern, mentioned in this answer, i came up with a very hacky solution that works without cluttering global namespace (another thing criticized by #Austin Philp).
The Solution
I got the idea to implement a method in each module that does nothing but create an instance of said module and each module is mentioned in services/__init__.py:
#services/__init__.py
from .service1 import service1
from .service2 import service2
__all__=["service1", "service2", ...]
#services/service1.py
class service1(object):
def __init__(self, input):
...
...
#
def create_instance(input):
return service1(input) # create the object and return it.
Then in main.py, I simply do this (it is extremely hacky, but it works)
#main.py
import services
import sys
# use the __all__ method to get module names. actually
for name in services.__all__:
service = sys.modules[f'services.{name}'].create_instance( input )
# do whatever with service
This way i can just happily do whatever needed without cluttering the global namespace but still iterating over or even individually calling the modules. The only thing i would have to edit to add/remove a module is another entry in the __all__ variable inside services/__init__.py. It even removed the need to have the serv_arr array, because services.__all__ already has all the names i am interested in and would have the same length as modules used.
I am creating a web app using web.py on python 2.7.3.
I have the following folder structure:
start_app.py
/app
__init__.py
/models
__init__.py
ActionModel.py
AreaModel.py
/controllers
__init__.py
world.py
/views
Whenever I freshly start the app using python start_app.py, and visit world/surrounding I get the following error
<type 'exceptions.ImportError'> at /world/surrounding
cannot import name AreaModel
Python /home/dev/app/models/ActionModel.py in <module>, line 13
Web GET http://localhost:5000/world/surrounding
Line 13 is simply: from app.models import AreaModel but I don't see why python is complaining here.
If I comment this importing line, it runs fine. However, if I call a different URL, e.g. world/view, I get an error that AreaModel is not defined. Once I uncomment the line, it works fine again for all cases (i.e. /surrounding and /view).
I am suspecting that this has something to do with the fact that I am "importing in circles", i.e. world.py imports AreaModel, AreaModel imports ActionModel and ActionModel imports AreaModel.
I doubt that this is 'the pythonic way' to do things or even the 'MVC way', so I would very much appreciate your enlightening me how to do this properly.
Note: app is not in my PYTHONPATH, but I don't think it is needed here, since start_app.py is in the top-level directory and according to this all modules should be available.
Basically, what it comes down to is:
I need the models' functionalities in both the controllers and the models. Is it good practice to "import in circles"? Or is there a nicer approach to do this?
Also, is this problem related to python in general or just web.py?
Update:
Added init.py files, I had them, but did not include in original question. Sorry for that.
Update:
ActionModel.py includes (among others) a class named BaseAction and a few functions, which return instances or subclasses of BaseAction depending on what type of Action we are dealing with. They are called using e.g. ActionModel.get_by_id()
#matthew-trevor : Are you suggesting in a) that I should move those functions get_by_id() into a class ActionModel?
#actionmodel.py
class ActionModel(object):
def __init__(arg1, arg2, area_class):
self.area = area_class()
def get_by_id(self, id):
return BaseAction(id)
class BaseAction(object):
def __init__(id):
pass
I don't see how this should remedy my import problems though.
The Immediate Problem
You cannot import from folders, but you can import from packages. You can turn any folder into a package by adding an __init__.py file to it:
start_app.py
/app
__init__.py
/models
__init__.py
ActionModel.py
AreaModel.py
/controllers
__init__.py
world.py
/views
__init__.py
I'm guessing that ActionModel.py includes a class of the same name. If so, I recommend renaming the file to actionmodel.py to distinguish it from the class.
Circular imports
Is it good practice to "import in circles"? Or is there a nicer
approach to do this?
It's not only bad practice, it just won't work. There are a couple of ways to get around this, which will mostly depend on what you're trying to do:
a. In AreaModel, import the ActionModel module and then reference anything you want to use in it via attribute lookup and vice versa:
# areamodel.py
import actionmodel
def foo():
action = actionmodel.ActionModel(...)
As long as the references are inside class or function definitions, it will only occur at run time and not during importing, so the circular reference is avoided.
b. Turn models into a module and put both ActionModel and AreaModel code inside it.
c. Move the shared code/functionality for ActionModel and AreaModel into a base module they both import from.
d. Make your ActionModel class (or whatever) accept a class as an input, then pass AreaModel into it in world.py (ditto for AreaModel). This way, ActionModel doesn't need to contain a reference to AreaModel, it just has to know what to do with it:
# actionmodel.py
class ActionModel(object):
def __init__(arg1, arg2, area_class):
self.area = area_class()
# areamodel.py
class AreaModel(object):
def __init__(action_class):
self.action = action_class()
# world.py
from actionmodel import ActionModel
from areamodel import AreaModel
action = ActionModel('foo', 'bar', AreaModel)
area = AreaModel(ActionModel)
This is known as object composition.