In this question
How can I separate functions of class into multiple files?
the top answer suggests to use
from method_file import method
Inside a class definition to have class methods defined in separate files. However, for a class like this
my_number.py
class MyNumber:
def __init__(self):
self.x = 5
from my_method import my_method
my_method.py
def my_method(self):
print(self.x)
It would not be clear to the IDE that self refers to a MyNumber object. As a consequence code completion (for e.g. self.x) is not available in my_method. A type hint for self could solve this, i.e.
my_method.py
from my_number import MyNumber
def my_method(self: MyNumber):
print(self.x)
but this leads to a circular import.
Is there any workaround or best practice for such a situation?
There is an approach that combines a __future__ import to disregard type annotations at runtime, with a if TYPE_CHECKING clause that "imports" the code from your IDE's point of view only, so that code completion is available.
Example:
my_number.py
class MyNumber:
def __init__(self):
self.x = 5
from my_method import my_method
my_method.py
from __future__ import annotations
from typing import TYPE_CHECKING
if TYPE_CHECKING:
from my_number import MyNumber
def my_method(self: MyNumber):
print(self.x)
With the from __future__ import annotations, we postpone the evaluation of type hints - in other words, we can type hint my_method even if we don't actually import MyNumber. This behavior was planned to be the default in Python 3.10, but it got postponed, so we need this import for now.
Now depending on your editor/IDE, you will still get a warning complaining that MyNumber isn't defined, and its methods/attributes may not show up on the autocomplete. Here's where the TYPE_CHECKING comes into play: this is simply a constant False value, which means that our clause is:
if False:
from my_number import MyNumber
In other words, we're "tricking" the IDE into thinking we're importing MyNumber, but in reality that line never executes. Thus we avoid the circular import altogether.
This might feel a little hacky, but it works :-) the whole point of the TYPE_CHECKING constant is to allow type checkers to do their job, while not actually importing code at runtime, and to do so in a clear way (Zen of Python: There should be one-- and preferably only one --obvious way to do it).
This approach has worked for me consistently in PyCharm, not sure about other IDEs/editors.
Is there any workaround or best practice for such a situation?
The best practice is not to do this. If a method implementation is specific to a class, it should be part of the class definition.
If a method is not specific to a class, it should be defined across all valid types. A Protocol is appropriate to express this :
from typing import Protocol, Any
class HasX(Protocol):
x: Any # might need a TypeVar for complex cases
def my_method(self: HasX):
print(self.x)
If a method extends a class separate of its definition, it should not be patched in. Use functools.singledispatch to externally define single dispatch functions, which are logically similar to methods:
from functools import singledispatch
from my_number import MyNumber
# not imported into MyNumber
#singledispatch
def my_method(self):
raise NotImplementedError(f"no dispatch for {type(self}")
#my_method.register
def _(self: MyNumber):
print(self.x)
Typically, the self keyword is used to represent an instance of the class. It doesn't make sense to type hint self. Second, you can't access the instance variable x via self if your function is not a method to a class that is a MyNumber object.
I would suggest two options to accomplish what you want to. You can accept a MyNumber object as a parameter to the my_method() function or you can create a new class and inherit the MyNumber class. Make sure the files are in the same directory, otherwise update the import statement in File 2.
Option #1
class MyNumber:
def __init__(self):
self.x = 5
def my_method(my_number: MyNumber):
print(my_number.x)
my_method(MyNumber())
Option #2
#my_number.py
class MyNumber:
def __init__(self):
self.x = 5
#my_method.py
from my_number import MyNumber
class MyMethod(MyNumber):
def __init__(self):
super().__init__()
def my_method(self):
print(self.x)
MyMethod().my_method()
I think you are having problems regarding object-oriented concepts in python. Your "my_method" function doesn't need the "self: MyNumber" as a parameter, in fact, you need to create an object of the MyNumber class and consequently this class will have an attribute that is the "x" since you defined the "x" in the constructor of the MyNumber class. It would look something like this:
#my_number.py
class MyNumber:
def __init__(self):
self.x = 5
#my_method.py
from my_number import MyNumber
def my_method():
mm = MyNumber()
print(mm.x)
Related
I'm trying to split my huge class into two; well, basically into the "main" class and a mixin with additional functions, like so:
main.py file:
import mymixin.py
class Main(object, MyMixin):
def func1(self, xxx):
...
mymixin.py file:
class MyMixin(object):
def func2(self: Main, xxx): # <--- note the type hint
...
Now, while this works just fine, the type hint in MyMixin.func2 of course can't work. I can't import main.py, because I'd get a cyclic import and without the hint, my editor (PyCharm) can't tell what self is.
I'm using Python 3.4, but I'm willing to move to 3.5 if a solution is available there.
Is there any way I can split my class into two files and keep all the "connections" so that my IDE still offers me auto-completion and all the other goodies that come from it knowing the types?
There isn't a hugely elegant way to handle import cycles in general, I'm afraid. Your choices are to either redesign your code to remove the cyclic dependency, or if it isn't feasible, do something like this:
# some_file.py
from typing import TYPE_CHECKING
if TYPE_CHECKING:
from main import Main
class MyObject(object):
def func2(self, some_param: 'Main'):
...
The TYPE_CHECKING constant is always False at runtime, so the import won't be evaluated, but mypy (and other type-checking tools) will evaluate the contents of that block.
We also need to make the Main type annotation into a string, effectively forward declaring it since the Main symbol isn't available at runtime.
If you are using Python 3.7+, we can at least skip having to provide an explicit string annotation by taking advantage of PEP 563:
# some_file.py
from __future__ import annotations
from typing import TYPE_CHECKING
if TYPE_CHECKING:
from main import Main
class MyObject(object):
# Hooray, cleaner annotations!
def func2(self, some_param: Main):
...
The from __future__ import annotations import will make all type hints be strings and skip evaluating them. This can help make our code here mildly more ergonomic.
All that said, using mixins with mypy will likely require a bit more structure then you currently have. Mypy recommends an approach that's basically what deceze is describing -- to create an ABC that both your Main and MyMixin classes inherit. I wouldn't be surprised if you ended up needing to do something similar in order to make Pycharm's checker happy.
For people struggling with cyclic imports when importing class only for Type checking: you will likely want to use a Forward Reference (PEP 484 - Type Hints):
When a type hint contains names that have not been defined yet, that definition may be expressed as a string literal, to be resolved later.
So instead of:
class Tree:
def __init__(self, left: Tree, right: Tree):
self.left = left
self.right = right
you do:
class Tree:
def __init__(self, left: 'Tree', right: 'Tree'):
self.left = left
self.right = right
The bigger issue is that your types aren't sane to begin with. MyMixin makes a hardcoded assumption that it will be mixed into Main, whereas it could be mixed into any number of other classes, in which case it would probably break. If your mixin is hardcoded to be mixed into one specific class, you may as well write the methods directly into that class instead of separating them out.
To properly do this with sane typing, MyMixin should be coded against an interface, or abstract class in Python parlance:
import abc
class MixinDependencyInterface(abc.ABC):
#abc.abstractmethod
def foo(self):
pass
class MyMixin:
def func2(self: MixinDependencyInterface, xxx):
self.foo() # ← mixin only depends on the interface
class Main(MixinDependencyInterface, MyMixin):
def foo(self):
print('bar')
Since Python 3.5, breaking your classes up into separate files is easy.
It's actually possible to use import statements inside of a class ClassName: block in order to import methods into a class. For instance,
class_def.py:
class C:
from _methods1 import a
from _methods2 import b
def x(self):
return self.a() + " " + self.b()
In my example,
C.a() will be a method which returns the string hello
C.b() will be a method which returns hello goodbye
C.x() will thus return hello hello goodbye.
To implement a and b, do the following:
_methods1.py:
from __future__ import annotations
from typing import TYPE_CHECKING
if TYPE_CHECKING:
from class_def import C
def a(self: C):
return "hello"
Explanation: TYPE_CHECKING is True when the type checker is reading the code. Since the type checker doesn't need to execute the code, circular imports are fine when they occur within the if TYPE_CHECKING: block. The __future__ import enables postponed annotations. This is an optional; without it you must quote the type annotations (i.e. def a(self: "C"):).
We define _methods2.py similarly:
from __future__ import annotations
from typing import TYPE_CHECKING
if TYPE_CHECKING:
from class_def import C
def b(self: C):
return self.a() + " goodbye"
In VS Code, I can see the type detected from self.a() when hovering:
And everything runs as expected:
>>> from class_def import C
>>> c = C()
>>> c.x()
'hello hello goodbye'
Notes on older Python versions
For Python versions ≤3.4, TYPE_CHECKING is not defined, so this solution won't work.
For Python versions ≤3.6, postponed annotations are not defined. As a workaround, omit from __future__ import annotations and quote the type declarations as mentioned above.
Turns out my original attempt was quite close to the solution as well. This is what I'm currently using:
# main.py
import mymixin.py
class Main(object, MyMixin):
def func1(self, xxx):
...
# mymixin.py
if False:
from main import Main
class MyMixin(object):
def func2(self: 'Main', xxx): # <--- note the type hint
...
Note the import within if False statement that never gets imported (but IDE knows about it anyway) and using the Main class as string because it's not known at runtime.
Rather than forcing oneself to engage in typing.TYPE_CHECKING shenanigans, there is a simple way to avoid circular type-hints: don't use from imports, and use either from __future__ import annotations or string annotations.
# foo.py
from __future__ import annotations
import bar
class Foo:
bar: bar.Bar
# bar.py
import foo
class Bar:
foo: "foo.Foo"
This style of import is "lazily evaluated", whereas using from foo import Foo would force Python to run the entire foo module to get the final value of Foo immediately at the import line. It's quite useful if you need to use it at runtime as well e.g. if foo.Foo or bar.Bar needs to be used within a function/method, since your functions/methods should only be called once foo.Foo and bar.Bar can be used.
I would advice refactoring your code, as some other persons suggested.
I can show you a circular error I recently faced:
BEFORE:
# person.py
from spell import Heal, Lightning
class Person:
def __init__(self):
self.life = 100
class Jedi(Person):
def heal(self, other: Person):
Heal(self, other)
class Sith(Person):
def lightning(self, other: Person):
Lightning(self, other)
# spell.py
from person import Person, Jedi, Sith
class Spell:
def __init__(self, caster: Person, target: Person):
self.caster: Person = caster
self.target: Person = target
class Heal(Spell):
def __init__(self, caster: Jedi, target: Person):
super().__init__(caster, target)
target.life += 10
class Lightning(Spell):
def __init__(self, caster: Sith, target: Person):
super().__init__(caster, target)
target.life -= 10
# main.py
from person import Jedi, Sith
Step by step:
# main starts to import person
from person import Jedi, Sith
# main did not reach end of person but ...
# person starts to import spell
from spell import Heal, Lightning
# Remember: main is still importing person
# spell starts to import person
from person import Person, Jedi, Sith
console:
ImportError: cannot import name 'Person' from partially initialized module
'person' (most likely due to a circular import)
A script/module can be imported only by one and only one script.
AFTER:
# person.py
class Person:
def __init__(self):
self.life = 100
# spell.py
from person import Person
class Spell:
def __init__(self, caster: Person, target: Person):
self.caster: Person = caster
self.target: Person = target
# jedi.py
from person import Person
from spell import Spell
class Jedi(Person):
def heal(self, other: Person):
Heal(self, other)
class Heal(Spell):
def __init__(self, caster: Jedi, target: Person):
super().__init__(caster, target)
target.life += 10
# sith.py
from person import Person
from spell import Spell
class Sith(Person):
def lightning(self, other: Person):
Lightning(self, other)
class Lightning(Spell):
def __init__(self, caster: Sith, target: Person):
super().__init__(caster, target)
target.life -= 10
# main.py
from jedi import Jedi
from sith import Sith
jedi = Jedi()
print(jedi.life)
Sith().lightning(jedi)
print(jedi.life)
order of executed lines:
from jedi import Jedi # start read of jedi.py
from person import Person # start AND finish read of person.py
from spell import Spell # start read of spell.py
from person import Person # start AND finish read of person.py
# finish read of spell.py
# idem for sith.py
console:
100
90
File composition is key
Hope it will help :D
I think the perfect way should be to import all the classes and dependencies in a file (like __init__.py) and then from __init__ import * in all the other files.
In this case you are
avoiding multiple references to those files and classes and
also only have to add one line in each of the other files and
the third would be the pycharm knowing about all of the classes that you might use.
How to get the filename of the subclass?
Example:
base.py:
class BaseClass:
def __init__(self):
# How to get the path "./main1.py"?
main1.py:
from base import BaseClass
class MainClass1(BaseClass):
pass
Remember that self in BaseClass.__init__ is an instance of the actual class that's being initialised. Therefore, one solution, is to ask that class which module it came from, and then from the path for that module:
import importlib
class BaseClass:
def __init__(self):
m = importlib.import_module(self.__module__)
print m.__file__
I think there are probably a number of way you could end up with a module that you can't import though; this doesn't feel like the most robust solution.
If all you're trying to do is identify where the subclass came from, then probably combining the module name and class name is sufficient, since that should uniquely identify it:
class BaseClass:
def __init__(self):
print "{}.{}".format(
self.__module__,
self.__class__.__name__
)
You could do it by reaching back through the calling stack to get the global namespace of the caller of the BaseClass.__init__() method, and from that you can extract the name of the file it is in by using the value of the __file__ key in that namespace.
Here's what I mean:
base.py:
import sys
class BaseClass(object):
def __init__(self):
print('In BaseClass.__init__()')
callers_path = sys._getframe(1).f_globals['__file__']
print(' callers_path:', callers_path)
main1.py:
from base import BaseClass
class MainClass1(BaseClass):
def __init(self):
super().__init__()
mainclass1 = MainClass1()
Sample output of running main1.py:
In BaseClass.__init__()
callers_path: the\path\to\main1.py
I think you're looking to the wrong mechanism for your solution. Your comments suggest that what you want is an exception handler with minimal trace-back capability. This is not something readily handled within the general class mechanism.
Rather, you should look into Python's stack inspection capabilities. Very simply, you want your __init__ method to report the file name of the calling sub-class. You can hammer this by requiring the caller to pass its own __file__ value. In automated fashion, you can dig back one stack frame and access __file__ via that context record. Note that this approach assumes that the only time you need this information is when __init__ is called is directly from a sub-class method.
Is that enough to get you to the right documentation?
I import a class Foo that has a default parameter dir upon which if performs a function doit. How can I change the default dir? Do I need to inherit this class and then change it, how?
class Foo(object):
def __init__(self, dir='xxx'):
self.bar = doit(dir) # fails because xxx is wrong
why don't you just provide a different argument when you construct an instance of the class:
foo = Foo(dir='something else')
btw: dir is a python built-in and therefore not the best choice as variable name.
if you want the default changed, you can inherit indeed:
class MyFoo(Foo):
def __init__(self, d='somethig else'):
super().__init__(d=d)
Just create a factory function for Foo objects and be done with it:
def create_foo():
return Foo(dir='my correct value goes here')
Since you're importing Foo, you could go a step further and just shadow Foo like this:
def Foo():
import foo
return foo.Foo(dir='my correct value goes here')
Of course you can inherit from Foo. Be sure to look up how to call the base
class constructors. I find that sooo hard to memorize that I just end up google it. Every. Single. Time.
BTW: Looks like #hiro protagonist already has the calling super figured out.
I'm not really sure how best to explain what I want, so I'll just show some code:
class Stuffclass():
def add(self, x, y):
return x + y
def subtract(self, x, y):
return x - y
# imagine that there are 20-30 other methods in here (lol)
class MyClass:
def __init__(self):
self.st = Stuffclass()
def doSomething(self):
return self.st.add(1, 2)
m = MyClass()
m.doSomething() # will print 3
# Now, what I want to be able to do is:
print m.add(2, 3) # directly access the "add" method of MyClass.st
print m.subtract(10, 5) # directly access the "subtract" method of MyClass.st
m.SomeMethod() # execute function MyClass.st.SomeMethod
I know I could do something like this:
class MyClass:
def __init__(self):
self.st = Stuffclass()
self.add = self.st.add
self.subtract = self.st.subtract
...but this requires manually assigning all possible attributes.
I'm writing all the classes so I can guarantee no name collisions.
Making MyClass a subclass of Stuffclass won't work, because I actually am using this in a plugin-based application, where MyClass loads other code dynamically using import. This means MyClass can't subclass from the plugin, because the plugin could be anything that follows my API.
Advice please?
I believe that writing a getattr function for your class will let you do what you want.
Called when an attribute lookup has not found the attribute in the usual places (i.e. it is not an instance attribute nor is it found in the class tree for self). name is the attribute name. This method should return the (computed) attribute value or raise an AttributeError exception
So something as simple as:
def __getattr__(self, name):
if hasattr(self.st, name):
return getattr(self.st, name)
else:
raise AttributeError
should do roughly what you're after.
But, having answered (I think) the question you asked, I'm going to move on to the question I think you should have asked.
I actually am using this in a plugin-based application, where MyClass loads other code dynamically using import. This means MyClass can't subclass from the plugin, because the plugin could be anything that follows my API
I can see why MyClass can't be a subclass of StuffClass; but couldn't StuffClass be a subclass of MyClass? If you defined the inheritance that way, you'd have a guarantee what StuffClass implements all the basic stuff in MyClass, and also that your instances of StuffClass have all the extra methods defined in StuffClass.
From your mention that the plugins need to "follows my API", I'm assuming that might be a case where you need to ensure that the plugins implement a set of methods in order to conform with the API; but since the implementation of the methods is going to depend on the specifics of the plugin, you can't provide those functions in MyClass. In that case, it sounds as though defining an Abstract Base Class that your plugins are required to inherit from might be useful for you.
Use __getattr__ to delegate the calls to Stuffclass's instance:
class MyClass:
def __init__(self):
self.st = Stuffclass()
def __getattr__(self,attr):
return getattr(self.st,attr)
Demo:
>>> from so import *
>>> m = MyClass()
>>> m.add(1,2)
3
>>> m.subtract(100,2)
98
I have the following code, which is a command line test
from cmd2 import Cmd
class App(Cmd, object):
def __init__(self, *args, **kwargs):
pass
def do_test(self, line):
'test'
print "parent test"
class App2():
def __init__(self, *args, **kwargs):
pass
def do_test2(self, line):
print "Test2"
app = App()
app.cmdloop()
Is there a possibility to extend the App class with extra functions?
I know there is the following solution
class App2(App):
....
app = App2()
app.cmdloop()
but in my case I would like to run only the App and extend it if it is possible.
In general this is not a good idea, because when people (including you in six months) read the code they will expect App to be the class they know about, and for an extended version of the class to have a different name. However, there's nothing preventing you from naming the subclass the same as the original:
class App(App):
# etc.
Python's smart enough to know that the App in parentheses means the one it already knows about, and then, since the new class has the same name, it replaces the original one. Don't worry, the new class contains a reference to the old one, so the old class doesn't go away entirely (if it did, nothing inherited by the subclass would work).
If the class came from some module that you've imported, you can even monkey-patch the replacement class back into the original module, so that all code that imports that module uses your replacement class. (Though I would recommend against it!)
import appmodule
class App(appmodule.App):
# etc.
appmodule.App = App
Of course, this gets tricky, because some modules may already have imported a reference to the original class, if you don't do this first thing in your script. And if other modules are also trying to patch the same class, all hell can break loose. Still, if you want to confuse yourself and those who will maintain your code, Python will let you do it!
It is worth noting that you can always augment the class dictionary, therefore extending it at runtime.
class App(...):
def __init__(self, a, b):
pass
def do_something(self, a):
pass
app_instance = App()
def do_something_else(self, b):
pass
App.do_something_else = do_something_else
app_instance.do_something_else('b')
You have to think how python does lookups at runtime. First looks at the instance of yourclass, then looks at the __mro__ (starting with type(yourclass)), and on up until it gets to object.
Since classes ARE objects, you can extend them by adding attributes, which will then be found during attribute lookups. Make sure you do this ONCE (eg, during an import of another file).
Here is a real example:
>>> class foo():
... pass
...
>>> x = foo()
>>>
>>> # Define a function and attach it
>>>
>>> def bar(self, a):
... print(a)
...
>>> foo.bar = bar
>>>
>>> x.bar('a')
a
this isnt an exact solution but it allows you to always access it by App name
rename App class to _App
then where you want to use it
from blah import _App as App
and when you extend it
from blah import App2 as App