I have a module which can be described as
python
class Symbol():
def __init__(data):
self.data = data
pass
def __add__(self,other):
return Add(self,other)
class Integer(Symbol):
pass
class Add(Symbol):
def __init__(a,b):
self.data = [a,b]
I want to split it into three files, which are symbol.py, integer.py and add.py; there are of course going to be a lot more details on those classes so having them in one files is ridiculous.
For some reason the imports never seem to work, while it's not even complaining of circular dependencies, can someone give me a little example?
Your circular dependency situation isn't unsolvable, because Symbol doesn't depend on Add at definition time, only when the __add__ method is called. There are two good ways to resolve it.
The first is to not have the module Symbol is in import Add at top level, but only do that within the __add__ method itself. For instance, if your modules were named after the classes (only lowercase), you'd use this in symbol.py:
class Symbol():
# ...
def __add__(self,other):
from add import Add
return Add(self,other)
The other approach is to import Add globally into the symbol module, but do so after the definintion of the Symbol class. This way, when the add module imports symbol back, it will always be able to see the Symbol class's definition, even if the rest of the module is not finished loading.
class Symbol():
# same as in your current code
from add import Add
If you go with this second approach and the Symbol class imports other stuff at the top of the file (where import statements normally are put), you might want to add a comment in that space about Add being imported later (and why).
Related
I am struggling with what seems to me a very basic and common problem, and the fact that I could not find any answer after hours of Internet searching tells me that I must be doing something very wrong...
I am simply trying to find an elegent way to handle imports with my package.
The background :
My package is structured like this :
mypackage/
__init__.py
model/
__init__.py
A.py
B.py
controllers/
__init__.py
A.py
B.py
# mypackage/model/A.py
class A:
def __init__(self):
print("This is A's model.")
# mypackage/model/B.py
from mypackage.model.A import A as AModel
class B:
def __init__(self):
self._a_model = AModel()
print("This is B's model.")
# mypackage/controllers/A.py
class A:
def __init__(self):
print("This is A's controller.")
# mypackage/controllers/B.py
from mypackage.controllers.A import A as AController
class B:
def __init__(self):
self._a = AController()
print("This is B's controller.")
The problem :
Two things are really bothering me with this design.
First : I want to use namespaces
I really don't like writing
from mypackage.controllers.A import A as AController
...
self._a = AController()
It feels cumbersome and not very pythonic...
I would prefer using namespaces like in :
from mypackage import controllers
...
self._a = controllers.A.A()
But if I try, I get a AttributeError: module 'mypackage.controllers' has no attribute 'A'
Second : I really don't like typing the class's filename
I really don't like writing
from mypackage.controllers.A import A as AController
I would prefer :
from mypackage.controllers import A as AController
What did not work
Putting everything in one file
I understand that I could get what I want by puting all controller's class (A and B) defininitions in a single file (controllers.py) and do the same with the models...
I read several time that putting several class definitions in a single file is a quite common thing to do in python... But for big separate classes I just can't. If A and B are hundreds of lines and have nothing to do with each other (other than being controllers and models), having their definitions in a single file is unusable.
Using imports in the init.py files
Indeed it would solve all my problems...
Except that :
It leads to circular imports. As you can see I need A's model in B's model... So if all models are imported when I need to access one of them, I'm stuck in vicious circle...
It does not seems very pythonic. If only because it forces the user to load every modules.
Here I am...
What is wrong with my reasoning ?
Using imports in the __init__.py file
That is the way to go.
Sorry if it looks too much boiler plate for you, but if yru project is big, that is what is needed.
As for the circular import problem: it will kick in wether you write your imports
in the __init__ files or not, and is just a matter of logistic.
If in your __init__ file, you write the imports in the correct order, there will be no circularity problem.
I.e. in your myproject/models/__init__.py you have:
from .A import A as AModel
from .B import B as BModel
Of course you naming the .py files the same names as the classes won't help you - if you will at least let go of the casing in the filename you can write:
from .a import A
from .b import B
Otherwise, you can do just:
import myproject.models.A
import myproject.models.B
A = myproject.models.A.A
B = myproject.models.B.B
To be able to use "myproject.models.A" as the class:
The name A inside __init__ will override the module object
with the same name.
One writting
import myproject.models.A will get to the module, but by doing
from myproject.models import A you get the module.
If that feels confusing... try not to use the same name for the module file than the classes. Even because in case-ignoring file systems, like Windows you would
ambiguities anyway. Stick with the convention: module names in snake_case, class names in CamelCase
Back to the circular-imports matter: the point is that in this design, b.py is only read after a.py has been already imported, and no circular-import problem.
That is not always possible - sometimes cross-reference between submodules are needed. What is possible to do in these cases is to move the import lines into
the functions or methods, instead of as a global statement. That way, when the import is executed, the referred module is already initialised.
In your example that would be:
mypackage.models.b.py
# mypackage/model/B.py
class B:
def __init__(self):
from mypackage.model.A import A as AModel
self._a_model = AModel()
print("This is B's model.")
# mypackage/controllers/__init__.py
from A import A
Then you can make a new file outside of mypackage with.
# check.py
from mypackage.controllers import A as AController
from mypackage import controllers
a = controllers.A()
>>> This is A's controller.
let us know if it works for you.
[Off the top of my head, without testing]
I really don't like writing
from mypackage.controllers.A import A as AController
#...
self._a = AController()
It feels cumbersome and not very pythonic... I would prefer using
namespaces like in :
from mypackage import controllers
# ...
self._a = controllers.A.A()
In mypackage/controllers/__init__.py you would need: from . import A.
I really don't like writing
from mypackage.controllers.A import A as AController
I would prefer :
from mypackage.controllers import A as AController
In mypackage/controllers/__init__.py you would need from A import A.
I have a code like below. My requirement is to fill BagA and BagB once and only once. So, I do not have a class but have written code in a module. I have __BagA and __BagB with underscores because I do not want it to get changed from outside. I have a getter as shown using #property decorator. But it does not work.
module.py:
import csv
import sys
import threading
__BagA = list()
__BagB = dict()
__lock = threading.Lock()
#property
def BagA():
if not __BagA:
raise("Call Initialize() first")
else:
return __BagA
#property
def BagB():
if not __BagB:
raise("Call lookup.Initialize() first")
else:
return __BagB
def __get__BagA(directory):
#logic to populate BagA
BagA=some list
def __get__BagB(directory):
#logic to populate BagB
BagB=dict after processing the logic
def initialize(directory):
__lock.acquire()
__get__BagA(directory)
__get__BagB(directory)
__lock.release()
Main script:
import module
module.initialize("dir")#this works. I see all bags getting populated
#Below does not work
module.BagA #This gives an object with some fget and fset and not the value of BagA
#below works
module.__BagA
One more problem is that instead of __BagA which could still be modified from outside, I could write BagA and have #Bag.setter decorator and #BagA.deleter decorator which returns an exception saying 'you cannot modify this'.
These are also not working and I do not want to write them separately but in a single line like below because they have same behavior. Both separate and single line do not work:
#BagA.setter,#BagA.deleter
def BagA(value):
print('cannot set or delete') #Is this possible?
I have __BagA and __BagB with underscores because I do not want it to get changed from outside.
that's not what it's for, the double underscore is a way to avoid name collisions in inheritance scenarios
that's not how it works, the double underscore just mangles names using a well-defined scheme, the double underscore does not make things private or protected, they're still completely public and visible to anyone who cares
that doesn't do anything at the module level, it's implemented at the class level
I have a getter as shown using #property decorator. But it does not work.
#property is a descriptor, it works through the attribute-resolution mechanism of classes.
Here's how you say that a symbol is "private" and shouldn't be touched in python: you prefix it with a single underscore. Then if others touch it that's their problem for being idiots or assholes.
TL; DR
Basically the question is about hiding from the user the fact that my modules have class implementations so that the user can use the module as if it has direct function definitions like my_module.func()
Details
Suppose I have a module my_module and a class MyThing that lives in it. For example:
# my_module.py
class MyThing(object):
def say():
print("Hello!")
In another module, I might do something like this:
# another_module.py
from my_module import MyThing
thing = MyThing()
thing.say()
But suppose that I don't want to do all that. What I really want is for my_module to create an instance of MyThing automatically on import such that I can just do something like the following:
# yet_another_module.py
import my_module
my_module.say()
In other words, whatever method I call on the module, I want it to be forwarded directly to a default instance of the class contained in it. So, to the user of the module, it might seem that there is no class in it, just direct function definitions in the module itself (where the functions are actually methods of a class contained therein). Does that make sense? Is there a short way of doing this?
I know I could do the following in my_module:
class MyThing(object):
def say():
print("Hello!")
default_thing = MyThing()
def say():
default_thing.say()
But then suppose MyThing has many "public" methods that I want to use, then I'd have to explicitly define a "forwarding" function for every method, which I don't want to do.
As an extension to my question above, is there a way to achieve what I want above, but also be able to use code like from my_module import * and be able to use methods of MyThing directly in another module, like say()?
In module my_module do the following:
class MyThing(object):
...
_inst = MyThing()
say = _inst.say
move = _inst.move
This is exactly the pattern used by the random module.
Doing this automatically is somewhat contrived. First, one needs to find out which of the instance/class attributes are the methods to export... perhaps export only names which do not start with _, something like
import inspect
for name, member in inspect.getmembers(Foo(), inspect.ismethod):
if not name.startswith('_'):
globals()[name] = member
However in this case I'd say that explicit is better than implicit.
You could just replace:
def say():
return default_thing.say()
with:
say = default_thing.say
You still have to list everything that's forwarded, but the boilerplate is fairly concise.
If you want to replace that boilerplate with something more automatic, note that (details depending on Python version), MyThing.__dict__.keys() is something along the lines of ['__dict__', '__weakref__', '__module__', 'say', '__doc__']. So in principle you could iterate over that, skip the __ Python internals, and call setattr on the current module (which is available as sys.modules[__name__]). You might later regret not listing this stuff explicitly in the code, but you could certainly do it.
Alternatively you could get rid of the class entirely as use the module as the unit of encapsulation. Wherever there is data on the object, replace it with global variables. "But", you might say, "I've been warned against using global variables because supposedly they cause problems". The bad news is that you've already created a global variable, default_thing, so the ship has sailed on that one. The even worse news is that if there is any data on the object, then the whole concept of what you want to do: module-level functions that mutate a shared global state, carries with it most of the problems of globals.
Not Sure why this wouldn't work.
say = MyClass().say()
from my_module import *
say
>>Hello!
I'm developing a PyQT4 application, and it's getting pretty hard for me to navigate through all of the code at once. I know of the import foo statement, but I can't figure out how to make it import a chunk of code directly into my script, like the BASH source foo statement.
I'm trying to do this:
# File 'functions.py'
class foo(asd.fgh):
def __init__(self):
print 'foo'
Here is the second file.
# File 'main.py'
import functions
class foo(asd.fgh):
def qwerty(self):
print 'qwerty'
I want to include code or merge class decelerations from two separate files. In PHP, there is import_once('foo.php'), and as I mentioned previously, BASH has source 'foo.sh', but can I accomplish this with Python?
Thanks!
For some reason, my first thought was multiple inheritance. But why not try normal inheritance?
class foo(functions.foo):
# All of the methods that you want to add go here.
Is there some reason that this wont work?
Since you just want to merge class definitions, why don't you do:
# main.py
import functions
# All of the old stuff that was in main.foo is now in this class
class fooBase(asd.fgh):
def qwerty(self):
print 'qwerty'
# Now create a class that has methods and attributes of both classes
class foo(FooBase, functions.foo): # Methods from FooBase take precedence
pass
or
class foo(functions.foo, FooBase): # Methods from functions.foo take precedence
pass
This takes advantage of pythons capability for multiple inheritance to create a new class with methods from both sources.
You want execfile(). Although you really don't, since redefining a class, uh... redefines it.
monkey patching in python doesn't work in nearly the same way. This is normally considered poor form, but if you want to do it anyways, you can do this:
# File 'functions.py'
class foo(asd.fgh):
def __init__(self):
print 'foo'
the imported module remains unchanged. In the importing module, we do things quite differently.
# File 'main.py'
import functions
def qwerty(self):
print 'qwerty'
functions.foo.qwerty = qwerty
Note that there is no additional class definition, just a bare function. we then add the function as an attribute of the class.
What are the best practices for extending an existing Python module – in this case, I want to extend the python-twitter package by adding new methods to the base API class.
I've looked at tweepy, and I like that as well; I just find python-twitter easier to understand and extend with the functionality I want.
I have the methods written already – I'm trying to figure out the most Pythonic and least disruptive way to add them into the python-twitter package module, without changing this modules’ core.
A few ways.
The easy way:
Don't extend the module, extend the classes.
exttwitter.py
import twitter
class Api(twitter.Api):
pass
# override/add any functions here.
Downside : Every class in twitter must be in exttwitter.py, even if it's just a stub (as above)
A harder (possibly un-pythonic) way:
Import * from python-twitter into a module that you then extend.
For instance :
basemodule.py
class Ball():
def __init__(self,a):
self.a=a
def __repr__(self):
return "Ball(%s)" % self.a
def makeBall(a):
return Ball(a)
def override():
print "OVERRIDE ONE"
def dontoverride():
print "THIS WILL BE PRESERVED"
extmodule.py
from basemodule import *
import basemodule
def makeBalls(a,b):
foo = makeBall(a)
bar = makeBall(b)
print foo,bar
def override():
print "OVERRIDE TWO"
def dontoverride():
basemodule.dontoverride()
print "THIS WAS PRESERVED"
runscript.py
import extmodule
#code is in extended module
print extmodule.makeBalls(1,2)
#returns Ball(1) Ball(2)
#code is in base module
print extmodule.makeBall(1)
#returns Ball(1)
#function from extended module overwrites base module
extmodule.override()
#returns OVERRIDE TWO
#function from extended module calls base module first
extmodule.dontoverride()
#returns THIS WILL BE PRESERVED\nTHIS WAS PRESERVED
I'm not sure if the double import in extmodule.py is pythonic - you could remove it, but then you don't handle the usecase of wanting to extend a function that was in the namespace of basemodule.
As far as extended classes, just create a new API(basemodule.API) class to extend the Twitter API module.
Don't add them to the module. Subclass the classes you want to extend and use your subclasses in your own module, not changing the original stuff at all.
Here’s how you can directly manipulate the module list at runtime – spoiler alert: you get the module type from types module:
from __future__ import print_function
import sys
import types
import typing as tx
def modulize(namespace: tx.Dict[str, tx.Any],
modulename: str,
moduledocs: tx.Optional[str] = None) -> types.ModuleType:
""" Convert a dictionary mapping into a legit Python module """
# Create a new module with a trivially namespaced name:
namespacedname: str = f'__dynamic_modules__.{modulename}'
module = types.ModuleType(namespacedname, moduledocs)
module.__dict__.update(namespace)
# Inspect the new module:
name: str = module.__name__
doc: tx.Optional[str] = module.__doc__
contents: str = ", ".join(sorted(module.__dict__.keys()))
print(f"Module name: {name}")
print(f"Module contents: {contents}")
if doc:
print(f"Module docstring: {doc}")
# Add to sys.modules, as per import machinery:
sys.modules.update({ modulename : module })
# Return the new module instance:
return module
… you could then use such a function like so:
ns = {
'func' : lambda: print("Yo Dogg"), # these can also be normal non-lambda funcs
'otherfunc' : lambda string=None: print(string or 'no dogg.'),
'__all__' : ('func', 'otherfunc'),
'__dir__' : lambda: ['func', 'otherfunc'] # usually this’d reference __all__
}
modulize(ns, 'wat', "WHAT THE HELL PEOPLE")
import wat
# Call module functions:
wat.func()
wat.otherfunc("Oh, Dogg!")
# Inspect module:
contents = ", ".join(sorted(wat.__dict__.keys()))
print(f"Imported module name: {wat.__name__}")
print(f"Imported module contents: {contents}")
print(f"Imported module docstring: {wat.__doc__}")
… You could also create your own module subclass, by specifying types.ModuleType as the ancestor of your newly declared class, of course; I have never personally found this necessary to do.
(Also, you don’t have to get the module type from the types module – you can always just do something like ModuleType = type(os) after importing os – I specifically pointed out this one source of the type because it is non-obvious; unlike many of its other builtin types, Python doesn’t offer up access to the module type in the global namespace.)
The real action is in the sys.modules dict, where (if you are appropriately intrepid) you can replace existing modules as well as adding your new ones.
Say you have an older module called mod that you use like this:
import mod
obj = mod.Object()
obj.method()
mod.function()
# and so on...
And you want to extend it, without replacing it for your users. Easily done. You can give your new module a different name, newmod.py or place it by same name at a deeper path and keep the same name, e.g. /path/to/mod.py. Then your users can import it in either of these ways:
import newmod as mod # e.g. import unittest2 as unittest idiom from Python 2.6
or
from path.to import mod # useful in a large code-base
In your module, you'll want to make all the old names available:
from mod import *
or explicitly name every name you import:
from mod import Object, function, name2, name3, name4, name5, name6, name7, name8, name9, name10, name11, name12, name13, name14, name15, name16, name17, name18, name19, name20, name21, name22, name23, name24, name25, name26, name27, name28, name29, name30, name31, name32, name33, name34, name35, name36, name37, name38, name39
I think the import * will be more maintainable for this use-case - if the base module expands functionality, you'll seamlessly keep up (though you might shade new objects with the same name).
If the mod you are extending has a decent __all__, it will restrict the names imported.
You should also declare an __all__ and extend it with the extended module's __all__.
import mod
__all__ = ['NewObject', 'newfunction']
__all__ += mod.__all__
# if it doesn't have an __all__, maybe it's not good enough to extend
# but it could be relying on the convention of import * not importing
# names prefixed with underscores, (_like _this)
Then extend the objects and functionality as you normally would.
class NewObject(object):
def newmethod(self):
"""this method extends Object"""
def newfunction():
"""this function builds on mod's functionality"""
If the new objects provide functionality you intend to replace (or perhaps you are backporting the new functionality into an older code base) you can overwrite the names
May I suggest not to reinvent the Wheel here? I'm building a >6k line Twitter Client for 2 month now, at first I checked python-twitter too, but it's lagging a lot behind the recent API changes,, Development doesn't seem to be that active either, also there was(at least when I last checked) no support for OAuth/xAuth).
So after searching around a bit more I discovered tweepy:
http://github.com/joshthecoder/tweepy
Pros: Active development, OAauth/xAuth and up to date with the API.
Chances are high that what you need is already in there.
So I suggest going with that, it's working for me, the only thing I had to add was xAuth(that got merge back to tweepy :)
Oh an a shameless plug, if you need to parse Tweets and/or format them to HTML use my python version of the twitter-text-* libraries:
http://github.com/BonsaiDen/twitter-text-python
This thing is unittestetd an guaranteed to parse Tweets just like Twitter.com does it.
Define a new class, and instead of inherit it from the class you want to extend from the original module, add an instance of the original class as an attribute to your new class.
And here comes the trick: intercept all non-existing method calls on your new class and try to call it on the instance of the old class.
In your NewClass just define new or overridden methods as you like:
import originalmodule
class NewClass:
def __init__(self, *args, **kwargs):
self.old_class_instance = originalmodule.create_oldclass_instance(*args, **kwargs)
def __getattr__(self, methodname):
"""This is a wrapper for the original OldClass class.
If the called method is not part of this NewClass class,
the call will be intercepted and replaced by the method
in the original OldClass instance.
"""
def wrapper(*args, **kwargs):
return getattr(self.old_class_instance, methodname)(*args, **kwargs)
return wrapper
def new_method(self, arg1):
"""Does stuff with the OldClass instance"""
thing = self.old_class_instance.get_somelist(arg1)
# returns the first element only
return thing[0]
def overridden_method(self):
"""Overrides an existing method, if OldClass has a method with the same name"""
print("This message is coming from the NewClass and not from the OldClass")
In my case I used this solution when simple inheritance from the old class was not possible, because an instance had to be created not by its constructor, but with an init script from an other class/module. (It is the originalmodule.create_oldclass_instance in the example above.)