Probably stupid question, and I've read many of the similar threads on here but still can't fathom the answer:
in main.py
from userMod import *
class Handler(webapp2.RequestHandler):
def write(self): #some code here etc
in userMod.py
class signup(Handler):
def get(self): #some code here etc
I get an error saying that Handler is not defined. My simple yet clearly stupid question is how can I access classes from a parent script within a loaded module? Or is it that I simply need to duplicate Handler in each module I create?
Bear in mind I've very new to Python and trying to make my code more modular by splitting out certain types of functions (in this case the user login and registration component of the site I'm building).
In usermod.py you need to import main, not the other way around.
Can't say for certain without seeing your code, but in that very basic snippet you basically have the imports reversed. #IgnacioVazquez-Abrams provided a link that will describe the whole process in general (and definitely better than I can), but at a basic level, each module exists in its own namespace and isn't aware of other modules unless you tell it about them.
So in your case, when you subclass Handler, the module has no idea what Handler is because it 1.) isn't a built-in, and 2.) hasn't been imported. Try this in usermod.py:
import main
class signup(main.Handler):
def get(self): #some code here etc
and see if it does what you want.
Related
As an example, let's say I am building an Rest API using Django Rest Framework. Now as part of the application, a few methods are common across all views. My approach is that in the root directory, I have created a services.py file. Inside that module, is a class (CommonUtils) containing all the common utility methods. In that same services.py module I have instantiated an object of CommonUtils.
Now across the application, in the different views.py files I am importing the object from the module and calling the methods on that object. So, essentially I am using a singleton object for the common utility methods.
I feel like this is not a good design approach. So, I want to get an explanation for why this approach is not a good idea and What would the best practice or best approach to achieve the same thing, i.e use a set of common utility methods across all views.py files.
Thanks in advance.
Is this the right design? Why? How to do better?
I feel like this is not a good design approach. So, I want to get an explanation for why this approach is not a good idea and What would the best practice or best approach to achieve the same thing, i.e use a set of common utility methods across all views.py files.
Like #Dmitry Belaventsev wrote above, there is no general rule to solve this problem. This is a typical case of cross-cutting-concerns.
Now across the application, in the different views.py files I am importing the object from the module and calling the methods on that object. So, essentially I am using a singleton object for the common utility methods.
Yes, your implementation is actually a singleton and there is nothing wrong with it. You should ask yourself what do you want to achieve or what do you really need. There are a lot of solutions and you can start with the most basic one:
A simple function in a python module
# file is named utils.py and lives in the root directory
def helper_function_one(param):
return transcendent_all_evil_of(param)
def helper_function_two(prename, lastname):
return 'Hello {} {}'.format(prename, lastname)
In Python it is not uncommon to use just plain functions in a module. You can upgrade it to a method (and a class) if this is really necessary and you need the advantages of classes and objects.
You also can use a class with static methods:
# utils.py
class Utils():
#staticmethod
def helper_one():
print('do something')
But you can see, this is nothing different than the solution with plain functions besides the extra layer of the class. But it has no further value.
You could also write a Singleton Class but in my opinion, this is not very pythonic, because you get the same result with a simple object instance in a module.
I have a lot of different child classes that inherit from one base class. However all the different child classes implement very similar methods. So if I want to change code in the child classes, I have to change it multiple times.
For me this sounds like bad practice and I would like to implement it correcty. But after a lot of googling I still didn't find a coherent way of how this should be done.
Here is an example of what I mean:
from ABC import ABC, abstractmethod
import logging.config
class BaseModel(ABC):
def __init__(self):
# initialize logging
logging.config.fileConfig(os.path.join(os.getcwd(),
'../myconfig.ini'))
self.logger = logging.getLogger(__name__)
#abstractmethod
def prepare_data(self):
"""
Prepares the needed data.
"""
self.logger.info('Data preparation started.\n')
pass
So this is my BaseClass. Now from this class multiple other classes inherit the init and prepare_data method. The prepare_data method is very similar for every class.
class Class_One(BaseModel):
def __init__(self):
super.__init()__
def prepare_data(self):
super().prepare_data()
# Some code that this method does
class Class_Two(BaseModel):
def __init__(self):
super.__init()__
def prepare_data(self):
super().prepare_data()
# Some code that this method does
# Code is almost the same as for Class_One
class Class_Three(BaseModel):
def __init__(self):
super.__init()__
def prepare_data(self):
super().prepare_data()
# Some code that this method does
# Code is almost the same as for Class_One and Class_Two
# etc.
I suppose you could refactor the methods into another file and then call them in each class. I would love to know how to do this correctly. Thanks a lot in advance!
I'm afraid there's no generic one-size-fits-all magic answer - it all really depend on the "almost" part AND on the forces that will drive change in those parts of the code. IOW, one can only really answer on a concrete example...
This being said, there are a couple lessons learned from experience, which are mostly summmarized in the famous (but unfortunately often misunderstood) GOF "Design Patterns" book. If you take time to first read the first part of the book, you understand that most of (if not all) the patterns in the catalog are based on the same principle: separate the variant from the invariant. Once you can tell one from the other in your code (warning: there's a trap here and beginner almost always fall into it), which pattern to apply is usually obvious (sometimes to the point you only realize you used this and that patterns after you refactored your code).
Now as I said, there IS a trap: accidental duplication. Just because two pieces of code look similar doesn't mean they are duplicates - quite often, they are only "accidentally" similar now but the forces that will make one or the other change are mostly unrelated. If you try to immediatly refactor this code, you'll soon find yourself making the "generic" case more and more complicated to support changes that are actually unrelated, and end up with an overcomplicated, undecipherable mess that only make your code unmaintainable. So the trick here is to carefully examine the whole context, ask yourself what would drive change in one or the other "similar" parts, and if in doubt, wait until you know more. If it happens than each time you change A you have to make the exact same change in B for the exact same reasons then you DO have real duplicate.
For a more practical, short-term advise based on what we can guess from your way too abstract example (and from experience), there are at least two patterns that are most often involved in factoring out duplication in a class hierarchy: the template method and the strategy.
NB : I said "unfortunately often misunderstood" because most people seem to jump to the patterns catalog and try to forcefit all of them in their code (whether it makes sense for the problem at hand or not), and usually by copy-pasting the canonical textbook _implementation_ (usually Java or C++ based) instead of understanding the _concept_ and implementing it in a way that's both idiomatic and adapted to the concrete use case (example: when functions are first class object, you don't necessarily need a Strategie class with abstract base and concrete subclasses - most often plain old callback functions JustWork(tm)).
EDIT totally unrelated but this:
def __init__(self):
# initialize logging
logging.config.fileConfig(os.path.join(os.getcwd(),
'../myconfig.ini'))
self.logger = logging.getLogger(__name__)
is NOT how to use logging. Library code can use loggers, but must not configure anything - this is the application's (your main script / function / whatever) responsability, the rational being that the proper logging config depends on the context - which type of application is using the lib (a CLI app, a local GUI app and a backend web app don't have the same needs at all) and in which kind of environment (a local dev env will want much more logs than a production one for example).
Also, with the logger created with __name__ in your base class module, all child classes will send their log to the same logger, which is certainly not what you want (you want them to have their own package / module specific loggers so you can fine tune the config per package / module).
And finally, this:
os.path.join(os.getcwd(), '../myconfig.ini')
certainly doesn't work as you expect - your cwd can be just anything at this point and you have no way of knowing in advance. If you want to reference a path relative to the current file's directory, you want os.path.dirname(os.path.realpath(__file__)). And of course adding system specific path stuff (ie "../") in a os.path.join() call totally defeats the whole point of using os.path.
Python is extremely elegant language. Well, except... except imports. I still can't get it work the way it seems natural to me.
I have a class MyObjectA which is in file mypackage/myobjecta.py. This object uses some utility functions which are in mypackage/utils.py. So in my first lines in myobjecta.py I write:
from mypackage.utils import util_func1, util_func2
But some of the utility functions create and return new instances of MyObjectA. So I need to write in utils.py:
from mypackage.myobjecta import MyObjectA
Well, no I can't. This is a circular import and Python will refuse to do that.
There are many question here regarding this issue, but none seems to give satisfactory answer. From what I can read in all the answers:
Reorganize your modules, you are doing it wrong! But I do not know
how better to organize my modules even in such a simple case as I
presented.
Try just import ... rather than from ... import ...
(personally I hate to write and potentially refactor all the full
name qualifiers; I love to see what exactly I am importing into
module from the outside world). Would that help? I am not sure,
still there are circular imports.
Do hacks like import something in the inner scope of a function body just one line before you use something from other module.
I am still hoping there is solution number 4) which would be Pythonic in the sense of being functional and elegant and simple and working. Or is there not?
Note: I am primarily a C++ programmer, the example above is so much easily solved by including corresponding headers that I can't believe it is not possible in Python.
There is nothing hackish about importing something in a function body, it's an absolutely valid pattern:
def some_function():
import logging
do_some_logging()
Usually ImportErrors are only raised because of the way import() evaluates top level statements of the entire file when called.
In case you do not have a logic circular dependency...
, nothing is impossible in python...
There is a way around it if you positively want your imports on top:
From David Beazleys excellent talk Modules and Packages: Live and Let Die! - PyCon 2015, 1:54:00, here is a way to deal with circular imports in python:
try:
from images.serializers import SimplifiedImageSerializer
except ImportError:
import sys
SimplifiedImageSerializer = sys.modules[__package__ + '.SimplifiedImageSerializer']
This tries to import SimplifiedImageSerializer and if ImportError is raised (due to a circular import error or the it not existing) it will pull it from the importcache.
PS: You have to read this entire post in David Beazley's voice.
Don't import mypackage.utils to your main module, it already exists in mypackage.myobjecta. Once you import mypackage.myobjecta the code from that module is being executed and you don't need to import anything to your current module, because mypackage.myobjecta is already complete.
What you want isn't possible. There's no way for Python to know in which order it needs to execute the top-level code in order to do what you ask.
Assume you import utils first. Python will begin by evaluating the first statement, from mypackage.myobjecta import MyObjectA, which requires executing the top level of the myobjecta module. Python must then execute from mypackage.utils import util_func1, util_func2, but it can't do that until it resolves the myobjecta import.
Instead of recursing infinitely, Python resolves this situation by allowing the innermost import to complete without finishing. Thus, the utils import completes without executing the rest of the file, and your import statement fails because util_func1 doesn't exist yet.
The reason import myobjecta works is that it allows the symbols to be resolved later, after the body of every module has executed. Personally, I've run into a lot of confusion even with this kind of circular import, and so I don't recommend using them at all.
If you really want to use a circular import anyway, and you want them to be "from" imports, I think the only way it can reliably work is this: Define all symbols used by another module before importing from that module. In this case, your definitions for util_func1 and util_func2 must be before your from mypackage.myobjecta import MyObjectA statement in utils, and the definition of MyObjectA must be before from mypackage.utils import util_func1, util_func2 in myobjecta.
Compiled languages like C# can handle situations like this because the top level is a collection of definitions, not instructions. They don't have to create every class and every function in the order given. They can work things out in whatever order is required to avoid any cycles. (C++ does it by duplicating information in prototypes, which I personally feel is a rather hacky solution, but that's also not how Python works.)
The advantage of a system like Python is that it's highly dynamic. Yes you can define a class or a function differently based on something you only know at runtime. Or modify a class after it's been created. Or try to import dependencies and go without them if they're not available. If you don't feel these things are worth the inconvenience of adhering to a strict dependency tree, that's totally reasonable, and maybe you'd be better served by a compiled language.
Pythonistas frown upon importing from a function. Pythonistas usually frown upon global variables. Yet, I saw both and don't think the projects that used them were any worse than others done by some strict Pythhonistas. The feature does exist, not going into a long argument over its utility.
There's an alternative to the problem of importing from a function: when you import from the top of a file (or the bottom, really), this import will take some time (some small time, but some time), but Python will cache the entire file and if another file needs the same import, Python can retrieve the module quickly without importing. Whereas, if you import from a function, things get complicated: Python will have to process the import line each time you call the function, which might, in a tiny way, slow your program down.
A solution to this is to cache the module independently. Okay, this uses imports inside function bodies AND global variables. Wow!
_MODULEA = None
def util1():
if _MODULEA is None:
from mymodule import modulea as _MODULEA
obj = _MODULEA.ClassYouWant
return obj
I saw this strategy adopted with a project using a flat API. Whether you like it or not (and I'm not sure about that myself), it works and is fast, because the import line is executed only once (when the function first executes). Still, I would recommend restructuring: problems with circular imports show a problem in structure, usually, and this is always worth fixing. I do agree, though, it would be nice if Python provided more useful errors when this kind of situation happens.
General Python Question
I'm importing a Python library (call it animals.py) with the following class structure:
class Animal(object): pass
class Rat(Animal): pass
class Bat(Animal): pass
class Cat(Animal): pass
...
I want to add a parent class (Pet) to each of the species classes (Rat, Bat, Cat, ...); however, I cannot change the actual source of the library I'm importing, so it has to be a run time change.
The following seems to work:
import animals
class Pet(object): pass
for klass in (animals.Rat, animals.Bat, animals.Cat, ...):
klass.__bases__ = (Pet,) + klass.__bases__
Is this the best way to inject a parent class into an inheritance tree in Python without making modification to the source definition of the class to be modified?
Motivating Circumstances
I'm trying to graft persistence onto the a large library that controls lab equipment. Messing with it is out of the question. I want to give ZODB's Persistent a try. I don't want to write the mixin/facade wrapper library because I'm dealing with 100+ classes and lots of imports in my application code that would need to be updated. I'm testing options by hacking on my entry point only: setting up the DB, patching as shown above (but pulling the species classes w/ introspection on the animals module instead of explicit listing) then closing out the DB as I exit.
Mea Culpa / Request
This is an intentionally general question. I'm interested in different approaches to injecting a parent and comments on the pros and cons of those approaches. I agree that this sort of runtime chicanery would make for really confusing code. If I settle on ZODB I'll do something explicit. For now, as a regular user of python, I'm curious about the general case.
Your method is pretty much how to do it dynamically. The real question is: What does this new parent class add? If you are trying to insert your own methods in a method chain that exists in the classes already there, and they were not written properly, you won't be able to; if you are adding original methods (e.g. an interface layer), then you could possibly just use functions instead.
I am one who embraces Python's dynamic nature, and would have no problem using the code you have presented. Make sure you have good unit tests in place (dynamic or not ;), and that modifying the inheritance tree actually lets you do what you need, and enjoy Python!
You should try really hard not to do this. It is strange, and will likely end in tears.
As #agf mentions, you can use Pet as a mixin. If you tell us more about why you want to insert a parent class, we can help you find a nicer solution.
Is there is a super global (like PHP) in Python? I have certain variables I want to use throughout my whole project in separate files, classes, and functions, and I don't want to have to keep declaring it throughout each file.
In theory yes, you can start spewing crud into __builtin__:
>>> import __builtin__
>>> __builtin__.rubbish= 3
>>> rubbish
3
But, don't do this; it's horrible evilness that will give your applications programming-cancer.
classes and functions and i don't want to have to keep declaring
Put them in modules and ‘import’ them when you need to use them.
I have certain variables i want to use throughout my whole project
If you must have unqualified values, just put them in a file called something like “mypackage/constants.py” then:
from mypackage.constants import *
If they really are ‘variables’ in that you change them during app execution, you need to start encapsulating them in objects.
Create empty superglobal.py module.
In your files do:
import superglobal
superglobal.whatever = loacalWhatever
other = superglobal.other
Even if there are, you should not use such a construct EVER. Consider using a borg pattern to hold this kind of stuff.
class Config:
"""
Borg singlton config object
"""
__we_are_one = {}
__myvalue = ""
def __init__(self):
#implement the borg patter (we are one)
self.__dict__ = self.__we_are_one
def myvalue(self, value=None):
if value:
self.__myvalue = value
return self.__myvalue
conf = Config()
conf.myvalue("Hello")
conf2 = Config()
print conf2.myvalue()
Here we use the borg pattern to create a singlton object. No matter where you use this in the code, the 'myvalue' will be the same, no matter what module or class you instantiate Config in.
in years of practice, i've grown quite disappointed with python's import system: it is complicated and difficult to handle correctly. also, i have to maintain scores of imports in each and every module i write, which is a pita.
namespaces are a very good idea, and they're indispensable---php doesn't have proper namespaces, and it's a mess.
conceptually, part of writing an application consists in defining a suitable vocabulary, the words that you'll use to do the things you want to. yet in the classical way, it's exactly these words that won't come easy, as you have to first import this, import that to obtain access.
when namespaces came into focus in the javascript community, john resig of jquery fame decided that providing a single $ variable in the global namespace was the way to go: it would only affect the global namespace minimally, and provide easy access to everything with jquery.
likewise, i experimented with a global variable g, and it worked to an extent. basically, you have two options here: either have a startup module that must be run prior to any other module in your app, which defines what things should be available in g, so it is ready to go when needed. the other approach which i tried was to make g lazy and to react with custom imports when a new name was required; so whenever you need to g.foo.frob(42) for the first time, the mechanism will try something like import foo; g.foo = foo behind the scenes. that was considerable more difficult to do right.
these days, i've ditched the import system almost completely except for standard library and site packages. most of the time i write wrappers for hose libraries, as 90% of those have inanely convoluted interfaces anyhow. those wrappers i then publish in the global namespace, using spelling conventions to keep the risk of collisions to a minimum.
i only tell this to alleviate the impression that modifying the global namespace is something that is inherently evil, which the other answers seem to state. not so. what is evil is to do it thoughtlessly, or be compelled by language or package design to do so.
let me add one remark, as i almost certainly will get some fire here: 99% of all imports done by people who religiously defend namespace purity are wrong. proof? you'll read in the beginning lines of any module foo.py that needs to do trigonometrics something like from math import sin. now when you correctly import foo and have a look at that namespace, what are you going to find? something named foo.sin. but that sin isn't part of the interface of foo, it is just a helper, it shouldn't clutter that namespace---hence, from math import sin as _sin or somesuch would've been correct. however, almost nobody does it that way.
i'm sure to arouse some heated comments with these views, so go ahead.
The reason it wasn't obvious to you is that Python intentionally doesn't try to support such a thing. Namespaces are a feature, and using them is to your advantage. If you want something you defined in another file, import it. This means from reading your source code you can figure out where everything came from, and also makes your code easier to test and modify.