I'm pretty new to Python as to OOP in general which is probably be the reason that I can't figure out the following:
I'm writing a python script which opens a text file and subsequently translates it into HTML, maintaining it's "own" mirrored directory-trees for the edit files and the html files.
Since directory creation and deletion is done automatically depending on, among other criteria, whether the file existed before or not, I need some kind of automatic and dynamic path adjustment. The script has to do several checks for files and associated directories before it can set the corresponding paths and start the processing.
I decided to put most of the more general functions (check file existence, show dialogs for duplicate filenames if found, etc) in a separate module, since they are quite specific and not depending on any state. Actually they create the state (path variables), so a class would not make sense if this is not a misconception.
On the other hand I'm using a class for the pure getting and setting the paths since I need the paths accessible from every module, so it's basically a global access point for paths.
This class is instantiated in the main module.
Now my problem is that I can't figure out how to manipulate the paths (using the path
setters) of that instance in the main module from a function inside the tools module. Importing the class instance or the main module into the tools module doesn't seem to work.
Generally speaking, is it possible to use a class instance across all module files and is this the way to go, or am I missing the point somehow?
I paste the relevant bits of code for illustration:
Setter/Getter class inside the main module
class SetEnv():
def __init__(self):
pass
def set_path_srcfile(self, path_srcfile):
self.path_srcfile = path_srcfile
def set_path_htmlfile(self):
self.path_htmlfile = self.path_srcfile.replace('source', 'html', 1).replace('.txt', '.html', 1)
def get_path_srcfile(self):
return self.path_srcfile
def get_path_htmlfile(self):
return self.path_htmlfile
Later in main_module:
env = SetEnv()
Part of tools module (inside a def acting upon user input):
import main_module as mm
path_srcfile = dict[int(user_option)][1] # dictionary holding the path we want to set
mm.env.set_path_srcfile(path_srcfile)
mm.env.set_path_htmlfile()
I might be misinterpreting your question, correct me if I am. As I understand it, you are using one single instance of a SetEnv object across an entire project to store and modify some path configuration.
If you really want a singleton like settings object, then use a module instead of a class.
# env.py
_src = ''
_html = ''
def set_path_srcfile(path_srcfile):
global _src
_src = path_srcfile
def get_path_srcfile():
return _src
...
Then everywhere you need it you can use import env; env.set_path_srcfile(myfile) and know that all other functions / modules / classes will be aware of the update.
If you don't want a singleton, then making a settings object available in the main module somewhere (as you have done) is a fine solution.
Related
I am making a tiny framework for games with pygame, on which I wish to implement basic code to quickly start new projects. This will be a module that whoever uses should just create a folder with subfolders for sprite classes, maps, levels, etc.
My question is, how should my framework module load these client modules? I was considering to design it so the developer could just pass to the main object the names of the directories, like:
game = Game()
game.scenarios = 'scenarios'
Then game will append 'scenarios' to sys.path and use __import__(). I've tested and it works.
But then I researched a little more to see if there were already some autoloader in python, so I could avoid to rewrite it, and I found this question Python modules autoloader?
Basically, it is not recommended to use a autoloader in python, since "explicit is better than implicit" and "Readability counts".
That way, I think, I should compel the user of my module to manually import each of his/her modules, and pass these to the game instance, like:
import framework.Game
import scenarios
#many other imports
game = Game()
game.scenarios = scenarios
#so many other game.whatever = whatever
But this doesn't looks good to me, not so confortable. See, I am used to work with php, and I love the way it works with it's autoloader.
So, the first exemple has some problability to crash or be some trouble, or is it just not 'pythonic'?
note: this is NOT an web application
I wouldn't consider letting a library import things from my current path or module good style. Instead I would only expect a library to import from two places:
Absolute imports from the global modules space, like things you have installed using pip. If a library does this, this library must also be found in its install_requires=[] list
Relative imports from inside itself. Nowadays these are explicitly imported from .:
from . import bla
from .bla import blubb
This means that passing an object or module local to my current scope must always happen explicitly:
from . import scenarios
import framework
scenarios.sprites # attribute exists
game = framework.Game(scenarios=scenarios)
This allows you to do things like mock the scenarios module:
import types
import framework
# a SimpleNamespace looks like a module, as they both have attributes
scenarios = types.SimpleNamespace(sprites='a', textures='b')
scenarios.sprites # attribute exists
game = framework.Game(scenarios=scenarios)
Also you can implement a framework.utils.Scenario() class that implements a certain interface to provide sprites, maps etc. The reason being: Sprites and Maps are usually saved in separate files: What you absolutely do not want to do is look at the scenarios's __file__ attribute and start guessing around in its files. Instead implement a method that provides a unified interface to that.
class Scenario():
def __init__(self):
...
def sprites(self):
# optionally load files from some default location
# If no such things as a default location exists, throw a NotImplemented error
...
And your user-specific scenarios will derive from it and optionally overload the loading methods
import framework.utils
class Scenario(framework.utils.Scenario):
def __init__(self):
...
def sprites(self):
# this method *must* load files from location
# accessing __file__ is OK here
...
What you can also do is have framework ship its own framework.contrib.scenarios module that is used in case no scenarios= keyword arg was used (i.e. for a square default map and some colorful default textures)
from . import contrib
class Game()
def __init__(self, ..., scenarios=None, ...):
if scenarios is None:
scenarios = contrib.scenarios
self.scenarios = scenarios
Not sure if there's a neat way of dealing with it, it just makes sense to me visually to lay out each object/class into it's own module under a common package.
For instance:
/Settings/
/Settings/__init__.py
/Settings/AbstractSetting.py
/Settings/Float.py
/Settings/String.py
Each class inside of every module has the same name as the module and at the moment I keep doing this:
import Settings
mysetting = Settings.Float.Float()
..which is giving me these double "Float" names.
I could do, in the __init__.py of the package:
from Float import Float
..so that I could then do:
import Settings
mysetting = Settings.Float()
But I'd like this package to be dynamically updating to whatever modules I put inside of it. So that the next day, when I've added "Knob.py" to this package, I could do:
import Settings
myknob = Settings.Knob()
Makes sense?
But again, I haven't worked with packages before and are still trying to wrap my head around it and try and make it as easy as possible. At this point, I found it easier having all classes inside one big master module which is getting increasingly cumbersome.
Maybe packages isn't the way to go? What alternatives do I have?
Thanks a bunch.
EDIT: Main reason I want to do this is to let users write their own modules that will integrate with the rest of the application. A native "plugin" architeture, if you will.
Each module will contain a class inherited by a superclass with default values. The app then has a browser with available modules that, when clicked, displays relevant information found under the modules attributes. Each class contained then has a similar interface with which the application can use.
I did some further reading and apparently this is not the way to go. I'd love to hear your ideas though on what the benefits/disadvantages of this approach could be.
You should be aware that this is not the Python way. "One class per file" is a Java philosphy that does not apply in the Python world. We usually name modules in lowercase and stick related classes into the same file (in your example, all of the classes would go into settings.py or would be explicitely imported from there). But I guess the fact that you want users to provide plugins is a legitimate reason for your approach (immdbg does it the same way, I think).
So, if you really want to do this, you could put something like this into your Settings/__init__.py:
import os
import glob
import imp
for f in glob.glob(os.path.join(os.path.dirname(__file__), '*.py')):
modname = os.path.basename(f)[:-3]
if modname.startswith('__'): continue
mod = imp.load_source(modname, f)
globals()[modname] = getattr(mod, modname)
# or if you just want to import everything (even worse):
#for name in dir(mod):
# if name.startswith('__'): continue
# globals()[name] = getattr(mod, name)
Can you feel how the Python developers don't want you to do this? :)
There are many plugin systems. It is exemplified by the name of one such system yapsy (yet another plugin system).
You could create an object that provides necessary interface:
class Settings(object):
def __getattr__(self, attr):
return load_plugin(attr)
settings = Settings()
In your code:
from settings import settings
knob = settings.Knob()
You can use whatever implementation you like for load_plugin() e.g., for the code from the question:
from importlib import import_module
def load_plugin(name):
m = import_module('Settings.'+name)
return getattr(m, name)
I want to improve a Python framework that I'm writing by having it enumerate and load modules from some specified folder, at runtime, based on certain properties that the modules may have.
Probably properties such as: only modules that contain a certain value (like a tag) in some metadata field, or perhaps only modules that contain a class that derives from a certain base class.
For example, let's say that the extensions are plug-ins that support different types of authentication--I'd like my framework to discover the possible plug-ins at run-time without requiring explicit configuration.
It seems like this sort of "extension loading" should be possible, and has probably been done a zillion times before, none of the search queries that have thought to try are turning anything up, and I don't know a good specific project that already implements this to start by reading someone else's approach.
Any pointers on approaches that would work to build such a thing (or even advice on a more Pythonic way to think about this problem) would be great.
(A good answer for this would give an overview and options, so don't rush to accept my quick answer.)
I do this with one of my projects to load classes all the modules in a package without using import * and hardcoding names. The code is viewable in context on Google Code.
SPECIES = []
'''An automatically generated list of the available species types.'''
def _do_import():
'''Automatically populates SPECIES with all the modules in this
folder.
:Note:
Written as a function to prevent local variables from being
imported.
'''
import os
for _, _, files in os.walk(__path__[0]):
for filename in (file for file in files if file[0] != '_' and file[-3:] == '.py'):
modname = filename[:filename.find('.')]
mod = __import__(modname, globals(), fromlist=[])
for cls in (getattr(mod, s) for s in dir(mod)):
if cls is not Species and type(cls) is type and issubclass(cls, Species):
if getattr(cls, '_include_automatically', True):
SPECIES.append(cls)
globals()[cls.__name__] = cls
_do_import()
I am writing a moderate-sized (a few KLOC) PyQt app. I started out writing it in nice modules for ease of comprehension but I am foundering on the rules of Python namespaces. At several points it is important to instantiate just one object of a class as a resource for other code.
For example: an object that represents Aspell attached as a subprocess, offering a check(word) method. Another example: the app features a single QTextEdit and other code needs to call on methods of this singular object, e.g. "if theEditWidget.document().isEmpty()..."
No matter where I instantiate such an object, it can only be referenced from code in that module and no other. So e.g. the code of the edit widget can't call on the Aspell gateway object unless the Aspell object is created in the same module. Fine except it is also needed from other modules.
In this question the bunch class is offered, but it seems to me a bunch has exactly the same problem: it's a unique object that can only be used in the module where it's created. Or am I completely missing the boat here?
OK suggested elsewhere, this seems like a simple answer to my problem. I just tested the following:
junk_main.py:
import junk_A
singularResource = junk_A.thing()
import junk_B
junk_B.handle = singularResource
print junk_B.look()
junk_A.py:
class thing():
def __init__(self):
self.member = 99
junk_B.py:
def look():
return handle.member
When I run junk_main it prints 99. So the main code can inject names into modules just by assignment. I am trying to think of reasons this is a bad idea.
You can access objects in a module with the . operator just like with a function. So, for example:
# Module a.py
a = 3
>>> import a
>>> print a.a
3
This is a trivial example, but you might want to do something like:
# Module EditWidget.py
theEditWidget = EditWidget()
...
# Another module
import EditWidget
if EditWidget.theEditWidget.document().isEmpty():
Or...
import * from EditWidget
if theEditWidget.document().isEmpty():
If you do go the import * from route, you can even define a list named __all__ in your modules with a list of the names (as strings) of all the objects you want your module to export to *. So if you wanted only theEditWidget to be exported, you could do:
# Module EditWidget.py
__all__ = ["theEditWidget"]
theEditWidget = EditWidget()
...
It turns out the answer is simpler than I thought. As I noted in the question, the main module can add names to an imported module. And any code can add members to an object. So the simple way to create an inter-module communication area is to create a very basic object in the main, say IMC (for inter-module communicator) and assign to it as members, anything that should be available to other modules:
IMC.special = A.thingy()
IMC.important_global_constant = 0x0001
etc. After importing any module, just assign IMC to it:
import B
B.IMC = IMC
Now, this is probably not the greatest idea from a software design standpoint. If you just limit IMC to holding named constants, it acts like a C header file. If it's just to give access to singular resources, it's like a link extern. But because of Python's liberal rules, code in any module can modify or add members to IMC. Used in an undisciplined way, "who changed that" could be a debugging issue. If there are multiple processes, race conditions are a danger.
At several points it is important to instantiate just one object of a class as a resource for other code.
Instead of trying to create some sort of singleton factory, can you not create the single-use object somewhere between the main point of entry for the program and instantiating the object that needs it? The single-use object can just be passed as a parameter to the other object. Logically, then, you won't create the single-use object more than once.
For example:
def main(...):
aspell_instance = ...
myapp = MyAppClass(aspell_instance)
or...
class SomeWidget(...):
def __init__(self, edit_widget):
self.edit_widget = edit_widget
def onSomeEvent(self, ...):
if self.edit_widget.document().isEmpty():
....
I don't know if that's clear enough, or if it's applicable to your situation. But to be honest, the only time I've found I can't do this is in a CherryPy-based webserver, where the points of entry were pretty much everywhere.
I have tasked to modify a wx python gui based program which has several .py files.
I would like to share some variables defined in a.py and use them in b.py
The 10 -15 variables are of this form:
Amode = [SINGLE]
Format = [A] etc...
I would like to use them in b.py.
How do I go about it? I read about Pickle but still not clear how to use it well.
import a
// do something with a.Amode
// do something with a.Format
Generally, the best idea, in this case, is to either place the variables on the module directly or use some shared dataStore. I like the Borg pattern for this.
Basically do this:
#in borg.py
class Borg:
__shared_state = {}
def __init__(self):
self.__dict__ = self.__shared_state
Everywhere else:
import borg
drone = borg.Borg()
drone.foo = 1;
Obviously, you can limit this by defining __set__.
As to placing variables on modules directly, well, I'm not really a fan of having stateful variables publicly accessible on modules, but that is probably mostly me.
Modules are singletons (no matter how many times it's imported, it's only actually imported once, and that once is shared), so what I often do for this use case is to create a modules named, say, "shared.py", and put the data I want shared across other modules in it. Then, in those other modules:
import shared
# Use a shared variable/object
print shared.Amode
# Changes to the shared data are seen in all modules where it's imported.
shared.Amode = aNewValue
This has the nice effect of keeping all my shared data in its own namespace, "shared".