Develop Reference

Python global variable in import * [duplicate]

I've run into a bit of a wall importing modules in a Python script. I'll do my best to describe the error, why I run into it, and why I'm tying this particular approach to solve my problem (which I will describe in a second):
Let's suppose I have a module in which I've defined some utility functions/classes, which refer to entities defined in the namespace into which this auxiliary module will be imported (let "a" be such an entity):
module1:
def f():
print a
And then I have the main program, where "a" is defined, into which I want to import those utilities:
import module1
a=3
module1.f()
Executing the program will trigger the following error:
Traceback (most recent call last):
File "Z:\Python\main.py", line 10, in <module>
module1.f()
File "Z:\Python\module1.py", line 3, in f
print a
NameError: global name 'a' is not defined
Similar questions have been asked in the past (two days ago, d'uh) and several solutions have been suggested, however I don't really think these fit my requirements. Here's my particular context:
I'm trying to make a Python program which connects to a MySQL database server and displays/modifies data with a GUI. For cleanliness sake, I've defined the bunch of auxiliary/utility MySQL-related functions in a separate file. However they all have a common variable, which I had originally defined inside the utilities module, and which is the cursor object from MySQLdb module.
I later realised that the cursor object (which is used to communicate with the db server) should be defined in the main module, so that both the main module and anything that is imported into it can access that object.
End result would be something like this:
utilities_module.py:
def utility_1(args):
code which references a variable named "cur"
def utility_n(args):
etcetera
And my main module:
program.py:
import MySQLdb, Tkinter
db=MySQLdb.connect(#blahblah) ; cur=db.cursor() #cur is defined!
from utilities_module import *
And then, as soon as I try to call any of the utilities functions, it triggers the aforementioned "global name not defined" error.
A particular suggestion was to have a "from program import cur" statement in the utilities file, such as this:
utilities_module.py:
from program import cur
#rest of function definitions
program.py:
import Tkinter, MySQLdb
db=MySQLdb.connect(#blahblah) ; cur=db.cursor() #cur is defined!
from utilities_module import *
But that's cyclic import or something like that and, bottom line, it crashes too. So my question is:
How in hell can I make the "cur" object, defined in the main module, visible to those auxiliary functions which are imported into it?
Thanks for your time and my deepest apologies if the solution has been posted elsewhere. I just can't find the answer myself and I've got no more tricks in my book.

Globals in Python are global to a module, not across all modules. (Many people are confused by this, because in, say, C, a global is the same across all implementation files unless you explicitly make it static.)
There are different ways to solve this, depending on your actual use case.
Before even going down this path, ask yourself whether this really needs to be global. Maybe you really want a class, with f as an instance method, rather than just a free function? Then you could do something like this:
import module1
thingy1 = module1.Thingy(a=3)
thingy1.f()
If you really do want a global, but it's just there to be used by module1, set it in that module.
import module1
module1.a=3
module1.f()
On the other hand, if a is shared by a whole lot of modules, put it somewhere else, and have everyone import it:
import shared_stuff
import module1
shared_stuff.a = 3
module1.f()
… and, in module1.py:
import shared_stuff
def f():
print shared_stuff.a
Don't use a from import unless the variable is intended to be a constant. from shared_stuff import a would create a new a variable initialized to whatever shared_stuff.a referred to at the time of the import, and this new a variable would not be affected by assignments to shared_stuff.a.
Or, in the rare case that you really do need it to be truly global everywhere, like a builtin, add it to the builtin module. The exact details differ between Python 2.x and 3.x. In 3.x, it works like this:
import builtins
import module1
builtins.a = 3
module1.f()

As a workaround, you could consider setting environment variables in the outer layer, like this.
main.py:
import os
os.environ['MYVAL'] = str(myintvariable)
mymodule.py:
import os
myval = None
if 'MYVAL' in os.environ:
myval = os.environ['MYVAL']
As an extra precaution, handle the case when MYVAL is not defined inside the module.

This post is just an observation for Python behaviour I encountered. Maybe the advices you read above don't work for you if you made the same thing I did below.
Namely, I have a module which contains global/shared variables (as suggested above):
#sharedstuff.py
globaltimes_randomnode=[]
globalist_randomnode=[]
Then I had the main module which imports the shared stuff with:
import sharedstuff as shared
and some other modules that actually populated these arrays. These are called by the main module. When exiting these other modules I can clearly see that the arrays are populated. But when reading them back in the main module, they were empty. This was rather strange for me (well, I am new to Python). However, when I change the way I import the sharedstuff.py in the main module to:
from globals import *
it worked (the arrays were populated).
Just sayin'

A function uses the globals of the module it's defined in. Instead of setting a = 3, for example, you should be setting module1.a = 3. So, if you want cur available as a global in utilities_module, set utilities_module.cur.
A better solution: don't use globals. Pass the variables you need into the functions that need it, or create a class to bundle all the data together, and pass it when initializing the instance.

The easiest solution to this particular problem would have been to add another function within the module that would have stored the cursor in a variable global to the module. Then all the other functions could use it as well.
module1:
cursor = None
def setCursor(cur):
global cursor
cursor = cur
def method(some, args):
global cursor
do_stuff(cursor, some, args)
main program:
import module1
cursor = get_a_cursor()
module1.setCursor(cursor)
module1.method()

Since globals are module specific, you can add the following function to all imported modules, and then use it to:
Add singular variables (in dictionary format) as globals for those
Transfer your main module globals to it
.
addglobals = lambda x: globals().update(x)
Then all you need to pass on current globals is:
import module
module.addglobals(globals())

Since I haven't seen it in the answers above, I thought I would add my simple workaround, which is just to add a global_dict argument to the function requiring the calling module's globals, and then pass the dict into the function when calling; e.g:
# external_module
def imported_function(global_dict=None):
print(global_dict["a"])
# calling_module
a = 12
from external_module import imported_function
imported_function(global_dict=globals())
>>> 12

The OOP way of doing this would be to make your module a class instead of a set of unbound methods. Then you could use __init__ or a setter method to set the variables from the caller for use in the module methods.

Update
To test the theory, I created a module and put it on pypi. It all worked perfectly.
pip install superglobals
Short answer
This works fine in Python 2 or 3:
import inspect
def superglobals():
_globals = dict(inspect.getmembers(
inspect.stack()[len(inspect.stack()) - 1][0]))["f_globals"]
return _globals
save as superglobals.py and employ in another module thusly:
from superglobals import *
superglobals()['var'] = value
Extended Answer
You can add some extra functions to make things more attractive.
def superglobals():
_globals = dict(inspect.getmembers(
inspect.stack()[len(inspect.stack()) - 1][0]))["f_globals"]
return _globals
def getglobal(key, default=None):
"""
getglobal(key[, default]) -> value
Return the value for key if key is in the global dictionary, else default.
"""
_globals = dict(inspect.getmembers(
inspect.stack()[len(inspect.stack()) - 1][0]))["f_globals"]
return _globals.get(key, default)
def setglobal(key, value):
_globals = superglobals()
_globals[key] = value
def defaultglobal(key, value):
"""
defaultglobal(key, value)
Set the value of global variable `key` if it is not otherwise st
"""
_globals = superglobals()
if key not in _globals:
_globals[key] = value
Then use thusly:
from superglobals import *
setglobal('test', 123)
defaultglobal('test', 456)
assert(getglobal('test') == 123)
Justification
The "python purity league" answers that litter this question are perfectly correct, but in some environments (such as IDAPython) which is basically single threaded with a large globally instantiated API, it just doesn't matter as much.
It's still bad form and a bad practice to encourage, but sometimes it's just easier. Especially when the code you are writing isn't going to have a very long life.

Detect circular imports in Python [duplicate]

I'm working with a project that contains about 30 unique modules. It wasn't designed too well, so it's common that I create circular imports when adding some new functionality to the project.
Of course, when I add the circular import, I'm unaware of it. Sometimes it's pretty obvious I've made a circular import when I get an error like AttributeError: 'module' object has no attribute 'attribute' where I clearly defined 'attribute'. But other times, the code doesn't throw exceptions because of the way it's used.
So, to my question:
Is it possible to programmatically detect when and where a circular import is occuring?
The only solution I can think of so far is to have a module importTracking that contains a dict importingModules, a function importInProgress(file), which increments importingModules[file], and throws an error if it's greater than 1, and a function importComplete(file) which decrements importingModules[file]. All other modules would look like:
import importTracking
importTracking.importInProgress(__file__)
#module code goes here.
importTracking.importComplete(__file__)
But that looks really nasty, there's got to be a better way to do it, right?

To avoid having to alter every module, you could stick your import-tracking functionality in a import hook, or in a customized __import__ you could stick in the built-ins -- the latter, for once, might work better, because __import__ gets called even if the module getting imported is already in sys.modules, which is the case during circular imports.
For the implementation I'd simply use a set of the modules "in the process of being imported", something like (benjaoming edit: Inserting a working snippet derived from original):
beingimported = set()
originalimport = __import__
def newimport(modulename, *args, **kwargs):
if modulename in beingimported:
print "Importing in circles", modulename, args, kwargs
print " Import stack trace -> ", beingimported
# sys.exit(1) # Normally exiting is a bad idea.
beingimported.add(modulename)
result = originalimport(modulename, *args, **kwargs)
if modulename in beingimported:
beingimported.remove(modulename)
return result
import __builtin__
__builtin__.__import__ = newimport

Not all circular imports are a problem, as you've found when an exception is not thrown.
When they are a problem, you'll get an exception the next time you try to run any of your tests. You can change the code when this happens.
I don't see any change required from this situation.
Example of when it's not a problem:
a.py
import b
a = 42
def f():
return b.b
b.py
import a
b = 42
def f():
return a.a

Circular imports in Python are not like PHP includes.
Python imported modules are loaded the first time into an import "handler", and kept there for the duration of the process. This handler assigns names in the local namespace for whatever is imported from that module, for every subsequent import. A module is unique, and a reference to that module name will always point to the same loaded module, regardless of where it was imported.
So if you have a circular module import, the loading of each file will happen once, and then each module will have names relating to the other module created into its namespace.
There could of course be problems when referring to specific names within both modules (when the circular imports occur BEFORE the class/function definitions that are referenced in the imports of the opposite modules), but you'll get an error if that happens.

import uses __builtin__.__import__(), so if you monkeypatch that then every import everywhere will pick up the changes. Note that a circular import is not necessarily a problem though.

Python: Modifying class variables outside module has no effect

I am trying to create a system where I can mutate the values of A as easily as possible. When I import A into various other modules and modify the fields of A
they seem to remain changed in the foreign module but are unchanged in A's native module.
# module name a.py
import b
class A:
x = 0
#classmethod
def call_mutate_A_from_B(cls):
b_object = b.B()
b_object.mutate_A() # change does not seem to stay in effect upon
# return
print(cls.x) # prints 0
def main():
A.call_mutate_A_from_B()
if __name__ == '__main__':
main()
# module name b.py
from a import A
class B: # in a different module
def mutate_A():
A.x = 2
Is this the expected behaviour? I don't think it is. Should I create an object representation of A not treat is as a static class?

To answer your first question: yes, this is the expected behavior. Let us say that a module A is imported by another module B. When we import A, two things happen:
All the object definitions of the imported module A are imported into B.
Any executable statements in A are executed (only once).
To elaborate on the first point, all the definitions of classes, functions, variables etc. from A are imported into B. It is safe to say that B now has its own working copies of module A's contents to work with. Changing any aspect of these copies will not have any effect on the original definitions, meaning that these changes will not be reflected in A (the imported module).
I hope this clears your question. You can read more about how modules work in the official Python docs.

Imported module goes out of scope (unbound local error)

I am getting a strange "unbound local" error in a python package that seems to defy all logic. I can't get a MWE to reproduce it, but will try to explain succinctly and hope that someone might be able to offer some explanation of what is going on.
For the sake of this example module is a package I developed, and Model is a class within module. The definition of the Model class (model.py) looks like:
import module
class Model:
def __init__(self):
print module
def run(self):
print module
Now, when I instantiate a Model in a script like this:
from model import Model
m = Model()
m.run()
module prints successfully within the __init__, but I get an unbound local error within the run function.
I attempted to diagnose this with pdb, which is where things got really weird, because if I add a pdb trace immediately prior to the print module line in the run() function, then I can successfully run print module without an unbound local error, but if I step to the next line then it throws the error. How can module be in the scope of __init__(), and in the scope of pdb, but not in the scope of run()?
I know this is not ideal since there is no MWE, but I cannot seem to reproduce this outside the context of the full code. I am hoping that someone will have an idea of what might possibly be going on and suggest some strategies to debug further.

Apparently you have a local variable named module somewhere in the function run. For example, the following code will throw UnboundLocalError
import sys
def run():
print sys
sys = None
run()
Here sys = None introduces a local name that shadows the imported sys inside run and at the time print invoked it is not yet defined, hence the error. To use the imported module inside run you have to find and rename the local variable.
More info on python scoping rules is here

Global variable is undefined at the module level

(There are many similar and more generic questions, been trying the solutions from them after reading through them, can't get them working so asking here as a more situation-specific version of what I'm seeing)
I think I am really miss-understanding how Python does OOP due to my more C#/C++ background. So here's what I'm trying to do right this moment.
I'm starting with two modules to set up the rest of my project, partially as a sanity-check and proof-of-concept. One module logs things to a file as I go while also storing data from multiple modules (to eventually package them all and dump them on request) Doing all this in PyCharm and mentioning the error warnings it suggests by the way, and using Python 2.7
Module 1:
src\helpers\logHelpers.py
class LogHelpers:
class log:
def classEnter():
#doing stuff
def __init__(self):
self.myLog = LogHelpers.log() #forgot to mention this was here initially
[..] various logging functions and variables to summarize what's happening
__builtin__.mylogger = LogHelpers
Module 2:
src\ULTs\myULTs.py
mylogger.myLog.classEnter()
(both the modules and the root src\ have an empty init.py file in them)
So according to the totally awesome response here ( Python - Visibility of global variables in imported modules ) at this stage this should be working, but 'mylogger' becomes an 'unresolved reference'
So that was one approach. I also tried the more straight forward global one ( Python: How to make a cross-module variable? )
Module 1:
src\helpers\logHelpers.py
class LogHelpers:
class log:
def classEnter(self):
#doing stuff
def __init__(self):
self.myLog = LogHelpers.log() #forgot to mention this was here initially
[..] various logging functions and variables to summarize what's happening
mylogger = LogHelpers
__init__.py
__all__ = ['LogHelpers', hexlogger]
from .logHelpers import *
Module 2:
src\ULTs\myULTs.py
from helpers import mylogger
mylogger.myLog.classEnter()
This version gets a "parameter 'self' unfilled" error on the classEnter, which various reports seem to indicate means that mylogger is un-initialized (misleading error code but that's what it seems to mean)
And then I tried this..
Module 1:
src\helpers\logHelpers.py
class LogHelpers:
class log:
def classEnter(self):
#doing stuff
def __init__(self):
self.myLog = LogHelpers.log() #forgot to mention this was here initially
[..] various logging functions and variables to summarize what's happening
__mylogger = LogHelpers
__init__.py
__all__ = ['LogHelpers', hexlogger]
from .logHelpers import *
Module 2:
src\ULTs\myULTs.py
from helpers import mylogger
def someFunction(self):
global mylogger
mylogger.myLog.classEnter()
And this version gets the 'Global variable is undefined at the module level' error when I hover of global mylogger.
Then there is the idea of each other module tracking its own instance of a class apparently, if I end up having to I can go with that method and coordinate them.. but that's kind of a hack considering what I'm trying to do.
That's kind of where I'm at, that's the gist of what I'm trying to do... I'm reading through as many similar questions as I can but all of them seem to come back to these kinda of solutions (which don't seem to be working) or saying 'don't do that' (which is generally good advice but I'm not really grocking the preferred Pythony way of keeping multiple ongoing non-static classes organized for a large project - other than shoving them all in one directory)
Thoughts? (How badly am I mangling Python here?)
[EDIT] Based on feedback tried a mini version that eliminated the inner classes completely:
Ok, so did a local mini-class based on what you said:
class testClass:
def __init__(self):
self.testVar = 2
def incrementVar(self):
self.testVar += 1
myClass = testClass()
Set it up via init.py
__all__ = [myClass]
from .logHelpers import myClass
Went to other module and
from helpers import myClass
class Test_LogHelpers(unittest.TestCase):
def test_mini(self):
myClass.incrementVar()
Ran it directly instead of looking at PyCharm, no Global anything.. NameError: name 'myClass is not defined
So still at square one :( (and still need to store state)
[EDIT] Adding Traceback:
Traceback (most recent call last):
File "C:\Program Files (x86)\JetBrains\PyCharm Community Edition 3.4.1\helpers\pycharm\utrunner.py", line 124, in <module> module = loadSource(a[0])
File "C:\Program Files (x86)\JetBrains\PyCharm Community Edition 3.4.1\helpers\pycharm\utrunner.py", line 40, in loadSource module = imp.load_source(moduleName, fileName)
File "C:\[...mylocation...]\py\src\ULTs\LogHelpers_ULT.py", line 3, in <module> from helpers import myClass
File "C:\[...mylocation...]\py\src\helpers\__init__.py", line 7, in <module>
__all__ = [myClass]
NameError: name 'myClass' is not defined
============================================================================
kk, I got it working with the miniclass. I don't know why the other approach / approaches was not working, but this seemed to fix things.
(Resources: http://docs.python-guide.org/en/latest/writing/structure/ , http://mikegrouchy.com/blog/2012/05/be-pythonic-__init__py.html )
**logHelpers.py**
[... some static logging functionality ...]
class testClass:
def __init__(self):
self.testVar = 2
def incrementVar(self, source):
self.testVar += 1
mylogger.myLog.info(source + " called, new val: " + str(self.testVar))
myClass = testClass()
**test_LogHelpers_ULT.py**
import unittest
from helpers.logHelpers import myClass
class Test_LogHelpers(unittest.TestCase):
def test_mini(self):
myClass.incrementVar("LogHelpers")
For some reason skipping the
init.py
(and leaving it blank) and going for the explicit importation worked. It also maintained state - I created a duplicate of the test file and my log output correctly had '3' for the first file to call the helper, and '4' for the second file to call the helper.
Thanks Daniel Roseman for the help and suggestions, they had me look a bit more in the right direction. If you can spot why the previous stuff wasn't working it would be much appreciate just to add to my understanding of this language, but I'm gonna go ahead and mark your answer as 'Answered' since it had some very useful feedback.

Before I start, note that the PyCharm warnings are not actual Python errors: if you ran your code, you would probably get more useful feedback (remember static analysis of a dynamic language like Python can only get you so far, many things can't be resolved until you actually run the code).
Firstly, it's really not clear why you have nested classes here. The outer class seems completely useless; you should remove it.
The reason for the error message about "self" is that you have defined an instance method, which can only be called on an instance of log. You could make mylogger (absolutely no need for the double-underscore prefix) an instance: mylogger = log() - and then import that, or import the class and instantiate it where it is used.
So in your first snippet, the error message is quite clear: you have not defined mylogger. Using my recommendation above, you can do from helpers import mylogger and then directly call mylogger.classEnter().
Finally, I can't see what that global statement is doing in someFunction. There's no need to declare a name as global unless you plan to reassign it within your scope and have that reassignment reflected in the global scope. You're not doing that here, so no need for global.
By the way, you should also question whether you even need the inner log class. Generally speaking, classes are only useful when you need to store some kind of state in the object. Here, as your docstring says, you have a collection of utility methods. So why put them in a class? Just make them top-level functions inside the logHelpers module (incidentally, Python style prefers lower_case_with_underscore for module names, so it should be "log_helpers.py").