Develop Reference

change global variable in different files

I do not really understand how python handles global variables if the code is split into different files.
Assuming I have 3 files: class1.py, class2.py, main.py
In main.py I define a global variable
from class1 import Class1
from class2 import Class2
global sys
sys = constructor()
This object contains information about the system which I simulate and is used and manipulated by the classes defined in class1.py and in class2.py.
One could of course argue that this is bad style and one should avoid exploiting global variables like this, but this is not the point here.
If now I use sys in either class, it is unknown. To use a global variable one could of course define them somewhere and then include this file. But then, the changes that are made by the classes would not effect each other, so I don't want to do this.
Another way would be to define a new Class SuperClass where sys is a member. If now Class1 and Class2 are inherited from SuperClass, I could probably do some stuff with the super keyword. I do not really want to do this...
Long story short... Is there a way to define an python object such, that it behaves similar to a C-style global variable?
Maybe it helps if I give an example:
sys includes the system frequency
a function of Class1 changes the system frequency
a function of Class2 simulates stuff and uses the system frequency
based on this the system power is changed in sys
a function of Class1 performs a task with updated system power

No, there's no way to have "C-style global variables", and that's by design. Explicitely pass your sys objects to Class1 and Class2 when instanciating them and you'll be done, with a clean, readable, testable, maintainable implementation:
# lib.py
class Class1(object):
def __init__(self, sys, whatever):
self.sys = sys
# ...
class Class2(object):
def __init__(self, sys, whateverelse):
self.sys = sys
# ...
# main.py
from lib import Class1, Class2
def main():
sys = constructor()
c1 = Class1(sys, 42)
c2 = Class(sys, "spam")
# now you can work with c1 and c2
if __name__ == "__main__":
main()
To answer your very first question:
I do not really understand how python handles global variables if the
code is split into different files.
Python's "globals" are module-level names. At runtime, modules are objects (instances of the module type), and all names defined at the module's top level become attributes of the module instance - "defined" being either by assignment, by a def or class statement or by an import statement.
From within the module, top-level names are accessible by their unqualified names in all the module's code - you only need the global keyword if you want to rebind a top-level name from within a function.
From outside the module, you can access those names using the qualified path, ie if module1 defines name foo, you can access it from module2 by importing module1 (making module1 a top-level name in module2) and using the usual attribute resolution (dotted.name) syntax.
You can also import foo directly using from module1 import foo, in which case a new name foo will be created in module2 and bound to module1.foo (more exactly: bound to the object which is at this point bound to module1.foo). The point here is that you know have two distinct names in two distinct namespaces, so rebinding either will only affect the namespace in which it's rebound.
You certainly want to read Ned Batcheler's famous article on Python's names, it's usually a good starting point to understand all this.

Python Multiple file program

For larger programs, in order to be more organized, I have been looking into dividing my code up into different .py files and having the main file that calls upon those files when needed. I have looked around and seen lots of remarks about creating a directory and a SystemPath for Python. Are those reasonable options for a program that could be distributed between a few computers? As a test, I tried to assemble an example:
This is the class named grades in the same directory as main
class student:
def __init__(self):
self.name = ""
self.score = 0
self.grade = 0
def update(self,name,score,grade):
self.score = score
self.name = name
self.grade = grade
print self.score,self.name,self.grade
s = student()
s.update(name,score,grade)
This is my main script currently:
from grades import score
import random
name = 'carl'
score = random.randrange(0,100)
grade = 11
s = student()
s.score(name,score,grade)
There are some questions I have generally about this method:
Is there a way to import all from different files or do I need to specify each individual class?
If I just had a function, is it possible to import it just as a function or can you only import via a class?
Why is it when I call upon a class, in general, I have to make a variable for it as in the example below?
# way that works
s = student()
s.update(name,score,grade)
# incorrect way
student.update(name,score,grade)
Thank you for your time and thought towards my question.

Yes.
You can import instance of student from other script to main script like this:
from grades import s
# if your first script is called grades.py
import random
name = 'carl'
score = random.randrange(0,100)
grade = 11
# you can directly use it without initializing it again.
s.score(name,score,grade)
2.
If you have a function called test() in grades.py, you can import it in this way:
from grades import test
# then invoke it
test()
3.
This variable stands for the instance of class student. You need this instance to invoke the function inside.

Generally, to divide the source code of a program, Python use module to do that, which corresponds to a *.py file. Then for your 3 questions:
You can import a whole "module content" (function, class, global variables, ...) through import module_name.*
for a function, if it is a function in a class(member method, class method or static method) you can not only import the function, you should import class to use the method; if it is a function under module, you can separately import the function through import module_name.function_name
update is a member function of the student class, so you should use it through an instance. if it is a class method or static method, you can use it through the class name you wrote.

1: Is there a way to import all from different file or do i need to
specify each individual class?
You can use the "wildcard import", but you probably shouldn't. See
Should wildcard import be avoided?
If i just had a function, is it possible to import it just as a
function or can you only import via a class?
Functions can be totally independent of classes in Python.
3.Why is it when i call upon a class in general i have to make a variable for it as in the example below?
You should read up on object-oriented programming. In the basic cases, you have to instantiate instances of a class in order to use that class's functionality. In your example, the class student describes what it means to be a student, and the statement
s = student()
creates a student and names it "s".
I think this should be clear after reading a bit about object-oriented programming.

First, you can use from module import * to import everything like:
hello.py:
def hello():
print 'hello'
def bye():
print 'Bye'
main.py:
from hello import *
hello()
bye()
But it's not a good way, if you have two files, two functions have the same name,
so use
from hello import hello, bye
hello()
bye()
is better, it an example for function ,as same as class.
Third before Second, student is a class, so you have to use an instance object to use the function which with self parameter. If you want to use student.function, the function must be a static method like this:
class Person:
def __init__():
pass
#staticmethod
def Count():
return 1
print Person.Count()
Second, you can import the function in a class file which is independent of the class.

Is there a way to import all from different file or do i need to specify each individual class?
the answer is yes , as python import statement use sys.path (A list of strings that specifies the search path for modules ) you need to add the patht of your modules in sys.path , for example if you want to interact between different computers you can put your modules in public folder and add the path of folder to sys.path :
import sys
sys.path.append( path to public )
If i just had a function, is it possible to import it just as a function or can you only import via a class?
you just need to use from ... import function_name.
Why is it when i call upon a class in general i have to make a variable for it as in the example below?
for this question you just need to read the python Class objects documentation :
Class objects support two kinds of operations: attribute references and instantiation.

Adding functions from other files to a Python class

I am having trouble with this setup mainly because I am not sure what I actually want in order to solve this problem.
This is the setup
- main.py
- lib
- __init__.py
- index.py
- test.py
__init__.py has this code
import os
for module in os.listdir(os.path.dirname(__file__)+"/."):
if module == '__init__.py' or module[-3:] != '.py':
continue
__import__(module[:-3], locals(), globals())
del module
main.py has this code as of now
from lib.index import *
print User.__dict__
index.py has this code
class User(object):
def test(self):
return "hi"
pass
test.py has this code
class User(object):
def tes2(self):
return "hello"
When I execute main.py it successfully prints the method test from index.py but what I am trying to do is figure out a way where I can just create a file in the lib folder where that while has only one function in the format
class User(object):
def newFunction(self):
return abc
and this function should automatically be available for me in main.py
I am sure that this is not a hard thing to do but I honestly don't know what I want (what to search for to solve this) which is preventing me from researching the solution.

You can use a metaclass to customize class creation and add functions defined elsewhere:
import types
import os
import os.path
import imp
class PluginMeta(type):
def __new__(cls, name, bases, dct):
modules = [imp.load_source(filename, os.path.join(dct['plugindir'], filename))
for filename in os.listdir(dct['plugindir']) if filename.endswith('.py')]
for module in modules:
for name in dir(module):
function = getattr(module, name)
if isinstance(function, types.FunctionType):
dct[function.__name__] = function
return type.__new__(cls, name, bases, dct)
class User(metaclass=PluginMeta):
plugindir = "path/to/the/plugindir"
def foo(self):
print "foo"
user = User()
print dir(user)
Then in the plugin files, just create functions not classes:
def newFunction(self, abc):
self.abc = abc
return self.abc
And the metaclass will find them, turn them into methods, and attach them to your class.

Classes are objects, and methods are nothing more than attributes on class-objects.
So if you want to add a method to an existing class, outside the original class block, all that is is the problem of adding an attribute to an object, which I would hope you know how to do:
class User(object):
pass
def newFunction(self):
return 'foo'
User.newFunction = newFunction
agf's metaclass answer is basically a nifty automatic way of doing this, although it works by adding extra definitions to the class block before the class is created, rather than adding extra attributes to the class object afterwards.
That should be basically all you need to develop a framework in which things defined in one module are automatically added to a class defined elsewhere. But you still need to make a number of design decisions, such as:
If your externally-defined functions need auxiliary definitions, how do you determine what's supposed to get added to the class and what was just a dependency?
If you have more than one class you're extending this way, how do you determine what goes in which class?
At what point(s) in your program does the auto-extension happen?
Do you want to say in your class "this class has extensions defined elsewhere", or say in your extensions "this is an extension to a class defined elsewhere", or neither and somewhere bind extensions to classes externally from both?
Do you need to be able to have multiple versions of the "same" class with different extensions active at the same time?
A metaclass such as proposed by agf can be a very good way of implementing this sort of framework, because it lets you put all the complex code in one place while still "tagging" every class that doesn't work the way classes normally work. It does fix the answers to some of the questions I posed above, though.

here a working code we used in a project, I'm not sure it's the best way but it worked and there is almost no additional code to add to other files
cpu.py:
from cpu_base import CPU, CPUBase
import cpu_common
import cpu_ext
cpu_base.py:
def getClass():
return __cpu__
def setClass(CPUClass):
global __cpu__
__cpu__ = CPUClass
__classes__.append(CPUClass)
def CPU(*kw):
return __cpu__(*kw)
class CPUBase:
def __init__(self):
your_init_Stuff
# optionally a method classname_constructor to mimic __init__ for each one
for c in __classes__:
constructor = getattr(c, c.__name__ + '_constructor', None)
if constructor is not None:
constructor(self)
setClass(CPUBase)
cpu_common.py:
from cpu_base import getClass, setClass
class CPUCommon(getClass()):
def CPUCommon_constructor(self):
pass
setClass(CPUCommon)
cpu_ext.py:
from cpu_base import getClass, setClass
class CPUExt(getClass()):
pass
setClass(CPUExt)
to use the class import CPU from cpu.py

Variable scope outside of classes

My text editor of choice is extensible through python plugins. It requires me to extend classes and override its methods. The general structure looks similar the snippet below. Note that the function signature is fixed.
ftp_client is supposed to be shared by instances of both classes.
ftp_client = None
class FtpFileCommand(sublime_plugin.TextCommand):
def run(self, args):
global ftp_client # does it reference the variable of the outer scope?
self.ftp_client = ftplib.FTP('foo')
# login and stuff
class FtpFileEventListener(sublime_plugin.EventListener):
def run(self, args):
global ftp_client # same for this
self.ftp_client.quit() #
Both of these classes are supposed to have one variable in common. What is the best practice in order to share variables?
Edit based on madjars answer:
FtpFileCommand.run is called first, instanciates ftp_client and works like a charm. FtpFileEventListener.run is called later and, can reference ftp_client perfectly but it is still None. Using the global keyword, does it add the variable as a member to self?

Yep, that's exactly how global works.
It seems to me you are doing it right, as it's done this way in some modules of the python standard library (fileinput, for example).

In this code:
global ftp_client # does it reference the variable of the outer scope?
self.ftp_client = ftplib.FTP('foo')
you declare ftp_client as a global variable. This means it lives at the module level (where your classes are for example).
The second line is wrong. You wanted to assign to the global variable but instead you set an instance attribute of the same name.
It should be:
global ftp_client
ftp_client = ftplib.FTP('foo')
But let me suggest a different approach. A common practice is to put such stuff inside the class, since it is shared by all instances of this class.
class FtpFileCommand(sublime_plugin.TextCommand):
ftp_client = None
def run(self, args):
FtpFileCommand.ftp_client = ftplib.FTP('foo')
# login and stuff
Notice that the method doesn't use self so it might as well be a class method:
class FtpFileCommand(sublime_plugin.TextCommand):
ftp_client = None
#classmethod
def run(cls, args):
cls.ftp_client = ftplib.FTP('foo')
# login and stuff
This way you will get the class as the first argument and you can use it to access the FTP client without using the class name.

If there's only a single shared variable, then a global is the simplest solution. But note that a variable only needs to be declared with global when it is being assigned to. If the global variable is an object, you can call its methods, modify its attributes, etc without declaring it as global first.
An alternative to using global variables is to use class attributes which are accessed using classmethods. For example:
class FtpFile(object):
_client = None
#classmethod
def client(cls):
return cls._client
#classmethod
def setClient(cls, client):
cls._client = client
class FtpFileCommand(FtpFile, sublime_plugin.TextCommand):
def run(self, args):
client = self.client()
class FtpFileEventListener(FtpFile, sublime_plugin.EventListener):
def run(self, args):
client = self.client()

Could you add a constructor to each class then pass ftp_client as an argument?
class FtpFileCommand(sublime_plugin.TextCommand):
...
def __init__(self, ftp_client):
self.ftp_client = ftp_client
...
class FtpFileEventListener(sublime_plugin.EventListener):
...
def __init__(self, ftp_client):
self.ftp_client = ftp_client
...

Yak... THANK YOU SO MUCH FOR THIS!!!
You declare ftp_client as a global variable. This means it lives at
the module level (where your classes are for example).
I was having a difficult time trying to write my program "properly" where I'm utilizing classes and functions and couldn't call any of the variables. I recognized that global would make it available outside of the class. When I read that I thought... If it lives outside of the class then the variable I need to retrieve from the py script that I'm importing that module from would be:
module.variable
And then within that module, I declared another global variable to call it from the main script... so example...
#Main Script main.py
import moduleA
print(moduleA.moduleA.variable)
#ModuleA code moduleA.py
import moduleB
class classA():
def functionA():
global moduleA_variable
call.something.from.moduleB.classB.functionB()
moduleA_variable = moduleB.moduleB_variable
ModuleB code moduleB.py
class classB():
def functionB():
global moduleB_variable
moduleB_variable = retrieve.tacos()
I hope my explanation also helps someone. I'm a beginner with python and struggled with this for a while. In case it wasn't clear... I had separate custom modules made up of a few different .py files. Main was calling moduleA and moduleA was calling moduleB. I had to return the variable up the chain to the main script. The point of me doing it this way, was to keep the main script clean for the most part, and set myself up for executing repetitive tasks without having to write pages of crap. Basically trying to reuse functions instead of writing a book.

How do I extend a python module? Adding new functionality to the `python-twitter` package

What are the best practices for extending an existing Python module – in this case, I want to extend the python-twitter package by adding new methods to the base API class.
I've looked at tweepy, and I like that as well; I just find python-twitter easier to understand and extend with the functionality I want.
I have the methods written already – I'm trying to figure out the most Pythonic and least disruptive way to add them into the python-twitter package module, without changing this modules’ core.

A few ways.
The easy way:
Don't extend the module, extend the classes.
exttwitter.py
import twitter
class Api(twitter.Api):
pass
# override/add any functions here.
Downside : Every class in twitter must be in exttwitter.py, even if it's just a stub (as above)
A harder (possibly un-pythonic) way:
Import * from python-twitter into a module that you then extend.
For instance :
basemodule.py
class Ball():
def __init__(self,a):
self.a=a
def __repr__(self):
return "Ball(%s)" % self.a
def makeBall(a):
return Ball(a)
def override():
print "OVERRIDE ONE"
def dontoverride():
print "THIS WILL BE PRESERVED"
extmodule.py
from basemodule import *
import basemodule
def makeBalls(a,b):
foo = makeBall(a)
bar = makeBall(b)
print foo,bar
def override():
print "OVERRIDE TWO"
def dontoverride():
basemodule.dontoverride()
print "THIS WAS PRESERVED"
runscript.py
import extmodule
#code is in extended module
print extmodule.makeBalls(1,2)
#returns Ball(1) Ball(2)
#code is in base module
print extmodule.makeBall(1)
#returns Ball(1)
#function from extended module overwrites base module
extmodule.override()
#returns OVERRIDE TWO
#function from extended module calls base module first
extmodule.dontoverride()
#returns THIS WILL BE PRESERVED\nTHIS WAS PRESERVED
I'm not sure if the double import in extmodule.py is pythonic - you could remove it, but then you don't handle the usecase of wanting to extend a function that was in the namespace of basemodule.
As far as extended classes, just create a new API(basemodule.API) class to extend the Twitter API module.

Don't add them to the module. Subclass the classes you want to extend and use your subclasses in your own module, not changing the original stuff at all.

Here’s how you can directly manipulate the module list at runtime – spoiler alert: you get the module type from types module:
from __future__ import print_function
import sys
import types
import typing as tx
def modulize(namespace: tx.Dict[str, tx.Any],
modulename: str,
moduledocs: tx.Optional[str] = None) -> types.ModuleType:
""" Convert a dictionary mapping into a legit Python module """
# Create a new module with a trivially namespaced name:
namespacedname: str = f'__dynamic_modules__.{modulename}'
module = types.ModuleType(namespacedname, moduledocs)
module.__dict__.update(namespace)
# Inspect the new module:
name: str = module.__name__
doc: tx.Optional[str] = module.__doc__
contents: str = ", ".join(sorted(module.__dict__.keys()))
print(f"Module name: {name}")
print(f"Module contents: {contents}")
if doc:
print(f"Module docstring: {doc}")
# Add to sys.modules, as per import machinery:
sys.modules.update({ modulename : module })
# Return the new module instance:
return module
… you could then use such a function like so:
ns = {
'func' : lambda: print("Yo Dogg"), # these can also be normal non-lambda funcs
'otherfunc' : lambda string=None: print(string or 'no dogg.'),
'__all__' : ('func', 'otherfunc'),
'__dir__' : lambda: ['func', 'otherfunc'] # usually this’d reference __all__
}
modulize(ns, 'wat', "WHAT THE HELL PEOPLE")
import wat
# Call module functions:
wat.func()
wat.otherfunc("Oh, Dogg!")
# Inspect module:
contents = ", ".join(sorted(wat.__dict__.keys()))
print(f"Imported module name: {wat.__name__}")
print(f"Imported module contents: {contents}")
print(f"Imported module docstring: {wat.__doc__}")
… You could also create your own module subclass, by specifying types.ModuleType as the ancestor of your newly declared class, of course; I have never personally found this necessary to do.
(Also, you don’t have to get the module type from the types module – you can always just do something like ModuleType = type(os) after importing os – I specifically pointed out this one source of the type because it is non-obvious; unlike many of its other builtin types, Python doesn’t offer up access to the module type in the global namespace.)
The real action is in the sys.modules dict, where (if you are appropriately intrepid) you can replace existing modules as well as adding your new ones.

Say you have an older module called mod that you use like this:
import mod
obj = mod.Object()
obj.method()
mod.function()
# and so on...
And you want to extend it, without replacing it for your users. Easily done. You can give your new module a different name, newmod.py or place it by same name at a deeper path and keep the same name, e.g. /path/to/mod.py. Then your users can import it in either of these ways:
import newmod as mod # e.g. import unittest2 as unittest idiom from Python 2.6
or
from path.to import mod # useful in a large code-base
In your module, you'll want to make all the old names available:
from mod import *
or explicitly name every name you import:
from mod import Object, function, name2, name3, name4, name5, name6, name7, name8, name9, name10, name11, name12, name13, name14, name15, name16, name17, name18, name19, name20, name21, name22, name23, name24, name25, name26, name27, name28, name29, name30, name31, name32, name33, name34, name35, name36, name37, name38, name39
I think the import * will be more maintainable for this use-case - if the base module expands functionality, you'll seamlessly keep up (though you might shade new objects with the same name).
If the mod you are extending has a decent __all__, it will restrict the names imported.
You should also declare an __all__ and extend it with the extended module's __all__.
import mod
__all__ = ['NewObject', 'newfunction']
__all__ += mod.__all__
# if it doesn't have an __all__, maybe it's not good enough to extend
# but it could be relying on the convention of import * not importing
# names prefixed with underscores, (_like _this)
Then extend the objects and functionality as you normally would.
class NewObject(object):
def newmethod(self):
"""this method extends Object"""
def newfunction():
"""this function builds on mod's functionality"""
If the new objects provide functionality you intend to replace (or perhaps you are backporting the new functionality into an older code base) you can overwrite the names

May I suggest not to reinvent the Wheel here? I'm building a >6k line Twitter Client for 2 month now, at first I checked python-twitter too, but it's lagging a lot behind the recent API changes,, Development doesn't seem to be that active either, also there was(at least when I last checked) no support for OAuth/xAuth).
So after searching around a bit more I discovered tweepy:
http://github.com/joshthecoder/tweepy
Pros: Active development, OAauth/xAuth and up to date with the API.
Chances are high that what you need is already in there.
So I suggest going with that, it's working for me, the only thing I had to add was xAuth(that got merge back to tweepy :)
Oh an a shameless plug, if you need to parse Tweets and/or format them to HTML use my python version of the twitter-text-* libraries:
http://github.com/BonsaiDen/twitter-text-python
This thing is unittestetd an guaranteed to parse Tweets just like Twitter.com does it.

Define a new class, and instead of inherit it from the class you want to extend from the original module, add an instance of the original class as an attribute to your new class.
And here comes the trick: intercept all non-existing method calls on your new class and try to call it on the instance of the old class.
In your NewClass just define new or overridden methods as you like:
import originalmodule
class NewClass:
def __init__(self, *args, **kwargs):
self.old_class_instance = originalmodule.create_oldclass_instance(*args, **kwargs)
def __getattr__(self, methodname):
"""This is a wrapper for the original OldClass class.
If the called method is not part of this NewClass class,
the call will be intercepted and replaced by the method
in the original OldClass instance.
"""
def wrapper(*args, **kwargs):
return getattr(self.old_class_instance, methodname)(*args, **kwargs)
return wrapper
def new_method(self, arg1):
"""Does stuff with the OldClass instance"""
thing = self.old_class_instance.get_somelist(arg1)
# returns the first element only
return thing[0]
def overridden_method(self):
"""Overrides an existing method, if OldClass has a method with the same name"""
print("This message is coming from the NewClass and not from the OldClass")
In my case I used this solution when simple inheritance from the old class was not possible, because an instance had to be created not by its constructor, but with an init script from an other class/module. (It is the originalmodule.create_oldclass_instance in the example above.)