Please be kind with me, I'm a Python beginner :-)
Now, I see that the 'best practice' for writing Python programs would be to wrap the main code inside a 'main' function, and do the if "__main__" == __name__: test to invoke the 'main' function.
This of course results in the necessity of using a series of global statements in the 'main' function to access the global variables.
I wonder if it is more proper (or 'Pythonic', if you will) to gather the global variables into a custom class, say _v, and refer to the variables using _v. prefix instead?
Also, as a corollary question, would that have any negative impact to, let's say, performance or exception handling?
EDIT : The following is the general structure of the program:
paramset = {
0: { ...dict of params... }
1: { ...dict of params... }
2: { ...dict of params... }
}
selector = 0
reset_requested = False
selector_change = False
def sighup_handler(signal,frame):
global reset_requested
logger.info('Caught SIGHUP, resetting to set #{0}'.format(new_selector))
reset_requested = True
selector = 0
def sigusr1_handler(signal,frame):
global selector
new_selector = (selector + 1) % len(paramset)
logger.info('Caught SIGHUP, changing parameters to set #{0}'.format(new_selector))
selector = new_selector
selector_change = True
signal.signal(signal.SIGHUP, sighup_handler)
signal.signal(signal.SIGUSR1, sigusr1_handler)
def main():
global reset_requested
global selector
global selector_change
keep_running = True
while keep_running
logger.info('Processing selector {0}'.format(selector))
for stage in [process_stage1, process_stage2, process_stage3]
err, result = stage(paramset[selector])
if err is not None:
logger.critical('Stage failure! Err {0} details: {0}'.format(err, result))
raise SystemError('Err {0} details: {0}'.format(err, result))
else:
logger.info('Stage success: {0}'.format(result))
if reset_requested:
stage_cleanup()
reset_requested = False
else:
inter_stage_pause()
if selector_change:
selector_change = False
break
selector = (selector + 1) % len(paramset)
There are enough pieces missing from the example code that reaching any firm conclusions is difficult.
Event-driven approach
The usual approach for this type of problem would be to make it entirely event-driven. As it stands, the code is largely polling. For example, sighup_handler sets reset_requested = True and the while loop in main processes that request. An event-driven approach would handle the reset, meaning the call to stage_cleanup, directly:
def sighup_handler(signal,frame):
logger.info('Caught SIGHUP, resetting to set #{0}'.format(new_selector))
stage_cleanup()
Class with shared variables
In the sample code, the purpose of all those process_stages and cycling through the stages is not clear. Can it all be put in an event-driven context? I don't know. If it can't and it does require shared variables, then your suggestion of a class would be a natural choice. The beginnings of such a class might look like:
class Main(object);
def __init__(self):
self.selector = 0
self.selector_change = False
signal.signal(signal.SIGHUP, self.sighup_handler)
signal.signal(signal.SIGUSR1, self.sigusr1_handler)
def sighup_handler(self, signal,frame):
logger.info('Caught SIGHUP, resetting to set #{0}'.format(new_selector))
stage_cleanup()
self.selector = 0
def sigusr1_handler(self, signal,frame):
new_selector = (selector + 1) % len(paramset)
logger.info('Caught SIGHUP, changing parameters to set #{0}'.format(new_selector))
self.selector = new_selector
self.selector_change = True
def mainloop(self):
# Do here whatever polling is actually required.
if __name__ == '__main__':
main = Main()
main.mainloop()
Again, because the true purpose of the polling loop is not clear to me, I didn't try to reproduce its functionality in the class above.
Generally, it is best practice to avoid global variables, and instead just pass variables to classes/methods that need them through method calls. Example: if you are making a calculator, make an addition method that takes 2 ints and returns an int. This is in contrast to making 2 input ints and 1 output int as global variables, and having the add method work on those.
Related
Having an issue calling function dynamic_exit from class dynamic, called from an imported python file within the project file. Only including enough code to depict an efficient example of my issue.
Call example below:
from lib.core import dynamic
import ...
if requests.get(url).status_code != 200:
clear()
print(" xxxxx \n\n\n")
print("[ !! | Invalid URL ] Status code: {0}".format(
str(requests.get(url).status_code)))
time.sleep(1)
print("\n\n Please enter a valid URL.\nExiting...")
dynamic.dynamic_exit(self=dynamic())
time.sleep(3)
exit()
Lib.core contains:
class dynamic:
def __init__(self):
self.loadSwitch = False
self.analyzeSwitch = False
self.exitSwitch = False
def dynamic_load(self, loadSwitch=True):
self.loadSwitch = loadSwitch
done = False
for c in itertools.cycle(['[ | ]', '[ / ]', '[ - ]', '[ \\ ]']):
if done:
break
sys.stdout.write('\rLoading ' + c)
sys.stdout.flush()
time.sleep(0.1)
# Further along...
if dynamic.dynamic_analyze(): # Separate function -- Irrelevant
t = threading.Thread(target=dynamic_analyze())
t.start()
elif dynamic_exit(): # Separate function -- Irrelevant
t2 = threading.Thread(target=dynamic_exit())
t2.start()
else: # dynamic_load -- Example function
t3 = threading.Thread(target=dynamic_load())
t3.start()
sys.stdout.write('\r[ ✓ ] Process Complete ')
time.sleep(4.5)
done = True
loadSwitch = False
exitSwitch = False
analyzeSwitch = False
Lord, I know it's a mess. First time actually working with classes like this.
Error is as follows:
File "~/test-delete.py", line 11, in <module>
from lib.core import dynamic
File "~/lib/core.py", line 55, in <module>
if dynamic.dynamic_analyze():
TypeError: dynamic_analyze() missing 1 required positional argument: 'self'
The IDE is wanting more than a simple self parameter, it is recommending self=. So unsure of how to handle this.
Basically, need help under the context of the __init__ function and using the self parameter. Trying to call the function setting either exitSwitch, analyzeSwitch, or loadSwitch = True, ifswitch == True, perform either function dynamic_load, dynamic_exit, or dynamic_analyze. Post-completion, set all switches back to False.
The problem is that you are calling instance methods as if they are static methods. In other words, you call the methods as dynamic.dynamic_analyse() where dynamic is a reference to the class, not to an instance of that class.
So proceed as follows:
Name your class with PascalCase -- a common practice to distinguish classes from other things. So yours should be named Dynamic.
Create an instance and assign it to a variable. This one could actually get the name dynamic with lower case initial letter.
Don't pass an instance as argument when calling methods on the instance. Because in the notation a.b(), b will be called with the value for self set to a.
So define the class as:
class Dynamic:
def __init__(self):
self.loadSwitch = False
self.analyzeSwitch = False
self.exitSwitch = False
# ...etc.
Import the class and create an instance like this:
from lib.core import Dynamic
# ...
dynamic = Dynamic() # Create instance
# ...
if dynamic.dynamic_analyze(): # Call method on the instance
# ..etc
Your exit code should have:
dynamic.dynamic_exit() # No argument.
I cannot comment on the rest of your code, as it is hard to tell what it is doing. For instance, I do wonder why you call dynamic_analyse() twice... but this at least will solve the problem with the error you got.
from threading import Thread
import time
def print_k():
while true:
if main.k % 2 == 1: # ditto
print(main.k, "is even.") # <-- my problem is HERE ( Ignore all the other stuff )
time.sleep(2)
def main():
k = 1
while k != 200:
k += 1
print k
time.sleep(0.5)
if __name__ == '__main__':
Thread(target=print_k).start()
Thread(target=main).start()
in this script (example only, ignore all realistic functionality) I am trying to run main(), which adds up to 200 and prints it, and in print_k, i am printing main's variable, k.
I have an exception raised, unsurprisingly, and am wondering how i can access a separate function's variable from a different function (they are both running at the same time, by the way, hence the Threading module.)
You can't print main's variable k. The whole point of local variables is that they're local. It doesn't matter whether they're running at the same time or not; they each have their own separate local environment. (In fact, if you call main 60 times, each of those 60 calls has its own local environment.)
But there are a number of things you can do.
The simplest, but generally worst, is to use global variables instead of local variables. Just add global k to the top of the main function, add some start value for k at the top level (before either thread starts), and you can now access the same global variable inside print_k.
Bundling the shared state and functions up together in a class, where both functions become methods that can access self.k, is a better solution. Passing in some kind of mutable "holder" to both main and print_k is also a better solution. Redesigning your app around explicit message passing (e.g., on a Queue.Queue) is even better.
I'll show how to do it with a class:
class KCounter(object):
def __init__(self):
self.k = 0
def print_k(self):
while True:
if self.k % 2 == 1:
print(self.k, "is even.")
time.sleep(2)
def main(self):
self.k = 1
while self.k != 200:
self.k += 1
print self.k
time.sleep(0.5)
if __name__ == '__main__':
kcounter = KCounter()
Thread(target=kcounter.print_k).start()
Thread(target=kcounter.main).start()
Now, because we're using self.k, an attribute of the KCounter instance, instead of k, a local variable, both methods see the same variable.
This is the code from my recent tool I have made, I am trying to write unittest, I have an idea of how to test a method that returns something but don't understand how should i test method that don't like below:
def buildUi(self):
self.label = QtGui.QLabel()
self.label.setAlignment(QtCore.Qt.AlignCenter)
self.setCentralWidget(self.label)
def nextImage(self):
""" switch to next image or previous image
"""
if self._imagesInList:
if self._count == len(self._imagesInList):
self._count = 0
self.showImageByPath(
self._imagesInList[self._count])
if self.animFlag:
self._count += 1
else:
self._count -= 1
def showImageByPath(self, path):
if path:
image = QtGui.QImage(path)
pp = QtGui.QPixmap.fromImage(image)
self.label.setPixmap(pp.scaled(
self.label.size(),
QtCore.Qt.KeepAspectRatio,
QtCore.Qt.SmoothTransformation))
def playPause(self):
if not self._pause:
self._pause = True
self.updateTimer.start(2500)
return self._pause
else:
self._pause = False
self.updateTimer.stop()
def keyPressEvent(self, keyevent):
""" Capture key to exit, next image, previous image,
on Escape , Key Right and key left respectively.
"""
event = keyevent.key()
if event == QtCore.Qt.Key_Escape:
self.close()
if event == QtCore.Qt.Key_Left:
self.animFlag = False
self.nextImage()
if event == QtCore.Qt.Key_Right:
self.animFlag = True
self.nextImage()
if event == 32:
self._pause = self.playPause()
the complete code for looking can be found here
is it possible to test these methods above or do I have to modify to make them testable ?
Edit: updated:
class TestSlideShow(unittest.TestCase):
""" docstring for TestSlideShow
"""
def setUp(self):
self.mox = mox.Mox()
self.imgLst = ['/folder/test/images/test1.jpg', '/folder/test/images/test2.JPG',
'/folder/test/images/test3.png', '/folder/test/images/test4.PNG']
app = QtGui.QApplication([])
self.show = slideShow.SlideShowPics(imgLst=self.imgLst, ui=False)
def tearDown(self):
self.mox.UnsetStubs()
self.mox.ResetAll()
def test_nextImage(self):
self.mox.StubOutWithMock(self.show, 'prepairWindow')
self.show.prepairWindow()
self.mox.StubOutWithMock(self.show, 'showImageByPath')
self.show.showImageByPath(self.imgLst[1])
self.show.nextImage()
# self.mox.ReplayAll()
self.assertEquals(1, self.show.count)
self.assertEquals(self.imgLst[1], self.show._imagesInList[1])
# self.mox.VerifyAll()
def test_nextImage_animFlag_False(self):
self.show.animFlag = False
self.show.count = 4
self.mox.StubOutWithMock(self.show, 'prepairWindow')
self.show.prepairWindow()
self.mox.StubOutWithMock(self.show, 'showImageByPath')
self.show.showImageByPath(self.imgLst[3])
print self.show.count
self.show.nextImage()
print self.show.count
# self.assertEquals(3, self.show.count)
self.assertEquals(self.imgLst[3], self.show._imagesInList[3])
if __name__ == '__main__':
unittest.main()
the first test when self.show.animFlag is True works fine and but the when I manually set the animFlag= False then second test fails.
This is the problem with writing unittest after the code - you then realize your code is difficult to test. Writing the tests before the code (well, really "along" the code - you don't write all tests before start coding, but still you dont write a line of code before you have a test for it) makes sure you don't have such a problem.
Now even with the "test first" approach you do have to test methods that don't return anything. The way to do so is to test for the expected side effects. Some of these side effects are easy to test - in your above case, you can test the value of self._count before and after the call to nextImage, depending on your object's state (_imagesInList and animFlag mostly). Where it gets more difficult is if you want to test that nextImage does actually call showImageByPath with the correct arguments, and with your current design the only way to do so is to monkeypatch showImageByPath for the tests. Testing showImageByPath will require patching / mocking self.label.setPixmap(), etc.
As others already pointed there are a couple mock/stub libs that can help, but they won't solve all possible testability issues and you may have to rethink your design to make things easier - like not hardcoding the call to QtGui.QLabel() in buildUI() but have some way to "inject" the desired componant (QtGui.QLabel() or a mock) instead. As a general rule, testable code has very few hard-coded dependencies, few side effects, and lot of small classes with short methods (instead of huge classes with long methods).
I'm creating a program that uses the Twisted module and callbacks.
However, I keep having problems because the asynchronous part goes wrecked.
I have learned (also from previous questions..) that the callbacks will be executed at a certain point, but this is unpredictable.
However, I have a certain program that goes like
j = calc(a)
i = calc2(b)
f = calc3(c)
if s:
combine(i, j, f)
Now the boolean s is set by a callback done by calc3. Obviously, this leads to an undefined error because the callback is not executed before the s is needed.
However, I'm unsure how you SHOULD do if statements with asynchronous programming using Twisted. I've been trying many different things, but can't find anything that works.
Is there some way to use conditionals that require callback values?
Also, I'm using VIFF for secure computations (which uses Twisted): VIFF
Maybe what you're looking for is twisted.internet.defer.gatherResults:
d = gatherResults([calc(a), calc2(b), calc3(c)])
def calculated((j, i, f)):
if s:
return combine(i, j, f)
d.addCallback(calculated)
However, this still has the problem that s is undefined. I can't quite tell how you expect s to be defined. If it is a local variable in calc3, then you need to return it so the caller can use it.
Perhaps calc3 looks something like this:
def calc3(argument):
s = bool(argument % 2)
return argument + 1
So, instead, consider making it look like this:
Calc3Result = namedtuple("Calc3Result", "condition value")
def calc3(argument):
s = bool(argument % 2)
return Calc3Result(s, argument + 1)
Now you can rewrite the calling code so it actually works:
It's sort of unclear what you're asking here. It sounds like you know what callbacks are, but if so then you should be able to arrive at this answer yourself:
d = gatherResults([calc(a), calc2(b), calc3(c)])
def calculated((j, i, calc3result)):
if calc3result.condition:
return combine(i, j, calc3result.value)
d.addCallback(calculated)
Or, based on your comment below, maybe calc3 looks more like this (this is the last guess I'm going to make, if it's wrong and you'd like more input, then please actually share the definition of calc3):
def _calc3Result(result, argument):
if result == "250":
# SMTP Success response, yay
return Calc3Result(True, argument)
# Anything else is bad
return Calc3Result(False, argument)
def calc3(argument):
d = emailObserver("The argument was %s" % (argument,))
d.addCallback(_calc3Result)
return d
Fortunately, this definition of calc3 will work just fine with the gatherResults / calculated code block immediately above.
You have to put if in the callback. You may use Deferred to structure your callback.
As stated in previous answer - the preocessing logic should be handled in callback chain, below is simple code demonstration how this could work. C{DelayedTask} is a dummy implementation of a task which happens in the future and fires supplied deferred.
So we first construct a special object - C{ConditionalTask} which takes care of storring the multiple results and servicing callbacks.
calc1, calc2 and calc3 returns the deferreds, which have their callbacks pointed to C{ConditionalTask}.x_callback.
Every C{ConditionalTask}.x_callback does a call to C{ConditionalTask}.process which checks if all of the results have been registered and fires on a full set.
Additionally - C{ConditionalTask}.c_callback sets a flag of wheather or not the data should be processed at all.
from twisted.internet import reactor, defer
class DelayedTask(object):
"""
Delayed async task dummy implementation
"""
def __init__(self,delay,deferred,retVal):
self.deferred = deferred
self.retVal = retVal
reactor.callLater(delay, self.on_completed)
def on_completed(self):
self.deferred.callback(self.retVal)
class ConditionalTask(object):
def __init__(self):
self.resultA=None
self.resultB=None
self.resultC=None
self.should_process=False
def a_callback(self,result):
self.resultA = result
self.process()
def b_callback(self,result):
self.resultB=result
self.process()
def c_callback(self,result):
self.resultC=result
"""
Here is an abstraction for your "s" boolean flag, obviously the logic
normally would go further than just setting the flag, you could
inspect the result variable and do other strange stuff
"""
self.should_process = True
self.process()
def process(self):
if None not in (self.resultA,self.resultB,self.resultC):
if self.should_process:
print 'We will now call the processor function and stop reactor'
reactor.stop()
def calc(a):
deferred = defer.Deferred()
DelayedTask(5,deferred,a)
return deferred
def calc2(a):
deferred = defer.Deferred()
DelayedTask(5,deferred,a*2)
return deferred
def calc3(a):
deferred = defer.Deferred()
DelayedTask(5,deferred,a*3)
return deferred
def main():
conditional_task = ConditionalTask()
dFA = calc(1)
dFB = calc2(2)
dFC = calc3(3)
dFA.addCallback(conditional_task.a_callback)
dFB.addCallback(conditional_task.b_callback)
dFC.addCallback(conditional_task.c_callback)
reactor.run()
For brevity, I'm just showing what can/must occur in states. I haven't run into any oddities in the state machine framework itself.
Here is a specific question:
Do you find it confusing that we have to return StateChange(...) and StateMachineComplete(...) whereas some of the of the other actions like some_action_1(...) and some_action_2(...) need not be returned - they're just direct method invocations?
I think that StateChange(...) needs to return because otherwise code beyond the StateChange(...) call will be executed. This isn't how a state machine should work! For example see the implementation of event1 in the ExampleState below
import abc
class State(metaclass=abc.ABCMeta):
# =====================================================================
# == events the state optionally or must implement ====================
# =====================================================================
# optional: called when the state becomes active.
def on_state_entry(self): pass
# optional: called when we're about to transition away from this state.
def on_state_exit(self): pass
#abc.abstractmethod
def event1(self,x,y,z): pass
#abc.abstractmethod
def event2(self,a,b): pass
#abc.abstractmethod
def event3(self): pass
# =====================================================================
# == actions the state may invoke =====================================
# =====================================================================
def some_action_1(self,c,d,e):
# implementation omitted for brevity
pass
def some_action_2(self,f):
# implementation omitted for brevity
pass
class StateChange:
def __init__(self,new_state_type):
# implementation omitted for brevity
pass
class StateMachineComplete: pass
class ExampleState(State):
def on_state_entry(self):
some_action_1("foo","bar","baz")
def event1(self,x,y,z):
if x == "asdf":
return StateChange(ExampleState2)
else:
return StateChange(ExampleState3)
print("I think it would be confusing if we ever got here. Therefore the StateChange calls above are return")
def event2(self,a,b):
if a == "asdf":
return StateMachineComplete()
print("As with the event above, the return above makes it clear that we'll never get here.")
def event3(self):
# Notice that we're not leaving the state. Therefore this can just be a method call, nothing need be returned.
self.some_action_1("x","y","z")
# In fact we might need to do a few things here. Therefore a return call again doesn't make sense.
self.some_action_2("z")
# Notice we don't implement on_state_exit(). This state doesn't care about that.
When I need a state machine in Python, I store it as a dictionary of functions. The indices into the dictionary are the current states, and the functions do what they need to and return the next state (which may be the same state) and outputs. Turning the crank on the machine is simply:
state, outputs = machine_states[state](inputs)
By putting the outgoing state changes in code you're obfuscating the whole process. A state machine should be driven by a simple set of tables. One axis is the current state, and the other is the possible events. You have two or three tables:
The "next-state" table that determines the exit state
The "action" table that determines what action to take
The "read" table that determines whether you stay on the current input event or move on to the next.
The third table may or may not be needed depending on the complexity of the input "grammar".
There are more esoteric variations, but I've never found a need for more than this.
I also struggled to find a good state_machine solution in python. So I wrote state_machine
It works like the following
#acts_as_state_machine
class Person():
name = 'Billy'
sleeping = State(initial=True)
running = State()
cleaning = State()
run = Event(from_states=sleeping, to_state=running)
cleanup = Event(from_states=running, to_state=cleaning)
sleep = Event(from_states=(running, cleaning), to_state=sleeping)
#before('sleep')
def do_one_thing(self):
print "{} is sleepy".format(self.name)
#before('sleep')
def do_another_thing(self):
print "{} is REALLY sleepy".format(self.name)
#after('sleep')
def snore(self):
print "Zzzzzzzzzzzz"
#after('sleep')
def big_snore(self):
print "Zzzzzzzzzzzzzzzzzzzzzz"
person = Person()
print person.current_state == person.sleeping # True
print person.is_sleeping # True
print person.is_running # False
person.run()
print person.is_running # True
person.sleep()
# Billy is sleepy
# Billy is REALLY sleepy
# Zzzzzzzzzzzz
# Zzzzzzzzzzzzzzzzzzzzzz
print person.is_sleeping # True