I am relatively new to Rx and RxPy - there is one fundamental thing that I am trying to do which is access a value that I store at the end of an Observer on_completed. I have a feeling I am either missing something very obvious, or I am perhaps bending the Observable concept into something that it's not meant to be. Either way, hoping I can get some guidance on this.
I've looked over the documentation at things like materialize but they don't quite seem to match up. Have also looked into Do and Disposable but can't find many examples that are close to what I need.
import rx
from rx import operators as ops
from rx.core import Observer
if __name__ == "__main__":
class PipelineObserver(Observer):
def __init__(self):
self.status = None
def on_next(self, payload):
print(payload)
def on_error(self, err):
print(err)
def on_completed(self):
self.status = "Done"
return self.status
## This returns a disposable, not the actual value I want. Which in this case is self.status
output = rx.from_([1, 2]).subscribe(
PipelineObserver()
)
print(output) ## Hoping for "Done" which is stored in self.status, not disposable class
Is there anyway to access a value a value from the on_completed method? Short of saving something as a global variable (which seems like a bad idea to me) I'm not sure it's possible? Basically whatever it output by on_completed or something similar. Maybe Do or Finally?
Ended up figuring it out, posting here in case anyone else runs into this. Due to this operation I am looking for being one that has to run after an observable is completed, it has to be a blocking operation. The run() function is used for this.
import rx
from rx import operators as ops
from rx.core import Observer
if __name__ == "__main__":
class PipelineObserver(Observer):
def __init__(self):
self.status = None
def on_next(self, payload):
print(payload)
def on_error(self, err):
print(err)
def on_completed(self):
self.status = "Done"
# First, seperate out the observer and observable:
my_list = rx.from_([1, 2])
my_list.subscribe(
PipelineObserver()
)
# Say I want to return an integer of the count of items, I can use this:
output = my_list.pipe(
ops.count()
).run()
print(output)
# Notice the run command at the end of the chain.
# Output: 2
There might be other/better ways of doing this but this will work for now!
Related
I would like to understand why using the following snippet leads me to an error:
a) I want to use the following class to create a context manager, as outlined in the link attached below: for me it is very important to keep the "class PrintStop(ExitStack)" form, so please bear in mind when trying to solve this issue, that I already know there are other ways to use ExitStack(), but I am interested in this specific way of using it:
class PrintStop(ExitStack):
def __init__(self, verbose: bool = False):
super().__init__()
self.verbose = verbose
def __enter__(self):
super().__enter__()
if not self.verbose:
sys.stdout = self.enter_context(open(os.devnull, 'w'))
b) when trying to use the class in the more appropriate way, I get the desired effect to stop all the printing within the "with" block, but when trying to print again after that block I get an error:
with PrintStop(verbose=False):
print('this shouldn't be printed') <------ok till here
print('this should be printed again as it is outside the with block) <-----ERROR
c) the error I get is "ValueError: I/O operation on closed file": the reason I guess is the fact that exit method of ExitStack() is not automatically called once we exit the 'with' block, so, how may I change the class to fix this bug?
Here is a quick reference to a similar topic,
Pythonic way to compose context managers for objects owned by a class
ExitStack.__exit__ simply ensures that each context you enter has its __exit__ method called; it does not ensure that any changes you made (like assigning to sys.stdout) inside the corresponding __enter__ is undone.
Also, the purpose of an exit stack is to make it easy to enter contexts that require information not known when the with statement is introduced, or to create a variable number of contexts without having to enumerate them statically.
If you really want to use an exit stack, you'll need something like
class PrintStop(ExitStack):
def __init__(self, verbose: bool = False):
super().__init__()
self.verbose = verbose
def __enter__(self):
rv = super().__enter__()
if not self.verbose:
sys.stdout = self.enter_context(open(os.devnull, 'w'))
return rv
def __exit__(self):
sys.stdout = sys.__stdout__ # Restore the original
return super().__exit__()
Keep in mind that contextlib already provides a context manager for temporarily replacing standard output with a different file, appropriately named redirect_stdout.
with redirect_stdout(open(os.devnull, 'w')):
...
Using this as the basis for PrintStop makes use of composition, rather than inheritance.
from contextlib import redirect_stdout, nullcontext
class PrintStop:
def __init__(self, verbose: bool = False):
super().__init__()
if verbose:
self.cm = redirect_stdout(open(os.devnull, 'w'))
else:
self.cm = nullcontext()
def __enter__(self):
return self.cm.__enter__()
def __exit__(self):
return self.cm.__exit__()
First of all, sorry for the wording of the question, I can't express it in a more compact form.
Let's say I have a code like this in Python:
something_happened = False
def main():
# 'procs' is a list of procedures
for proc in procs:
try:
# Any of these can set the 'something_happened'
# global var to True
proc()
except as e:
handle_unexpected_exception(e)
continue
# If some procedure found some problem,
# print a remainder to check the logging files
if something_happened:
print('Check the logfile, just in case.')
Any of the involved procedures may encounter some problem but execution MUST continue, the problem is properly logged and that's the ONLY handling needed, really, because the problems that may arise while running the procedures shouldn't stop the program, this shouldn't involve raising an exception and stopping the execution.
The reason why the logfile should be checked is that some of the problems may need further human action, but the program can't do anything about them, other than logging them and keep running (long story).
Right now the only way of achieving this that I can think about is to make each procedure to set something_happened == True after logging a potential problem, but using a global variable which may be set from any of the procedures, or returning a status code from the procedures.
And yes, I know I can raise an exception from the procedures instead of setting a global or returning an error code, but that would only work because I'm running them in a loop, and this may change in the future (and then raising an exception will jump out the try-block), so that's my last resort.
Can anyone suggest a better way of dealing with this situation? Yes, I know, this is a very particular use case, but that's the reason why I'm not raising an exception in the first place, and I'm just curious because I didn't find anything after googling for hours...
Thanks in advance :)
You have a variable that may be set to True by any of the procs. It looks like a common OOP schema:
class A():
"""Don't do that"""
def __init__(self, logger):
self._logger = logger
self._something_happened = False
def proc1(self):
try:
...
except KeyError as e:
self._something_happened = True
self._logger.log(...)
def proc2(self):
...
def execute(self):
for proc in [self.proc1, self.proc2, ...]:
try:
proc()
except as e:
self._handle_unexpected_exception(e)
continue
if self._something_happened:
print('Check the logfile, just in case.')
But that's a very bad idea, because you're violating the Single Responsibility Principle: your classs has to know about proc1, proc2, ... You have to reverse the idea:
class Context:
def __init__(self):
self.something_happened = False
def main():
ctx = Context()
for proc in procs:
try:
proc(ctx) # proc may set ctx.something_happened to True
except as e:
handle_unexpected_exception(e)
continue
if ctx.something_happened:
print('Check the logfile, just in case.')
Creating a void class like that is not attracting. You can take the idea further:
class Context:
def __init__(self, logger):
self._logger = logger
self._something_happened = False
def handle_err(self, e):
self._something_happened = True
self._logger.log(...)
def handle_unexpected_exception(self, e):
...
self._logger.log(...)
def after(self):
if self._something_happened:
print('Check the logfile, just in case.')
def proc1(ctx):
try:
...
except KeyError as e:
ctx.handle_err(e) # you delegate the error handling to ctx
def proc2(ctx):
...
def main():
ctx = Context(logging.gerLogger("main"))
for proc in procs:
try:
proc(ctx)
except as e:
ctx.handle_unexpected_exception(e)
ctx.after()
The main benefit here is you that can use another Context if you want:
def StrictContext():
def handle_err(self, e):
raise e
def handle_unexpected_exception(self, e):
raise e
def after(self):
pass
Or
class LooseContext:
def handle_err(self, e):
pass
def handle_unexpected_exception(self, e):
pass
def after(self):
pass
Or whatever you need.
Looks like the cleaner solution is to raise an exception, and I will change the code accordingly. They only problem is what will happen if in the future the loop goes away, but I suppose I'll cross that bridge when I arrive to it ;) and then I'll use another solution or I'll try to change the main code miself.
#cglacet, #Phydeaux, thanks for your help and suggestions.
Assuming I have the following code
IDLE = 0
STARTED = 1
STOPPED = 2
ERRORED = 3
# additional states as needed
class StateMachine:
def __init__(self)
self.state = IDLE
def start(self):
self.state = STARTED
# do something
def stop(self):
self.state = STOPPED
# do something
def reset(self):
self.state = IDLE
# do something
Our current interface allows a client to change the state of an instance by stating the desired target state, at which point we run certain validation checks and then the appropriate method. I would ideally like to keep a dictionary mapping of the desired target state to the correct method to avoid massive and pointless if-statement blocks. i.e.
if target_state = STARTED:
instance.start()
elif target_state = STOPPED:
instance.stop()
...
But I'm uncertain as to whether or not the following solution is considered good practice or not (it feels a bit whacky calling methods from the class using an instance as arg).
state_mapping = {
IDLE: StateMachine.reset,
STARTED: StateMachine.start,
....
}
And then calling using:
action = state_mapping[target_state]
action(instance)
....
Any thoughts?
Not so whacky.
However, the only thing one has to bear in mind is that action is an unbound method, which may not be very obvious at a first glance of the method call; except I know first-hand how that dictionary is defined.
I think a more readable alternative is to call the method from the instance:
state_mapping = {
IDLE: "reset",
STARTED: "start",
....
}
action = state_mapping[target_state]
getattr(instance, action)()
This will equally improve readability in the case when the method takes more than one argument.
One other alternative.
As your class is called "StateMachine", perhaps it should have a method to execute the state change?
In which case, you can use bound methods in your map
class StateMachine:
...
def ChangeState(self, target):
state_mapping = { IDLE: self.reset, STARTED: self.start, ... }
state_mapping[target]()
You may want to deal with invalid target states, or just let it raise a KEY_ERROR exception.
I have a simple code problem and do not know what I do wrong. The import part is OK, when I get as an error message is that, I guess I make a mistake with the classes.
status_listener = SessionStatusListener()
TypeError: interface takes exactly one argument
So the code is:
import clr
clr.AddReference ("fxcore2")
from fxcore2 import O2GTransport, IO2GSessionStatus
class SessionStatusListener(IO2GSessionStatus):
def __init__(self):
IO2GSessionStatus.__init__(self)
self.connected = False
def onLoginFailed(self, error):
print ("*** LOGIN FAILED: %s" % error)
def onSessionStatusChanged(self, status):
print ("NEW STATUS: %s" % status)
if status == O2GSessionStatusCode.Connected:
self.connected = True
The main Application starts here
if __name__ == "__main__":
session = O2GTransport.createSession()
status_listener = SessionStatusListener()
Any advice is appreciated.
Pass an argument to SessionStatusListener, like it's telling you to. I would imagine that you need to change the __init__ body to something like
super().__init__(self)
instead of
IO2GSessionStatus.__init__(self)
I believe it is saying that
status_listener = SessionStatusListener()
Needs one argument, like this:
status_listener = SessionStatusListener(1)
I'm not sure exactly what types of data it's expecting, but you need to pass in an argument.
I have python TCP client and need to send media(.mpg) file in a loop to a 'C' TCP server.
I have following code, where in separate thread I am reading the 10K blocks of file and sending it and doing it all over again in loop, I think it is because of my implementation of thread module, or tcp send. I am using Queues to print the logs on my GUI ( Tkinter ) but after some times it goes out of memory..
UPDATE 1 - Added more code as requested
Thread class "Sendmpgthread" used to create thread to send data
.
.
def __init__ ( self, otherparams,MainGUI):
.
.
self.MainGUI = MainGUI
self.lock = threading.Lock()
Thread.__init__(self)
#This is the one causing leak, this is called inside loop
def pushlog(self,msg):
self.MainGUI.queuelog.put(msg)
def send(self, mysocket, block):
size = len(block)
pos = 0;
while size > 0:
try:
curpos = mysocket.send(block[pos:])
except socket.timeout, msg:
if self.over:
self.pushlog(Exit Send)
return False
except socket.error, msg:
print 'Exception'
return False
pos = pos + curpos
size = size - curpos
return True
def run(self):
media_file = None
mysocket = None
try:
mysocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
mysocket.connect((self.ip, string.atoi(self.port)))
media_file = open(self.file, 'rb')
while not self.over:
chunk = media_file.read(10000)
if not chunk: # EOF Reset it
print 'resetting stream'
media_file.seek(0, 0)
continue
if not self.send(mysocket, chunk): # If some error or thread is killed
break;
#disabling this solves the issue
self.pushlog('print how much data sent')
except socket.error, msg:
print 'print exception'
except Exception, msg:
print 'print exception'
try:
if media_file is not None:
media_file.close()
media_file = None
if mysocket is not None:
mysocket.close()
mysocket = None
finally:
print 'some cleaning'
def kill(self):
self.over = True
I figured out that it is because of wrong implementation of Queue as commenting that piece resolves the issue
UPDATE 2 - MainGUI class which is called from above Thread class
class MainGUI(Frame):
def __init__(self, other args):
#some code
.
.
#from the above thread class used to send data
self.send_mpg_status = Sendmpgthread(params)
self.send_mpg_status.start()
self.after(100, self.updatelog)
self.queuelog = Queue.Queue()
def updatelog(self):
try:
msg = self.queuelog.get_nowait()
while msg is not None:
self.printlog(msg)
msg = self.queuelog.get_nowait()
except Queue.Empty:
pass
if self.send_mpg_status: # only continue when sending
self.after(100, self.updatelog)
def printlog(self,msg):
#print in GUI
Since printlog is adding to a tkinter text control, the memory occupied by that control will grow with each message (it has to store all the log messages in order to display them).
Unless storing all the logs is critical, a common solution is to limit the maximum number of log lines displayed.
A naive implementation is to eliminate extra lines from the begining after the control reaches a maximum number of messages. Add a function to get the number of lines in the control and then, in printlog something similar to:
while getnumlines(self.edit) > self.maxloglines:
self.edit.delete('1.0', '1.end')
(above code not tested)
update: some general guidelines
Keep in mind that what might look like a memory leak does not always mean that a function is wrong, or that the memory is no longer accessible. Many times there is missing cleanup code for a container that is accumulating elements.
A basic general approach for this kind of problems:
form an opinion on what part of the code might be causing the problem
check it by commenting that code out (or keep commenting code until you find a candidate)
look for containers in the responsible code, add code to print their size
decide what elements can be safely removed from that container, and when to do it
test the result
I can't see anything obviously wrong with your code snippet.
To reduce memory usage a bit under Python 2.7, I'd use buffer(block, pos) instead of block[pos:]. Also I'd use mysocket.sendall(block) instead of your send method.
If the ideas above don't solve your problem, then the bug is most probably elsewhere in your code. Could you please post the shortest possible version of the full Python script which still grows out-of-memory (http://sscce.org/)? That increases your change of getting useful help.
Out of memory errors are indicative of data being generated but not consumed or released. Looking through your code I would guess these two areas:
Messages are being pushed onto a Queue.Queue() instance in the pushlog method. Are they being consumed?
The MainGui printlog method may be writing text somewhere. eg. Is it continually writing to some kind of GUI widget without any pruning of messages?
From the code you've posted, here's what I would try:
Put a print statement in updatelog. If this is not being continually called for some reason such as a failed after() call, then the queuelog will continue to grow without bound.
If updatelog is continually being called, then turn your focus to printlog. Comment the contents of this function to see if out of memory errors still occur. If they don't, then something in printlog may be holding on to the logged data, you'll need to dig deeper to find out what.
Apart from this, the code could be cleaned up a bit. self.queuelog is not created until after the thread is started which gives rise to a race condition where the thread may try to write into the queue before it has been created. Creation of queuelog should be moved to somewhere before the thread is started.
updatelog could also be refactored to remove redundancy:
def updatelog(self):
try:
while True:
msg = self.queuelog.get_nowait()
self.printlog(msg)
except Queue.Empty:
pass
And I assume the the kill function is called from the GUI thread. To avoid thread race conditions, the self.over should be a thread safe variable such as a threading.Event object.
def __init__(...):
self.over = threading.Event()
def kill(self):
self.over.set()
There is no data piling up in your TCP sending loop.
Memory error is probably caused by logging queue, as you have not posted complete code try using following class for logging:
from threading import Thread, Event, Lock
from time import sleep, time as now
class LogRecord(object):
__slots__ = ["txt", "params"]
def __init__(self, txt, params):
self.txt, self.params = txt, params
class AsyncLog(Thread):
DEBUGGING_EMULATE_SLOW_IO = True
def __init__(self, queue_max_size=15, queue_min_size=5):
Thread.__init__(self)
self.queue_max_size, self.queue_min_size = queue_max_size, queue_min_size
self._queuelock = Lock()
self._queue = [] # protected by _queuelock
self._discarded_count = 0 # protected by _queuelock
self._pushed_event = Event()
self.setDaemon(True)
self.start()
def log(self, message, **params):
with self._queuelock:
self._queue.append(LogRecord(message, params))
if len(self._queue) > self.queue_max_size:
# empty the queue:
self._discarded_count += len(self._queue) - self.queue_min_size
del self._queue[self.queue_min_size:] # empty the queue instead of creating new list (= [])
self._pushed_event.set()
def run(self):
while 1: # no reason for exit condition here
logs, discarded_count = None, 0
with self._queuelock:
if len(self._queue) > 0:
# select buffered messages for printing, releasing lock ASAP
logs = self._queue[:]
del self._queue[:]
self._pushed_event.clear()
discarded_count = self._discarded_count
self._discarded_count = 0
if not logs:
self._pushed_event.wait()
self._pushed_event.clear()
continue
else:
# print logs
if discarded_count:
print ".. {0} log records missing ..".format(discarded_count)
for log_record in logs:
self.write_line(log_record)
if self.DEBUGGING_EMULATE_SLOW_IO:
sleep(0.5)
def write_line(self, log_record):
print log_record.txt, " ".join(["{0}={1}".format(name, value) for name, value in log_record.params.items()])
if __name__ == "__main__":
class MainGUI:
def __init__(self):
self._async_log = AsyncLog()
self.log = self._async_log.log # stored as bound method
def do_this_test(self):
print "I am about to log 100 times per sec, while text output frequency is 2Hz (twice per second)"
def log_100_records_in_one_second(itteration_index):
for i in xrange(100):
self.log("something happened", timestamp=now(), session=3.1415, itteration=itteration_index)
sleep(0.01)
for iter_index in range(3):
log_100_records_in_one_second(iter_index)
test = MainGUI()
test.do_this_test()
I have noticed that you do not sleep() anywhere in the sending loop, this means data is read as fast as it can and is sent as fast as it can. Note that this is not desirable behavior when playing media files - container time-stamps are there to dictate data-rate.