Develop Reference

How to make a python script stopable from another script?

TL;DR: If you have a program that should run for an undetermined amount of time, how do you code something to stop it when the user decide it is time? (Without KeyboardInterrupt or killing the task)
--
I've recently posted this question: How to make my code stopable? (Not killing/interrupting)
The answers did address my question, but from a termination/interruption point of view, and that's not really what I wanted. (Although, my question didn't made that clear)
So, I'm rephrasing it.
I created a generic script for example purposes. So I have this class, that gathers data from a generic API and write the data into a csv. The code is started by typing python main.py on a terminal window.
import time,csv
import GenericAPI
class GenericDataCollector:
def __init__(self):
self.generic_api = GenericAPI()
self.loop_control = True
def collect_data(self):
while self.loop_control: #Can this var be changed from outside of the class? (Maybe one solution)
data = self.generic_api.fetch_data() #Returns a JSON with some data
self.write_on_csv(data)
time.sleep(1)
def write_on_csv(self, data):
with open('file.csv','wt') as f:
writer = csv.writer(f)
writer.writerow(data)
def run():
obj = GenericDataCollector()
obj.collect_data()
if __name__ == "__main__":
run()
The script is supposed to run forever OR until I command it to stop. I know I can just KeyboardInterrupt (Ctrl+C) or abruptly kill the task. That isn't what I'm looking for. I want a "soft" way to tell the script it's time to stop, not only because interruption can be unpredictable, but it's also a harsh way to stop.
If that script was running on a docker container (for example) you wouldn't be able to Ctrl+C unless you happen to be in the terminal/bash inside the docker.
Or another situation: If that script was made for a customer, I don't think it's ok to tell the customer, just use Ctrl+C/kill the task to stop it. Definitely counterintuitive, especially if it's a non tech person.
I'm looking for way to code another script (assuming that's a possible solution) that would change to False the attribute obj.loop_control, finishing the loop once it's completed. Something that could be run by typing on a (different) terminal python stop_script.py.
It doesn't, necessarily, needs to be this way. Other solutions are also acceptable, as long it doesn't involve KeyboardInterrupt or Killing tasks. If I could use a method inside the class, that would be great, as long I can call it from another terminal/script.
Is there a way to do this?
If you have a program that should run for an undetermined amount of time, how do you code something to stop it when the user decide it is time?

In general, there are two main ways of doing this (as far as I can see). The first one would be to make your script check some condition that can be modified from outside (like the existence or the content of some file/socket). Or as #Green Cloak Guy stated, using pipes which is one form of interprocess communication.
The second one would be to use the built in mechanism for interprocess communication called signals that exists in every OS where python runs. When the user presses Ctrl+C the terminal sends a specific signal to the process in the foreground. But you can send the same (or another) signal programmatically (i.e. from another script).
Reading the answers to your other question I would say that what is missing to address this one is a way to send the appropriate signal to your already running process. Essentially this can be done by using the os.kill() function. Note that although the function is called 'kill' it can send any signal (not only SIGKILL).
In order for this to work you need to have the process id of the running process. A commonly used way of knowing this is making your script save its process id when it launches into a file stored in a common location. To get the current process id you can use the os.getpid() function.
So summarizing I'd say that the steps to achieve what you want would be:
Modify your current script to store its process id (obtainable by using os.getpid()) into a file in a common location, for example /tmp/myscript.pid. Note that if you want your script to be protable you will need to address this in a way that works in non-unix like OSs like Windows.
Choose one signal (typically SIGINT or SIGSTOP or SIGTERM) and modify your script to register a custom handler using signal.signal() that addresses the graceful termination of your script.
Create another (note that it could be the same script with some command line paramater) script that reads the process id from the known file (aka /tmp/myscript.pid) and sends the chosen signal to that process using os.kill().
Note that an advantage of using signals to achieve this instead of an external way (files, pipes, etc.) is that the user can still press Ctrl+C (if you chose SIGINT) and that will produce the same behavior as the 'stop script' would.

What you're really looking for is any way to send a signal from one program to another, independent, program. One way to do this would be to use an inter-process pipe. Python has a module for this (which does, admittedly, seem to require a POSIX-compliant shell, but most major operating systems should provide that).
What you'll have to do is agree on a filepath beforehand between your running-program (let's say main.py) and your stopping-program (let's say stop.sh). Then you might make the main program run until someone inputs something to that pipe:
import pipes
...
t = pipes.Template()
# create a pipe in the first place
t.open("/tmp/pipefile", "w")
# create a lasting pipe to read from that
pipefile = t.open("/tmp/pipefile", "r")
...
And now, inside your program, change your loop condition to "as long as there's no input from this file - unless someone writes something to it, .read() will return an empty string:
while not pipefile.read():
# do stuff
To stop it, you put another file or script or something that will write to that file. This is easiest to do with a shell script:
#!/usr/bin/env sh
echo STOP >> /tmp/pipefile
which, if you're containerizing this, you could put in /usr/bin and name it stop, give it at least 0111 permissions, and tell your user "to stop the program, just do docker exec containername stop".
(using >> instead of > is important because we just want to append to the pipe, not to overwrite it).
Proof of concept on my python console:
>>> import pipes
>>> t = pipes.Template()
>>> t.open("/tmp/file1", "w")
<_io.TextIOWrapper name='/tmp/file1' mode='w' encoding='UTF-8'>
>>> pipefile = t.open("/tmp/file1", "r")
>>> i = 0
>>> while not pipefile.read():
... i += 1
...
At this point I go to a different terminal tab and do
$ echo "Stop" >> /tmp/file1
then I go back to my python tab, and the while loop is no longer executing, so I can check what happened to i while I was gone.
>>> print(i)
1704312

Add time delay between every statement of python code

Is there an easy way to execute time delay (like time.sleep(3)) between every statement of Python code without having to explicitly write between every statement?
Like in the below Python Script which performs certain action on SAP GUI window. Sometimes, the script continues to the next statement before the previous statement is complete. So, I had to add a time delay between every statement so that it executes correctly. It is working with time delay, but I end up adding time.sleep(3) between every line. Just wondering if there is a better way?
import win32com.client
import time
sapgui = win32com.client.GetObject("SAPGUI").GetScriptingEngine
session = sapgui.FindById("ses[0]")
def add_record(employee_num, start_date, comp_code):
try:
time.sleep(3)
session.findById("wnd[0]/tbar[0]/okcd").text = "/npa40"
time.sleep(3)
session.findById("wnd[0]").sendVKey(0)
time.sleep(3)
session.findById("wnd[0]/usr/ctxtRP50G-PERNR").text = employee_num
time.sleep(3)
session.findById("wnd[0]").sendVKey(0)
time.sleep(3)
session.findById("wnd[0]/usr/ctxtRP50G-EINDA").text = start_date
time.sleep(3)
session.findById("wnd[0]/usr/tblSAPMP50ATC_MENU_EVENT/ctxtRP50G-WERKS[1,0]").text = comp_code
time.sleep(3)
session.findById("wnd[0]/usr/tblSAPMP50ATC_MENU_EVENT/ctxtRP50G-PERSG[2,0]").text = "1"
time.sleep(3)
session.findById("wnd[0]/usr/tblSAPMP50ATC_MENU_EVENT/ctxtRP50G-PERSK[3,0]").text = "U1"
time.sleep(3)
session.findById("wnd[0]/usr/tblSAPMP50ATC_MENU_EVENT").getAbsoluteRow(0).selected = True
time.sleep(3)
return "Pass"
except:
return "failed"

The right way to do what you asked for is almost certainly to use the debugger, pdb.
The right way to do what you want is probably something completely different: find some signal that tells you that the step is done, and wait for that signal. With problems like this, almost any time you pick will be way, way too long 99% of the time, but still too short 1% of the time. That signal may be joining a thread, or waiting on a (threading or multiprocessing) Condition, or getting from a queue, or awaiting a coroutine or future, or setting the sync flag on an AppleEvent, or… It really depends on what you're doing.
But if you really want to do this, you can use settrace:
def sleeper(frame, event, arg):
if event == 'line':
time.sleep(2)
return sleeper
sys.settrace(sleeper)
One small problem is that the notion of line used by the interpreter may well not be what you want. Briefly, a 'line' trace event is triggered whenever the ceval loop jumps to a different lnotab entry (see lnotab_notes.txt in the source to understand what that means—and you'll probably need at least a passing understanding of how bytecode is interpreted, at least from reading over the dis docs, to understand that). So, for example, a multiline expression is a single line; the line of a with statement may appear twice, etc.1
And there's probably an even bigger problem.
Sometimes, the script continues to next step before the previous step is fully complete.
I don't know what those steps are, but if you put the whole thread to sleep for 2 seconds, there's a good chance the step you're waiting for won't make any progress, because the thread is asleep. (For example, you're not looping through any async or GUI event loops, because you're doing nothing at all.) If so, then after 2 seconds, it'll still be just as incomplete as it was before, and you'll have wasted 2 seconds for nothing.
1. If your notion of "line" is closer to what's described in the reference docs on lexing and parsing Python, you could create an import hook that walks the AST and adds an expression statement with a Call to time.sleep(2) after each list element in each body with a module, definition, or compound statement (and then compiles and execs the result as usual).

Anything you want to happen in a program has to be explicitly stated - this is the nature of programming. This is like asking if you can print hello world without calling print("hello world").

I think the best advice to give you here is: don't think in terms of "lines", but think in term of functions.

use debugging mode and watch each and every line executing line by line.

Executing commands at breakpoint in gdb via python interface

Not a duplicate of this question, as I'm working through the python interface to gdb.
This one is similar but does not have an answer.
I'm extending a gdb.breakpoint in python so that it writes certain registers to file, and then jumps to an address: at 0x4021ee, I want to write stuff to file, then jump to 0x4021f3
However, nothing in command is ever getting executed.
import gdb
class DebugPrintingBreakpoint(gdb.Breakpoint):
def __init__(self, spec, command):
super(DebugPrintingBreakpoint, self).__init__(spec, gdb.BP_BREAKPOINT, internal = False)
self.command = command
def stop(self):
with open('tracer', 'a') as f:
f.write(chr(gdb.parse_and_eval("$rbx") ^ 0x71))
f.close()
return False
gdb.execute("start")
DebugPrintingBreakpoint("*0x4021ee", "jump *0x4021f3")
gdb.execute("continue")
If I explicitly add gdb.execute(self.command) to the end of stop(), I get Python Exception <class 'gdb.error'> Cannot execute this command while the selected thread is running.:
Anyone have a working example of command lists with breakpoints in python gdb?

A couple options to try:
Use gdb.post_event from stop() to run the desired command later. I believe you'll need to return True from your function then call continue from your event.
Create a normal breakpoint and listen to events.stop to check if your breakpoint was hit.

The Breakpoint.stop method is called when, in gdb terms, the inferior is still "executing". Partly this is a bookkeeping oddity -- of course the inferior isn't really executing, it is stopped while gdb does a bit of breakpoint-related processing. Internally it is more like gdb hasn't yet decided to report the stop to other interested parties inside gdb. This funny state is what lets stop work so nicely vis a vis next and other execution commands.
Some commands in gdb can't be invoked while the inferior is running, like jump, as you've found.
One thing you could try -- I have never tried this and don't know if it would work -- would be to assign to the PC in your stop method. This might do the right thing; but of course you should know that the documentation warns against doing weird stuff like this.
Failing that I think the only approach is to fall back to using commands to attach the jump to the breakpoint. This has the drawback that it will interfere with next.
One final way would be to patch the running code to insert a jump or just a sequence of nops.

Changes to a Gmail notification script

After a long search I've found this python script that does what I need in order to get a real time notification to my iOS app when a new email arrives. I usually write in Objective-c and this is the first time I'm dealing with Python. Before I'll try to setup and run the script I'd like to understand it a bit better.
This is the part that I'm not sure about:
# Because this is just an example, exit after 8 hours
time.sleep(8*60*60)
#finally:
# Clean up.
idler.stop()
idler.join()
M.close()
# This is important!
M.logout()
My questions:
Should I comment out time.sleep(8*60*60) If I want to keep the connection active at all times?
What's the use for the Clean up section? Do I need it if I want to keep the connection?
Why M.logout() is important?
The main question that includes all the above is What changes (if any) I need to do to this script in order for it to function without stoping or timing out.
Thanks

The script has started another thread, the actual work is done in this other thread.
For some reason the main thread is left without anything to do, that's why the author has put the time.sleep(8*60*60) to occupy it for a while.
If you want to keep the connection active at all times you need to uncomment the try:/finally:, see bellow.
If you are new to python beware that indentation is used to define blocks of code. The cleanup part might actually not be useful if you don't plan to stop the program, but with the try:/finally: the cleanup code will be executed even if you stop the program with Ctrl+C.
Not tested:
# Had to do this stuff in a try-finally, since some testing
# went a little wrong.....
try:
# Set the following two lines to your creds and server
M = imaplib2.IMAP4_SSL("imap.gmail.com")
M.login(USER, PASSWORD)
# We need to get out of the AUTH state, so we just select
# the INBOX.
M.select("INBOX")
numUnseen = getUnseen()
sendPushNotification(numUnseen)
#print M.status("INBOX", '(UNSEEN)')
# Start the Idler thread
idler = Idler(M)
idler.start()
# Sleep forever, one minute at a time
while True:
time.sleep(60)
finally:
# Clean up.
idler.stop()
idler.join()
M.close()
# This is important!
M.logout()

Drop into an Interpreter anytime in Python

I know how to drop into an interpreter with pdb and IPython, but this requires me knowing beforehand exactly where I want to stop. However, I often run number crunching scripts that take minutes to hours, and I would like to know exactly what it's progress is. One solution is to simply put lots of logging statements everywhere, but then I either inundate myself with too much information or fail to log exactly what I want to know.
Is there a way to initialize a listener loop that under some key combination will drop me into the code wherever it currently is? Think CTRL+Z but leaving me in Python rather than Bash.

You can use the signal module to setup a handler that will launch the debugger when you hit control-C or control-Z or whatever.. SIGINTR, SIGSUSP.
For example, define a module instant_debug.py that overrides SIGQUIT,
import signal
import pdb
def handler(signum, frame):
pdb.set_trace()
signal.signal(signal.SIGQUIT, handler)
Then make a script
import instant_debug
import time
for i in xrange(1000000):
print i
time.sleep(0.1)
At any point during execution, you can jump into the code by typing CTRL+\, examine the stack with u and d as in normal pdb, then continue with c as if nothing ever happened. Note that you will only jump in at the end of the next "atomic" operation -- that means no stopping in the middle of a giant C module.

You could do this
def main():
i = 1000
while True:
print "Count Down %s" % i
time.sleep(1)
i -= 1
try:
main()
except KeyboardInterrupt:
pass # Swallow ctrl-c
finally:
code.interact("Dropped into interpreter", local=globals())