I have a Python-based app that can accept a few commands in a simple read-eval-print-loop. I'm using raw_input('> ') to get the input. On Unix-based systems, I also import readline to make things behave a little better. All this is working fine.
The problem is that there are asynchronous events coming in, and I'd like to print output as soon as they happen. Unfortunately, this makes things look ugly. The "> " string doesn't show up again after the output, and if the user is halfway through typing something, it chops their text in half. It should probably redraw the user's text-in-progress after printing something.
This seems like it must be a solved problem. What's the proper way to do this?
Also note that some of my users are Windows-based.
TIA
Edit: The accepted answer works under Unixy platforms (when the readline module is available), but if anyone knows how to make this work under Windows, it would be much appreciated!
Maybe something like this will do the trick:
#!/usr/bin/env python2.6
from __future__ import print_function
import readline
import threading
PROMPT = '> '
def interrupt():
print() # Don't want to end up on the same line the user is typing on.
print('Interrupting cow -- moo!')
print(PROMPT, readline.get_line_buffer(), sep='', end='')
def cli():
while True:
cli = str(raw_input(PROMPT))
if __name__ == '__main__':
threading.Thread(target=cli).start()
threading.Timer(2, interrupt).start()
I don't think that stdin is thread-safe, so you can end up losing characters to the interrupting thread (that the user will have to retype at the end of the interrupt). I exaggerated the amount of interrupt time with the time.sleep call. The readline.get_line_buffer call won't display the characters that get lost, so it all turns out alright.
Note that stdout itself isn't thread safe, so if you've got multiple interrupting threads of execution, this can still end up looking gross.
Why are you writing your own REPL using raw_input()? Have you looked at the cmd.Cmd class? Edit: I just found the sclapp library, which may also be useful.
Note: the cmd.Cmd class (and sclapp) may or may not directly support your original goal; you may have to subclass it and modify it as needed to provide that feature.
run this:
python -m twisted.conch.stdio
You'll get a nice, colored, async REPL, without using threads. While you type in the prompt, the event loop is running.
look into the code module, it lets you create objects for interpreting python code also (shameless plug) https://github.com/iridium172/PyTerm lets you create interactive command line programs that handle raw keyboard input (like ^C will raise a KeyboardInterrupt).
It's kind of a non-answer, but I would look at IPython's code to see how they're doing it.
I think you have 2 basic options:
Synchronize your output (i.e. block until it comes back)
Separate your input and your (asyncronous) output, perhaps in two separate columns.
Related
I have a c exectuable that I want to exploit.
The output of that file looks like this:
$ ./vuln_nostack
Enter some text:
enteringTEXT
You entered: enteringTEXT
You enter some text, and the program spits it back.
I want to run this prorgam (and later exploit it) with python and pwntools.
So far, the functioning part of my pwntools program looks like this:
from concurrent.futures import process
from sys import stdout
from pwn import *
import time
pty = process.PTY
p = process("./vuln_nostack", stdin=pty, stdout=pty)
ss = p.recv()
p.clean()
asstring = ss.decode("utf-8")
print(asstring)
This works fine, it gets the first line and then prints it.
What I want to do now is to send a message to the program and then get the final line.
I have tried something along these lines:
p.send(b"dong")
p.clean()
print(p.recv())
I'm not sure whether or not the send actually ever sends anything, but as soon as I add the recv function, the prorgam just hangs and never finishes.
My guess is that the input to the executable is never given properly, and therefore it's still just waiting.
How do I actually get a message delivered to the exectuable so that it can move on and srever me the last line?
You can also use p.sendline():
p.sendline("payload")
This automatically adds a breakline after your bytes.
Moreover, to know whether your exploit is sending/receiving messages to/from the program, you can use debug context by adding this assignment:
context.log_level = 'debug'
The answer was a lot more simple than formerly presumed.
I just needed a breakline in the send:
p.send("payload \n")
TL;DR: If you have a program that should run for an undetermined amount of time, how do you code something to stop it when the user decide it is time? (Without KeyboardInterrupt or killing the task)
--
I've recently posted this question: How to make my code stopable? (Not killing/interrupting)
The answers did address my question, but from a termination/interruption point of view, and that's not really what I wanted. (Although, my question didn't made that clear)
So, I'm rephrasing it.
I created a generic script for example purposes. So I have this class, that gathers data from a generic API and write the data into a csv. The code is started by typing python main.py on a terminal window.
import time,csv
import GenericAPI
class GenericDataCollector:
def __init__(self):
self.generic_api = GenericAPI()
self.loop_control = True
def collect_data(self):
while self.loop_control: #Can this var be changed from outside of the class? (Maybe one solution)
data = self.generic_api.fetch_data() #Returns a JSON with some data
self.write_on_csv(data)
time.sleep(1)
def write_on_csv(self, data):
with open('file.csv','wt') as f:
writer = csv.writer(f)
writer.writerow(data)
def run():
obj = GenericDataCollector()
obj.collect_data()
if __name__ == "__main__":
run()
The script is supposed to run forever OR until I command it to stop. I know I can just KeyboardInterrupt (Ctrl+C) or abruptly kill the task. That isn't what I'm looking for. I want a "soft" way to tell the script it's time to stop, not only because interruption can be unpredictable, but it's also a harsh way to stop.
If that script was running on a docker container (for example) you wouldn't be able to Ctrl+C unless you happen to be in the terminal/bash inside the docker.
Or another situation: If that script was made for a customer, I don't think it's ok to tell the customer, just use Ctrl+C/kill the task to stop it. Definitely counterintuitive, especially if it's a non tech person.
I'm looking for way to code another script (assuming that's a possible solution) that would change to False the attribute obj.loop_control, finishing the loop once it's completed. Something that could be run by typing on a (different) terminal python stop_script.py.
It doesn't, necessarily, needs to be this way. Other solutions are also acceptable, as long it doesn't involve KeyboardInterrupt or Killing tasks. If I could use a method inside the class, that would be great, as long I can call it from another terminal/script.
Is there a way to do this?
If you have a program that should run for an undetermined amount of time, how do you code something to stop it when the user decide it is time?
In general, there are two main ways of doing this (as far as I can see). The first one would be to make your script check some condition that can be modified from outside (like the existence or the content of some file/socket). Or as #Green Cloak Guy stated, using pipes which is one form of interprocess communication.
The second one would be to use the built in mechanism for interprocess communication called signals that exists in every OS where python runs. When the user presses Ctrl+C the terminal sends a specific signal to the process in the foreground. But you can send the same (or another) signal programmatically (i.e. from another script).
Reading the answers to your other question I would say that what is missing to address this one is a way to send the appropriate signal to your already running process. Essentially this can be done by using the os.kill() function. Note that although the function is called 'kill' it can send any signal (not only SIGKILL).
In order for this to work you need to have the process id of the running process. A commonly used way of knowing this is making your script save its process id when it launches into a file stored in a common location. To get the current process id you can use the os.getpid() function.
So summarizing I'd say that the steps to achieve what you want would be:
Modify your current script to store its process id (obtainable by using os.getpid()) into a file in a common location, for example /tmp/myscript.pid. Note that if you want your script to be protable you will need to address this in a way that works in non-unix like OSs like Windows.
Choose one signal (typically SIGINT or SIGSTOP or SIGTERM) and modify your script to register a custom handler using signal.signal() that addresses the graceful termination of your script.
Create another (note that it could be the same script with some command line paramater) script that reads the process id from the known file (aka /tmp/myscript.pid) and sends the chosen signal to that process using os.kill().
Note that an advantage of using signals to achieve this instead of an external way (files, pipes, etc.) is that the user can still press Ctrl+C (if you chose SIGINT) and that will produce the same behavior as the 'stop script' would.
What you're really looking for is any way to send a signal from one program to another, independent, program. One way to do this would be to use an inter-process pipe. Python has a module for this (which does, admittedly, seem to require a POSIX-compliant shell, but most major operating systems should provide that).
What you'll have to do is agree on a filepath beforehand between your running-program (let's say main.py) and your stopping-program (let's say stop.sh). Then you might make the main program run until someone inputs something to that pipe:
import pipes
...
t = pipes.Template()
# create a pipe in the first place
t.open("/tmp/pipefile", "w")
# create a lasting pipe to read from that
pipefile = t.open("/tmp/pipefile", "r")
...
And now, inside your program, change your loop condition to "as long as there's no input from this file - unless someone writes something to it, .read() will return an empty string:
while not pipefile.read():
# do stuff
To stop it, you put another file or script or something that will write to that file. This is easiest to do with a shell script:
#!/usr/bin/env sh
echo STOP >> /tmp/pipefile
which, if you're containerizing this, you could put in /usr/bin and name it stop, give it at least 0111 permissions, and tell your user "to stop the program, just do docker exec containername stop".
(using >> instead of > is important because we just want to append to the pipe, not to overwrite it).
Proof of concept on my python console:
>>> import pipes
>>> t = pipes.Template()
>>> t.open("/tmp/file1", "w")
<_io.TextIOWrapper name='/tmp/file1' mode='w' encoding='UTF-8'>
>>> pipefile = t.open("/tmp/file1", "r")
>>> i = 0
>>> while not pipefile.read():
... i += 1
...
At this point I go to a different terminal tab and do
$ echo "Stop" >> /tmp/file1
then I go back to my python tab, and the while loop is no longer executing, so I can check what happened to i while I was gone.
>>> print(i)
1704312
Not a duplicate of this question, as I'm working through the python interface to gdb.
This one is similar but does not have an answer.
I'm extending a gdb.breakpoint in python so that it writes certain registers to file, and then jumps to an address: at 0x4021ee, I want to write stuff to file, then jump to 0x4021f3
However, nothing in command is ever getting executed.
import gdb
class DebugPrintingBreakpoint(gdb.Breakpoint):
def __init__(self, spec, command):
super(DebugPrintingBreakpoint, self).__init__(spec, gdb.BP_BREAKPOINT, internal = False)
self.command = command
def stop(self):
with open('tracer', 'a') as f:
f.write(chr(gdb.parse_and_eval("$rbx") ^ 0x71))
f.close()
return False
gdb.execute("start")
DebugPrintingBreakpoint("*0x4021ee", "jump *0x4021f3")
gdb.execute("continue")
If I explicitly add gdb.execute(self.command) to the end of stop(), I get Python Exception <class 'gdb.error'> Cannot execute this command while the selected thread is running.:
Anyone have a working example of command lists with breakpoints in python gdb?
A couple options to try:
Use gdb.post_event from stop() to run the desired command later. I believe you'll need to return True from your function then call continue from your event.
Create a normal breakpoint and listen to events.stop to check if your breakpoint was hit.
The Breakpoint.stop method is called when, in gdb terms, the inferior is still "executing". Partly this is a bookkeeping oddity -- of course the inferior isn't really executing, it is stopped while gdb does a bit of breakpoint-related processing. Internally it is more like gdb hasn't yet decided to report the stop to other interested parties inside gdb. This funny state is what lets stop work so nicely vis a vis next and other execution commands.
Some commands in gdb can't be invoked while the inferior is running, like jump, as you've found.
One thing you could try -- I have never tried this and don't know if it would work -- would be to assign to the PC in your stop method. This might do the right thing; but of course you should know that the documentation warns against doing weird stuff like this.
Failing that I think the only approach is to fall back to using commands to attach the jump to the breakpoint. This has the drawback that it will interfere with next.
One final way would be to patch the running code to insert a jump or just a sequence of nops.
I wrote a script in python that takes a few files, runs a few tests and counts the number of total_bugs while writing new files with information for each (bugs+more).
To take a couple files from current working directory:
myscript.py -i input_name1 input_name2
When that job is done, I'd like the script to 'return total_bugs' but I'm not sure on the best way to implement this.
Currently, the script prints stuff like:
[working directory]
[files being opened]
[completed work for file a + num_of_bugs_for_a]
[completed work for file b + num_of_bugs_for_b]
...
[work complete]
A bit of help (notes/tips/code examples) could be helpful here.
Btw, this needs to work for windows and unix.
If you want your script to return values, just do return [1,2,3] from a function wrapping your code but then you'd have to import your script from another script to even have any use for that information:
Return values (from a wrapping-function)
(again, this would have to be run by a separate Python script and be imported in order to even do any good):
import ...
def main():
# calculate stuff
return [1,2,3]
Exit codes as indicators
(This is generally just good for when you want to indicate to a governor what went wrong or simply the number of bugs/rows counted or w/e. Normally 0 is a good exit and >=1 is a bad exit but you could inter-prate them in any way you want to get data out of it)
import sys
# calculate and stuff
sys.exit(100)
And exit with a specific exit code depending on what you want that to tell your governor.
I used exit codes when running script by a scheduling and monitoring environment to indicate what has happened.
(os._exit(100) also works, and is a bit more forceful)
Stdout as your relay
If not you'd have to use stdout to communicate with the outside world (like you've described).
But that's generally a bad idea unless it's a parser executing your script and can catch whatever it is you're reporting to.
import sys
# calculate stuff
sys.stdout.write('Bugs: 5|Other: 10\n')
sys.stdout.flush()
sys.exit(0)
Are you running your script in a controlled scheduling environment then exit codes are the best way to go.
Files as conveyors
There's also the option to simply write information to a file, and store the result there.
# calculate
with open('finish.txt', 'wb') as fh:
fh.write(str(5)+'\n')
And pick up the value/result from there. You could even do it in a CSV format for others to read simplistically.
Sockets as conveyors
If none of the above work, you can also use network sockets locally *(unix sockets is a great way on nix systems). These are a bit more intricate and deserve their own post/answer. But editing to add it here as it's a good option to communicate between processes. Especially if they should run multiple tasks and return values.
Certain applications like hellanzb have a way of printing to the terminal with the appearance of dynamically refreshing data, kind of like top().
Whats the best method in python for doing this? I have read up on logging and curses, but don't know what to use. I am creating a reimplementation of top. If you have any other suggestions I am open to them as well.
The simplest way, if you only ever need to update a single line (for instance, creating a progress bar), is to use '\r' (carriage return) and sys.stdout:
import sys
import time
for i in range(10):
sys.stdout.write("\r{0}>".format("="*i))
sys.stdout.flush()
time.sleep(0.5)
If you need a proper console UI that support moving the pointer etc., use the curses module from the standard library:
import time
import curses
def pbar(window):
for i in range(10):
window.addstr(10, 10, "[" + ("=" * i) + ">" + (" " * (10 - i )) + "]")
window.refresh()
time.sleep(0.5)
curses.wrapper(pbar)
It's highly advisable to use the curses.wrapper function to call your main function, it will take care of cleaning up the terminal in case of an error, so it won't be in an unusable state afterwards.
If you create a more complex UI, you can create multiple windows for different parts of the screen, text input boxes and mouse support.
As most of the answers have already stated, you really have little option on Linux but to use ncurses. But what if you aren't on Linux, or want something a little more high-level for creating your terminal UI?
I personally found the lack of a modern, cross-platform terminal API in Python frustrating, so wrote asciimatics to solve this. Not only does it give you a simple cross-platform API, it also provides a lot of higher level abstractions for UI widgets and animations which could be easily used to create a top-like UI.
Sending output to the terminal via the print() command can be done without scrolling if you use the attribute "end".
The default is end='\n' which is a new line.
To suppress scrolling and overwrite the whole previous line, you can use the RETURN escape which is '\r'.
If you only want to rewrite the last four characters, you can use a few back-spaces.
print(value, "_of_", total, end='\r')
NOTE
This works for the standard system terminal. The terminal emulator in some tools like IDLE has an error and the '\r' does not work properly, the output is simply concatenated with some non-printable character between.
BONUS INFORMATION FOR print()
In the example above, the spaces on each side of "of" are meant to insure white-space between my values and the word "of". However, the default separater of the print() is a " " (space) so we end up with white space between the value and underscore of "_of_".
>> print (value, "_of_", total, end='\r')
8 _of_ 17
The sepparator attribute, sep, can be used to set character between printed items. In my example, I will change it to a null string ('') to make my output suit my needs.
>> print (value, "_of_", total, sep='', end='\r')
8_of_17
I hacked this script using curses. Its really a ad-hoc solution I did for a fun. It does not support scrolling but I think its a good starting point if you are looking to build a live updating monitor with multiple rows on the terminal.
https://gist.github.com/tpandit/b2bc4f434ee7f5fd890e095e79283aec
Here is the main:
if __name__ == "__main__":
stdscr = curses.initscr()
curses.noecho()
curses.cbreak()
curses.start_color()
curses.init_pair(1, curses.COLOR_GREEN, curses.COLOR_BLACK)
curses.init_pair(2, curses.COLOR_RED, curses.COLOR_BLACK)
curses.init_pair(3, curses.COLOR_YELLOW, curses.COLOR_BLACK)
curses.init_pair(4, curses.COLOR_CYAN, curses.COLOR_BLACK)
try:
while True:
resp = get_data()
report_progress(get_data())
time.sleep(60/REQUESTS_PER_MINUTE)
finally:
curses.echo()
curses.nocbreak()
curses.endwin()
When I do this in shell scripts on Unix, I tend to just use the clear program. You can use the Python subprocess module to execute it. It will at least get you what you're looking for quickly.
import time
for i in range(10):
print('\r{}>'.format('='*i), end='')
time.sleep(0.5)
I don't think that including another libraries in this situation is really good practice. So, solution:
print("\rCurrent: %s\t%s" % (str(<value>), <another_value>), end="")