Develop Reference

print information in a corner when using python/ipython REPL -- and quitting it while thread is running

I am using a python library I wrote to interact with a custom USB device. The library needs to send and receive data. I also need to interactively call methods. At the moment I utilize two shells, one receiving only and the other sending only. The latter is in the (i)python REPL. It works but it is clumsy, so I want to consolidate the two things in a single shell, which will have the advantage to have access to data structures from both sides in one context. That works fine. The problem is in the UI.
In fact, the receiving part needs to asynchronously print some information. So I wrote something like the following:
#!/usr/bin/python
import threading
import time
import blessings
TOTAL=5
def print_time( thread_id, delay):
count = 0
t=blessings.Terminal()
while count < TOTAL:
time.sleep(delay)
count += 1
stuff = "Thread " + str(thread_id) + " " + str(time.ctime(time.time())) + " -- " + str(TOTAL - count) + " to go"
with t.location(t.width - len(stuff) - 1, thread_id):
print (stuff, end=None )
print("", end="") # just return the cursor
try:
t1 = threading.Thread( target = print_time, args = (1, 2, ) )
t1.start()
print ("Thread started")
except:
print ("Error: unable to start thread")
That is my __init__.py file for the module. It somewhat works, but it has two problems:
While the thread is running, you cannot exit the REPL neither with CTRL-D nor with sys.exit() (that is the reason I am using TOTAL=5 above, so your life is easier if you try this code). This is a problem since my actual thread needs to be an infinite loop. I guess one solution could be to exit via a custom call which will cause a break into that infinite loop, but is there anything better?
The cursor does not return correctly to its earlier position
if I remove the end="" in the line with the comment # just return the cursor, it sort of works, but obviously print an unwanted newline in the place the cursor was (which messes us other input and/or output which might be happening there, in addition to add that unwanted newline)
if I leave the end="" it does not return the cursor, not even if I add something to print, e.g. print(".", end="") -- the dots . are printed at the right place, but the blinking cursor and the input is printed at the top
I know these are two unrelated problem and I could have asked two separate questions, but I need an answer to both, or otherwise it's a moot point. Or alternatively, I am open to other solutions. I thought of a separate GTK window, and that might work, but it's a subpar solution, since I really would like this to work in CLI only (to keep it possible in a ssh-without-X-tunneling setup).

Using blessed instead of blessing does not have the problem with the cursor not returning to the previous position, even without anything outside of the with context.
Making the thread a daemon solves the other problem.

Does `print()` delay in some consoles where `stdout.flush` doesn't?

I'm working on an open source python library that uses a verbose_print command to log outputs in the console. Currently it looks like this:
def sys_write_flush(s):
""" Writes and flushes without delay a text in the console """
sys.stdout.write(s)
sys.stdout.flush()
def verbose_print(verbose, s):
""" Only prints s (with sys_write_flush) if verbose is True."""
if verbose:
sys_write_flush(s)
I proposed a change that looks like this:
def verbose_print(verbose, *args):
""" Prints everything passed except the first argument if verbose is True."""
if verbose:
print(*args)
Apart from the fact that it fails on Python 2 (bonus point for fixing this!), I thought that this would be better and more idiomatic. The advantages are, that you can treat verbose_print exactly like print, except that the first argument has to be True or False.
The repo owner replied with this message:
I should have documented this one, but basically the issue was that in
some consoles (and in the IPython notebook, at least at the time),
"print" commands get delayed, while stdout.flush are instantaneous, so
my method was better at providing feedback.
I would be against changing it to print unless it solves some known
issues.
Is this still a valid concern? Would print() followed by sys.stdout.flush() avoid the delay? Are there any better ways to write this?
Source

Quote from the docs:
print evaluates each expression in turn and writes the resulting
object to standard output.
Standard output is defined as the file object named stdout in the
built-in module sys. If no such object exists, or if it does not
have a write() method, a RuntimeError exception is raised.
According to this, print writes to sys.stdout, so, yes, doing a sys.stdout.flush() after printing will have the same effect as flushing after sys.stdout.write-ing.
The syntax print(*a) fails in Python 2 because print isn't a function, but a statement, and that fun(*stuff) construct is only applicable to functions.
In Python 3 print(*a) passes whatever a contains to the function print as separate arguments, but this is equal to passing a big string:
separator = ' '
print separator.join(map(str, iterable))
So, your code could look like this:
def verbose_print(verbose, *args):
""" Prints everything passed except the first argument if verbose is True."""
if verbose:
print " ".join(map(str, args))
sys.stdout.flush()
Although I don't see why this can be faster or more readable than the original.

How to test while-loop (once) with nosetest (Python 2.7)

I'm pretty new to this whole "programming thing" but at age 34 I thought that I'd like to learn the basics.
I unfortunately don't know any python programmers. I'm learning programming due to personal interest (and more and more for the fun of it) but my "social habitat" is not "where the programmers roam" ;) .
I'm almost finished with Zed Shaws "Learn Python the Hard Way" and for the first time I can't figure out a solution to a problem. The last two days I didn't even stumble upon useful hints where to look when I repeatedly rephrased (and searched for) my question.
So stackoverflow seems to be the right place.
Btw.: I lack also the correct vocabular quite often so please don't hesitate to correct me :) . This may be one reason why I can't find an answer.
I use Python 2.7 and nosetests.
How far I solved the problem (I think) in the steps I solved it:
Function 1:
def inp_1():
s = raw_input(">>> ")
return s
All tests import the following to be able to do the things below:
from nose.tools import *
import sys
from StringIO import StringIO
from mock import *
import __builtin__
# and of course the module with the functions
Here is the test for inp_1:
import __builtin__
from mock import *
def test_inp_1():
__builtin__.raw_input = Mock(return_value="foo")
assert_equal(inp_1(), 'foo')
This function/test is ok.
Quite similar is the following function 2:
def inp_2():
s = raw_input(">>> ")
if s == '1':
return s
else:
print "wrong"
Test:
def test_inp_2():
__builtin__.raw_input = Mock(return_value="1")
assert_equal(inp_1(), '1')
__builtin__.raw_input = Mock(return_value="foo")
out = StringIO()
sys.stdout = out
inp_1()
output = out.getvalue().strip()
assert_equal(output, 'wrong')
This function/test is also ok.
Please don't assume that I really know what is happening "behind the scenes" when I use all the stuff above. I have some layman-explanations how this is all functioning and why I get the results I want but I also have the feeling that these explanations may not be entirely true. It wouldn't be the first time that how I think sth. works turns out to be different after I've learned more. Especially everything with "__" confuses me and I'm scared to use it since I don't really understand what's going on. Anyway, now I "just" want to add a while-loop to ask for input until it is correct:
def inp_3():
while True:
s = raw_input(">>> ")
if s == '1':
return s
else:
print "wrong"
The test for inp_3 I thought would be the same as for inp_2 . At least I am not getting error messages. But the output is the following:
$ nosetests
......
# <- Here I press ENTER to provoke a reaction
# Nothing is happening though.
^C # <- Keyboard interrupt (is this the correct word for it?)
----------------------------------------------------------------------
Ran 7 tests in 5.464s
OK
$
The other 7 tests are sth. else (and ok).
The test for inp_3 would be test nr. 8.
The time is just the times passed until I press CTRL-C.
I don't understand why I don't get error- or "test failed"-meassages but just an "ok".
So beside the fact that you may be able to point out bad syntax and other things that can be improved (I really would appreciate it, if you would do this), my question is:
How can I test and abort while-loops with nosetest?

So, the problem here is when you call inp_3 in test for second time, while mocking raw_input with Mock(return_value="foo"). Your inp_3 function runs infinite loop (while True) , and you're not interrupting it in any way except for if s == '1' condition. So with Mock(return_value="foo") that condition is never satisfied, and you loop keeps running until you interrupt it with outer means (Ctrl + C in your example). If it's intentional behavior, then How to limit execution time of a function call in Python will help you to limit execution time of inp_3 in test. However, in cases of input like in your example, developers often implement a limit to how many input attempts user have. You can do it with using variable to count attempts and when it reaches max, loop should be stopped.
def inp_3():
max_attempts = 5
attempts = 0
while True:
s = raw_input(">>> ")
attempts += 1 # this is equal to "attempts = attempts + 1"
if s == '1':
return s
else:
print "wrong"
if attempts == max_attempts:
print "Max attempts used, stopping."
break # this is used to stop loop execution
# and go to next instruction after loop block
print "Stopped."
Also, to learn python I can recommend book "Learning Python" by Mark Lutz. It greatly explains basics of python.
UPDATE:
I couldn't find a way to mock python's True (or a builtin.True) (and yea, that sounds a bit crazy), looks like python didn't (and won't) allow me to do this. However, to achieve exactly what you desire, to run infinite loop once, you can use a little hack.
Define a function to return True
def true_func():
return True
, use it in while loop
while true_func():
and then mock it in test with such logic:
def true_once():
yield True
yield False
class MockTrueFunc(object):
def __init__(self):
self.gen = true_once()
def __call__(self):
return self.gen.next()
Then in test:
true_func = MockTrueFunc()
With this your loop will run only once. However, this construction uses a few advanced python tricks, like generators, "__" methods etc. So use it carefully.
But anyway, generally infinite loops considered to be bad design solutions. Better to not getting used to it :).

It's always important to remind me that infinite loops are bad. So thank you for that and even more so for the short example how to make it better. I will do that whenever possible.
However, in the actual program the infinite loop is how I'd like to do it this time. The code here is just the simplified problem.
I very much appreciate your idea with the modified "true function". I never would have thought about that and thus I learned a new "method" how tackle programming problems :) .
It is still not the way I would like to do it this time, but this was the so important clue I needed to solve my problem with existing methods. I never would have thought about returning a different value the 2nd time I call the same method. It's so simple and brilliant it's astonishing me :).
The mock-module has some features that allows a different value to be returned each time the mocked method is called - side effect .
side_effect can also be set to […] an iterable.
[when] your mock is going to be
called several times, and you want each call to return a different
value. When you set side_effect to an iterable every call to the mock
returns the next value from the iterable:
The while-loop HAS an "exit" (is this the correct term for it?). It just needs the '1' as input. I will use this to exit the loop.
def test_inp_3():
# Test if input is correct
__builtin__.raw_input = Mock(return_value="1")
assert_equal(inp_1(), '1')
# Test if output is correct if input is correct two times.
# The third time the input is corrct to exit the loop.
__builtin__.raw_input = Mock(side_effect=['foo', 'bar', '1'])
out = StringIO()
sys.stdout = out
inp_3()
output = out.getvalue().strip()
# Make sure to compare as many times as the loop
# is "used".
assert_equal(output, 'wrong\nwrong')
Now the test runs and returns "ok" or an error e.g. if the first input already exits the loop.
Thank you very much again for the help. That made my day :)

Execute a CLI command and store in a variable using telnetlib

I am using Python telnetlib for taking the "running config" output from router.How to store the "show running-config" output in a variable.And print the variable.My requirement is the each and every output will display in the console when executing each and every line of the code.Is there any option to aviod these print statements.
import telnetlib
#import getpass
ipaddr = "10.1.1.1"
passwd = "abcd123"
tn = telnetlib.Telnet(ipaddr)
if Password:
try:
print (tn.write (password + "\n"))
print(tn.read_until("Router>"))
print(tn.write("enable\n"))
print(tn.read_until("Password:"))
print(tn.write(passwd + "\n"))
print(tn.read_until("Router#"))
print(tn.write("show clock\n"))
print(tn.read_until("#"))
print(tn.write("show running-config\n"))
print(tn.write("\040\n"))
print(tn.write("\040\n"))
print(tn.write("\040\n"))
print(tn.read_until("#"))
print(tn.write("logout\n"))
print(tn.read_until(">"))
print tn.close

If I understand you correctly you wish to print out to your local console the output of each command which you run on the remote console. I am not sure why it needs to be synchronous except you say that is a requirement. You might want to make sure you understand the requirements. In any case, since your requirement is that the output be printed, you don't need to print your input...
I highly recommend storing the output into a variable even if you need to print it immediately simply because I see no benefit of retrieving the data unless you are going to act on the data and if you merely print the data you cannot act on it. Store it in a variable and then it can be printed as well as acted upon. I doubt the human eye would be able to tell the difference in storing it and then writing it all at once rather than piecemeal.
Your try block, as written, will never happen because you have to read from the telnet session first before you can evaluate if 'Password:' is on the remote console.
Some further suggestions:
First, write terminal length 0, that will avoid having to handle paused output.
Second, since I am lazy, any variables I know I am only using to pass to the remote unit I save with a newline character.
Third, always give it a timeout or else it runs the risk of waiting forever for a match that might never come.
Fourth, have a look at Telnet.expect(list, [timeout]). I find it far more useful than a simple read_until; it allows you to look for multiple responses and act on each of them accordingly. It is quite useful for handling errors. It returns a three item tuple that represents the index of the matched item (-1 if no match) as well as the matched text (or everything in the buffer if no match).
Fifth, write a decorator for your telnet session to log in. You know it will be used at least once every time you interact with a remote unit, and more if you are loading new firmware. Develop a library of functions that you can reuse rather than writing it out each time. Lazy is good.
import telnetlib
import sys
ipaddr = "10.1.1.1"
passwd = "abcd123"
def login(tn):
global passwd
passwd=passwd+'\n'
def error_check(tmp):
if tmp[0]==-1:
print "Unexpected response"
print "tmp[2]
sys.exit()
tmp=tn.expect(["Password:",], 5)
error_check(tmp)
tn.write(passwd)
tmp=expect([">",],5)
error_check(tmp)
tn.write('en\n')
tmp=expect(["Password", "#'],5)
error_check(tmp)
if tmp(0)==0: #if someone left enable unlocked, don't send a password
tn.write(passwd)
tmp=expect(["#',],5)
error_check(tmp)
tn = telnetlib.Telnet(ipaddr)
login(tn)
tn.write('terminal length 0')
tmp=expect(["#',],5)
tn.write('sho clock')
now=tn.expect(["#",], 5)
print now
tn.write('sho run')
print run
cfg=tn.expect(["#",], 5)
tn.write("logout\n"))
bye=tn.expect([">",], 5)
print bye
tn.close()

Python: how to modify/edit the string printed to screen and read it back?

I'd like to print a string to command line / terminal in Windows and then edit / change the string and read it back. Anyone knows how to do it? Thanks
print "Hell"
Hello! <---Edit it on the screen
s = raw_input()
print s
Hello!

You could do some ANSI trickery to make it look like you are editing on screen. Check out this link (also similar to this SO post on colors).
This would only work on certain terminals and configurations. ymmv.
This python script worked in my Cygwin terminal on Win7:
print 'hell'
print '\033[1A\033[4CO!'
Ends up printing hellO! on one line. The 2nd print moves the cursor up one line (Esc[1A) then over 4 characters (Esc[4C]) and then prints the 'O!'.
It wouldn't let you read it back though... only a 1/2 answer.

I had this same use-case for a command-line application.
Finally found a hack to do this.
# pip install pyautogui gnureadline
import pyautogui
import readline
from threading import Thread
def editable_input(text):
Thread(target=pyautogui.write, args=(text,)).start()
modified_input = input()
return modified_input
a = editable_input("This is a random text")
print("Received input : ", a)
The trick here is use pyautogui to send the text from keyboard. But we want to do this immediately after the input(). Since input() is a blocking call, we can run the pyautogui command in a different thread. And have an input function immediately after that in the main thread.
gnureadline is for making sure we can press left and right arrow keys to move the cursor in a terminal without printing escape characters.
Tested this on Ubuntu 20, python 3.7

raw_input accepts a parameter for a "prompt message", so use that to output the message, and then prepend it to what you get back. However, this won't allow you to backspace into the prompt, because it's a prompt and not really part of the input.
s = "Hell" + raw_input("Hell")
print s

os.sys.stdout is write only, but you can erase some characters of the last line with \b or the whole line with \r, as long as you did not write a carriage return.
(however, see also my question about limitations to the standard python console/terminal)
I once made some output exercise (including a status bar) to write,erase or animate if you will, perhaps it is helpfull:
from __future__ import print_function
import sys, time
# status generator
def range_with_status(total):
n=0
while n<total:
done = '#'*(n+1)
todo = '-'*(total-n-1)
s = '<{0}>'.format(done+todo)
if not todo:
s+='\n'
if n>0:
s = '\r'+s
sys.stdout.write(s)
sys.stdout.flush()
yield n
n+=1
print ('doing something ...')
for i in range_with_status(10):
time.sleep(0.1)
print('ready')
time.sleep(0.4)
print ('And now for something completely different ...')
time.sleep(0.5)
msg = 'I am going to erase this line from the console window.'
sys.stdout.write(msg); sys.stdout.flush()
time.sleep(1)
sys.stdout.write('\r' + ' '*len(msg))
sys.stdout.flush()
time.sleep(0.5)
print('\rdid I succeed?')
time.sleep(4)

If it's for your own purposes, then here's a dirty wee hack using the clipboard without losing what was there before:
def edit_text_at_terminal(text_to_edit):
import pyperclip
# Save old clipboard contents so user doesn't lose them
old_clipboard_contents = pyperclip.paste()
#place text you want to edit in the clipboard
pyperclip.copy(text_to_edit)
# If you're on Windows, and ctrl+v works, you can do this:
import win32com.client
shell = win32com.client.Dispatch("WScript.Shell")
shell.SendKeys("^v")
# Otherwise you should tell the user to type ctrl+v
msg = "Type ctrl+v (your old clipboard contents will be restored):\n"
# Get the new value, the old value will have been pasted
new_value= str(raw_input(msg))
# restore the old clipboard contents before returning new value
pyperclip.copy(old_clipboard_contents )
return new_value
Note that ctrl+v doesn't work in all terminals, notably the Windows default (there are ways to make it work, though I recommend using ConEmu instead).
Automating the keystrokes for other OSs will involve a different process.
Please remember this is a quick hack and not a "proper" solution. I will not be held responsible for loss of entire PhD dissertations momentarily stored on your clipboard.
For a proper solution there are better approaches such as curses for Linux, and on Windows it's worth looking into AutHotKey (perhaps throw up an input box, or do some keystrokes/clipboard wizardry).