I'm writing a command line tool (let's call it interactive_rm), that is supposed to read file paths line by line from STDIN so that it can be used with Unix pipes like in the following example:
$ find . | interactive_rm
I'm currently reading each path from STDIN like this:
def _parse_stdin():
for line in sys.stdin:
yield prepare_line(line)
Now the problem... Before the tool removes a path it should ask the user for confirmation. To do so I would use input() like this:
for path in _parse_stdin():
print('Do you want to delete this path: [y/n]\n' + path)
answer = input()
if answer == 'y':
delete(path)
But this doesn't work, because STDIN is already occupied by the pipe, so the input() function is skipped or I end up with an "EOFError: EOF when reading a line".
Does anyone know a solution to this?
When stdin is redirected, it needs to reopen the terminal to be able to read from it, e.g.:
from __future__ import print_function
import os
def main():
tty = os.open("/dev/tty", os.O_RDONLY)
while True:
r = os.read(tty, 1024)
if not r: # Wait for Ctrl-D.
break
print("----", r)
if __name__ == "__main__":
main()
And run it like the following to test it:
python t.py < /dev/null
Note that the terminal may be unavailable. For example, if the command is run through a ssh session without allocating a terminal as ssh -T ... command.
Related
I'd like to change my current password from Python script using PowerShell. For this I need to run PowerShell command from the code bellow. What would be the best way to do that?
import datetime
#generated password is always changing. There might be different quotations and slashes so it's tricky to put it into PowerShell command bellow
generated_password=r'/#\n\9{S;-l2H~'
#password to set is changing depending on date. Let's say today it has become 'xxx180220'
password_to_set="xxx"+datetime.datetime.now().strftime("%y%m%d")
#I'd like to run this PS command to change my generated password to value stored in password_to_set variable. How do I do it?
PowerShell.exe -ExecutionPolicy ByPass ([adsi]'WinNT://valueaddco/wadmin,user').ChangePassword('/#\n\9{S;-l2H~','xxx180220')
I've created new1.ps file:
echo "changing password from $($args[0]) to $($args[1])"
([adsi]'WinNT://valueaddco/wadmingolyaa1,user').ChangePassword($($args[0]), $($args[1]))
It could be launched with Python code:
from subprocess import Popen, PIPE
cmd = ['powershell.exe', '-ExecutionPolicy', 'ByPass', '-File', 'new.ps1', generated_password, password_to_set]
proc= Popen(cmd, stdout=PIPE, stderr=PIPE)
while True:
line = proc.stdout.readline()
if line != b'':
print(line.strip())
else:
break
I've been trying to run a Java program and capture it's STDOUT output to a file from the Python script. The idea is to run test files through my program and check if it matches the answers.
Per this and this SO questions, using subprocess.call is the way to go. In the code below, I am doing subprocess.call(command, stdout=f) where f is the file I opened.
The resulted file is empty and I can't quite understand why.
import glob
test_path = '/path/to/my/testfiles/'
class_path = '/path/to/classfiles/'
jar_path = '/path/to/external_jar/'
test_pattern = 'test_case*'
temp_file = 'res'
tests = glob.glob(test_path + test_pattern) # find all test files
for i, tc in enumerate(tests):
with open(test_path+temp_file, 'w') as f:
# cd into directory where the class files are and run the program
command = 'cd {p} ; java -cp {cp} package.MyProgram {tc_p}'
.format(p=class_path,
cp=jar_path,
tc_p=test_path + tc)
# execute the command and direct all STDOUT to file
subprocess.call(command.split(), stdout=f, stderr=subprocess.STDOUT)
# diff is just a lambda func that uses os.system('diff')
exec_code = diff(answers[i], test_path + temp_file)
if exec_code == BAD:
scream(':(')
I checked the docs for subprocess and they recommended using subprocess.run (added in Python 3.5). The run method returns the instance of CompletedProcess, which has a stdout field. I inspected it and the stdout was an empty string. This explained why the file f I tried to create was empty.
Even though the exit code was 0 (success) from the subprocess.call, it didn't mean that my Java program actually got executed. I ended up fixing this bug by breaking down command into two parts.
If you notice, I initially tried to cd into correct directory and then execute the Java file -- all in one command. I ended up removing cd from command and did the os.chdir(class_path) instead. The command now contained only the string to run the Java program. This did the trick.
So, the code looked like this:
good_code = 0
# Assume the same variables defined as in the original question
os.chdir(class_path) # get into the class files directory first
for i, tc in enumerate(tests):
with open(test_path+temp_file, 'w') as f:
# run the program
command = 'java -cp {cp} package.MyProgram {tc_p}'
.format(cp=jar_path,
tc_p=test_path + tc)
# runs the command and redirects it into the file f
# stores the instance of CompletedProcess
out = subprocess.run(command.split(), stdout=f)
# you can access useful info now
assert out.returncode == good_code
Let us consider the following Python code, to be executed by cpython on a Linux system (warning: it will try to create or overwrite files in /tmp/first, /tmp/second and /tmp/third).
#!/usr/bin/env python2
# -*- coding: utf-8 -*-
import subprocess
import os
import sys
import threading
class ThreadizedPopen(threading.Thread):
def __init__(self, command, stdin_name, stdout_name):
super(ThreadizedPopen, self).__init__()
self.command = command
self.stdin_name = stdin_name
self.stdout_name = stdout_name
self.returncode = None
def run(self):
with open(self.stdin_name, 'rb') as fin:
with open(self.stdout_name, 'wb') as fout:
popen = subprocess.Popen(self.command, stdin=fin, stdout=fout, stderr=None)
popen.communicate()
self.returncode = popen.returncode
def main():
os.system('mkfifo /tmp/first')
os.system('mkfifo /tmp/second')
os.system('mkfifo /tmp/third')
popen1 = ThreadizedPopen(['cat'], '/tmp/first', '/tmp/second')
popen2 = ThreadizedPopen(['cat'], '/tmp/second', '/tmp/third')
popen1.start()
popen2.start()
with open('/tmp/third') as fin:
print fin.read()
popen1.join()
popen2.join()
if __name__ == '__main__':
main()
I execute it then, on another shell, I write something in /tmp/first (say with echo test > /tmp/first). I would expect the Python program to quickly exit and print the same thing I fed to the first FIFO.
In theory it should happen that the string I wrote in /tmp/first gets copied over by the two cat processes spawned by my program to the other two FIFOs and then picked up by the main Python program to be wrote on its stdout. As soon as every cat process finished, it should close its end of the writing FIFO, making the corresponding reading end return EOF and triggering the termination of the following cat process. Looking at the program with strace reveals that the test string is copied correctly through all the three FIFOs and is read by the main Python program. The first FIFO is also correctly closed (and the first cat process exits, together with its manager Python thread). However the second cat process is stuck in a read() call, expecting data from its reading FIFO.
I do not understand why this happens. From the pipe(t) man page (which, I understand, covers also this kind of FIFOs) it seems that a read on a FIFO is returned EOF as soon as the writing end (and all its duplicates) are closed. According to strace this appears to be the trace (in particular, the cat process is dead, thus all its file descriptors are closed; its managing thread has closed its descriptors as well, I can see it in the strace output).
Can you suggest me why that happens? I can post the strace output if it can be useful.
I found this question
and simply added close_fds=True to your subprocess call. Your code now reads:
#!/usr/bin/env python2
# -*- coding: utf-8 -*-
import subprocess
import os
import sys
import threading
class ThreadizedPopen(threading.Thread):
def __init__(self, command, stdin_name, stdout_name):
super(ThreadizedPopen, self).__init__()
self.command = command
self.stdin_name = stdin_name
self.stdout_name = stdout_name
self.returncode = None
def run(self):
with open(self.stdin_name, 'rb') as fin:
with open(self.stdout_name, 'wb') as fout:
popen = subprocess.Popen(self.command, stdin=fin, stdout=fout, stderr=None, close_fds=True)
popen.communicate()
self.returncode = popen.returncode
def main():
os.system('mkfifo /tmp/first')
os.system('mkfifo /tmp/second')
os.system('mkfifo /tmp/third')
popen1 = ThreadizedPopen(['cat'], '/tmp/first', '/tmp/second')
popen2 = ThreadizedPopen(['cat'], '/tmp/second', '/tmp/third')
popen1.start()
popen2.start()
with open('/tmp/third') as fin:
print fin.read()
popen1.join()
popen2.join()
if __name__ == '__main__':
main()
I placed your code in a script called fifo_issue.py and ran it in a terminal. The script was idling as you'd expect (ignore mkfifo: cannot create fifo):
$ python fifo_issue.py
mkfifo: cannot create fifo ‘/tmp/first’: File exists
mkfifo: cannot create fifo ‘/tmp/second’: File exists
mkfifo: cannot create fifo ‘/tmp/third’: File exists
Then, in a second terminal, I typed:
$ echo "I was echoed to /tmp/first!" > /tmp/first
Back to the first terminal that's still running your idling threads:
$ python fifo_issue.py
mkfifo: cannot create fifo ‘/tmp/first’: File exists
mkfifo: cannot create fifo ‘/tmp/second’: File exists
mkfifo: cannot create fifo ‘/tmp/third’: File exists
I was echoed to /tmp/first!
After which python exited correctly
I am trying to run my python module as a command, however I am always getting the error: command not found.
#!/usr/bin/env python
import sys
import re
from sys import stdin
from sys import stdout
class Grepper(object):
def __init__(self, pattern):
self.pattern = pattern
def pgreper(self):
y = (str(self.pattern))
for line in sys.stdin:
regex = re.compile(y)
x = re.search(regex, line)
if x:
sys.stdout.write(line)
if __name__ == "__main__":
print("hello")
pattern = str(sys.argv[1])
Grepper(pattern).pgreper()
else:
print("nope")
I am sure whether it has something to do with the line:
if __name__ == "__main__":
However I just can't figure it out, this is a new area for me, and it's a bit stressful.
Your script name should have a .py extension, so it should be named something like pgreper.py.
To run it, you need to do either python pgreper.py pattern_string or if it has executable permission, as explained by Gabriel, you can do ./pgreper.py pattern_string. Note that you must give the script path (unless the current directory is in your command PATH); pgreper.py pattern_string will cause bash to print the "command not found" error message.
You can't pass the pattern data to it by piping, IOW, cat input.txt | ./pgreper.py "pattern_string" won't work: the pattern has to be passed as an argument on the command line. I guess you could do ./pgreper.py "$(cat input.txt)" but it'd be better to modify the script to read from stdin if you need that functionality.
Sorry, I didn't read the body of your script properly. :embarrassed:
I now see that your pgreper() method reads data from stdin. Sorry if the paragraph above caused any confusion.
By way of apology for my previous gaffe, here's a slightly cleaner version of your script.
#! /usr/bin/env python
import sys
import re
class Grepper(object):
def __init__(self, pattern):
self.pattern = pattern
def pgreper(self):
regex = re.compile(self.pattern)
for line in sys.stdin:
if regex.search(line):
sys.stdout.write(line)
def main():
print("hello")
pattern = sys.argv[1]
Grepper(pattern).pgreper()
if __name__ == "__main__":
main()
else:
print("nope")
Make sure you have something executable here : /usr/bin/env.
When you try to run your python module as a command, it will call this as an interpreter. You may need to replace it with /usr/bin/python or /usr/bin/python3 if you don't have an env command.
Also, make sure your file is executable : chmod +x my_module.py and try to run it with ./my_module.py.
I have a script that I want to run from within Python (2.6.5) that follows the logic below:
Prompts the user for a password. It looks like ("Enter password: ") (*Note: Input does not echo to screen)
Output irrelevant information
Prompt the user for a response ("Blah Blah filename.txt blah blah (Y/N)?: ")
The last prompt line contains text which I need to parse (filename.txt). The response provided doesn't matter (the program could actually exit here without providing one, as long as I can parse the line).
My requirements are somewhat similar to Wrapping an interactive command line application in a Python script, but the responses there seem a bit confusing, and mine still hangs even when the OP mentions that it doesn't for him.
Through looking around, I've come to the conclusion that subprocess is the best way of doing this, but I'm having a few issues. Here is my Popen line:
p = subprocess.Popen("cmd", shell=True, stdout=subprocess.PIPE,
stderr=subprocess.STDOUT, stdin=subprocess.PIPE)
When I call a read() or readline() on stdout, the prompt is printer to the screen and it hangs.
If I call a write("password\n") for stdin, the prompt is written to the screen and it hangs. The text in write() is not written (I don't the cursor move the a new line).
If I call p.communicate("password\n"), same behavior as write()
I was looking for a few ideas here on the best way to input to stdin and possibly how to parse the last line in the output if your feeling generous, though I could probably figure that out eventually.
If you are communicating with a program that subprocess spawns, you should check out A non-blocking read on a subprocess.PIPE in Python. I had a similar problem with my application and found using queues to be the best way to do ongoing communication with a subprocess.
As for getting values from the user, you can always use the raw_input() builtin to get responses, and for passwords, try using the getpass module to get non-echoing passwords from your user. You can then parse those responses and write them to your subprocess' stdin.
I ended up doing something akin to the following:
import sys
import subprocess
from threading import Thread
try:
from Queue import Queue, Empty
except ImportError:
from queue import Queue, Empty # Python 3.x
def enqueue_output(out, queue):
for line in iter(out.readline, b''):
queue.put(line)
out.close()
def getOutput(outQueue):
outStr = ''
try:
while True: # Adds output from the Queue until it is empty
outStr+=outQueue.get_nowait()
except Empty:
return outStr
p = subprocess.Popen("cmd", stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=False, universal_newlines=True)
outQueue = Queue()
errQueue = Queue()
outThread = Thread(target=enqueue_output, args=(p.stdout, outQueue))
errThread = Thread(target=enqueue_output, args=(p.stderr, errQueue))
outThread.daemon = True
errThread.daemon = True
outThread.start()
errThread.start()
try:
someInput = raw_input("Input: ")
except NameError:
someInput = input("Input: ")
p.stdin.write(someInput)
errors = getOutput(errQueue)
output = getOutput(outQueue)
Once you have the queues made and the threads started, you can loop through getting input from the user, getting errors and output from the process, and processing and displaying them to the user.
Using threading it might be slightly overkill for simple tasks.
Instead os.spawnvpe can be used. It will spawn script shell as a process. You will be able to communicate interactively with the script.
In this example I passed password as an argument, obviously that is not a good idea.
import os
import sys
from getpass import unix_getpass
def cmd(cmd):
cmd = cmd.split()
code = os.spawnvpe(os.P_WAIT, cmd[0], cmd, os.environ)
if code == 127:
sys.stderr.write('{0}: command not found\n'.format(cmd[0]))
return code
password = unix_getpass('Password: ')
cmd_run = './run.sh --password {0}'.format(password)
cmd(cmd_run)
pattern = raw_input('Pattern: ')
lines = []
with open('filename.txt', 'r') as fd:
for line in fd:
if pattern in line:
lines.append(line)
# manipulate lines
If you just want a user to enter a password without it being echoed to the screen just use the standard library's getpass module:
import getpass
print("You entered:", getpass.getpass())
NOTE:The prompt for this function defaults to "Password: " also this will only work on command lines where echoing can be controlled. So if it doesn't work try running it from terminal.