Develop Reference

Tracking 7zip progress on Windows 10 with Python

I understand 7zip has some issue, where it masks its progress from code that tries to call it (not sure why).
I saw here that -bsp1 flag should show the hidden progress, but still nothing in Python:
from subprocess import Popen, PIPE
from time import sleep
cmd = Popen('7z.exe e D:\stuff.rar -od:\stuff -aoa -bsp1'.split(), stdout=PIPE, stderr=PIPE)
while cmd.poll() !=0: # Not sure this helps anything
out = cmd.stdout.read()
print(out)
sleep(1)
Running the 7z command in the command line gives me a nice percentage until unpacking is done.
In Python, I get 7z's prelude printout (Path, Type etc.) and after that just b'' until I press Ctrl-c
How does 7z know I'm calling it not from the "real" terminal? Can I somehow make it look like I am, maybe using ctypes and some windows kernel call / API?
I saw the term "pseudo terminal" mentioned in regards to this, but I'm not sure it's relevant, and if it is, Windows' ConPTY API is hidden

There is no need to use pseudo-terminal.
I am working on windows 10.
Get the output could be easy but it is hard to get the progress immediately if you use stdout.readline() directly.(Because it contains \r and it will put the cursor in
start of the line, then 7zip use space to fill them.).But readline() use \r\n as the seperator.
In my example, I use stdout.read(1) to get the output directly.
Due to the progress line is 12.So I use a number to check it.
import subprocess
s = "D:/7-Zip/7z.exe e E:/work/Compile/python/python_project/temp/test.zip -oE:/work/Compile/python/python_project/temp/test -aoa -bsp1"
p = subprocess.Popen(s.split(), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
i = 0
while True:
line = p.stdout.readline()
if line:
if i == 11:
s = b""
while True:
char = p.stdout.read(1)
if char == b"E": # "Everything is ok" means end
break
s += char
if char == b"%":
print(s.decode("gbk"))
s = b""
print(line.decode("gbk"))
i += 1
This give me:
You could improve it:
The condition of end.In my code, I used if char == b"E".I don't think it is good.Also if you remove the .decode("gbk") in each print line, you will see the file name and the number,like:
Though the char split is different from the cmd(Normally it should be x% xx - filename)So there is one line delay:

I am a bit late for this question, but I have searched for an answer like jizhihaoSAMA for the last 3 days I was searching this problem. It is a great answer and I just wanted to share a simpler one that I manage to produce.
So to print all the lines that 7zip produces this simple script is enough:
from subprocess import Popen, PIPE
cmd = [r'C:\Program Files\7-Zip\7z.exe', 'a', r'C:\ola\cenas.exe', "-sfx", r'C:\ola' + "\\*", '-mmt', '-mx5', "-bsp1"]
final_str = ["Files read from disk:", "Archive size", "Everything is Ok"]
i = 0
with Popen(cmd, stdout=PIPE, bufsize=1,
universal_newlines=True) as p:
for line in p.stdout:
line = line.replace("\n", "")
print(line)
But then maybe you don't want to see the blank lines that 7zip output produces. In this case you have this possible code (the first three lines are always the same):
i = 0
with Popen(cmd, stdout=PIPE, bufsize=1, universal_newlines=True) as p:
for line in p.stdout:
i = i + 1
if "+" in line or i < 15 or any(s in line for s in final_str):
if "Files read from disk:" in line:
print("\n", line)
else:
print(line, end="")
And now because you like the the original way 7zip output looks in cmd where it prints in the same line you can use this code:
i = 0
with Popen(cmd, stdout=PIPE, bufsize=1,
universal_newlines=True) as p:
for line in p.stdout:
i = i + 1
if "+" in line or i < 14 or any(s in line for s in final_str):
if "+" in line:
line = line.replace("\n", "")
print("\r", line, end="")
elif final_str[0] in line:
print("\n\n", line, end="")
else:
print(line, end="")
The code probably can be improved but I am also no expert in python . This is my first answer so if anything is wrong just let me know :)

Running vulture from a python script

I'm trying to find a way to run vulture (which finds unused code in python projects) inside a python script.
vulture documentation can be found here:
https://pypi.org/project/vulture/
Does anyone know how to do it?
The only way I know to use vulture is by shell commands.
I tried to tun the shell commands from the script, using module subprocess, something like this:
process = subprocess.run(['vulture', '.'], check=True,
stdout=subprocess.PIPE, stderr=subprocess.STDOUT,universal_newlines=True)
which I though would have the same effect as running the shell command "vulture ."
but it doesn't work.
Can anyone help?
Thanks

Vulture dev here.
The Vulture package exposes an API, called scavenge - which it uses internally for running the analysis after parsing command line arguments (here in vulture.main).
It takes in a list of Python files/directories. For each directory, Vulture analyzes all contained *.py files.
To analyze the current directory:
import vulture
v = vulture.Vulture()
v.scavenge(['.'])
If you just want to print the results to stdout, you can call:
v.report()
However, it's also possible to perform custom analysis/filters over Vulture's results. The method vulture.get_unused_code returns a list of vulture.Item objects - which hold the name, type and location of unused code.
For the sake of this answer, I'm just gonna print the name of all unused objects:
for item in v.get_unused_code():
print(item.name)
For more info, see - https://github.com/jendrikseipp/vulture

I see you want to capture the output shown at console:
Below code might help:
import tempfile
import subprocess
def run_command(args):
with tempfile.TemporaryFile() as t:
try:
out = subprocess.check_output(args,shell=True, stderr=t)
t.seek(0)
console_output = '--- Provided Command: --- ' + '\n' + args + '\n' + t.read() + out + '\n'
return_code = 0
except subprocess.CalledProcessError as e:
t.seek(0)
console_output = '--- Provided Command: --- ' + '\n' + args + '\n' + t.read() + e.output + '\n'
return_code = e.returncode
return return_code, console_output
Your expected output will be displayed in console_output
Link:
https://docs.python.org/3/library/subprocess.html

How to intercept output to windows 10 cmd.exe and modify to add color?

I'm invoking another program from the command line to create visual studio solutions and build them. This program outputs the results of those commands.
I want to print warning lines that are output in yellow text rather than the default grey and error lines in red.
Let's assume that my cmd.exe console has already been modified to support rendering ascii2 escape codes to color output.
I've done quite a bit of searching for solutions, but most of the things I've found are made for linux/osx. I did find a script that given regex as input, could replace text using the specified rules.
regex script
Is it possible for me to run this script in the background, but still connected to the cmd.exe, such that it will run on all the text that is output to the cmd.exe, to run the regex search and replace before the text is displayed in the cmd.exe window? I could put this into a batch file or python script.
I wanted to lay out the specific application, but to make this question potentially more generic, how do I apply an existing script/program to a running cmd.exe prompt in the background, such that the user can still run commands in the cmd prompt, but have the background program apply to the commands run by the user?
I'm open to trying powershell if there are no other performant viable solutions that exist.
The regular expression to detect if a line is an error just searches for the word error
"\berror\b"
It's the same search for a warning
"\bwarning\b"

Edit: Adding the better solution first. This solution sets up a Pipe so it can receive the output from the external program, then prints the colorized result in realtime.
#Python 2
from subprocess import Popen, PIPE
def invoke(command):
process = Popen(command, stdout=PIPE, bufsize=1)
with process.stdout:
#b'' is byte. Per various other SO posts, we use this method to
#iterate to workaround bugs in Python 2
for line in iter(process.stdout.readline, b''):
line = line.rstrip()
if not line:
continue
line = line.decode()
if "error" in line:
print (bcolors.FAIL + line + bcolors.ENDC)
elif "warning" in line:
print (bcolors.WARNING + line + bcolors.ENDC)
else:
print (line)
error_code = process.wait()
return error_code
To accomplish this, I pipped the output of the build command to a file. I then wrote this python script to install a required dependency, loop through the file contents, then print the data with appropriate coloring.
I will now look into a solution which colors the output in real time, as this solution requires the user to wait for the build to complete before seeing the colored output.
#Python 2
import pip
def install(package):
if hasattr(pip, 'main'):
pip.main(['install', package])
else:
pip._internal.main(['install', package])
class bcolors:
WARNING = '\033[93m'
FAIL = '\033[91m'
ENDC = '\033[0m'
def print_text():
install('colorama')
try:
import colorama
colorama.init()
except:
print ("could not import colorama")
if len(sys.argv) != 2:
print ("usage: python pretty_print \"file_name\"")
return 0
else:
file_name = sys.argv[1]
with open(sys.argv[1], "r") as readfile:
for line in readfile:
line = line.rstrip()
if not line:
continue
if "error" in line:
print (bcolors.FAIL + line + bcolors.ENDC)
elif "warning" in line:
print (bcolors.WARNING + line + bcolors.ENDC)
else:
print (line)
return 0
if __name__ == "__main__":
ret = print_text()
sys.exit(ret)

Python console not printing before subprocess is called

Here is a snippet of my code:
for directory in (projects + non_promote_projects):
os.chdir(directory)
print_color("Running " + " ".join(ant_command) + " in " + os.getcwd(), "cyan", newline=False)
print_color(" ... ", "gray", newline=False)
ant_process = subprocess.call(ant_command, stdout=subprocess.PIPE)
if ant_process != 0:
print_color("Failure!", "red")
print_color("Error running " + " ".join(ant_command) + " in " + os.getcwd(), "red")
sys.exit(1)
else:
print_color("Success!", "gold")
os.chdir(current_dir)
I am basically just running some ant commands in some directories. The problem is, my first print_color statement and the second are not printing before the subprocess is called. I get the feeling there is not enough time for it to print to the console before the subprocess starts. If that's the case how can I ensure it prints before the subprocess call to ant begins?

I'm making a guess about your print_color function, but you set newline=False. Console output is line buffered meaning you either need to send a new line or flush it. Assuming you really do want your print to end a line, change it to newline=True (or likely remove that param completely) to get the line.

You need to manually call sys.stdout.flush()
import sys
import subprocess
sys.stdout.write('\033[1;32;40m111 \033[0m')
sys.stdout.flush() #try comment this line and see the difference
subprocess.call(["sleep","10"], stdout=subprocess.PIPE)
print 222

Subprocess popen: Why do all writes happen at once in child process?

I have two scripts, one controlling the other and communicating to it via stdin. The parent script:
import subprocess
import time
p = subprocess.Popen(['python','read_from_stdin.py'], stdout=subprocess.PIPE, stdin=subprocess.PIPE, stderr=subprocess.PIPE)
for i in range(0,10):
p.stdin.write(str(i))
p.stdin.write('\r\n') # \n is not sufficient on Windows
p.stdin.flush()
print i
time.sleep(1)
p.stdin.close()
The child script (called 'read_from_stdin.py'):
import sys
import datetime
with open(datetime.datetime.now().strftime('%Y-%m-%d-%H-%M-%S') + '.txt','w') as f:
for line in sys.stdin:
f.write(datetime.datetime.now().isoformat() + ' ' + line)
In the file that's created by the child script all the inputs have the same timestamp, despite being written a second apart by the parent script, and despite the use for flush().

It is the read-ahead bug in Python 2: for line in sys.stdin: doesn't yield anything until its internal buffer is full. Use for line in iter(sys.stdin.readline, ''):, to workaround it.

EDIT: As per Karoly Horvath's comment below, it's not the case that it waits for for EOF, but there's buffering. The different child script below does work as expected.
I found this question on the subject: How do you read from stdin in Python?
A fair way down in the answers is:
The answer proposed by others:
for line in sys.stdin:
print line
is very simple and pythonic, but it must be noted that the script will
wait until EOF before starting to iterate on the lines of input.
This child script behaves as expected:
import sys
import datetime
with open(datetime.datetime.now().strftime('%Y-%m-%d-%H-%M-%S') + '.txt','w') as f:
line = sys.stdin.readline()
while line:
f.write(datetime.datetime.now().isoformat() + ' ' + line)
line = sys.stdin.readline()
f.write('Finished')