Develop Reference

Getting stdout from console

I'm trying to make a python script that would clone(ISO image) one usb stick to another using dd if=/dev/sda of=/dev/sdb
Here's my problem:
I want to create progress bar showing what is done.
I tried:
Looking at storage space at second usb stick, but this doesn't work beacause ISO image scans also unused space
By adding status=progress to dd command I can get progress in terminal but I can't figure how to access stdout from python. I tried subprocess.Popen,run(stdout = PIPE) with and without shell = True
reading process.stdout with .read(), .read(1), .readline() or communicate(). Nothing worked for me. (https://www.endpointdev.com/blog/2015/01/getting-realtime-output-using-python/)
I can see progress going on in python shell but .read() function always get stuck.
Part of code I am concerned about:
comm = 'sudo dd if=/dev/sda of=/dev/sdb'
cloning = subprocess.Popen(shlex.split(comm),stdout = PIPE,text = True)
while True:
print(cloning.stdout.read())
I want something that would work like:
while True:
progress = cloning.stdout.read()
update_bar(progress)
I'm using python 3.7 on Raspberry
Thanks for help

You were on the right track with status=progress, but it outputs to stderr, not stdout. If you do stderr = PIPE and then read from cloning.stderr instead of cloning.stdout, it will work.

Print l2ping output in real time using Python

I'm trying to integrate l2ping with Python (2.7). l2ping is a tool from blueZ package, running on linux, that performs echo request to a bluetooth device.
I want to show (or store in a variable) the output of the echo request in real time.
I have read a lot of discussion in this community, but (in my case) all the solutions show the result only at the end of the pinging process.
This is my code:
#!/usr/bin/python
import subprocess
mac_addr= 'XX:XX:XX:XX:XX:XX' //my real bluetooth mac address
process = subprocess.Popen(['unbuffer', 'l2ping', '-c', '3', mac_addr], bufsize=1, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
while True:
print "debug"
line = process.stdout.readline()
if not line:
break
else:
print line
As I previously said, my code show the output but all in one time at the end of the process.
The program goes till the "debug" line and then it stops, wait the end of l2ping and prints out all the output.
I've also try to use bufsize=0, or stdout.flush() but no output are shown in real time. Also, communicate() nor check_output() don't work for me.
I think that l2ping command is quite different from some others, in fact it does not write immediately on stdout and this can cause the specific problem.
What I'm doing wrong? How can I read in real time the output of l2ping?
EDIT: I've found a solution that works fine for me. I use the command unbuffer when I call l2ping. I've edit the code.

How to use Python subprocess to write multiple inputs into custom exe program

I am trying to open a executable that opens a HEC .dss database file. However, I can only seem to get it to read one argument after opening the exe and then it doesn't read anything else. Is there any way to force it to keep inserting commands.
This exe has some unique features to it, which include that the first command asks what DSS file you are going to read. Then you can input a command to create the output txt file that it will write to for the rest of the commands. What I've been able to do so far is to start the program and run one command into the exe (the mydss variable). However, after that first command is read, none of the other commands are used in the command prompt. I feel like I'm missing something here. Here is the code:
##Testing on how to run and use the DSSUTL program
import subprocess
from subprocess import PIPE, STDOUT
DSSUTL = "C:\Users\sduncan\Documents\HEC-DSS\HEC-DSSVue-2_0_1\FromSivaSel\DSSUTL.exe"
mydss = "C:\Users\sduncan\Documents\HEC-DSS\HEC-DSSVue-2_0_1\FromSivaSel\\forecast.dss"
firstLine = "WR.T TO=PythonTextOutput.txt"
commandLine = "WR.T B=SHAVER RESERVOIR-POOL C=FLOW-IN E=1HOUR F=10203040"
myList = [firstLine, commandLine]
ps = subprocess.Popen([DSSUTL, mydss, myList[1], myList[0]], shell=True)
I've also tried including stdin=subprocess.PIPE, but that only leads to the exe opening and it is blank (when I open it with the code above I can read it and see that the mydss variable was read correctly). When I used stdout or sterr, the program only opens and closes.
I've also tried using the code when the stdin=PIPE was turned on with:
ps.stdin.write(myList[1])
ps.stdin.write(myList[0])
ps.communicate()[0]
However, it did not read anything in the program. This program runs like a command prompt, however, it's not the typical cmd as it was made to read the DSS filetype and produce a text file with the list from searches like in the commandLine variable
It would be nice to know what I could do to fix the code so that I could input the extra commands. Any help to know how to event check if the commands were being sent or processed by this exe. Eventually, I will be adding many more commands to the exe file to print to the text file, so if there is any way to get python to write to the exe that would help.

#tdelaney, #eryksun Thank you for commenting, your comments about the pipes and delay really helped. I was able to fix the problem by using this code:
##Testing on how to run and use the DSSUTL program
import subprocess
from subprocess import PIPE, STDOUT
import time
DSSUTL = "C:\Users\sduncan\Documents\HEC-DSS\HEC-DSSVue-2_0_1\FromSivaSel\DSSUTL.exe"
mydss = "C:\Users\sduncan\Documents\HEC-DSS\HEC-DSSVue-2_0_1\FromSivaSel\\forecast.dss"
location = "WR.T TO=PythonTextOutput.txt" + " WR.T B=SHAVER RESERVOIR-POOL C=FLOW-IN E=1HOUR F=10203040" + "\n"
filecontent1 = "WR.T B=FLORENCE RESERVOIR-POOL C=FLOW-IN E=1HOUR F=10203040" + "\n"
filecontent2 = "WR.T B=HUNTINGTON LAKE-POOL C=FLOW-IN E=1HOUR F=10203040" + "\n"
filecontentList = [filecontent1, filecontent2]
myList = [DSSUTL, mydss] # commandLine, location
ps = subprocess.Popen(myList , shell=False, stdin=PIPE, stdout=PIPE, stderr=STDOUT)
time.sleep(1)
# input into stdin
ps.stdin.write(location)
time.sleep(1)
ps.stdin.write(filecontent1)
time.sleep(1)
ps.stdin.write(filecontent2)
time.sleep(1)
print ps.communicate()[0]
# End Script
By using the pipes to talk to the program and putting a time delay seemed to fix the problem and allowed me to talk to the console. Even though the console display is blank, by printing the communicate() command, it outputs what the console did and produce the text file with the wanted series.
Thanks for pushing me in the right direction!

python: getting a response from check_output before the command is finished

I am using python to run Youtube-dl, a software used to download videos from YouTube and other sites. I am using it specifically to download the audio from YouTube videos. I am able to download the audio from a YouTube video using python by using check_output() to execute the youtube-dl.exe file saved on my computer.
Below is the method I wrote to execute the youtube-dl file and download a song. It returns a huge string of text that is the output from the youtube-dl program.
from subprocess import check_output
#downloads the audio from a song to wherever the python script is
def getSong(songUrl):
return check_output('C:\Users\Ben\Desktop\youtube-dl\youtube-dl.exe --max-quality FORMAT --extract-audio ' + songUrl, shell=True)
When I run this method in python, it will wait for the command to be completed, then return a huge response containing lots of information. This includes generic info outputted at the end of the download, but it also (if run through command prompt) will give out info about the download progress every few seconds. The problem is, when running it in python, I get all of this progress information after it is finished downloading.
Is there any way to get the output from the check_output() method while it is still running? that way I would be able to make a progress bar for the download.
EDIT:
update: below is the current code that I have gotten to work as I want to except for one small exception...
import subprocess
songUrl = "http://www.youtube.com/watch?v=uO3lN-hugaQ"
process = subprocess.Popen(["C:\Users\Ben\Desktop\youtube-dl\youtube-dl.exe", "--max quality", "FORMAT", "--extract-audio", songUrl], shell=True, stdout = subprocess.PIPE)
for line in iter(process.stdout.readline, b''):
print line,
process.communicate()
When I run this new above code, it will start to print out information line by line as it is generated in the executed command, but all of the information regarding percent downloaded is all printed out at the very end. When I tried running the same command in command prompt, I discovered that the text that gives information about percent downloaded is actually changed every second or so, rather than a new line being created with updated information, as I suspected was true based on the output in python. Do you think there is any way around this, so I can get the information necessary for a progress bar?

Use Popen instead. Then you can do something like the following:
import subprocess
process = subprocess.Popen(["C:\Users\Ben\Desktop\youtube-dl\youtube-dl.exe", "--max-quality", "FORMAT", "--extract-audio", "songUrl"], shell=True, stdout = subprocess.PIPE)
while process.poll() is None:
result = process.stdout.read()
# do some calculations for percentage done
Also as a general rule of thumb, you should avoid using shell=True to prevent potential security vulnerabilities.

Printing progress bar on a console without the use of for -loop

I have a script written in python, where I have a statement:
Process.open() //some parameters
Which executes a command and puts the output on the console ,
where I do not know the time taken to execute the above statement , the above statement will be called by a function to complete a series of command execution and there is no for loop used at all like in typical examples in “progress bar in python examples”.
Now , my question is it possible to print the progress bar to show the completion of progress like 1..100% in python for this scenario?

The subprocess module allows you to attach pipes to the stdout and stderr of your spawned process. It's a bit complicated but if you know the output of that process very well, you can poll the output generated through the pipe at specific intervals and then increment a progressbar accordingly. Then again, if that process generates console output anyway, why not implement a progressbar there?
Edit to show how this could be done. args is a list with your command and its arguments.
import subprocess
sub = subprocess.Popen(args, bufsize=-1, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True)
while sub.poll() is None:
output = sub.stdout.read() # get all of current contents
if output:
# ... process output and decide how far the process has advanced
# ... advance your progressbar accordingly
else:
time.sleep(1E-01) # decide on a reasonable number of seconds to wait before polling again
if sub.returncode != 0:
err = sub.stderr.read()
# ... decide what to do
If you can modify the sub process I suggest you do the progress there.