Develop Reference

How to write to the beginning of a file when using Python subprocess.call()?

I run the following script:
with open("logfile.txt", 'w') as log_file:
cmd = path + '/somebinary'
log_file.write("Running the command: %s\n" % cmd)
subprocess.call(cmd, shell=True, stdout=log_file, stderr=log_file)
However, the cmd variable gets written to the end of the file and not to the beginning (which I was expecting / hoping for). Can somebody explain to me why and how to prevent this, please?

The operating system performs buffering which can cause output to occur in unexpected order. To force it, flush the file handle after writing.
with open("logfile.txt", 'w') as log_file:
cmd = path + '/somebinary'
log_file.write("Running the command: %s\n" % cmd)
log_file.flush()
subprocess.call(cmd, shell=True, stdout=log_file, stderr=log_file)
Demo: https://ideone.com/U8RPBy
As an aside, you generally want to avoid shell=True; try
cmd = [path + '/somebinary']
...
subprocess.call(cmd, stdout=log_file, stderr=log_file)
and perhaps make sure your PATH is correct instead of hardcoding a specific path.

Running a C executable from Python with command line arguments

I have a C file say, myfile.c.
Now to compile I am doing : gcc myfile.c -o myfile
So now to run this I need to do : ./myfile inputFileName > outputFileName
Where inputFileName and outputFileName are 2 command line inputs.
Now I am trying to execute this within a python program and I am trying this below approach but it is not working properly may be due to the >
import subprocess
import sys
inputFileName = sys.argv[1];
outputFileName = sys.argv[2];
subprocess.run(['/home/dev/Desktop/myfile', inputFileName, outputFileName])
Where /home/dev/Desktop is the name of my directory and myfile is the name of the executable file.
What should I do?

The > that you use in your command is a shell-specific syntax for output redirection. If you want to do the same through Python, you will have to invoke the shell to do it for you, with shell=True and with a single command line (not a list).
Like this:
subprocess.run(f'/home/dev/Desktop/myfile "{inputFileName}" > "{outputFileName}"', shell=True)
If you want to do this through Python only without invoking the shell (which is what shell=True does) take a look at this other Q&A: How to redirect output with subprocess in Python?

You can open the output file in Python, and pass the file object to subprocess.run().
import subprocess
import sys
inputFileName = sys.argv[1];
outputFileName = sys.argv[2];
with open(outputFileName, "w") as out:
subprocess.run(['/home/dev/Desktop/myfile', inputFileName], stdout=out)

Subprocess.run() inside loop

I would like to loop over files using subprocess.run(), something like:
import os
import subprocess
path = os.chdir("/test")
files = []
for file in os.listdir(path):
if file.endswith(".bam"):
files.append(file)
for file in files:
process = subprocess.run("java -jar picard.jar CollectHsMetrics I=file", shell=True)
How do I correctly call the files?

shell=True is insecure if you are including user input in it. #eatmeimadanish's answer allows anybody who can write a file in /test to execute arbitrary code on your machine. This is a huge security vulnerability!
Instead, supply a list of command-line arguments to the subprocess.run call. You likely also want to pass in check=True – otherwise, your program would finish without an exception if the java commands fails!
import os
import subprocess
os.chdir("/test")
for file in os.listdir("."):
if file.endswith(".bam"):
subprocess.run(
["java", "-jar", "picard.jar", "CollectHsMetrics", "I=" + file], check=True)

Seems like you might be over complicating it.
import os
import subprocess
path = os.chdir("/test")
for file in os.listdir(path):
if file.endswith(".bam"):
subprocess.run("java -jar picard.jar CollectHsMetrics I={}".format(file), shell=True)

List only files and directory names from subprocess ls -l

In python subprocess using Popen or check_output, I need to list files and directories in a given source directory. But I can only use the command ls -l.
Sample code
cmd = ["ls", "-l", source]
proc = subprocess.Popen(cmd, stdout=subprocess.PIPE)
stdout, stderr = proc.communicate()
exitcode = proc.returncode
if exitcode != 0:
raise SystemError("Exitcode '{}', stderr: '{}', stdout: '{}' for command: '{}'".format(
exitcode, stderr, stdout, cmd))
From above proc, by using grep or any other way, can I get only a list of files and directory names inside source directory without other information?

If you insist on using subprocess please try:
[x.split(' ')[-1] for x in stdout.decode().split('\n')[1:-1]]
Obviously this is a pretty "hacky" way of doing this. Instead I can suggest the standard library glob
import glob
glob.glob(source + '/*')
returns a list of all file/directory names in source.
Edit:
cmd = ["ls", source]
proc = subprocess.Popen(cmd, stdout=subprocess.PIPE)
stdout, stderr = proc.communicate()
exitcode = proc.returncode
stdout.decode("utf-8").split('\n')[:-1]
Should also do it. -l option is not necessary here.

Parsing the output of ls is a bad idea for a few reasons. If your file name has a trailing space, then ls will display it as 'trailing space ' and if you try to open("'trailing space '") it won't work. Also file names can contain newlines.
Use pathlib instead:
from pathlib import Path
source = Path("/path/to/some/directory")
[x.name for x in source.iterdir()]
# ['a_file', 'some_other_file.txt', 'a_directory']

As Charles Duffy mentioned, you can use os. Like this.
import os
directory=#wherever you want to search
files_and_directories=os.listdir(directory)
Out: ['Directories and file names in a list']

"No such file or directory" sp.Popen module

I am trying to take a video and convert it in audio , for this I am using ffmpeg in python. I run the following command but it gives me " No such file or directory" for the input file. Here's the code-
FFMPEG_BIN = "ffmpeg"
import subprocess as sp
command = [FFMPEG_BIN, '-i', '/home/suryansh/Downloads/t.mp4']
pipe = sp.Popen(command, stdout = sp.PIPE)
After executing this code I get home/suryansh/Downloads/t.mp4: No such file or directory but the file is their in the path specified.

try add
shell=True
after the "command", and i think that you could guide for the next example
https://stackoverflow.com/a/26312166/4941927