I have a jar file that I can send data to for processing, the data is in json format.
data_path is a path to a file that has the data. Below works great.. However the data I have is not going to be a file, but in a variable. Below command does not work with a variable, it tries to read the data passed as a literal directory path to file.. Would it be a different bash command ? or something I can do with the subprocess module? Thanks!
import subprocess as sub
cmd = "java -jar %s < %s" % (jar_path, data_path)
# send data in a var
# cmd = "java -jar %s < %s" % (jar_path, data)
proc = sub.Popen(cmd, stdin=sub.PIPE, stdout=sub.PIPE, shell=True)
(out, err) = proc.communicate()
You can write it to a temporary file and pass that:
import tempfile
with tempfile.NamedTemporaryFile() as f:
f.write(data)
f.flush()
cmd = "java -jar %s < %s" % (jar_path, f.name)
...
The temp file will delete itself when the context ends.
#FedorGogolev had deleted answers going for a Popen stdin approach that weren't quite working for your specific needs. But it was a good approach so I credit him, and thought I would add the working version of what he was going for...
import tempfile
with tempfile.TemporaryFile() as f:
f.write(data)
f.flush()
f.seek(0)
cmd = "java -jar %s" % jar_path
p = subprocess.Popen(cmd, shell=True, stdin=f, stdout=subprocess.PIPE)
...
If you are passing the file object as the stdin arg you have to make sure to seek it to 0 position first.
Related
I'm trying to read the content of a file with python 3.8.5 but the output is empty, I don't understand what I'm doing wrong.
Here is the code:
import subprocess
import os
filename = "ls.out"
ls_command = "ls -la"
file = open(filename, "w")
subprocess.Popen(ls_command, stdout=file, shell=True)
file.close()
# So far, all is ok. The file "ls.out" is correctly created and filled with the output of "ls -la" command"
file = open(filename, "r")
for line in file:
print(line)
file.close()
The output of this script is empty, it doesn't print anything. I'm not able to see the content of ls.out.
What is not correct here ?
Popen creates a new process and launches it but returns immediately. So the end result is that you've forked your code and have both processes running at once. Your python code in executing faster than the start and finish of ls. Thus, you need to wait for the process to finish by adding a call to wait():
import subprocess
import os
filename = "ls.out"
ls_command = "ls -la"
file = open(filename, "w")
proc = subprocess.Popen(ls_command, stdout=file, shell=True)
proc.wait()
file.close()
file = open(filename, "r")
for line in file:
print(line)
file.close()
Popen merely starts the subprocess. Chances are the file is not yet populated when you open it.
If you want to wait for the Popen object to finish, you have to call its wait method, etc; but a much better and simpler solution is to use subprocess.check_call() or one of the other higher-level wrappers.
If the command prints to standard output, why don't you read it drectly?
import subprocess
import shlex
result = subprocess.run(
shlex.split(ls_command), # avoid shell=True
check=True, text=True, capture_output=True)
line = result.stdout
Problem
The code below is a simulation of a real terminal, in this case, a CMD terminal. The problem is that the "cls" don't clear the STDOUT of CMD. So, the string STDOUT start to stay so much extensive.
Example of problem
Microsoft Windows [versÆo 10.0.19042.746]
(c) 2020 Microsoft Corporation. Todos os direitos reservados.
C:\Users\Lsy\PycharmProjects\Others>chdir
C:\Users\Lsy\PycharmProjects\Others
C:\Users\Lsy\PycharmProjects\Others>echo test
test
C:\Users\Lsy\PycharmProjects\Others>cls
Type:
Question
How to clear the STDOUT?
Script
import subprocess
f = open('output.txt', 'w')
proc = subprocess.Popen('cmd.exe', stderr=subprocess.STDOUT, stdin=subprocess.PIPE, stdout=f, shell=True)
while True:
command = input('Type:')
command = command.encode('utf-8') + b'\n'
proc.stdin.write(command)
proc.stdin.flush()
with open('output.txt', 'r') as ff:
print(ff.read())
ff.close()
This is not how I recommend using sub processes - but I'm assuming you have some reason for doing things this way...
Given:
You've directed the CMD sub process to STDOUT to a file called "output.txt".
The CLS character is captured in the output.txt.
Your terminal then displaying the contents of the "output.txt" file (which is not ever cleared) and leaves a mess.
Therefore: If you want to "clear" your sub process terminal, then you will have to flush your "output.txt" file.
You can trivially do this by processing on the "command" variable before encoding and sending it to the sub process.
e.g:
import subprocess
import os
f = open('output.txt', 'w')
proc = subprocess.Popen('cmd.exe', stderr=subprocess.STDOUT, stdin=subprocess.PIPE, stdout=f, shell=True)
while True:
command = input('Type:')
if command == "cls":
open('output.txt', 'w').close()
os.system('cls' if os.name == 'nt' else 'clear')
else:
command = command.encode('utf-8') + b'\n'
proc.stdin.write(command)
proc.stdin.flush()
with open('output.txt', 'r+') as ff:
print(ff.read())
You could maybe also not redirect the output to a text file...
I use the following code to read a csv file
f = csv.reader(open(filename, 'rb'))
Then there's no way I can close filename, right? Is there any harm of doing so or is there a better way of reading it?
There is, use context managers:
with open(filename, 'rb') as handle:
f = csv.reader(handle)
In general an open unused file descriptor is a resource leak and should be avoided.
Interestingly in the case of files, at least the file descriptor is released, as soon as there is no reference to the file any more (see also this answer):
#!/usr/bin/env python
import gc
import os
import subprocess
# GC thresholds (http://docs.python.org/3/library/gc.html#gc.set_threshold)
print "Garbage collection thresholds: {}".format(gc.get_threshold())
if __name__ == '__main__':
pid = os.getpid()
print('------- No file descriptor ...')
subprocess.call(['lsof -p %s' % pid], shell=True)
x = open('/tmp/test', 'w')
print('------- Reference to a file ...')
subprocess.call(['lsof -p %s' % pid], shell=True)
x = 2
print('------- FD is freed automatically w/o GC')
subprocess.call(['lsof -p %s' % pid], shell=True)
I use external library, like this:
from some_lib import runThatProgram
infile = '/tmp/test'
outfile = '/tmp/testout'
runThatProgram(infile, outfile)
while runThatProgram is:
def runThatProgram(infile, outfile):
os.system("%s %s > %s" % ('thatProgram', infile, outfile))
The problem is that 'thatProgram' returns lots of stuff on STDERR, I want to redirect it to a file, but I cannot edit runThatProgram code because it is in third party lib!
To illustrate what Rosh Oxymoron said, you can hack the code like this :
from some_lib import runThatProgram
infile = '/tmp/test'
outfile = '/tmp/testout 2>&1'
runThatProgram(infile, outfile)
with this, it will call
thatProgram /tmp/test > /tmp/testout 2>&1
that will redirected stderr (2) to stdout (1), and everything will be logged in your outfile.
To elaborate on using subprocess, you can open it, give it a pipe and then work from there so
import subprocess
program = "runthatprogram.py".split()
process = subprocess.Popen(program, stdout = subprocess.PIPE, stderr = open('stderr','w')) #stderr to fileobj
process.communicate()[0] #display stdout
I am using subprocess module, which Popen class output some results like:
063.245.209.093.00080-128.192.076.180.01039:HTTP/1.1 302 Found
063.245.209.093.00080-128.192.076.180.01040:HTTP/1.1 302 Found
and here is the script I wrote:
import subprocess, shlex, fileinput,filecmp
proc = subprocess.Popen('egrep \'^HTTP/\' *', shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE,)
stdout_value = proc.communicate()[0]
print 'results:'
print stdout_value
My question is: how to convert/record the results from stdout into a file?
I appreciate all your responses and helps!
import subprocess
import glob
def egrep(pattern, *files):
""" runs egrep on the files and returns the entire result as a string """
cmd = ['egrep', pattern]
for filespec in files:
cmd.extend(glob.glob(filespec))
proc = subprocess.Popen(cmd, stdout=subprocess.PIPE)
return proc.communicate()[0]
results = egrep(r'^HTTP/', '*')
print 'results:'
print results
# write to file
with open('result_file.txt', 'w') as f:
f.write(results)
One or any of the stdin, stdout, and stderr arguments to subprocess.Popen() can be file objects (or a file descriptor), which will cause the program to read from or write to the given files.