I am trying to execute this command using Python:
findSyntax = "find . -maxdepth 2 -name '.config' | cpio -updm ../test1/"
subprocess.Popen(findSyntax.split(' '))
But this command just would not work. When I execute this command, it will start listing all the files (not just .config) under the . directory beyond the maxdepth 2... which is a long list.
What am I missing here! Can someone point it out? Thanks.
NOTE: I've tried running subProcess.run as well with same results. I was able to get just the find part working using os.system() command.
EDIT: I just wanted to clarify that this command will copy the files found with the exact directory structure intact to the new location (creating subdirectories if necessary). I've tried this command on bash terminal, and it works fine. But I couldn't get it to work with Python.
EDIT2: So, the whole command works with os.system(), but I couldn't figure out how to make it work with subprocess. os.system() is supposed to be deprecated, so I would be very interested in figuring out the solution using subprocess instead.
Please look at this good answer and this also helps
But in essence, you can't use your above subprocess command with a pipe.
Lets run through the simple example of getting all py files in the current directory: (ls | grep py)
This is broken:
import subprocess
subprocess.call(['ls', '|', 'grep', 'py'])
Because subprocess does only one process at a time, and by piping you are really creating 2 processes.
The simple but limited (to platform) way is to use os.system
import os
os.system('ls | grep py')
This literally just passes a shell command to the system to execute.
However, you should do it with subprocess by defining your pipes:
# Get all files and pass the stdout to a pipe
p1 = subprocess.Popen(['ls'], stdout=subprocess.PIPE)
# then pass that pipe to another process as stdin and do part 2
output = subprocess.check_output(['grep', 'py'], stdin=p1.stdout)
print(output)
So, a copy paste for your example:
import subprocess
p1 = subprocess.Popen("find . -maxdepth 2 -name '.config'".split(), stdout=subprocess.PIPE)
output = subprocess.check_output("cpio -updm ../test1/".split(), stdin=p1.stdout)
Or with os:
os.system("find . -maxdepth 2 -name '.config' | cpio -updm ../test1/")
Related
I am aware that many similar questions have been posted here but none of them seems to work in my case. I have a few commands in my bash profile like below
export HEADAS=/Users/heasoft/x86_64-apple-darwin18.7.0
alias heainit=". $HEADAS/headas-init.sh"
. $HEADAS/headas-init.sh
export SAS_DIR=/Users/sas-Darwin-16.7.0-64/xmmsas
alias sas=". $SAS_DIR/setsas.sh"
sit='source ~/.bash_profile'
in which I created an alias to run them consecutively: alias prep1='sit; heainit; sas. This works just fine when I execute it in the command line. But I want to insert in a python script and run it from there. I am running Python (v 3.7.4). So, as suggested in here, I tried
import subprocess
command = "prep1"
process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=None, shell=True)
output = process.communicate()
print(output[0].decode())
But I get an error saying command not found. I tried to export it in bash profile but got an error stating -bash: export: prep1: not a function
I also tried the method suggested in here, but still nothing. Related to this, I couldn't even run a shell command like below in python
epatplot set=evli.FTZ plotfile="pn_filtered_pat.ps" 2>&1 | tee pn_filtered_pat.txt
Here is my Python script attempt
command = "epatplot set=evli.FTZ plotfile="pn_filtered_pat.ps" 2>&1 | tee pn_filtered_pat.txt"
process = subprocess.Popen(command.split(), stdout=subprocess.PIPE)
output, error = process.communicate()
I get SyntaxError: invalid syntax. I know where this syntax error is rising from but don't know how to fix.
I am a beginner in python so I appreciate any help/guidance.
Please see this answer: https://askubuntu.com/a/98791/1100014
The recommendation is to convert your aliases to bash functions and then export them with -f to be available in subshells.
When you call Popen, execute "bash -c <functionname>".
As for your last script attempt, you have a conflict in quotation marks. Replace the outer quotes with single quotes like this:
command = 'epatplot set=evli.FTZ plotfile="pn_filtered_pat.ps" 2>&1 | tee pn_filtered_pat.txt'
process = subprocess.Popen(command.split(), stdout=subprocess.PIPE)
output, error = process.communicate()
This is a beginner level question for anyone pro in subprocess.
In Windows, is it possible for me to send the following CMD commands using subprocesssuch that they are executed one after another in a single shell:
cd C:\Users\User\myvirtualenvs\project1
Scripts\activate.bat
Hello.py
Effectively, I am trying to load the Virtualenv without having to manually myself touch CMD prompt.
Thanks in advance :)
Just like mentioned in the Comment with &&:
from subprocess import call
call(r'cd C:\ && echo 123 && dir', shell=True)
Please notice the shell=True argument.
Edit due to comment:
Shell=True is an security issue, if you're passing raw input values to the call. See this example from the docs:
from subprocess import call
filename = input("What file would you like to display?\n")
>>> What file would you like to display?
>>> non_existent; rm -rf / #
call("cat " + filename, shell=True) # Uh-oh. This will end badly...
In initially thought you want to make a small script for personal purposes. If you want to give this code away, think about packaging your code via distutils or setuptools.
I have the following function that is used to execute system commands in Python:
def engage_command(
command = None
):
#os.system(command)
return os.popen(command).read()
I am using the os module instead of the subprocess module because I am dealing with a single environment in which I am interacting with many environment variables etc.
How can I use Bash with this type of function instead of the default sh shell?
output = subprocess.check_output(command, shell=True, executable='/bin/bash')
os.popen() is implemented in terms of subprocess module.
I am dealing with a single environment in which I am interacting with many environment variables etc.
each os.popen(cmd) call creates a new /bin/sh process, to run cmd shell command.
Perhaps, it is not obvious from the os.popen() documentation that says:
Open a pipe to or from command cmd
"open a pipe" does not communicate clearly: "start a new shell process with a redirected standard input or output" -- your could report a documentation issue.
If there is any doubt; the source confirms that each successful os.popen() call creates a new child process
the child can't modify its parent process environment (normally).
Consider:
import os
#XXX BROKEN: it won't work as you expect
print(os.popen("export VAR=value; echo ==$VAR==").read())
print(os.popen("echo ==$VAR==").read())
Output:
==value==
====
==== means that $VAR is empty in the second command because the second command runs in a different /bin/sh process from the first one.
To run several bash commands inside a single process, put them in a script or pass as a string:
output = check_output("\n".join(commands), shell=True, executable='/bin/bash')
Example:
#!/usr/bin/env python
from subprocess import check_output
output = check_output("""
export VAR=value; echo ==$VAR==
echo ==$VAR==
""", shell=True, executable='/bin/bash')
print(output.decode())
Output:
==value==
==value==
Note: $VAR is not empty here.
If you need to generate new commands dynamically (based on the output from the previous commands); it creates several issues and some of the issues could be fixed using pexpect module: code example.
I'm trying to run a python script that will open a command prompt(OSGeo4W.bat is a command prompt line). I can get it to open but now I would like to send the command prompt commands.
import subprocess
myProcess = subprocess.Popen(['C:\OSGeo4W64\OSGeo4W.bat'],shell = False) #opens command prompt
myProcess.communicate('gdal2tiles -p raster -z 0-1 new.jpg abc')
myProcess.wait()
print("my process has terminated")
I've also tried
subprocess.check_call('gdal2tiles -p raster -z 0-1 new.jpg abc', shell=False)
I keep getting errors that say "WindowsError: [Error 2] The system cannot find the file specified"
although, if I were to keep the command prompt that it opens and type in " 'gdal2tiles -p raster -z 0-1 new.jpg abc' " then it will work just as I wanted. Help would be great, thanks!
Try:
check_call('gdal2tiles -p raster -z 0-1 new.jpg abc', shell=True)
shell=True changes how the executable is searched on Windows.
Or if gdal2tiles works only in the environment created by OSGeo4W.bat:
shell = Popen(r'C:\OSGeo4W64\OSGeo4W.bat', stdin=subprocess.PIPE)
shell.communicate('gdal2tiles -p raster -z 0-1 new.jpg abc')
# you don't need shell.wait() here
Notice: r"" literal. It is necessary to avoid escaping the backslashes in the path.
For those of you that are still trying to figure this one out, this is what I found. The "stdin=subprocess.PIPE" method above works with some command line tool in OSGeo4W64 but not all. It works with gdal_translate but not pdal translate for example. Not sure why;(
My Solution:
OSGeo4Wenv = r'CALL "C:/OSGeo4W64/bin/o4w_env.bat" '
pdal_translate_String = r'c:/OSGeo4W64/bin/pdal translate c:\inputFile c:\outputFile radiusoutlier --filters.radiusoutlier.min_neighbors=2 --filters.radiusoutlier.radius=8.0 --filters.radiusoutlier.extract=true'
Cmd = str(OSGeo4Wenv)+' & '+str(pdal_translateCmd)
shell = subprocess.call(Cmd, stdout=None, shell=True)
What is going on?
1) Open shell and set up the OSGeo4W environment by calling "OSGeo4Wenv". This is normally called by the OSGeo4W.bat file. Without this, the command line programs don't know where to look for the libraries.
2) The pdal_translate command is then sent to the dos shell because in Windows, multiple commands can be separated by the "&" symbol. I use the .call method of python 2.7. It has the advantage that it waits for the end of the process. That is nice if you use the multiprocessing map.pool method to launch multiple processes at the same time.
Hope this help others!
Nicolas
I need to execute this script from my Python script.
Is it possible? The script generate some outputs with some files being written. How do I access these files? I have tried with subprocess call function but without success.
fx#fx-ubuntu:~/Documents/projects/foo$ bin/bar -c somefile.xml -d text.txt -r aString -f anotherString >output
The application "bar" also references to some libraries, it also create the file "bar.xml" besides the output. How do I get access to these files? Just by using open()?
Thank you,
Edit:
The error from Python runtime is only this line.
$ python foo.py
bin/bar: bin/bar: cannot execute binary file
For executing the external program, do this:
import subprocess
args = ("bin/bar", "-c", "somefile.xml", "-d", "text.txt", "-r", "aString", "-f", "anotherString")
#Or just:
#args = "bin/bar -c somefile.xml -d text.txt -r aString -f anotherString".split()
popen = subprocess.Popen(args, stdout=subprocess.PIPE)
popen.wait()
output = popen.stdout.read()
print output
And yes, assuming your bin/bar program wrote some other assorted files to disk, you can open them as normal with open("path/to/output/file.txt"). Note that you don't need to rely on a subshell to redirect the output to a file on disk named "output" if you don't want to. I'm showing here how to directly read the output into your python program without going to disk in between.
The simplest way is:
import os
cmd = 'bin/bar --option --otheroption'
os.system(cmd) # returns the exit status
You access the files in the usual way, by using open().
If you need to do more complicated subprocess management then the subprocess module is the way to go.
For executing a unix executable file. I did the following in my Mac OSX and it worked for me:
import os
cmd = './darknet classifier predict data/baby.jpg'
so = os.popen(cmd).read()
print so
Here print so outputs the result.