I want to execute a script inside a subdirectory/superdirectory (I need to be inside this sub/super-directory first). I can't get subprocess to enter my subdirectory:
tducin#localhost:~/Projekty/tests/ve$ python
Python 2.7.4 (default, Sep 26 2013, 03:20:26)
[GCC 4.7.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> import os
>>> os.getcwd()
'/home/tducin/Projekty/tests/ve'
>>> subprocess.call(['cd ..'])
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/subprocess.py", line 524, in call
return Popen(*popenargs, **kwargs).wait()
File "/usr/lib/python2.7/subprocess.py", line 711, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1308, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
Python throws OSError and I don't know why. It doesn't matter whether I try to go into an existing subdir or go one directory up (as above) - I always end up with the same error.
What your code tries to do is call a program named cd ... What you want is call a command named cd.
But cd is a shell internal. So you can only call it as
subprocess.call('cd ..', shell=True) # pointless code! See text below.
But it is pointless to do so. As no process can change another process's working directory (again, at least on a UNIX-like OS, but as well on Windows), this call will have the subshell change its dir and exit immediately.
What you want can be achieved with os.chdir() or with the subprocess named parameter cwd which changes the working directory immediately before executing a subprocess.
For example, to execute ls in the root directory, you either can do
wd = os.getcwd()
os.chdir("/")
subprocess.Popen("ls")
os.chdir(wd)
or simply
subprocess.Popen("ls", cwd="/")
To run your_command as a subprocess in a different directory, pass cwd parameter, as suggested in #wim's answer:
import subprocess
subprocess.check_call(['your_command', 'arg 1', 'arg 2'], cwd=working_dir)
A child process can't change its parent's working directory (normally). Running cd .. in a child shell process using subprocess won't change your parent Python script's working directory i.e., the code example in #glglgl's answer is wrong. cd is a shell builtin (not a separate executable), it can change the directory only in the same process.
subprocess.call and other methods in the subprocess module have a cwd parameter.
This parameter determines the working directory where you want to execute your process.
So you can do something like this:
subprocess.call('ls', shell=True, cwd='path/to/wanted/dir/')
Check out docs subprocess.popen-constructor
You want to use an absolute path to the executable, and use the cwd kwarg of Popen to set the working directory. See the docs.
If cwd is not None, the child’s current directory will be changed to
cwd before it is executed. Note that this directory is not considered
when searching the executable, so you can’t specify the program’s path
relative to cwd.
I guess these days you would do:
import subprocess
subprocess.run(["pwd"], cwd="sub-dir")
Another option based on this answer: https://stackoverflow.com/a/29269316/451710
This allows you to execute multiple commands (e.g cd) in the same process.
import subprocess
commands = '''
pwd
cd some-directory
pwd
cd another-directory
pwd
'''
process = subprocess.Popen('/bin/bash', stdin=subprocess.PIPE, stdout=subprocess.PIPE)
out, err = process.communicate(commands.encode('utf-8'))
print(out.decode('utf-8'))
just use os.chdir
Example:
>>> import os
>>> import subprocess
>>> # Lets Just Say WE want To List The User Folders
>>> os.chdir("/home/")
>>> subprocess.run("ls")
user1 user2 user3 user4
If you want to have cd functionality (assuming shell=True) and still want to change the directory in terms of the Python script, this code will allow 'cd' commands to work.
import subprocess
import os
def cd(cmd):
#cmd is expected to be something like "cd [place]"
cmd = cmd + " && pwd" # add the pwd command to run after, this will get our directory after running cd
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True) # run our new command
out = p.stdout.read()
err = p.stderr.read()
# read our output
if out != "":
print(out)
os.chdir(out[0:len(out) - 1]) # if we did get a directory, go to there while ignoring the newline
if err != "":
print(err) # if that directory doesn't exist, bash/sh/whatever env will complain for us, so we can just use that
return
If you need to change directory, run a command and get the std output as well:
import os
import logging as log
from subprocess import check_output, CalledProcessError, STDOUT
log.basicConfig(level=log.DEBUG)
def cmd_std_output(cd_dir_path, cmd):
cmd_to_list = cmd.split(" ")
try:
if cd_dir_path:
os.chdir(os.path.abspath(cd_dir_path))
output = check_output(cmd_to_list, stderr=STDOUT).decode()
return output
except CalledProcessError as e:
log.error('e: {}'.format(e))
def get_last_commit_cc_cluster():
cd_dir_path = "/repos/cc_manager/cc_cluster"
cmd = "git log --name-status HEAD^..HEAD --date=iso"
result = cmd_std_output(cd_dir_path, cmd)
return result
log.debug("Output: {}".format(get_last_commit_cc_cluster()))
Output: "commit 3b3daaaaaaaa2bb0fc4f1953af149fa3921e\nAuthor: user1<user1#email.com>\nDate: 2020-04-23 09:58:49 +0200\n\n
Related
I want to execute a script inside a subdirectory/superdirectory (I need to be inside this sub/super-directory first). I can't get subprocess to enter my subdirectory:
tducin#localhost:~/Projekty/tests/ve$ python
Python 2.7.4 (default, Sep 26 2013, 03:20:26)
[GCC 4.7.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> import os
>>> os.getcwd()
'/home/tducin/Projekty/tests/ve'
>>> subprocess.call(['cd ..'])
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/subprocess.py", line 524, in call
return Popen(*popenargs, **kwargs).wait()
File "/usr/lib/python2.7/subprocess.py", line 711, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1308, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
Python throws OSError and I don't know why. It doesn't matter whether I try to go into an existing subdir or go one directory up (as above) - I always end up with the same error.
What your code tries to do is call a program named cd ... What you want is call a command named cd.
But cd is a shell internal. So you can only call it as
subprocess.call('cd ..', shell=True) # pointless code! See text below.
But it is pointless to do so. As no process can change another process's working directory (again, at least on a UNIX-like OS, but as well on Windows), this call will have the subshell change its dir and exit immediately.
What you want can be achieved with os.chdir() or with the subprocess named parameter cwd which changes the working directory immediately before executing a subprocess.
For example, to execute ls in the root directory, you either can do
wd = os.getcwd()
os.chdir("/")
subprocess.Popen("ls")
os.chdir(wd)
or simply
subprocess.Popen("ls", cwd="/")
To run your_command as a subprocess in a different directory, pass cwd parameter, as suggested in #wim's answer:
import subprocess
subprocess.check_call(['your_command', 'arg 1', 'arg 2'], cwd=working_dir)
A child process can't change its parent's working directory (normally). Running cd .. in a child shell process using subprocess won't change your parent Python script's working directory i.e., the code example in #glglgl's answer is wrong. cd is a shell builtin (not a separate executable), it can change the directory only in the same process.
subprocess.call and other methods in the subprocess module have a cwd parameter.
This parameter determines the working directory where you want to execute your process.
So you can do something like this:
subprocess.call('ls', shell=True, cwd='path/to/wanted/dir/')
Check out docs subprocess.popen-constructor
You want to use an absolute path to the executable, and use the cwd kwarg of Popen to set the working directory. See the docs.
If cwd is not None, the child’s current directory will be changed to
cwd before it is executed. Note that this directory is not considered
when searching the executable, so you can’t specify the program’s path
relative to cwd.
I guess these days you would do:
import subprocess
subprocess.run(["pwd"], cwd="sub-dir")
Another option based on this answer: https://stackoverflow.com/a/29269316/451710
This allows you to execute multiple commands (e.g cd) in the same process.
import subprocess
commands = '''
pwd
cd some-directory
pwd
cd another-directory
pwd
'''
process = subprocess.Popen('/bin/bash', stdin=subprocess.PIPE, stdout=subprocess.PIPE)
out, err = process.communicate(commands.encode('utf-8'))
print(out.decode('utf-8'))
just use os.chdir
Example:
>>> import os
>>> import subprocess
>>> # Lets Just Say WE want To List The User Folders
>>> os.chdir("/home/")
>>> subprocess.run("ls")
user1 user2 user3 user4
If you want to have cd functionality (assuming shell=True) and still want to change the directory in terms of the Python script, this code will allow 'cd' commands to work.
import subprocess
import os
def cd(cmd):
#cmd is expected to be something like "cd [place]"
cmd = cmd + " && pwd" # add the pwd command to run after, this will get our directory after running cd
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True) # run our new command
out = p.stdout.read()
err = p.stderr.read()
# read our output
if out != "":
print(out)
os.chdir(out[0:len(out) - 1]) # if we did get a directory, go to there while ignoring the newline
if err != "":
print(err) # if that directory doesn't exist, bash/sh/whatever env will complain for us, so we can just use that
return
If you need to change directory, run a command and get the std output as well:
import os
import logging as log
from subprocess import check_output, CalledProcessError, STDOUT
log.basicConfig(level=log.DEBUG)
def cmd_std_output(cd_dir_path, cmd):
cmd_to_list = cmd.split(" ")
try:
if cd_dir_path:
os.chdir(os.path.abspath(cd_dir_path))
output = check_output(cmd_to_list, stderr=STDOUT).decode()
return output
except CalledProcessError as e:
log.error('e: {}'.format(e))
def get_last_commit_cc_cluster():
cd_dir_path = "/repos/cc_manager/cc_cluster"
cmd = "git log --name-status HEAD^..HEAD --date=iso"
result = cmd_std_output(cd_dir_path, cmd)
return result
log.debug("Output: {}".format(get_last_commit_cc_cluster()))
Output: "commit 3b3daaaaaaaa2bb0fc4f1953af149fa3921e\nAuthor: user1<user1#email.com>\nDate: 2020-04-23 09:58:49 +0200\n\n
I have a cmd file "file.cmd" containing 100s of lines of command.
Example
pandoc --extract-media -f docx -t gfm "sample1.docx" -o "sample1.md"
pandoc --extract-media -f docx -t gfm "sample2.docx" -o "sample2.md"
pandoc --extract-media -f docx -t gfm "sample3.docx" -o "sample3.md"
I am trying to run these commands using a script so that I don't have to go to a file and click on it.
This is my code, and it results in no output:
file1 = open('example.cmd', 'r')
Lines = file1.readlines()
# print(Lines)
for i in Lines:
print(i)
os.system(i)
You don't need to read the cmd file line by line. you can simply try the following:
import os
os.system('myfile.cmd')
or using the subprocess module:
import subprocess
p = subprocess.Popen(['myfile.cmd'], shell = True, close_fds = True)
stdout, stderr = proc.communicate()
Example:
myfile.cmd:
#ECHO OFF
ECHO Grettings From Python!
PAUSE
script.py:
import os
os.system('myfile.cmd')
The cmd will open with:
Greetings From Python!
Press any key to continue ...
You can debug the issue by knowing the return exit code by:
import os
return_code=os.system('myfile.cmd')
assert return_code == 0 #asserts that the return code is 0 indicating success!
Note: os.system works by calling system() in C can only take up to 65533 arguments after a command (so it is a 16 bit issue). Giving one more argument will result in the return code 32512 (which implies the exit code 127).
The subprocess module provides more powerful facilities for spawning new processes and retrieving their results; using that module is preferable to using this function (os.system('command')).
since it is a command file (cmd), and only the shell can run it, then shell argument must set to be true. since you are setting the shell argument to true, the command needs to be string form and not a list.
use the Popen method for spawn a new process and the communicte for waiting on that process (you can time it out as well). if you whish to communicate with the child process, provide the PIPES (see mu example, but you dont have to!)
the code below for python 3.3 and beyond
import subprocess
try:
proc=subprocess.Popen('myfile.cmd', shell=True, stderr=subprocess.PIPE, stdout=subprocess.PIPE)
outs, errs = proc.communicate(timeout=15) #timing out the execution, just if you want, you dont have to!
except TimeoutExpired:
proc.kill()
outs, errs = proc.communicate()
for older python versions
proc = subprocess.Popen('myfile.cmd', shell=True)
t=10
while proc.poll() is None and t >= 0:
print('Still waiting')
time.sleep(1)
t -= 1
proc.kill()
In both cases (python versions) if you dont need the timeout feature and you dont need to interact with the child process, then just, use:
proc = subprocess.Popen('myfile.cmd', shell=True)
proc.communicate()
I am trying to use bash functions inside my python script to allow me to locate a specific directory and then grep a given file inside the directory. The catch is that I only have part of the directory name, so I need to use the bash function find to get the rest of the directory name (names are unique and will only ever return one folder)
The code I have so far is as follows:
def get_tag(part_of_foldername):
import subprocess
import os
p1 = subprocess.Popen(["find", "/path/to/directory", "-maxdepth", "1", "-name", "%s.*" % part_of_foldername, "-type", "d"], stdout=subprocess.PIPE)
directory = p1.communicate()[0].strip('\n')
os.chdir(directory)
p2 = subprocess.Popen(["grep", "STUFF_", ".hgtags"], stdout=subprocess.PIPE)
tag = p2.comminucate()[0].strip('\n')
return tag
Here is what's really strange. This code works when you enter it line by line into interactive, but not when it's run thru a script. It also works when you import the script file into interactive and call the function, but not when it's called by the main function. The traceback I get from running the script straight is as follows:
Traceback (most recent call last):
File "./integration.py", line 64, in <module>
main()
File "./integration.py", line 48, in main
tag = get_tag(folder)
File "./integration.py", line 9, in get_date
os.chdir(directory)
OSError: [Errno 2] No such file or directory: ''
And it's called in the main function like this:
if block_dict[block][0]=='0':
tag = get_tag(folder)
with "folder" being previously defined as a string.
Please note we use python 2.6 so I can't use the module check_output unfortunately.
Have you tried using the glob module as opposed to find?
import glob
glob.glob("/path/to/directory/*/SomeDir/path/*")
You can look past multiple dirctories using **:
glob.glob("/path/**/SomeDir/path/*")
and that would match /path/to/your/SomeDir/path/file.
evidently p1.communicate()[0].strip('\n') is returning an empty string. are you really using the hardcoded value "/path/to/directory" as in your example?
Check the result of p1.communicate()[0]. It maybe empty string.
. in "%s.*" % part_of_foldername seems to be the cause.
UPDATE
Found typo: comminucate -> comminucate
def get_tag(part_of_foldername):
p1 = subprocess.Popen(["find", "/path/to/directory", "-maxdepth", "1", "-name", "*%s*" % part_of_foldername, "-type", "d"], stdout=subprocess.PIPE)
out, err = p1.communicate()
directory = out.split('\n')[0]
p1.wait()
if directory:
os.chdir(directory)
p2 = subprocess.Popen(["grep", "STUFF_", ".hgtags"], stdout=subprocess.PIPE)
out, err = p2.communicate()
p2.wait()
return out.rstrip('\n')
Sorry if this question is dumb. I am using python subprocess statement to call a .bat file in Ubuntu (Natty 11.04), however, I got error messages:
Traceback (most recent call last):
File "pfam_picloud.py", line 40, in <module>
a=subprocess.Popen(src2, shell=0)
File "/usr/lib/python2.7/subprocess.py", line 672, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1213, in _execute_child
raise child_exception
run this python file
$python pfam_picloud.py
Python code (pfam_picloud.py)
#!/usr/bin/python
#
met="wTest.dvf"
run="run_pfam.bat"
inp="pfam_input.PFA"
import os
import stat
import shutil
import subprocess
import string
import random
# Generate a random ID for file save
def id_generator(size=6, chars=string.ascii_uppercase + string.digits):
return ''.join(random.choice(chars) for x in range(size))
name_temp=id_generator()
cwd=os.getcwd()
src=cwd
src1=cwd+'/'+name_temp
if not os.path.exists(src1):
os.makedirs(src1)
else:
shutil.rmtree(src1)
os.makedirs(src1)
##
shutil.copy(src+"/"+run,src1)
shutil.copy(src+"/"+met,src1)
shutil.copy(cwd+"/pfam_pi.exe",src1)
shutil.copy(src+"/"+inp,src1)
#
src2=src1+"/run_pfam.bat"
os.chdir(src1)
a=subprocess.Popen(src2, shell=0)
a.wait()
bash file (run_pfam.bat)
#!/bin/sh
./pfam_pi.exe pfam_input.PFA
I can successfully run this bash file in Ubuntu. So I guess, I messed up something in my Python script. Could anyone give me some suggestions? Thanks for any inputs.
EDIT
the file pfam_pi.exe is a Linux executable. I compiled it in Ubuntu. Sorry for the confusion.
update
Well, I got different types of error now.
1. With #!/bin/sh, it said No such file or directory.
2. With /bin/sh, it said exec format error.
3. If I sent everything as arguments a=subprocess.Popen(['./pfam_pi.exe', 'inp', 'src1'], shell=0), it said end of line symbol error
Since feature requests to mark a comment as an answer remain declined, I copy the above solution here.
#Ellioh: Thanks for your comments. I found once I changed the shell=1, problem is solved. – tao.hong
Try running wine (you should have it installed) and pass pfam_pi.exe to it as a parameter. Maybe pfam_pi.exe is not a Linux executable. :-) Certainly, executable file extensions are not meaningful on Linux, but probably it really is a Windows program, otherwise I hardly can imagine it named pfam_pi.exe.
However, if it is a Linux executable, note subprocess.Popen accepts a list of args (the first element is the program itself), not a command line:
>>> import shlex, subprocess
>>> command_line = raw_input()
/bin/vikings -input eggs.txt -output "spam spam.txt" -cmd "echo '$MONEY'"
>>> args = shlex.split(command_line)
>>> print args
['/bin/vikings', '-input', 'eggs.txt', '-output', 'spam spam.txt', '-cmd', "echo '$MONEY'"]
>>> p = subprocess.Popen(args) # Success!
printbob.py:
import sys
for arg in sys.argv:
print arg
getbob.py
import subprocess
#printbob.py will always be in root of getbob.py
#a sample of sending commands to printbob.py is:
#printboby.py arg1 arg2 arg3 (commands are seperated by spaces)
print subprocess.Popen(['printbob.py', 'arg1 arg2 arg3 arg4']).wait()
x = raw_input('done')
I get:
File "C:\Python27\lib\subprocess.py", line 672, in __init__
errread, errwrite)
File "C:\Python27\lib\subprocess.py", line 882, in _execute_child
startupinfo)
WindowsError: [Error 193] %1 is not a valid Win32 application
What am I doing wrong here?
I just want to get the output of another python script inside of another python script.
Do I need to call cmd.exe or can I just run printbob.py and send commands to it?
proc = subprocess.Popen(['python', 'printbob.py', 'arg1 arg2 arg3 arg4'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
print proc.communicate()[0]
There must be a better way of doing it though, since the script is also in Python. It's better to find some way to leverage that than what you're doing.
This is the wrong approach.
You should refactor printbob.py so that it can be imported by other python modules. This version can be imported and called from the command-line:
#!/usr/bin/env python
import sys
def main(args):
for arg in args:
print(arg)
if __name__ == '__main__':
main(sys.argv)
Here it is called from the command-line:
python printbob.py one two three four five
printbob.py
one
two
three
four
five
Now we can import it in getbob.py:
#!/usr/bin/env python
import printbob
printbob.main('arg1 arg2 arg3 arg4'.split(' '))
Here it is running:
python getbob.py
arg1
arg2
arg3
arg4
The shell argument (which defaults to False) specifies whether to use
the shell as the program to execute. If shell is True, it is
recommended to pass args as a string rather than as a sequence
Just wrap all arguments in a string and give shell=True
proc = subprocess.Popen("python myScript.py --alpha=arg1 -b arg2 arg3" ,stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True)
print proc.communicate()[0]
The subprocess.run() function was added in Python 3.5.
import subprocess
cmd = subprocess.run(["ls", "-ashl"], capture_output=True)
stdout = cmd.stdout.decode() # bytes => str
refer: PEP 324 – PEP proposing the subprocess module