I am trying to use bash functions inside my python script to allow me to locate a specific directory and then grep a given file inside the directory. The catch is that I only have part of the directory name, so I need to use the bash function find to get the rest of the directory name (names are unique and will only ever return one folder)
The code I have so far is as follows:
def get_tag(part_of_foldername):
import subprocess
import os
p1 = subprocess.Popen(["find", "/path/to/directory", "-maxdepth", "1", "-name", "%s.*" % part_of_foldername, "-type", "d"], stdout=subprocess.PIPE)
directory = p1.communicate()[0].strip('\n')
os.chdir(directory)
p2 = subprocess.Popen(["grep", "STUFF_", ".hgtags"], stdout=subprocess.PIPE)
tag = p2.comminucate()[0].strip('\n')
return tag
Here is what's really strange. This code works when you enter it line by line into interactive, but not when it's run thru a script. It also works when you import the script file into interactive and call the function, but not when it's called by the main function. The traceback I get from running the script straight is as follows:
Traceback (most recent call last):
File "./integration.py", line 64, in <module>
main()
File "./integration.py", line 48, in main
tag = get_tag(folder)
File "./integration.py", line 9, in get_date
os.chdir(directory)
OSError: [Errno 2] No such file or directory: ''
And it's called in the main function like this:
if block_dict[block][0]=='0':
tag = get_tag(folder)
with "folder" being previously defined as a string.
Please note we use python 2.6 so I can't use the module check_output unfortunately.
Have you tried using the glob module as opposed to find?
import glob
glob.glob("/path/to/directory/*/SomeDir/path/*")
You can look past multiple dirctories using **:
glob.glob("/path/**/SomeDir/path/*")
and that would match /path/to/your/SomeDir/path/file.
evidently p1.communicate()[0].strip('\n') is returning an empty string. are you really using the hardcoded value "/path/to/directory" as in your example?
Check the result of p1.communicate()[0]. It maybe empty string.
. in "%s.*" % part_of_foldername seems to be the cause.
UPDATE
Found typo: comminucate -> comminucate
def get_tag(part_of_foldername):
p1 = subprocess.Popen(["find", "/path/to/directory", "-maxdepth", "1", "-name", "*%s*" % part_of_foldername, "-type", "d"], stdout=subprocess.PIPE)
out, err = p1.communicate()
directory = out.split('\n')[0]
p1.wait()
if directory:
os.chdir(directory)
p2 = subprocess.Popen(["grep", "STUFF_", ".hgtags"], stdout=subprocess.PIPE)
out, err = p2.communicate()
p2.wait()
return out.rstrip('\n')
Related
I want to execute a script inside a subdirectory/superdirectory (I need to be inside this sub/super-directory first). I can't get subprocess to enter my subdirectory:
tducin#localhost:~/Projekty/tests/ve$ python
Python 2.7.4 (default, Sep 26 2013, 03:20:26)
[GCC 4.7.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> import os
>>> os.getcwd()
'/home/tducin/Projekty/tests/ve'
>>> subprocess.call(['cd ..'])
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/subprocess.py", line 524, in call
return Popen(*popenargs, **kwargs).wait()
File "/usr/lib/python2.7/subprocess.py", line 711, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1308, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
Python throws OSError and I don't know why. It doesn't matter whether I try to go into an existing subdir or go one directory up (as above) - I always end up with the same error.
What your code tries to do is call a program named cd ... What you want is call a command named cd.
But cd is a shell internal. So you can only call it as
subprocess.call('cd ..', shell=True) # pointless code! See text below.
But it is pointless to do so. As no process can change another process's working directory (again, at least on a UNIX-like OS, but as well on Windows), this call will have the subshell change its dir and exit immediately.
What you want can be achieved with os.chdir() or with the subprocess named parameter cwd which changes the working directory immediately before executing a subprocess.
For example, to execute ls in the root directory, you either can do
wd = os.getcwd()
os.chdir("/")
subprocess.Popen("ls")
os.chdir(wd)
or simply
subprocess.Popen("ls", cwd="/")
To run your_command as a subprocess in a different directory, pass cwd parameter, as suggested in #wim's answer:
import subprocess
subprocess.check_call(['your_command', 'arg 1', 'arg 2'], cwd=working_dir)
A child process can't change its parent's working directory (normally). Running cd .. in a child shell process using subprocess won't change your parent Python script's working directory i.e., the code example in #glglgl's answer is wrong. cd is a shell builtin (not a separate executable), it can change the directory only in the same process.
subprocess.call and other methods in the subprocess module have a cwd parameter.
This parameter determines the working directory where you want to execute your process.
So you can do something like this:
subprocess.call('ls', shell=True, cwd='path/to/wanted/dir/')
Check out docs subprocess.popen-constructor
You want to use an absolute path to the executable, and use the cwd kwarg of Popen to set the working directory. See the docs.
If cwd is not None, the child’s current directory will be changed to
cwd before it is executed. Note that this directory is not considered
when searching the executable, so you can’t specify the program’s path
relative to cwd.
I guess these days you would do:
import subprocess
subprocess.run(["pwd"], cwd="sub-dir")
Another option based on this answer: https://stackoverflow.com/a/29269316/451710
This allows you to execute multiple commands (e.g cd) in the same process.
import subprocess
commands = '''
pwd
cd some-directory
pwd
cd another-directory
pwd
'''
process = subprocess.Popen('/bin/bash', stdin=subprocess.PIPE, stdout=subprocess.PIPE)
out, err = process.communicate(commands.encode('utf-8'))
print(out.decode('utf-8'))
just use os.chdir
Example:
>>> import os
>>> import subprocess
>>> # Lets Just Say WE want To List The User Folders
>>> os.chdir("/home/")
>>> subprocess.run("ls")
user1 user2 user3 user4
If you want to have cd functionality (assuming shell=True) and still want to change the directory in terms of the Python script, this code will allow 'cd' commands to work.
import subprocess
import os
def cd(cmd):
#cmd is expected to be something like "cd [place]"
cmd = cmd + " && pwd" # add the pwd command to run after, this will get our directory after running cd
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True) # run our new command
out = p.stdout.read()
err = p.stderr.read()
# read our output
if out != "":
print(out)
os.chdir(out[0:len(out) - 1]) # if we did get a directory, go to there while ignoring the newline
if err != "":
print(err) # if that directory doesn't exist, bash/sh/whatever env will complain for us, so we can just use that
return
If you need to change directory, run a command and get the std output as well:
import os
import logging as log
from subprocess import check_output, CalledProcessError, STDOUT
log.basicConfig(level=log.DEBUG)
def cmd_std_output(cd_dir_path, cmd):
cmd_to_list = cmd.split(" ")
try:
if cd_dir_path:
os.chdir(os.path.abspath(cd_dir_path))
output = check_output(cmd_to_list, stderr=STDOUT).decode()
return output
except CalledProcessError as e:
log.error('e: {}'.format(e))
def get_last_commit_cc_cluster():
cd_dir_path = "/repos/cc_manager/cc_cluster"
cmd = "git log --name-status HEAD^..HEAD --date=iso"
result = cmd_std_output(cd_dir_path, cmd)
return result
log.debug("Output: {}".format(get_last_commit_cc_cluster()))
Output: "commit 3b3daaaaaaaa2bb0fc4f1953af149fa3921e\nAuthor: user1<user1#email.com>\nDate: 2020-04-23 09:58:49 +0200\n\n
I have the following static method in Python.
import subprocess
class ProcessUtility:
#staticmethod
def execute_command(url):
process = None
process = subprocess.check_output(["phantomas " + url + " --har=test.har"])
return process
It command basically writes the output to test.har file. I have created the test.har file in the same directory as that of the script and given it read,write and execute permissions.
Upon executing I get the error.
OSError: [Errno 2] No such file or directory
Any ideas why I keep getting this.
Most subprocess functions take a list of arguments as opposed to a string, if shell=False (the default). Try
process = subprocess.check_output(['phantomas', url, '--har=test.har'])
I want to execute a script inside a subdirectory/superdirectory (I need to be inside this sub/super-directory first). I can't get subprocess to enter my subdirectory:
tducin#localhost:~/Projekty/tests/ve$ python
Python 2.7.4 (default, Sep 26 2013, 03:20:26)
[GCC 4.7.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> import os
>>> os.getcwd()
'/home/tducin/Projekty/tests/ve'
>>> subprocess.call(['cd ..'])
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/subprocess.py", line 524, in call
return Popen(*popenargs, **kwargs).wait()
File "/usr/lib/python2.7/subprocess.py", line 711, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1308, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
Python throws OSError and I don't know why. It doesn't matter whether I try to go into an existing subdir or go one directory up (as above) - I always end up with the same error.
What your code tries to do is call a program named cd ... What you want is call a command named cd.
But cd is a shell internal. So you can only call it as
subprocess.call('cd ..', shell=True) # pointless code! See text below.
But it is pointless to do so. As no process can change another process's working directory (again, at least on a UNIX-like OS, but as well on Windows), this call will have the subshell change its dir and exit immediately.
What you want can be achieved with os.chdir() or with the subprocess named parameter cwd which changes the working directory immediately before executing a subprocess.
For example, to execute ls in the root directory, you either can do
wd = os.getcwd()
os.chdir("/")
subprocess.Popen("ls")
os.chdir(wd)
or simply
subprocess.Popen("ls", cwd="/")
To run your_command as a subprocess in a different directory, pass cwd parameter, as suggested in #wim's answer:
import subprocess
subprocess.check_call(['your_command', 'arg 1', 'arg 2'], cwd=working_dir)
A child process can't change its parent's working directory (normally). Running cd .. in a child shell process using subprocess won't change your parent Python script's working directory i.e., the code example in #glglgl's answer is wrong. cd is a shell builtin (not a separate executable), it can change the directory only in the same process.
subprocess.call and other methods in the subprocess module have a cwd parameter.
This parameter determines the working directory where you want to execute your process.
So you can do something like this:
subprocess.call('ls', shell=True, cwd='path/to/wanted/dir/')
Check out docs subprocess.popen-constructor
You want to use an absolute path to the executable, and use the cwd kwarg of Popen to set the working directory. See the docs.
If cwd is not None, the child’s current directory will be changed to
cwd before it is executed. Note that this directory is not considered
when searching the executable, so you can’t specify the program’s path
relative to cwd.
I guess these days you would do:
import subprocess
subprocess.run(["pwd"], cwd="sub-dir")
Another option based on this answer: https://stackoverflow.com/a/29269316/451710
This allows you to execute multiple commands (e.g cd) in the same process.
import subprocess
commands = '''
pwd
cd some-directory
pwd
cd another-directory
pwd
'''
process = subprocess.Popen('/bin/bash', stdin=subprocess.PIPE, stdout=subprocess.PIPE)
out, err = process.communicate(commands.encode('utf-8'))
print(out.decode('utf-8'))
just use os.chdir
Example:
>>> import os
>>> import subprocess
>>> # Lets Just Say WE want To List The User Folders
>>> os.chdir("/home/")
>>> subprocess.run("ls")
user1 user2 user3 user4
If you want to have cd functionality (assuming shell=True) and still want to change the directory in terms of the Python script, this code will allow 'cd' commands to work.
import subprocess
import os
def cd(cmd):
#cmd is expected to be something like "cd [place]"
cmd = cmd + " && pwd" # add the pwd command to run after, this will get our directory after running cd
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True) # run our new command
out = p.stdout.read()
err = p.stderr.read()
# read our output
if out != "":
print(out)
os.chdir(out[0:len(out) - 1]) # if we did get a directory, go to there while ignoring the newline
if err != "":
print(err) # if that directory doesn't exist, bash/sh/whatever env will complain for us, so we can just use that
return
If you need to change directory, run a command and get the std output as well:
import os
import logging as log
from subprocess import check_output, CalledProcessError, STDOUT
log.basicConfig(level=log.DEBUG)
def cmd_std_output(cd_dir_path, cmd):
cmd_to_list = cmd.split(" ")
try:
if cd_dir_path:
os.chdir(os.path.abspath(cd_dir_path))
output = check_output(cmd_to_list, stderr=STDOUT).decode()
return output
except CalledProcessError as e:
log.error('e: {}'.format(e))
def get_last_commit_cc_cluster():
cd_dir_path = "/repos/cc_manager/cc_cluster"
cmd = "git log --name-status HEAD^..HEAD --date=iso"
result = cmd_std_output(cd_dir_path, cmd)
return result
log.debug("Output: {}".format(get_last_commit_cc_cluster()))
Output: "commit 3b3daaaaaaaa2bb0fc4f1953af149fa3921e\nAuthor: user1<user1#email.com>\nDate: 2020-04-23 09:58:49 +0200\n\n
Sorry if this question is dumb. I am using python subprocess statement to call a .bat file in Ubuntu (Natty 11.04), however, I got error messages:
Traceback (most recent call last):
File "pfam_picloud.py", line 40, in <module>
a=subprocess.Popen(src2, shell=0)
File "/usr/lib/python2.7/subprocess.py", line 672, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1213, in _execute_child
raise child_exception
run this python file
$python pfam_picloud.py
Python code (pfam_picloud.py)
#!/usr/bin/python
#
met="wTest.dvf"
run="run_pfam.bat"
inp="pfam_input.PFA"
import os
import stat
import shutil
import subprocess
import string
import random
# Generate a random ID for file save
def id_generator(size=6, chars=string.ascii_uppercase + string.digits):
return ''.join(random.choice(chars) for x in range(size))
name_temp=id_generator()
cwd=os.getcwd()
src=cwd
src1=cwd+'/'+name_temp
if not os.path.exists(src1):
os.makedirs(src1)
else:
shutil.rmtree(src1)
os.makedirs(src1)
##
shutil.copy(src+"/"+run,src1)
shutil.copy(src+"/"+met,src1)
shutil.copy(cwd+"/pfam_pi.exe",src1)
shutil.copy(src+"/"+inp,src1)
#
src2=src1+"/run_pfam.bat"
os.chdir(src1)
a=subprocess.Popen(src2, shell=0)
a.wait()
bash file (run_pfam.bat)
#!/bin/sh
./pfam_pi.exe pfam_input.PFA
I can successfully run this bash file in Ubuntu. So I guess, I messed up something in my Python script. Could anyone give me some suggestions? Thanks for any inputs.
EDIT
the file pfam_pi.exe is a Linux executable. I compiled it in Ubuntu. Sorry for the confusion.
update
Well, I got different types of error now.
1. With #!/bin/sh, it said No such file or directory.
2. With /bin/sh, it said exec format error.
3. If I sent everything as arguments a=subprocess.Popen(['./pfam_pi.exe', 'inp', 'src1'], shell=0), it said end of line symbol error
Since feature requests to mark a comment as an answer remain declined, I copy the above solution here.
#Ellioh: Thanks for your comments. I found once I changed the shell=1, problem is solved. – tao.hong
Try running wine (you should have it installed) and pass pfam_pi.exe to it as a parameter. Maybe pfam_pi.exe is not a Linux executable. :-) Certainly, executable file extensions are not meaningful on Linux, but probably it really is a Windows program, otherwise I hardly can imagine it named pfam_pi.exe.
However, if it is a Linux executable, note subprocess.Popen accepts a list of args (the first element is the program itself), not a command line:
>>> import shlex, subprocess
>>> command_line = raw_input()
/bin/vikings -input eggs.txt -output "spam spam.txt" -cmd "echo '$MONEY'"
>>> args = shlex.split(command_line)
>>> print args
['/bin/vikings', '-input', 'eggs.txt', '-output', 'spam spam.txt', '-cmd', "echo '$MONEY'"]
>>> p = subprocess.Popen(args) # Success!
I have the following code where I am trying to check for directory "Gerrits/HEAD/wlan" and then do some operations,for some reason if condition to check for the directory keeps failing even thought the directory exists?
anything wrong with the if condition#if (os.path.isdir(SCRIPT_ROOT + "/Gerrits/HEAD/wlan")): below
import os
import subprocess
from subprocess import check_call
SCRIPT_ROOT=subprocess.Popen(['pwd'], stdout=subprocess.PIPE).communicate()[0]
print SCRIPT_ROOT
def main ():
if (os.path.isdir(SCRIPT_ROOT + "/Gerrits/HEAD/wlan")):
print "SCRIPT_ROOT/Gerrits/HEAD/wlan already exists,cloning it again to the tip"
check_call("rm -rf $SCRIPT_ROOT/Gerrits/HEAD/wlan ", shell=True)
check_call("cd Gerrits/HEAD",shell=True)
else:
print "SCRIPT_ROOT/Gerrits/HEAD/wlan doesn't exist,cloning it"
os.makedirs("Gerrits/HEAD/wlan")
check_call("cd Gerrits/HEAD",shell=True)
currdir=subprocess.Popen(['pwd'], stdout=subprocess.PIPE).communicate()[0]
if __name__ == '__main__':
main()
Error:-
SCRIPT_ROOT/Gerrits/HEAD/wlan doesn't exist,cloning it
Traceback (most recent call last):
File "test.py", line 21, in <module>
main()
File "test.py", line 16, in main
os.makedirs("Gerrits/HEAD/wlan")
File "/usr/lib/python2.6/os.py", line 157, in makedirs
mkdir(name, mode)
OSError: [Errno 17] File exists: 'Gerrits/HEAD/wlan'
Add .strip() to your communicate()[0] calls, the code as is includes trailing newline in the output.
Just to be sure, your script that I just tested on a linux box with Python 2.5.
import os
import subprocess
from subprocess import check_call
SCRIPT_ROOT=subprocess.Popen(['pwd'], stdout=subprocess.PIPE).communicate()[0].strip()
print SCRIPT_ROOT
def main ():
if (os.path.isdir(SCRIPT_ROOT + "/Gerrits/HEAD/wlan")):
print "SCRIPT_ROOT/Gerrits/HEAD/wlan already exists,cloning it again to the tip"
check_call("rm -rf %s/Gerrits/HEAD/wlan" % SCRIPT_ROOT, shell=True)
check_call("cd Gerrits/HEAD",shell=True)
else:
print "SCRIPT_ROOT/Gerrits/HEAD/wlan doesn't exist,cloning it"
os.makedirs("Gerrits/HEAD/wlan")
check_call("cd Gerrits/HEAD",shell=True)
currdir=subprocess.Popen(['pwd'], stdout=subprocess.PIPE).communicate()[0].strip()
if __name__ == '__main__':
main()
And its output:
vlazarenko#xx:~$ python o.py
/media/home/vlazarenko
SCRIPT_ROOT/Gerrits/HEAD/wlan already exists,cloning it again to the tip
When I execute this line of code here:
SCRIPT_ROOT=subprocess.Popen(['pwd'], stdout=subprocess.PIPE).communicate()[0]
...the value of SCRIPT_ROOT has a trailing newline
>>> import os
>>> import subprocess
>>> ROOT = subprocess.Popen(['pwd'], stdout=subprocess.PIPE).communicate()[0]
>>> ROOT
'/Users/bgporter/personal\n'
...which makes this call
if (os.path.isdir(SCRIPT_ROOT + "/Gerrits/HEAD/wlan")):
behave differently than you'd like it to. You can either call strip() on that value, or if you always want to get the current working directory, you can do that much more easily by calling os.getcwd()
Similarly, you can use the os.removedirs() function to recursively remove the directories you don't want instead of shelling out.