I know that I can create a new directory with the os module. But I was trying to create a new directory with the subprocess module as follows:
p=subprocess.Popen("mkdir extractions", shell=True)
os.chdir("extractions")
When the script executes, I notice that the directory extractions is created but the next os.chdir call fails saying the directory extractions does not exist. I know that I am missing something in terms of using subprocess that makes the next line unaware of the directory created. Please help!
You probably want to call p.wait() to wait for the mkdir to complete, before calling os.chdir. Or even better, use (stdout, stderr) = p.communicate(), and check the result.
Why dont you use os.mkdir("extractions")?
You could even use subprocess.call("mkdir extractions")
Both of those methods will work
After Popen, you have to do something like communicate()
p1 = subprocess.Popen('mkdir extractions', shell=True)
p1.communicate()
However, this is the same as just using subprocess.call("mkdir extractions", shell=True).
>>> import os
>>> import subprocess
>>> p=subprocess.Popen("mkdir extractions", shell=True)
>>> os.chdir("extractions")
This worked for me (on OSX). What OS are you running? Have you tried os.popen?
It will be faster to call os.mkdir, and—while the difference is unlikely to actually have any noticeable effect on your system—you are generating the overhead of creating a whole new process and running a whole separate program, and then (assuming you take the advice of any of the other [correct] answers) waiting for it to notify to tell you that it's finished instead of just calling a function in the kernel.
Related
This question already has answers here:
What is the best way to call a script from another script? [closed]
(16 answers)
Closed 7 years ago.
I want to run a Python script from another Python script. I want to pass variables like I would using the command line.
For example, I would run my first script that would iterate through a list of values (0,1,2,3) and pass those to the 2nd script script2.py 0 then script2.py 1, etc.
I found Stack Overflow question 1186789 which is a similar question, but ars's answer calls a function, where as I want to run the whole script, not just a function, and balpha's answer calls the script but with no arguments. I changed this to something like the below as a test:
execfile("script2.py 1")
But it is not accepting variables properly. When I print out the sys.argv in script2.py it is the original command call to first script "['C:\script1.py'].
I don't really want to change the original script (i.e. script2.py in my example) since I don't own it.
I figure there must be a way to do this; I am just confused how you do it.
Try using os.system:
os.system("script2.py 1")
execfile is different because it is designed to run a sequence of Python statements in the current execution context. That's why sys.argv didn't change for you.
This is inherently the wrong thing to do. If you are running a Python script from another Python script, you should communicate through Python instead of through the OS:
import script1
In an ideal world, you will be able to call a function inside script1 directly:
for i in range(whatever):
script1.some_function(i)
If necessary, you can hack sys.argv. There's a neat way of doing this using a context manager to ensure that you don't make any permanent changes.
import contextlib
#contextlib.contextmanager
def redirect_argv(num):
sys._argv = sys.argv[:]
sys.argv=[str(num)]
yield
sys.argv = sys._argv
with redirect_argv(1):
print(sys.argv)
I think this is preferable to passing all your data to the OS and back; that's just silly.
Ideally, the Python script you want to run will be set up with code like this near the end:
def main(arg1, arg2, etc):
# do whatever the script does
if __name__ == "__main__":
main(sys.argv[1], sys.argv[2], sys.argv[3])
In other words, if the module is called from the command line, it parses the command line options and then calls another function, main(), to do the actual work. (The actual arguments will vary, and the parsing may be more involved.)
If you want to call such a script from another Python script, however, you can simply import it and call modulename.main() directly, rather than going through the operating system.
os.system will work, but it is the roundabout (read "slow") way to do it, as you are starting a whole new Python interpreter process each time for no raisin.
I think the good practice may be something like this;
import subprocess
cmd = 'python script.py'
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, shell=True)
out, err = p.communicate()
result = out.split('\n')
for lin in result:
if not lin.startswith('#'):
print(lin)
according to documentation
The subprocess module allows you to spawn new processes, connect to their input/output/error pipes, and obtain their return codes. This module intends to replace several older modules and functions:
os.system
os.spawn*
os.popen*
popen2.*
commands.*
Use communicate() rather than .stdin.write, .stdout.read or .stderr.read to avoid deadlocks due to any of the other OS pipe buffers filling up and blocking the child process.
Read Here
SubProcess module:
http://docs.python.org/dev/library/subprocess.html#using-the-subprocess-module
import subprocess
subprocess.Popen("script2.py 1", shell=True)
With this, you can also redirect stdin, stdout, and stderr.
import subprocess
subprocess.call(" python script2.py 1", shell=True)
I'm running an application from within my code, and it rewrites files which I need to read later on in the code. There is no output the goes directly into my program. I can't get my code to wait until the subprocess has finished, it just goes ahead and reads the unchanged files.
I've tried subprocess.Popen.wait(), subprocess.call(), and subprocess.check_call(), but none of them work for my problem. Does anyone have any idea how to make this work? Thanks.
Edit: Here is the relevant part of my code:
os.chdir('C:\Users\Jeremy\Documents\FORCAST\dusty')
t = subprocess.Popen('start dusty.exe', shell=True)
t.wait()
os.chdir('C:\Users\Jeremy\Documents\FORCAST')
Do you use the return object of subprocess.Popen()?
p = subprocess.Popen(command)
p.wait()
should work.
Are you sure that the command does not end instantly?
If you execute a program with
t = subprocess.Popen(prog, Shell=True)
Python won't thrown an error, regardless whether the program exists or not. If you try to start an non-existing program with Popen and Shell=False, you will get an error. My guess would be that your program either doesn't exist in the folder or doesn't execute. Try to execute in the Python IDLE environment with Shell=False and see if you get a new window.
From what I can tell, execv overtakes the current process, and once the called executable finishes, the program terminates. I want to call execv multiple times within the same script, but because of this, that cannot be done.
Is there an alternative to execv that runs within the current process (i.e. prints to same stdout) and won't terminate my program? If so, what is it?
Yes, use subprocess.
os.execv* is not approporiate for your task, from doc:
These functions all execute a new program, replacing the current
process; they do not return. On Unix, the new executable is loaded
into the current process, and will have the same process id as the
caller.
So, as you want the external exe to print to the same output, this is what you might do:
import subprocess
output = subprocess.check_output(['your_exe', 'arg1'])
By default, check_output() only returns output written to standard output. If you want both standard output and error collected, use the stderr argument.
output = subprocess.check_output(['your_exe', 'arg1'], stderr=subprocess.STDOUT)
The subprocess module in the stdlib is the best way to create processes.
This question already has answers here:
What is the best way to call a script from another script? [closed]
(16 answers)
Closed 8 years ago.
I want to run a Python script from another Python script. I want to pass variables like I would using the command line.
For example, I would run my first script that would iterate through a list of values (0,1,2,3) and pass those to the 2nd script script2.py 0 then script2.py 1, etc.
I found Stack Overflow question 1186789 which is a similar question, but ars's answer calls a function, where as I want to run the whole script, not just a function, and balpha's answer calls the script but with no arguments. I changed this to something like the below as a test:
execfile("script2.py 1")
But it is not accepting variables properly. When I print out the sys.argv in script2.py it is the original command call to first script "['C:\script1.py'].
I don't really want to change the original script (i.e. script2.py in my example) since I don't own it.
I figure there must be a way to do this; I am just confused how you do it.
Try using os.system:
os.system("script2.py 1")
execfile is different because it is designed to run a sequence of Python statements in the current execution context. That's why sys.argv didn't change for you.
This is inherently the wrong thing to do. If you are running a Python script from another Python script, you should communicate through Python instead of through the OS:
import script1
In an ideal world, you will be able to call a function inside script1 directly:
for i in range(whatever):
script1.some_function(i)
If necessary, you can hack sys.argv. There's a neat way of doing this using a context manager to ensure that you don't make any permanent changes.
import contextlib
#contextlib.contextmanager
def redirect_argv(num):
sys._argv = sys.argv[:]
sys.argv=[str(num)]
yield
sys.argv = sys._argv
with redirect_argv(1):
print(sys.argv)
I think this is preferable to passing all your data to the OS and back; that's just silly.
Ideally, the Python script you want to run will be set up with code like this near the end:
def main(arg1, arg2, etc):
# do whatever the script does
if __name__ == "__main__":
main(sys.argv[1], sys.argv[2], sys.argv[3])
In other words, if the module is called from the command line, it parses the command line options and then calls another function, main(), to do the actual work. (The actual arguments will vary, and the parsing may be more involved.)
If you want to call such a script from another Python script, however, you can simply import it and call modulename.main() directly, rather than going through the operating system.
os.system will work, but it is the roundabout (read "slow") way to do it, as you are starting a whole new Python interpreter process each time for no raisin.
I think the good practice may be something like this;
import subprocess
cmd = 'python script.py'
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, shell=True)
out, err = p.communicate()
result = out.split('\n')
for lin in result:
if not lin.startswith('#'):
print(lin)
according to documentation
The subprocess module allows you to spawn new processes, connect to their input/output/error pipes, and obtain their return codes. This module intends to replace several older modules and functions:
os.system
os.spawn*
os.popen*
popen2.*
commands.*
Use communicate() rather than .stdin.write, .stdout.read or .stderr.read to avoid deadlocks due to any of the other OS pipe buffers filling up and blocking the child process.
Read Here
SubProcess module:
http://docs.python.org/dev/library/subprocess.html#using-the-subprocess-module
import subprocess
subprocess.Popen("script2.py 1", shell=True)
With this, you can also redirect stdin, stdout, and stderr.
import subprocess
subprocess.call(" python script2.py 1", shell=True)
I want to spawn (fork?) multiple Python scripts from my program (written in Python as well).
My problem is that I want to dedicate one terminal to each script, because I'll gather their output using pexpect.
I've tried using pexpect, os.execlp, and os.forkpty but neither of them do as I expect.
I want to spawn the child processes and forget about them (they will process some data, write the output to the terminal which I could read with pexpect and then exit).
Is there any library/best practice/etc. to accomplish this job?
p.s. Before you ask why I would write to STDOUT and read from it, I shall say that I don't write to STDOUT, I read the output of tshark.
See the subprocess module
The subprocess module allows you to spawn new processes, connect to their input/output/error pipes, and obtain their return codes. This module intends to replace several other, older modules and functions, such as:
os.system
os.spawn*
os.popen*
popen2.*
commands.*
From Python 3.5 onwards you can do:
import subprocess
result = subprocess.run(['python', 'my_script.py', '--arg1', val1])
if result.returncode != 0:
print('script returned error')
This also automatically redirects stdout and stderr.
I don't understand why you need expect for this. tshark should send its output to stdout, and only for some strange reason would it send it to stderr.
Therefore, what you want should be:
import subprocess
fp= subprocess.Popen( ("/usr/bin/tshark", "option1", "option2"), stdout=subprocess.PIPE).stdout
# now, whenever you are ready, read stuff from fp
You want to dedicate one terminal or one python shell?
You already have some useful answers for Popen and Subprocess, you could also use pexpect if you're already planning on using it anyways.
#for multiple python shells
import pexpect
#make your commands however you want them, this is just one method
mycommand1 = "print 'hello first python shell'"
mycommand2 = "print 'this is my second shell'"
#add a "for" statement if you want
child1 = pexpect.spawn('python')
child1.sendline(mycommand1)
child2 = pexpect.spawn('python')
child2.sendline(mycommand2)
Make as many children/shells as you want and then use the child.before() or child.after() to get your responses.
Of course you would want to add definitions or classes to be sent instead of "mycommand1", but this is just a simple example.
If you wanted to make a bunch of terminals in linux, you just need to replace the 'python' in the pextpext.spawn line
Note: I haven't tested the above code. I'm just replying from past experience with pexpect.