I am new to Python (C++ fluent) and am learning on an as-need basis. I wrote a script that takes several arguments and creates and saves a matplotlib graph. It has no functions, methods, classes, etc. It is just a series of instructions that results in a graph. I would like to write another script that would execute this script with its arguments as simply as possible.
Is this possible in Python?
Take a look at the stdlib's subprocess module: http://docs.python.org/2/library/subprocess.html
from subprocess import call
call([sys.executable, 'script.py', arg1, arg2])
For a complete list of your options take a look at this similar question: Calling an external command in Python
Read the docs on link I provided above, specially if you need this call to be secure (make sure you trust or validate those params).
UPDATE:
As an alternative (and better) option would be to run this code by just importing it.
If you clean it up and put it in a function and then import and call it from your main program you dont need to execute that module as a script and, if you need to, you could still be able to run it as standalone script easily:
# script.py
def func(param1, param2, param3)
#...
if __name__=="__main__":
# get params...
func(param1, param2, param3)
# handle output etc...
# main.py
# ...
from script import func
# ...
func(param1, param2, param3)
# ...
You can use the subprocess module.
import subprocess
subprocess.Popen([sys.executable, "save_graph.py", parameter1, parameter2])
Upon execution, this code snippet will wait for the "save_graph.py" to fully complete execution before continuing.
If you want to spawn a child process that's detached from it's parent, you will have to do something like this:
DETACHED_PROCESS = 0x00000008
subprocess.Popen([sys.executable, "smtp_email.py", email, vcode], creationflags = DETACHED_PROCESS).pid
Related
This question already has answers here:
What is the best way to call a script from another script? [closed]
(16 answers)
Closed 7 years ago.
I want to run a Python script from another Python script. I want to pass variables like I would using the command line.
For example, I would run my first script that would iterate through a list of values (0,1,2,3) and pass those to the 2nd script script2.py 0 then script2.py 1, etc.
I found Stack Overflow question 1186789 which is a similar question, but ars's answer calls a function, where as I want to run the whole script, not just a function, and balpha's answer calls the script but with no arguments. I changed this to something like the below as a test:
execfile("script2.py 1")
But it is not accepting variables properly. When I print out the sys.argv in script2.py it is the original command call to first script "['C:\script1.py'].
I don't really want to change the original script (i.e. script2.py in my example) since I don't own it.
I figure there must be a way to do this; I am just confused how you do it.
Try using os.system:
os.system("script2.py 1")
execfile is different because it is designed to run a sequence of Python statements in the current execution context. That's why sys.argv didn't change for you.
This is inherently the wrong thing to do. If you are running a Python script from another Python script, you should communicate through Python instead of through the OS:
import script1
In an ideal world, you will be able to call a function inside script1 directly:
for i in range(whatever):
script1.some_function(i)
If necessary, you can hack sys.argv. There's a neat way of doing this using a context manager to ensure that you don't make any permanent changes.
import contextlib
#contextlib.contextmanager
def redirect_argv(num):
sys._argv = sys.argv[:]
sys.argv=[str(num)]
yield
sys.argv = sys._argv
with redirect_argv(1):
print(sys.argv)
I think this is preferable to passing all your data to the OS and back; that's just silly.
Ideally, the Python script you want to run will be set up with code like this near the end:
def main(arg1, arg2, etc):
# do whatever the script does
if __name__ == "__main__":
main(sys.argv[1], sys.argv[2], sys.argv[3])
In other words, if the module is called from the command line, it parses the command line options and then calls another function, main(), to do the actual work. (The actual arguments will vary, and the parsing may be more involved.)
If you want to call such a script from another Python script, however, you can simply import it and call modulename.main() directly, rather than going through the operating system.
os.system will work, but it is the roundabout (read "slow") way to do it, as you are starting a whole new Python interpreter process each time for no raisin.
I think the good practice may be something like this;
import subprocess
cmd = 'python script.py'
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, shell=True)
out, err = p.communicate()
result = out.split('\n')
for lin in result:
if not lin.startswith('#'):
print(lin)
according to documentation
The subprocess module allows you to spawn new processes, connect to their input/output/error pipes, and obtain their return codes. This module intends to replace several older modules and functions:
os.system
os.spawn*
os.popen*
popen2.*
commands.*
Use communicate() rather than .stdin.write, .stdout.read or .stderr.read to avoid deadlocks due to any of the other OS pipe buffers filling up and blocking the child process.
Read Here
SubProcess module:
http://docs.python.org/dev/library/subprocess.html#using-the-subprocess-module
import subprocess
subprocess.Popen("script2.py 1", shell=True)
With this, you can also redirect stdin, stdout, and stderr.
import subprocess
subprocess.call(" python script2.py 1", shell=True)
I have a function in a python script which should launch another python script multiple times, I am assuming this can be done like this(Script is just my imagination of how this would work.)
iterations = input("Enter the number of processes to run")
for x in range(0, iterations):
subprocess.call("python3 /path/to/the/script.py", shell=True)
but, I also need to pass over some defined variables into the other script, for example, if
x = 1
in the first script, then, I need x to have the same value in the second script without defining it there, I have NO idea how to do that.
And then also killing them, I have read about some method using PIDs, but don't those change every time?
Most of the methods I found on Google looked overly complex and what I want to do is really simple. Can anyone guide me in the right direction as to what to use and how I should go at accomplishing it?
I have a function in a python script which should launch another python script multiple times, I am assuming this can be done like this(Script is just my imagination of how this would work.)
**
Here is the subprocess manual page which contains everything I will be talking about
https://docs.python.org/2/library/subprocess.html
One of the way to call one script from other is using subprocess.Popen
something on the lines
import subprocess
for i in range(0,100):
ret = subprocess.Popen("python3 /path/to/the/script.py",stdout=subprocess.PIPE,stderr=subprocess.PIPE,shell=True)
you can use the return value from Open to make the call synchronous using the communicate method.
out,err = ret.communicate()
This would block the calling script until the subprocess finishes.
I also need to pass over some defined variables into the other script??
There are multiple ways to do this.
1. Pass parameters to the called script and parse it using OptionPraser or sys.args
in the called script have something like
from optparse import OptionParser
parser = OptionParser()
parser.add_option("-x","--variable",action="store_true",dest="xvalue",default=False)
(options,args) = parser.parse_args()
if options.xvalue == True:
###do something
in the callee script use subprocess as
ret = subprocess.Popen("python3 /path/to/the/script.py -x",stdout=subprocess.PIPE,stderr=subprocess.PIPE,shell=True)
Note the addition of -x parameter
You can use args parse
https://docs.python.org/2/library/argparse.html#module-argparse
Pass the subprocess a environment variable which can be used to configure the subprocess. This is fast but this only works one way, i.e. from parent process to child process.
in called script
import os
x = int(os.enviorn('xvalue'))
in callee script set the environment variable
import os
int x = 1
os.environ['xvalue'] = str(x)
Use sockets or pipes or some other IPC method
And then also killing them, I have read about some method using PIDs, but don't those change every time?
again you can use subprocess to hold the process id and terminate it
this will give you the process id
ret.pid
you can then use .terminate to terminate the process if it is running
ret.terminate()
to check if the process is running you can use the poll method from subprocess Popen. I would suggest you to check before you terminate the process
ret.poll()
poll will return a None if the process is running
If you just need to pass some values to second script, and you need to run that
by means of subprocess module, then you may simply pass the variables as command line arguments:
for x in range(0, iterations):
subprocess.call('python3 /path/to/second_script.py -x=%s'%x, shell=True)
And recieve the -x=1 via sys.argv list inside second_script.py (using argparse module)
On the other hand, If you need to exchange something between the two scripts dynamically (while both are running), You can use the pipe mechanism or even better, use the multiprocessing (wich requires some changes in your current code), it would make communication with and controlling it (terminating it) much cleaner.
You can pass variables to subprocesses via the command line, environment variables or passing data in on stdin. Command line is easy for simple strings that aren't too long and don't themselves have shell meta characters in them. The target script would pull them from sys.argv.
script.py:
import sys
import os
import time
x = sys.argv[1]
print(os.getpid(), "processing", x)
time.sleep(240)
subprocess.Popen starts child processes but doesn't wait for them to complete. You could start all of the children, put their popen objects in a list and finish with them later.
iterations = input("Enter the number of processes to run")
processes = []
for x in range(0, iterations):
processes.append(subprocess.Popen([sys.executable, "/path/to/the/script.py", str(x)])
time.sleep(10)
for proc in processes:
if proc.poll() is not None:
proc.terminate()
for proc in processes:
returncode = proc.wait()
I have python code that takes a bunch of tasks and distributes them to either different threads or different nodes on a cluster. I always end up writing a main script driver.py, that takes two command line arguments: --run-all and --run-task. The first is just a wrapper that iterates through all tasks and then calls driver.py --run-task with each task passed as argument. Example:
== driver.py ==
# Determine the current script
DRIVER = os.path.abspath(__file__)
(opts, args) = parser.parse_args()
if opts.run_all is not None:
# Run all tasks
for task in opts.run_all.split(","):
# Call driver.py again with a specific task
cmd = "python %s --run-task %s" %(DRIVER, task)
# Execute on system
distribute_cmd(cmd)
elif opts.run_task is not None:
# Run on an individual task
# code here for processing a task...
The user would then call:
$ driver.py --run-all task1,task2,task3,task4
And each task would get distributed.
The function distribute_cmd takes a shell executable command and sends in a system-specific way to either a node or a thread. The reason driver.py has to find its own name and call itself is because distribute_cmd needs an executable shell command; it cannot take a function name for example.
This consideration led me to this design, of a driver script having two modes and having to call itself. This has two complications: (1) the script has to find out its own path via __file__ and (2) when making this into a Python package, it's unclear where driver.py should go. It's meant to be an executable scripts, but if I put it in setup.py's scripts=, then I will have to find out where the scripts live (see correct way to find scripts directory from setup.py in Python distutils?). This does not seem to be a good solution.
What's an alternative design to this? Keep in mind that the distribution of tasks has to result in an executable command that can be passed as a string to distribute_cmd. thanks.
are you looking for is a library that already does exactly what you need, e.g. Fabric or Celery.
if you were not using nodes, I would suggest using multiprocessing.
this is a slightly similar question to this one
To be able to execute remotely, you either need:
ssh access to the box, in that case you can use Fabric to send your commands.
a server, SocketServer, tcp server, or anything that will accept connections.
an agent, or client, that will wait for data, if you are using a agent, you may as well use a broker for your messages. Celery allows you to do some of the plumbing, one end puts messages on the queue while the other end gets message from the queue. If the message is a command to execute, then the agent can do an os.system() call, or call subprocess.Popen()
celery example:
import os
from celery import Celery
celery = Celery('tasks', broker='amqp://guest#localhost//')
#celery.task
def run_command(command):
return os.system(command)
You will then need a worker that binds on the queue and waits for tasks to execute. More info in the documentation.
fabric example:
the code:
from fabric.api import run
def exec_remotely(command):
run(command)
the invocation:
$ fab exec_remotely:command='ls -lh'
More info in the documentation.
batch system case:
To go back to the question...
distribute_cmd is something that would call bsub somescript.sh
you need to find file only because you are going to re-execute the same script with other parameters
because of the above, you might have a problem providing a correct distutils script.
Let's question this design.
Why do you need to use the same script?
Can your driver write scripts then call bsub?
Can you use temporary files?
Do all the nodes actually share a filesystem?
How do you know file is going to exist on the node?
example:
TASK_CODE = {
'TASK1': '''#!/usr/bin/env python
#... actual code for task1 goes here ...
''',
'TASK2': '''#!/usr/bin/env python
#... actual code for task2 goes here ...
'''}
# driver portion
(opts, args) = parser.parse_args()
if opts.run_all is not None:
for task in opts.run_all.split(","):
task_path = '/tmp/taskfile_%s' % task
with open(task_path, 'w') as task_file:
task_file.write(TASK_CODE[task])
# note: should probably do better error handling.
distribute_cmd(task_path)
import subprocess
subprocess.Popen(['C:\Program Files\Nuke6.1v3\\Nuke6.1.exe', '-t', 'C:\Users\user\Desktop\\server.py'
I am currently using this to call server.py, which is setup in the following way
import sys
import os
import distutils
import shutil
import nuke
def imageScale(source,destination,scale):
myRead = nuke.createNode("Read")
#SET VALUES
myRead["file"].setValue(source)
myRead["selected"].setValue(True)
#CREATE REFORMAT
myReformat = nuke.createNode("Reformat")
#SET VALUES
myReformat["type"].setValue("scale")
myReformat["scale"].setValue(scale)#.75)
#SELECTION
myRead["selected"].setValue(False)
myReformat["selected"] .setValue(True)
#CREATE WRITE
myWrite = nuke.createNode("Write")
#SET VALUES
myWrite["file"].setValue(destination)
myWrite["file_type"].setValue("png")
myWrite["channels"].setValue("rgba")
myWrite["name"].setValue("temp")
nuke.execute("temp",1,1,1)
Would I be able to use subprocess.popen to pass in arguments to my imageScale function i.e. I pass in arguments source,destination,and scale directly from subprocess
Short answer is yes. You want to read arguments from sys.argv in your server.py.
However, If you know you have a python function and know you will be calling it from python, I would have a look at
import multiprocessing
import server
p = multiprocessing.Process(target=server.imageScale, args=(source,destination,scale))
p.start()
p.join()
Apart from being more esthetic, you are free to do other stuff between start and join.
EDIT: Actually, I am just assuming that Nuke will do the right thing with arguments. No guarantee that it will, so the second strategy is probably better for that reason too.
subprocess just executes a process and allows communication with the new process with arguments and pipes, it can't do anything else. If you want to achieve that you need a wrapper to communicate with the parent program and call the imageScale() function.
This question already has answers here:
What is the best way to call a script from another script? [closed]
(16 answers)
Closed 8 years ago.
I want to run a Python script from another Python script. I want to pass variables like I would using the command line.
For example, I would run my first script that would iterate through a list of values (0,1,2,3) and pass those to the 2nd script script2.py 0 then script2.py 1, etc.
I found Stack Overflow question 1186789 which is a similar question, but ars's answer calls a function, where as I want to run the whole script, not just a function, and balpha's answer calls the script but with no arguments. I changed this to something like the below as a test:
execfile("script2.py 1")
But it is not accepting variables properly. When I print out the sys.argv in script2.py it is the original command call to first script "['C:\script1.py'].
I don't really want to change the original script (i.e. script2.py in my example) since I don't own it.
I figure there must be a way to do this; I am just confused how you do it.
Try using os.system:
os.system("script2.py 1")
execfile is different because it is designed to run a sequence of Python statements in the current execution context. That's why sys.argv didn't change for you.
This is inherently the wrong thing to do. If you are running a Python script from another Python script, you should communicate through Python instead of through the OS:
import script1
In an ideal world, you will be able to call a function inside script1 directly:
for i in range(whatever):
script1.some_function(i)
If necessary, you can hack sys.argv. There's a neat way of doing this using a context manager to ensure that you don't make any permanent changes.
import contextlib
#contextlib.contextmanager
def redirect_argv(num):
sys._argv = sys.argv[:]
sys.argv=[str(num)]
yield
sys.argv = sys._argv
with redirect_argv(1):
print(sys.argv)
I think this is preferable to passing all your data to the OS and back; that's just silly.
Ideally, the Python script you want to run will be set up with code like this near the end:
def main(arg1, arg2, etc):
# do whatever the script does
if __name__ == "__main__":
main(sys.argv[1], sys.argv[2], sys.argv[3])
In other words, if the module is called from the command line, it parses the command line options and then calls another function, main(), to do the actual work. (The actual arguments will vary, and the parsing may be more involved.)
If you want to call such a script from another Python script, however, you can simply import it and call modulename.main() directly, rather than going through the operating system.
os.system will work, but it is the roundabout (read "slow") way to do it, as you are starting a whole new Python interpreter process each time for no raisin.
I think the good practice may be something like this;
import subprocess
cmd = 'python script.py'
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, shell=True)
out, err = p.communicate()
result = out.split('\n')
for lin in result:
if not lin.startswith('#'):
print(lin)
according to documentation
The subprocess module allows you to spawn new processes, connect to their input/output/error pipes, and obtain their return codes. This module intends to replace several older modules and functions:
os.system
os.spawn*
os.popen*
popen2.*
commands.*
Use communicate() rather than .stdin.write, .stdout.read or .stderr.read to avoid deadlocks due to any of the other OS pipe buffers filling up and blocking the child process.
Read Here
SubProcess module:
http://docs.python.org/dev/library/subprocess.html#using-the-subprocess-module
import subprocess
subprocess.Popen("script2.py 1", shell=True)
With this, you can also redirect stdin, stdout, and stderr.
import subprocess
subprocess.call(" python script2.py 1", shell=True)