I frequently find that I need to run a series of python scripts sequentially. The order and number of these scripts may vary, but instead of executing each script individually and sequentially, is there a preferred way of batch automating a series of Python scripts? My programs require multiple and complex file path and other argument combinations.
Refactor them into modules, then make a new script that imports the relevant functions from each module and runs those.
The easiest way to do this is to write a main function per module and run all the main function serially.
I think larsmans' option is the best by far. But if there is a reasons you cannot or should not turn them into easily imported modules, it is quite easy to just make a .bat file in Windows (or .ps1 if you have powershell) which calls each of them in turn. In Linux, the same thing can be done with bash or csh as long as you remember to make the resulting batch executable.
If you want another python script that calls them as though they were being called from a command line (in order to pass in the argument combinations without refactoring them into modules) you can readily use os.system or the more powerful (but sligtly more complicated) subprocess module.
Related
I have a Main script in Spyder that calls several functions contained in 6 different scripts (.py). I had to this way because the scripts are also used in different projects.
Currently, I have to run each script (containing several functions each) separately by hand, which is tiring, by clicking in the "green triangle" before launching the Main Script so that the functions contained in each script are stored in the working environment.
My question is: Would it be possible to automatically run each script directly from the Main Script and not running one after the another by hand?
Try
from filename import *
instead of
import filename
No .py extension in the import.
When you execute an import statement, the source file being imported is executed. So, for example, if you have thing.py and you execute import thing, all the code in thing.py will be run.
Also, as noted in a comment by Sven Krüger: you can use runpy.run_path, which I think is overall a better solution than my original suggestion.
I have a long-running python script script.py.
Will it cause any issues if I invoke that script one-after the another through terminal:
python script.py -----first invocation
python script.py -----second invocation before the first gets over.
Since python is a interpreted language, will there be any interference between these two scripts?
Or is it safer to make a copy of the script and then run it?
That depends entirely on what the script does.
In the simplest sense, the answer is no - though the two invocations run the same code, they don't inherently share any state and they can run side by side. Just like any program on your computer, ( for example, bash shell in separate terminals ), independent invocations have their own process space.
The only case your scripts might interfere with each other is if they both use shared resources. For example, if script.py created a file called /tmp/state.py then obviously the two invocations would conflict.
There is no danger from the source code; each invocation will read the file separately, allocate its own local variables, etc. However, there may be interference if the script uses any external references, such as writing to a common file.
I've got a bash script that's starting to cross that grey area where it should probably be rewritten in a scripting language. Since I'm constantly tweaking the script bash is perfect because of how concise things are. Mostly I'm manipulating output from one program to put into another program (i.e. using cat, grep, sed, tail, head). I'm familiar with Python so I'm looking for a module that can essentially perform these commands. I've found some stuff that uses subprocess to call bash commands but I don't want a wrapper for bash commands. I also realize I could simply take the time to write most of these commands in python and even chain them fairly easily if I encapsulated them all into one class. It seems like such an obvious thing though I find it hard to believe these utilities don't exist in a module that already exists.
You can use the os module in python to have a command executed exactly as it would be in a command shell.
For example, if you have a file named "junk.txt" and want to use the "head" command on it, you would write head junk.txt in your shell.
In Python, it's
import os
os.system("head junk.txt")
I have a python script which takes a while to finish its executing depending on the passed argument. So if I run them from two terminals with different arguments, do they get their own version of the code? I can't see two .pyc files being generated.
Terminal 1 runs: python prog.py 1000 > out_1000.out
Before the script running on terminal 1 terminate, i start running an another; thus terminal 2 runs: python prog.py 100 > out_100.out
Or basically my question is could they interfere with each other?
If you are writing the output to the same file in disk, then yes, it will be overwritten. However, it seems that you're actually printing to the stdout and then redirect it to a file. So that is not the case here.
Now answer to your question is simple: there is no interaction between two different executions of the same code. When you execute a program or a script OS will load the code to the memory and execute it and subsequent changes to code has nothing to do with the code that is already running. Technically a program that is running is called a process. Also when you run a code on two different terminals there will be two different processes on the OS one for each of them and there is no way for two process to interfere unless you explicitly do that (IPC or inter-process communication) which you are doing here.
So in summary you can run your code simultaneously on different terminals they will be completely independent.
Each Python interpreter process is independent. How the script reacts to itself being run multiple times depends on the exact code in use, but in general they should not interfere.
.pyc file reference http://effbot.org/pyfaq/how-do-i-create-a-pyc-file.htm
Python automatically compiles your script to compiled code, so called
byte code, before running it. When a module is imported for the first
time, or when the source is more recent than the current compiled
file, a .pyc file containing the compiled code will usually be created
in the same directory as the .py file.
If you afraid your code get overwritten due to whatever mistake, you should learn to put your code under VERSION CONTROL. Register github and use git to do that.
bigger sign ">" will send the output to the right handler. It you specify file name, it will push the output to that file name. Even in different terminal, if you run the code inside the same folder, use the ">" point to SAME file name, the file on the right of the ">" definitely get overwrite.
Program SOURCE CODE ARE NOT mutable during execution. Unless you acquire high level program hacking skill.
Each program will run inside its "execution workspace". Unless you make a code that tap into same resources(like change same file,shared reources ), otherwise there is no interference. (except if one exhaust all CPU, Memory resources, the second one will be interfere, but that is other story)
I am developing a Python package using a text editor and IPython. Each time I change any of the module code I have to restart the interpreter to test this. This is a pain since the classes I am developing rely on a context that needs to be re-established on each reload.
I am aware of the reload() function, but this appears to be frowned upon (also since it has been relegated from a built-in in Python 3.0) and moreover it rarely works since the modules almost always have multiple references.
My question is - what is the best/accepted way to develop a Python module/package so that I don't have to go through the pain of constantly re-establishing my interpreter context?
One idea I did think of was using the if __name__ == '__main__': trick to run a module directly so the code is not imported. However this leaves a bunch of contextual cruft (specific to my setup) at the bottom of my module files.
Ideas?
A different approach may be to formalise your test driven development, and instead of using the interpreter to test your module, save your tests and run them directly.
You probably know of the various ways to do this with python, I imagine the simplest way to start in this direction is to copy and paste what you do in the interpreter into the docstring as a doctest and add the following to the bottom of your module:
if __name__ == "__main__":
import doctest
doctest.testmod()
Your informal test will then be repeated every time the module is called directly. This has a number of other benefits. See the doctest docs for more info on writing doctests.
Ipython does allow reloads see the magic function %run iPython doc
or if modules under the one have changed the recursive dreloadd() function
If you have a complex context is it possible to create it in another module? or assign it to a global variable which will stay around as the interpreter is not restarted
How about using nose with nosey to run your tests automatically in a separate terminal every time you save your edits to disk? Set up all the state you need in your unit tests.
you could create a python script that sets up your context and run it with
python -i context-setup.py
-i When a script is passed as first argument or the -c option is
used, enter interactive mode after executing the script or the
command. It does not read the $PYTHONSTARTUP file. This can be
useful to inspect global variables or a stack trace when a
script raises an exception.