There is a python script start_test.py.
There is a second python script siple_test.py.
# pseudo code:
start_test.py --calls--> subprocess(python.exe simple_test.py, args_simple_test[])
The python interpreter for both scripts is the same. So instead of opening a new instance, I want to run simple_test.py directly from start_test.py. I need to preserve the sys.args environment. A nice to have would be to actually enter following code section in simple_test.py:
# file: simple_test.py
if __name__ == '__main__':
some_test_function()
Most important is, that the way should be a universal one, not depending on the content of the simple_test.py.
This setup would provide two benefits:
The call is much less resource intensive
The whole stack of simple_test.py can be debugged with pycharm
So, how do I execute the call of a python script, from a python script, without starting a new subprocess?
"Executing a script" is a somewhat blurry term.
Typically the if __name__== "__main__": part does the argument (sys.argv) decoding and then calls a worker function with explicit parameters. For clarity: It should not do anything else, since this additional work can't be called without creating a new process causing all the overhead you are trying to avoid.
You simply bypass that and call this implementing routine directly.
So you end up with start_test.py containing something like:
from simple_test import worker
# ...
worker(typed_arg1, typed_arg2)
Related
I had this script working for me, before I decided I'm gonna rewrite everything and make it portable.
Without delving too much into the details, there's a central Bash script, which calls 5 other Bash scripts in their own respective folders. I have no intention of porting to Windows anytime soon, as of current this is just for Linux.
The execution path of the central Bash script is:
dos.1/1-init.sh dos.1/
dos.2/1-trace-to-file.sh dos.2/ dos.1/
dos.3/1-recognize-categories.sh dos.3/
dos.4/1-ping-in-groups.sh dos.4/ dos.3/
dos.5/init.sh dos.5/ dos.4/
I run with ./init.sh
Before the script was 'portable' I was using explicit file paths inside each respective script. All was well and good. The program itself is a combination of Bash and Python, and writes to files in one directory, so that they can be manipulated in various ways, before being read back into different parts of the program.
I understand that the fastest way to do this would be to write a monolithic Python script, using subprocess calls for the Bash side of things... However, I am doing it this way to ease maintenance, and (before I started making it 'portable') it was lightning fast.
My issue now is this: each time I have to read text into Python (either from SQL or from file) there's always this added garbage. Up until this point, I have been using sed, awk and Python's .rstrip() function to manage this... Which is all well and good, but this one damn function will not play nice... And I feel there must be a better way.
In bash I call it with:
$prog_dir=$1
$data_dir=$2
$prog_dir/2fast-ping.py $data_dir/group0.txt > $prog_dir/group0_averages.txt
$prog_dir/2fast-ping.py $data_dir/group1.txt > $prog_dir/group1_averages.txt
...
Now I know that I could write to file from within Python, but in this instance I have other reasons not to.
The issue, is that when the 2fast-ping.py script is ran, it reads the text file in with commas and a newline char. I have vigorously checked and I can confirm that the group#.txt files 100% do not contain commas. Here's the Python:
import sys
import subprocess
import select
from concurrent.futures import ThreadPoolExecutor
filename = sys.argv[1]
f = open(filename, "r")
ips = [elem.rstrip('\n') for elem in f]
print(ips)
f.close()
The script goes on to do some work on the IPs afterwards, but this is the painful part. If I call the script direct from CLI: ./2fast-ping.py ../dos.3/group0.txt, the text is processed PROPERLY and the superseding instructions actually function. But, when called from the first init script, the program basically sh*ts itself because each line is read in with commas. It works until the point where it starts to use the processed info, then:
<actual IP would be here>
ping: ('##.###.###.###',): Name or service not known
Of course, the issue is the ('',) But, Python is adding that in, and I don't know how to stop it :(
Any ideas?
Python code was okay, just passing an additional / with the argument :(
Please excuse what I know is an incredibly basic question that I have nevertheless been unable to resolve on my own.
I'm trying to switch over my data analysis from Matlab to Python, and I'm struggling with something very basic: in Matlab, I write a function in the editor, and to use that function I simply call it from the command line, or within other functions. The function that I compose in the matlab editor is given a name at the function definition line, and it's generally best for the function name to match the .m file name to avoid confusion.
I don't understand how functions differ in Python, because I have not been successful translating the same approach there.
For instance, if I write a function in the Python editor (I'm using Python 2.7 and Spyder), simply saving the .py file and calling it by its name from the Python terminal does not work. I get a "function not defined" error. However, if I execute the function within Spyder's editor (using the "run file" button), not only does the code execute properly, from that point on the function is also call-able directly from the terminal.
So...what am I doing wrong? I fully appreciate that using Python isn't going to be identical to Matlab in every way, but it seems that what I'm trying to do isn't unreasonable. I simply want to be able to write functions and call them from the python command line, without having to run each and every one through the editor first. I'm sure my mistake here must be very simple, yet doing quite a lot of reading online hasn't led me to an answer.
Thanks for any information!
If you want to use functions defined in a particular file in Python you need to "import" that file first. This is similar to running the code in that file. Matlab doesn't require you to do this because it searches for files with a matching name and automagically reads in the code for you.
For example,
myFunction.py is a file containing
def myAdd(a, b):
return a + b
In order to access this function from the Python command line or another file I would type
from myFunction import myAdd
And then during this session I can type
myAdd(1, 2)
There are a couple of ways of using import, see here.
You need to a check for __main__ to your python script
def myFunction():
pass
if __name__ == "__main__":
myFunction()
then you can run your script from terminal like this
python myscript.py
Also if your function is in another file you need to import it
from myFunctions import myFunction
myFunction()
Python doesn't have MATLAB's "one function per file" limitation. You can have as many functions as you want in a given file, and all of them can be accessed from the command line or from other functions.
Python also doesn't follow MATLAB's practice of always automatically making every function it can find usable all the time, which tends to lead to function name collisions (two functions with the same name).
Instead, Python uses the concept of a "module". A module is just a file (your .py file). That file can have zero or more functions, zero or more variables, and zero or more classes. When you want to use something from that file, you just import it.
So say you have a file 'mystuff.py':
X = 1
Y = 2
def myfunc1(a, b):
do_something
def myfunc2(c, d):
do_something
And you want to use it, you can just type import mystuff. You can then access any of the variables or functions in mystuff. To call myfunc2, you can just do mystuff.myfunc2(z, w).
What basically happens is that when you type import mystuff, it just executes the code in the file, and makes all the variables that result available from mystuff.<varname>, where <varname> is the name of the variable. Unlike in MATLAB, Python functions are treated like any other variable, so they can be accessed just like any other variable. The same is true with classes.
There are other ways to import, too, such as from mystuff import myfunc.
You run python programs by running them with
python program.py
This question already has answers here:
Why doesn't the main() function run when I start a Python script? Where does the script start running (what is its entry point)?
(5 answers)
Closed 5 months ago.
Consider:
#! /usr/bin/python
def main():
print("boo")
This code does nothing when I try to run it in Python 3.3. No error or anything.
What’s wrong?
gvim script
chmod 775 script
./script
You still have to call the function.
def main(): # declaring a function just declares it - the code doesn't run
print("boo")
main() # here we call the function
I assumed you wanted to call the print function when the script was executed from the command line.
In Python you can figure out if the script containing a piece of code is the same as the script which was launched initially by checking the __name__ variable against __main__.
#! /usr/bin/python
if __name__ == '__main__':
print("boo")
With just these lines of code:
def main():
print("boo")
you're defining a function and not actually invoking it. To invoke the function main(), you need to call it like this:
main()
You need to call that function. Update the script to:
#! /usr/bin/python
def main():
print("boo")
# Call it
main()
In Python, if you want to write a script to perform a series of small tasks sequentially, then there is absolutely no need to write a function to contain them.
Just put each on a line on its own; or use an expression delimiter like ; (not really recommended, but you can do is you so desire), likewise:
task1
task2
task3
task4
or
task1; task2; task3; (again **not** really recommended, and certainly not pythonic)
In your case your code could be turned to something like:
print('boo')
print('boo2')
print('boo3')
and it would still act as you expect it to, without the main() method, as they get evaluated sequentially.
Please note that the reason you might want to create a function for these series of tasks is:
to present a nice interface (to clients of the code),
or to encapsulate repeated logic
There might be more uses, but that's the first I can come up with, and serve to prove my point.
Now, if you feel compelled to write code that resembles the main() method in other programming languages, then please use the following Python idiom (as stated by other users so far):
if __name__ == '__main__':
doSomething()
The above is working as follows:
When you import a Python module, it gets a string (usually, the name under which it was imported) assigned as its __name__ attribute.
When you execute a script directly (by invoking the Python vm and passing it the script's name as an argument), the __name__ attribute is set to __main__
So when you use the above idiom, you can both use the script as a pluggable module by importing it at will, or just execute it directly to have the series of expressions under the if __name__ == '__main__': be evaluated directly.
Should you feel the need to dig through more information, my sources were the following:
Python documentation: Modules
Python documentation: Executing modules as scripts
Python documentation: The data model (search for __name__)
If you find the other answers confusing or intimidating, here's a parable which should hopefully help. Look at the following Python program:
a = 34
When it runs, it does something: before exiting the script, Python learns that there is a variable a and that its value is the integer 34. It doesn't do anything with this information, but it's a complete program which does something. In order for it to produce some actual value, it needs to interact with its environment, though. What does it do with this value? It could create 34 directories, or ping the 34th server in your data center, or check the strength of the passwords of the newest 34 users in your database, or whatever; or just print something.
a = 34
print(a)
The following program is in some sense very similar to the first one.
def b():
a = 34
print(a)
When you run this program, it does something: Python now knows that there is a function named b, and that it doesn't take any arguments, and that it contains some syntactically valid Python code which will be run when some other code calls it, but it doesn't actually do anything with this code yet. In order to observe any value being produced by the code in the function, you have to actually call it:
b()
(As an aside, maybe also note that the local variable a inside the function declaration b is distinct from the global variable with the same name.)
I have been giving some huge command line tool from a colleague. The main reads a bunch of arguments, parses those using the elegant import OptionParser later on and does the job.
if __name__ == '__main__':
main(sys.argv)
I can either dig into the code and copy paste loads of code, or find a way to use a "command line" call from my python script. I guess the second option is preferrable as it prevents me from randomly extracting code. Would you agree ?
You don't need to do cut and paste or launch a new Python interpreter. You should be able to import the other script.
For example, if your colleague's script is called somescript.py you could do:
import somescript
args = ['one','two']
somescript.main(args)
Sorry for the beginner question, but I can't figure out cProfile (I'm really new to Python)
I can run it via my terminal with:
python -m cProfile myscript.py
But I need to run it on a webserver, so I'd like to put the command within the script it will look at. How would I do this? I've seen stuff using terms like __init__ and __main__ but I dont really understand what those are.
I know this is simple, I'm just still trying to learn everything and I know there's someone who will know this.
Thanks in advance! I appreciate it.
I think you've been seeing ideas like this:
if __name__ == "__main__":
# do something if this script is invoked
# as python scriptname. Otherwise, gets ignored.
What happens is when you call python on a script, that file has an attribute __name__ set to "__main__" if it is the file being directly called by the python executable. Otherwise, (if it is not directly called) it is imported.
Now, you can use this trick on your scripts if you need to, for example, assuming you have:
def somescriptfunc():
# does something
pass
if __name__ == "__main__":
# do something if this script is invoked
# as python scriptname. Otherwise, gets ignored.
import cProfile
cProfile.run('somescriptfunc()')
This changes your script. When imported, its member functions, classes etc can be used as normal. When run from the command-line, it profiles itself.
Is this what you're looking for?
From the comments I've gathered more is perhaps needed, so here goes:
If you're running a script from CGI changes are it is of the form:
# do some stuff to extract the parameters
# do something with the parameters
# return the response.
When I say abstract out, you can do this:
def do_something_with_parameters(param1, param2):
pass
if __name__ = "__main__":
import cProfile
cProfile.run('do_something_with_parameters(param1=\'sometestvalue\')')
Put that file on your python path. When run itself, it will profile the function you want profiling.
Now, for your CGI script, create a script that does:
import {insert name of script from above here}
# do something to determine parameter values
# do something with them *via the function*:
do_something_with_parameters(param1=..., param2=...)
# return something
So your cgi script just becomes a little wrapper for your function (which it is anyway) and your function is now self-testing.
You can then profile the function using made up values on your desktop, away from the production server.
There are probably neater ways to achieve this, but it would work.