I have been giving some huge command line tool from a colleague. The main reads a bunch of arguments, parses those using the elegant import OptionParser later on and does the job.
if __name__ == '__main__':
main(sys.argv)
I can either dig into the code and copy paste loads of code, or find a way to use a "command line" call from my python script. I guess the second option is preferrable as it prevents me from randomly extracting code. Would you agree ?
You don't need to do cut and paste or launch a new Python interpreter. You should be able to import the other script.
For example, if your colleague's script is called somescript.py you could do:
import somescript
args = ['one','two']
somescript.main(args)
Related
I am using a notebook (in Google Colab) in python 3 and really need to execute some python2 code with some data that is generated in my notebook !
So what I did was
Generate the data in my cells AND save it to a file (data.txt)
Write a python 2 script (myScript.py) with a main() function that parses the file from sys.argv[1] into data and than calls my python2 functions and do all the stuff then it ends with return results (which is a dictionary)
In my notebook I run
!python2 myScript.py ./data.txt
(They are of course all in the same directory)
The command runs with no errors but no output ! How do I catch the results that are returned in a variable that I could use later ?
Not important but could be helpful :
Is there a better way to actually achieve what I am willing to achieve ?
Thanks to #Manuel's comment on the question, I figured out this solution and it worked :
In myScript.py, I changed return results by sys.stdout.write(json.dumps(results))
In my notebook, I changed the cell that executes the script to this:
results = !python2 test_langdetect.py ./tmp_comments.txt
myVar = json.loads(results[0])
Of course you need to import json
I had this script working for me, before I decided I'm gonna rewrite everything and make it portable.
Without delving too much into the details, there's a central Bash script, which calls 5 other Bash scripts in their own respective folders. I have no intention of porting to Windows anytime soon, as of current this is just for Linux.
The execution path of the central Bash script is:
dos.1/1-init.sh dos.1/
dos.2/1-trace-to-file.sh dos.2/ dos.1/
dos.3/1-recognize-categories.sh dos.3/
dos.4/1-ping-in-groups.sh dos.4/ dos.3/
dos.5/init.sh dos.5/ dos.4/
I run with ./init.sh
Before the script was 'portable' I was using explicit file paths inside each respective script. All was well and good. The program itself is a combination of Bash and Python, and writes to files in one directory, so that they can be manipulated in various ways, before being read back into different parts of the program.
I understand that the fastest way to do this would be to write a monolithic Python script, using subprocess calls for the Bash side of things... However, I am doing it this way to ease maintenance, and (before I started making it 'portable') it was lightning fast.
My issue now is this: each time I have to read text into Python (either from SQL or from file) there's always this added garbage. Up until this point, I have been using sed, awk and Python's .rstrip() function to manage this... Which is all well and good, but this one damn function will not play nice... And I feel there must be a better way.
In bash I call it with:
$prog_dir=$1
$data_dir=$2
$prog_dir/2fast-ping.py $data_dir/group0.txt > $prog_dir/group0_averages.txt
$prog_dir/2fast-ping.py $data_dir/group1.txt > $prog_dir/group1_averages.txt
...
Now I know that I could write to file from within Python, but in this instance I have other reasons not to.
The issue, is that when the 2fast-ping.py script is ran, it reads the text file in with commas and a newline char. I have vigorously checked and I can confirm that the group#.txt files 100% do not contain commas. Here's the Python:
import sys
import subprocess
import select
from concurrent.futures import ThreadPoolExecutor
filename = sys.argv[1]
f = open(filename, "r")
ips = [elem.rstrip('\n') for elem in f]
print(ips)
f.close()
The script goes on to do some work on the IPs afterwards, but this is the painful part. If I call the script direct from CLI: ./2fast-ping.py ../dos.3/group0.txt, the text is processed PROPERLY and the superseding instructions actually function. But, when called from the first init script, the program basically sh*ts itself because each line is read in with commas. It works until the point where it starts to use the processed info, then:
<actual IP would be here>
ping: ('##.###.###.###',): Name or service not known
Of course, the issue is the ('',) But, Python is adding that in, and I don't know how to stop it :(
Any ideas?
Python code was okay, just passing an additional / with the argument :(
There is a python script start_test.py.
There is a second python script siple_test.py.
# pseudo code:
start_test.py --calls--> subprocess(python.exe simple_test.py, args_simple_test[])
The python interpreter for both scripts is the same. So instead of opening a new instance, I want to run simple_test.py directly from start_test.py. I need to preserve the sys.args environment. A nice to have would be to actually enter following code section in simple_test.py:
# file: simple_test.py
if __name__ == '__main__':
some_test_function()
Most important is, that the way should be a universal one, not depending on the content of the simple_test.py.
This setup would provide two benefits:
The call is much less resource intensive
The whole stack of simple_test.py can be debugged with pycharm
So, how do I execute the call of a python script, from a python script, without starting a new subprocess?
"Executing a script" is a somewhat blurry term.
Typically the if __name__== "__main__": part does the argument (sys.argv) decoding and then calls a worker function with explicit parameters. For clarity: It should not do anything else, since this additional work can't be called without creating a new process causing all the overhead you are trying to avoid.
You simply bypass that and call this implementing routine directly.
So you end up with start_test.py containing something like:
from simple_test import worker
# ...
worker(typed_arg1, typed_arg2)
I have a python script (we'll say "script.py") that I want to grab values from using a separate GUI python script ("GUI.py"). When I run my GUI.py script, I want to have these text fields in the GUI sent over to script.py after clicking a button in the GUI. I am thinking that my best solution might be to have the GUI.py create another script ("lib.py") that holds all these values and it just writes to it. It then runs the "script.py" script. So all in all the GUI.py will have a function that when called will look something like this:
def on_button(self):
username = self.usernameInput.get()
file = open(“lib.py”,”w”)
file.write(“username = ” + username)
os.system("script.py")
I think this will work, but I am just wondering, does this seem like a practical solution? Let me know what you guys think.
No, I don't think this is the practical solution.
Do you consider instead making the python script you want to run into a module or package that you can call directly inside your GUI? I think that is the cleanest approach. For using your scripts as modules, see the docs or for 2.7.
Basically a module is a python file, script.py, and as long as it is in the python path (say, your current directory), you can import it:
from script import action
So you could try:
def on_button(self):
username = self.usernameInput.get()
result = action(username) # and any other args you want to pass
print(result)
That is, if the script in question uses a if __name__ == "__main__": statement (or can otherwise be run from the command line), try putting the operations in some def action(args): function and importing it into your GUI.
Sorry for the beginner question, but I can't figure out cProfile (I'm really new to Python)
I can run it via my terminal with:
python -m cProfile myscript.py
But I need to run it on a webserver, so I'd like to put the command within the script it will look at. How would I do this? I've seen stuff using terms like __init__ and __main__ but I dont really understand what those are.
I know this is simple, I'm just still trying to learn everything and I know there's someone who will know this.
Thanks in advance! I appreciate it.
I think you've been seeing ideas like this:
if __name__ == "__main__":
# do something if this script is invoked
# as python scriptname. Otherwise, gets ignored.
What happens is when you call python on a script, that file has an attribute __name__ set to "__main__" if it is the file being directly called by the python executable. Otherwise, (if it is not directly called) it is imported.
Now, you can use this trick on your scripts if you need to, for example, assuming you have:
def somescriptfunc():
# does something
pass
if __name__ == "__main__":
# do something if this script is invoked
# as python scriptname. Otherwise, gets ignored.
import cProfile
cProfile.run('somescriptfunc()')
This changes your script. When imported, its member functions, classes etc can be used as normal. When run from the command-line, it profiles itself.
Is this what you're looking for?
From the comments I've gathered more is perhaps needed, so here goes:
If you're running a script from CGI changes are it is of the form:
# do some stuff to extract the parameters
# do something with the parameters
# return the response.
When I say abstract out, you can do this:
def do_something_with_parameters(param1, param2):
pass
if __name__ = "__main__":
import cProfile
cProfile.run('do_something_with_parameters(param1=\'sometestvalue\')')
Put that file on your python path. When run itself, it will profile the function you want profiling.
Now, for your CGI script, create a script that does:
import {insert name of script from above here}
# do something to determine parameter values
# do something with them *via the function*:
do_something_with_parameters(param1=..., param2=...)
# return something
So your cgi script just becomes a little wrapper for your function (which it is anyway) and your function is now self-testing.
You can then profile the function using made up values on your desktop, away from the production server.
There are probably neater ways to achieve this, but it would work.