Is it possible to wrap the standard python interpreter inside a bash script or similar, so the interpreter is called within slurm or any other program, and this script can be used as a remote interpreter in PyCharm?
The situation is I need to call the python interpreter remotely using slurm (and not as standalone), with a command like:
srun ---mem=10G python
A simple script doesn't seem to do the job. I can use remotely the python interpreter without slurm in PyCharm, but I need to use it within slurm. I've also tried to do it within PyCharm, but I haven't managed to do so.
Thanks in advance.
Related
1) I am new to Python and love to learn its core. I have downloaded the python software package and discovered the python.exe application inside. I double clicked it and a balck and white window popped up.
Should I call it a python Interpreter or python Shell?
2) I am learning python online. I came across the terms python tty, python shell and python interpreter. I am satisfied by calling that screen inside the window as a tty(TeleTYpewriter) because we could use only keyboard to work inside and no mouse. But actually that screen has got some intelligence responding to our request. Is python tty an apt term for it?
3) In UNIX, shell is an user interface and command line interpreter, so does python interpreter and python shell are the same.
Python shell lets you use the Python interpreter in interactive mode, just as an OS shell, such as bash, lets you use the OS in interactive mode. You can use the Python interpreter in script mode or batch mode wherein you let the interpreter execute all lines of code in one sequence. It is comparable to writing shell scripts (or batch files in Microsoft Windows).
In your case the screenshot is of a python "shell".
You shouldn't really pay attention to this distinction because in the end everything runs through the python interpreter be it in interactive mode or not.
It is both the python shell and the python interpreter. The shell is where you write your code directly in the CLI, whereas the interpreter is the program that will interpret your code and execute it. Therefore, the interpreter is called in the shell when you write some code, it may also be called when you execute some python code directly from a file.
The customary term for the interactive Python shell is the Python REPL. Many modern interpreters enter a Read-Eval-Print-Loop when you run them interactively, and this term has stuck.
The program which interprets and executes your Python code is the Python interpreter; it can act as a shell, as described above, or run silently and just execute your Python code without any visible user interface of its own, like when you run a script of yours with
python scriptname.py
I am using Python 2.7
I have interactive Python script, named A, written using Cmd module.
However, sometimes there is a need to activate script A from other Python scripts.
It also could be nice if the interactive script A wouldn't exit in between commands given to them.
I tried to search for solutions and I found couple here
One is to pass a file from which the interactive script will read commands. It works well for me but it is not much liked.
Another is to use echo which doesn't work for me (still trying).
I wonder whether there are other solutions.....
I'd like to call a separate non-child python program from a python script and have it run externally in a new shell instance. The original python script doesn't need to be aware of the instance it launches, it shouldn't block when the launched process is running and shouldn't care if it dies. This is what I have tried which returns no error but seems to do nothing...
import subprocess
python_path = '/usr/bin/python'
args = [python_path, '&']
p = subprocess.Popen(args, shell=True)
What should I be doing differently
EDIT
The reason for doing this is I have an application with a built in version of python, I have written some python tools that should be run separately alongside this application but there is no assurance that the user will have python installed on their system outside the application with the builtin version I'm using. Because of this I can get the python binary path from the built in version programatically and I'd like to launch an external version of the built in python. This eliminates the need for the user to install python themselves. So in essence I need a simple way to call an external python script using my current running version of python programatically.
I don't need to catch any output into the original program, in fact once launched I'd like it to have nothing to do with the original program
EDIT 2
So it seems that my original question was very unclear so here are more details, I think I was trying to over simplify the question:
I'm running OSX but the code should also work on windows machines.
The main application that has a built in version of CPython is a compiled c++ application that ships with a python framework that it uses at runtime. You can launch the embedded version of this version of python by doing this in a Terminal window on OSX
/my_main_app/Contents/Frameworks/Python.framework/Versions/2.7/bin/python
From my main application I'd like to be able to run a command in the version of python embedded in the main app that launches an external copy of a python script using the above python version just like I would if I did the following command in a Terminal window. The new launched orphan process should have its own Terminal window so the user can interact with it.
/my_main_app/Contents/Frameworks/Python.framework/Versions/2.7/bin/python my_python_script
I would like the child python instance not to block the main application and I'd like it to have its own terminal window so the user can interact with it. The main application doesn't need to be aware of the child once its launched in any way. The only reason I would do this is to automate launching an external application using a Terminal for the user
If you're trying to launch a new terminal window to run a new Python in (which isn't what your question asks for, but from a comment it sounds like it's what you actually want):
You can't. At least not in a general-purpose, cross-platform way.
Python is just a command-line program that runs with whatever stdin/stdout/stderr it's given. If those happen to be from a terminal, then it's running in a terminal. It doesn't know anything about the terminal beyond that.
If you need to do this for some specific platform and some specific terminal program—e.g., Terminal.app on OS X, iTerm on OS X, the "DOS prompt" on Windows, gnome-terminal on any X11 system, etc.—that's generally doable, but the way to do it is by launching or scripting the terminal program and telling it to open a new window and run Python in that window. And, needless to say, they all have completely different ways of doing that.
And even then, it's not going to be possible in all cases. For example, if you ssh in to a remote machine and run Python on that machine, there is no way it can reach back to your machine and open a new terminal window.
On most platforms that have multiple possible terminals, you can write some heuristic code that figures out which terminal you're currently running under by just walking os.getppid() until you find something that looks like a terminal you know how to deal with (and if you get to init/launchd/etc. without finding one, then you weren't running in a terminal).
The problem is that you're running Python with the argument &. Python has no idea what to do with that. It's like typing this at the shell:
/usr/bin/python '&'
In fact, if you pay attention, you're almost certainly getting something like this through your stderr:
python: can't open file '&': [Errno 2] No such file or directory
… which is exactly what you'd get from doing the equivalent at the shell.
What you presumably wanted was the equivalent of this shell command:
/usr/bin/python &
But the & there isn't an argument at all, it's part of sh syntax. The subprocess module doesn't know anything about sh syntax, and you're telling it not to use a shell, so there's nobody to interpret that &.
You could tell subprocess to use a shell, so it can do this for you:
cmdline = '{} &'.format(python_path)
p = subprocess.Popen(cmdline, shell=True)
But really, there's no good reason to. Just opening a subprocess and not calling communicate or wait on it already effectively "puts it in the background", just like & does on the shell. So:
args = [python_path]
p = subprocess.Popen(args)
This will start a new Python interpreter that sits there running in the background, trying to use the same stdin/stdout/stderr as your parent. I'm not sure why you want that, but it's the same thing that using & in the shell would have done.
Actually I think there might be a solution to your problem, I found a useful solution at another question here.
This way subprocess.popen starts a new python shell instance and runs the second script from there. It worked perfectly for me on Windows 10.
You can try using screen command
with this command a new shell instance created and the current instance runs in the background.
# screen; python script1.py
After running above command, a new shell prompt will be seen where we can run another script and script1.py will be running in the background.
Hope it helps.
Recently, I came across the Linux command source and then found this answer on what it does.
My understanding was that source executes the file that is passed to it, and it did work for a simple shell script. Then I tried using source on a Python script–but it did not work.
The Python script has a shebang (e.g. #!/usr/bin/python) and I am able to do a ./python.py, as the script has executable permission. If that is possible, source python.py should also be possible, right? The only difference is ./ executes in a new shell and source executes in the current shell. Why is it not working on a .py script? Am I missing something here?
You're still not quite on-target understanding what source does.
source does indeed execute commands from a file in the current shell process. It does this effectively as if you had typed them directly into your current shell.
The reason this is necessary is because when you run a shell script without sourcing it, it will spawn a subshell—a new process. When this process exits, any changes made within that script are lost as you return to the shell from which it spawned.
It follows, then, that you cannot source Python into a shell, because the Python interpreter is always a different process from your shell. Running a Python script spawns a brand-new process, and when that process exits, its state is lost.
Of course, if your shell is actually Python (which I would not recommend!), you can still "source" into it—by using import.
source executes the files and places whatever functions/aliases/environment variables created in that script within the shell that called it. It does this by not spawning a new process, but instead executing the script in the current process.
The shabang is used by the shell to indicate what to use to spawn the new process, so for source it is ignored, and the file is interpreted as the language of the current process (bash in this case). This is why using source on a python file failed for you.
I'd like to create python programs that use mpi4py and thus I'd like to run them using the following command:
mpirun -np 4 python script.py
I tried to create a shell script which does this and use it as a python interpreter but eclipse rejects the shell script. I tried to redirect the output (so that it doesen't show the mpi-stuff but soley prints the python-output of the first node).
If I run the script in the console using the interpreterinfo.py script to test the interpreter it gives exactly the same output as if I run it only through python.
It somehow seems that the script isn't executed properly by eclipse or that the output is not going into stdout.
Can anyone help?
I don't think you should try to configure mpirun as the python interpreter... Instead, configure the python interpreter as usual and just create a python module that'll do the launching for you and launch that module instead... (or create an external launch in run > external tools)
It'd be strange that mpirun is the actual python interpreter, because that way when requesting a code completion for builtins, pydev would launch the mpirun and it'd create 4 processes for code-completion? The same would apply to other things such as debug, coverage, etc...