I am trying to compile a C program using Python and want to give input using "<" operator but it's not working as expected.
If I compile the C program and run it by giving input though a file it works; for example
./a.out <inp.txt works
But similarly if I try to do this using a Python script, it did not quite work out as expected.
For example:
import subprocess
subprocess.call(["gcc","a.c","-o","x"])
subprocess.call(["./x"])
and
import subprocess
subprocess.call(["gcc","a.c","-o","x"])
subprocess.call(["./x","<inp.txt"])
Both script ask for input though terminal. But I think in the second script it should read from file. why both the programs are working the same?
To complement #Jonathan Leffler's and #alastair's helpful answers:
Assuming you control the string you're passing to the shell for execution, I see nothing wrong with using the shell for convenience. [1]
subprocess.call() has an optional Boolean shell parameter, which causes the command to be passed to the shell, enabling I/O redirection, referencing environment variables, ...:
subprocess.call("./x <inp.txt", shell = True)
Note how the entire command line is passed as a single string rather than an array of arguments.
[1]
Avoid use of the shell in the following cases:
If your Python code must run on platforms other than Unix-like ones, such as Windows.
If performance is paramount.
If you find yourself "outsourcing" tasks better handled on the Python side.
If you're concerned about lack of predictability of the shell environment (as #alastair is):
subprocess.call with shell = True always creates non-interactive non-login instances of /bin/sh - note that it is NOT the user's default shell that is used.
sh does NOT read initialization files for non-interactive non-login shells (neither system-wide nor user-specific ones).
Note that even on platforms where sh is bash in disguise, bash will act this way when invoked as sh.
Every shell instance created with subprocess.call with shell = True is its own world, and its environment is neither influenced by previous shell instances nor does it influence later ones.
However, the shell instances created do inherit the environment of the python process itself:
If you started your Python program from an interactive shell, then that shell's environment is inherited. Note that this only pertains to the current working directory and environment variables, and NOT to aliases, shell functions, and shell variables.
Generally, that's a feature, given that Python (CPython) itself is designed to be controllable via environment variables (for 2.x, see https://docs.python.org/2/using/cmdline.html#environment-variables; for 3.x, see https://docs.python.org/3/using/cmdline.html#environment-variables).
If needed, you can supply your own environment to the shell via the env parameter; note, however, that you'll have to supply the entire environment in that event, potentially including variables such as USER and HOME, if needed; simple example, defining $PATH explicitly:
subprocess.call('echo $PATH', shell = True, \
env = { 'PATH': '/sbin:/bin:/usr/bin' })
The shell does I/O redirection for a process. Based on what you're saying, the subprocess module does not do I/O redirection like that. To demonstrate, run:
subprocess.call(["sh","-c", "./x <inp.txt"])
That runs the shell and should redirect the I/O. With your code, your program ./x is being given an argument <inp.txt which it is ignoring.
NB: the alternative call to subprocess.call is purely for diagnostic purposes, not a recommended solution. The recommended solution involves reading the (Python 2) subprocess module documentation (or the Python 3 documentation for it) to find out how to do the redirection using the module.
import subprocess
i_file = open("inp.txt")
subprocess.call("./x", stdin=i_file)
i_file.close()
If your script is about to exit so you don't have to worry about wasted file descriptors, you can compress that to:
import subprocess
subprocess.call("./x", stdin=open("inp.txt"))
By default, the subprocess module does not pass the arguments to the shell. Why? Because running commands via the shell is dangerous; unless they're correctly quoted and escaped (which is complicated), it is often possible to convince programs that do this kind of thing to run unwanted and unexpected shell commands.
Using the shell for this would be wrong anyway. If you want to take input from a particular file, you can use subprocess.Popen, setting the stdin argument to a file descriptor for the file inp.txt (you can get the file descriptor by calling fileno() a Python file object).
Related
In bash or C, exec will terminate the current process and replace it with something new. Can this functionality be accomplished in Python? I don't wish to simply execute some code then continue running the python script (even just to immediately exit), or spawn a child process.
My specific situation is the following. I'm developing a command line application (python + curses) to manage data generation/analysis in the context of scientific computing. It will sometimes be necessary for me to terminate the application and go wrangle with the data in a given subdirectory manually. It would be convenient if I could do something like:
# within python script; d=<some directory>
if exit_and_goto_dir:
exec("pushd {}".format(d)) # C-style exec -- terminate program and execute new command
The following do not work, for example:
# attempt 1
if exit_and_goto_dir:
os.system("pushd {}".format(d))
exit(0) # pushd does not outlast the python script
# attempt 2
if exit_and_goto_dir:
os.chdir(d)
exit(0)
This behavior isn't really critical. There are plenty of work arounds (e.g. print the directory I care about to terminal then cd manually). Mostly I'm curious if it's possible. Thanks!
The os module contains Python wrappers for the various exec* functions in the C standard library:
>>> [method for method in dir(os) if method.startswith("exec")]
['execl', 'execle', 'execlp', 'execlpe', 'execv', 'execve', 'execvp', 'execvpe']
However, pushd is not an executable that you can exec but rather a bash builtin (and the same is true for cd).
What you could do would be to change directory inside the python process and then exec an interactive shell:
import os
os.chdir(d)
os.execvp("bash", ["bash", "-login"])
Python's current directory will be inherited by the shell that you exec. When you later exit from that shell, control will then return to the original login shell from which you invoked python (unless you used that shell's exec command to launch python in the first place).
What you can't do is to modify the current directory of the calling shell from inside python, in order to return to the shell prompt but in a different working directory from when python was invoked. (At least there's no straightforward way. There is a hack involving attaching gdb to it, described here, but which only worked as root when I tried it on Ubuntu.)
i'm calling a python script inside my bash script and I was wondering if there is a simple way to set my bash variables within my python script.
Example:
My bash script:
#!/bin/bash
someVar=""
python3 /some/folder/pythonScript.py
My python script:
anotherVar="HelloWorld"
Is there a way I can set my someVar to the value of anotherVar? I was thinking of printing properties in a file inside the python script and then read them from my bash script but maybe there is another way. Also I don't know and don't think it makes any difference but I can name both variable with the same name (someVar/someVar instead of someVar/anotherVar)
No, when you execute python, you start a new process, and every process has access only to their own memory. Imagine what would happen if a process could influence another processes memory! Even for parent/child processes like this, this would be a huge security problem.
You can make python print() something and use that, though:
#!/usr/bin/env python3
print('Hello!')
And in your shell script:
#!/usr/bin/env bash
someVar=$(python3 myscript.py)
echo "$someVar"
There are, of course, many others IPC techniques you could use, such as sockets, pipes, shared memory, etc... But without context, it's difficult to make a specific recommendation.
shlex.quote() in Python 3, or pipes.quote() in Python 2, can be used to generate code which can be evaled by the calling shell. Thus, if the following script:
#!/usr/bin/env python3
import sys, shlex
print('export foobar=%s' % (shlex.quote(sys.argv[1].upper())))
...is named setFoobar and invoked as:
eval "$(setFoobar argOne)"
...then the calling shell will have an environment variable set with the name foobar and the value argOne.
Assume using Linux:
In Perl, the exec function executes an external program and immediately exits itself, leaving the external program in same shell session.
A very close answer using Python is https://stackoverflow.com/a/13256908
However, the Python solution using start_new_session=True starts an external program using setsid method, that means that solution is suitable for making a daemon, not an interactive program.
Here is an simple example of using perl:
perl -e '$para=qq(-X --cmd ":vsp");exec "vim $para"'
After vim is started, the original Perl program has exited and the vim is still in the same shell session(vim is not sent to new session group).
How to get the same solution with Python.
Perl is just wrapping the exec* system call functions here. Python has the same wrappers, in the os module, see the os.exec* documentation:
These functions all execute a new program, replacing the current process; they do not return. On Unix, the new executable is loaded into the current process, and will have the same process id as the caller.
To do the same in Python:
python -c 'import os; para="-X --cmd \":vsp\"".split(); os.execlp("vim", *para)'
os.execlp accepts an argument list and looks up the binary in $PATH from the first argument.
The subprocess module is only ever suitable for running processes next to the Python process, not to replace the Python process. On POSIX systems, the subprocess module uses the low-level exec* functions to implement it's functionality, where a fork of the Python process is then replaced with the command you wanted to run with subprocess.
I have some instrument which requires environment variable which I want to set automatically from python code. So I tried several ways to make it happen, but none of them were successful.
Here are some examples:
I insert following code in my python script
import os
os.system("export ENV_VAR=/some_path")
I created bash script(env.sh) and run it from python:
#!/bin/bash
export ENV_VAR=some_path
#call it from python
os.system("source env.sh")
I also tried os.putenv() and os.environ*["ENV_VAR"] = "some_path"
Is it possible to set(export) environment variable using python, i.e
without directly exporting it to shell?
Setting an environment variable sets it only for the current process and any child processes it launches. So using os.system will set it only for the shell that is running to execute the command you provided. When that command finishes, the shell goes away, and so does the environment variable. Setting it using os.putenv or os.environ has a similar effect; the environment variables are set for the Python process and any children of it.
I assume you are trying to have those variables set for the shell that you launch the script from, or globally. That can't work because the shell (or other process) is not a child of the Python script in which you are setting the variable.
You'll have better luck setting the variables in a shell script. If you then source that script (so that it runs in the current instance of the shell, rather than in a subshell) then they will remain set after the script ends.
As long as you start the "instrument" (a script I suppose) from the very same process it should work:
In [1]: os.putenv("VARIABLE", "123")
In [2]: os.system("echo $VARIABLE")
123
You can't change an environment variable of a different process or a parent process.
A shell function may do this. You need to print your export statement and eval that.
set_shell_env() {
output=$(python print_export_env.py $*)
eval $output
}
Depending on how you execute your instrument, you might be able to change environment specifically for the child process without affecting the parent. See documentation for os.spawn*e or subprocess.Popen which accept separate argument denoting child environment. For example, Replacing the os.spawn family in subprocess module documentation which provides both usages:
Environment example:
os.spawnlpe(os.P_NOWAIT, "/bin/mycmd", "mycmd", "myarg", env)
==>
Popen(["/bin/mycmd", "myarg"], env={"PATH": "/usr/bin"})
By some need I was forced to correct os.environ['PATH'] to be able to run dir\to\fake\python.cmd script which adds some extra parameters to the original one before execution.
Also I have two python scripts:
test1.py:
# ...
p = subprocess.call("test2.py") # shell=True works fine
# ...
test2.py:
# ...
print "Hello from test2.py"
# ...
When I run python test1.py my "fake" python.cmd doing its stuff, refers to the original python in c:\Python25 and runs test1.py with my extra arguments. But, sadly, test2.py, script is never called. If I put shell=True as subprocess.call argument - everythin's fine, test2.py is called.
I know, Windows is trying to find python interpreter to use for the call in the real c:\Python25 working directory when shell=False is by default.
The question to you is: how can I achieve the goal without changing my code in test1.py and test2.py? Maybe virtualenv library may be very useful in this case?
Thank you very much for your help
As stated in the docs:
The shell argument (which defaults to False) specifies whether to use the shell as the program to execute.
and
On Windows with shell=True, the COMSPEC environment variable specifies the default shell. The only time you need to specify shell=True on Windows is when the command you wish to execute is built into the shell (e.g. dir or copy). You do not need shell=True to run a batch file or console-based executable.
So when you call subprocess.call("test2.py"), the system tries to call test2.py as an executable, which it is not, so it fails. However, you don't capture the return value from subprocess.open to check for error conditions, so it fails silently. When you call it with shell=True, it calls the system shell with the argument test2.py, which in turn looks up the default executable for .py files on your system and then executes the file that way.
All that said though, the deeper problem here is that your code is very poorly engineered. The probability that you have a legitimate reason to dispatch a python script via the OS from another python script is vanishingly small. Rather than patching another kludgy hack over the top of what you already have, save yourself headaches in the future and refactor the system so that it does things in an easier, more sensible, more naturally integrated way.