Develop Reference

Pass file paths from Python to shell script [duplicate]

This question already has answers here:
Actual meaning of 'shell=True' in subprocess
(7 answers)
Closed 8 months ago.
I would like to run a shell script from Python 3 in Linux passing two arguments that contain file paths to two different files. The shell script then calls a programme written in Python 2.
In Python 3, I call the shell script like this:
import os
import sys
os.chmod('/path/to/sh', 0o777)
subprocess.call(['/path/to/sh', '/path/to/file1', '/path/to/file2'], shell=True)
My shell script looks like this:
#!/bin/sh
set -x
path1=$1
path2=$2
python2 path/to/programme "$path1" "$path2"
Now, the file paths are empty, and the shell script returns something like python2 path/to/programme '' ''. Does someone know how I could correctly pass the file paths so that the programme written in Python 2 can read them?
Or is there even an easier solution such as using subprocess to directly call the programme written in Python 2?

There is no need for the shell script. You can use subprocess to run python2 directly.
a.py
#!/usr/bin/env python3
import subprocess
subprocess.call(['python2', './b.py', 'foo', 'bar'])
b.py
#!/usr/bin/env python2
import sys
print sys.argv
Running ./a.py outputs ['./b.py', 'foo', 'bar'].
You could also try using past.translation instead:
The past module provides an experimental translation package to help with importing and using old Python 2 modules in a Python 3 environment.

shell=True is only needed if you do something like
subprocess.run("/path/to/sh /path/to/file1 /path/to/file2", shell=True)
where the shell will split the single string into arguments that will identify as the program name and its arguments. But you already have the program name and its arguments identified, so
subprocess.run(['/path/to/sh', '/path/to/file1', '/path/to/file2'])
is all you need.
By using a list and shell=True, you are essentially asking Python to execute
sh -c /path/to/sh /path/to/file1 /path/to/file2
which uses /path/to/file1 to set the value of $0, not $1, in the command to execute.

Subprocess module in python claims 'command not found' for 'module list'?

I am currently using the subprocess module in python for scripting purposes, and have been unable to execute the command 'module list' despite this working when I run it in shell and despite any other kind of command working fine when using the subprocess module to execute commands.
Two variations I have tried:
p = subprocess.Popen('module list', shell=True)
print(p.communicate()[0])
and
p = Popen(["module", "list"], stdout=subprocess.PIPE)
print(p.communicate()[0])
For commands such as 'echo hello world' and even longer commands with multiple arguments, either of these formats works just fine. Is the terminal I run commands from different from the shell used to execute commands using subprocess? The error I get is as follows:
/bin/bash: line 1: module: command not found

Based on what you've said in the comments, I believe you're going about using environment modules in Python the wrong way: There is actually a method in Modules itself to import module functionality into Python, as explained here:
>>> execfile('/usr/local/Modules/default/init/python.py')
>>> module('list')
No Modulefiles Currently Loaded.
>>> module('load','foo')
>>> module('list')
Currently Loaded Modulefiles:
1) foo/1.0
Of course, it's not very safe to use execfile(), so I slightly prefer the import method described here (slightly altered for Python 3 support):
import os
if 'PYTHONPATH' in os.environ:
os.environ['PYTHONPATH'] +=':'+os.environ['MODULESHOME']+"/init"
else:
os.environ['PYTHONPATH'] = os.environ['MODULESHOME']+"/init"
from python import module

The documentation of the Environment Modules software provides a recommendation on how to initialize the module command in Python (that should work on either Python 2 or 3):
import os
exec(open('/usr/share/Modules/init/python.py').read())
Once initialized, the module function is available and could be used in the following way:
module('sub-command', 'arg1', 'arg2', ...)
For example:
module('load', 'foo', 'bar')
module('list')
module('avail')

Executing Linux command in Python with special characters

I just want to execute the command rm /tmp/*.idx from a python script. I have read that os.system is deprecated (IT IS NOT, see the comments), so I'm trying with Popen the following:
proc = subprocess.Popen(shlex.split('rm /tmp/*.idx'))
proc.communicate()
after of course importing shlex and subprocess, but it doesn't erase the files.
Thanks.

Glob patterns are shell syntax. So:
subprocess.Popen("rm /tmp/*.idx", shell=True)

Importing OS Commands into Python; how to pipe with a variable in the middle?

So here's an example of the terminal line I'm trying to run after importing the OS into a Python Script of mine:
user$ echo variable | thecommand
Even though OS imports have been working for me lately, the fact that the variable is in the MIDDLE of the imported OS command is not allowing my code to run:
#! /bin/python
import os
variable = 'thevariable'
os.system ("echo "+variable +" | thecommand")
the above is what I have tried in a few different syntax's with no success. Is there a way to accomplish what I'm looking to do using the os.system method?

Don't use os.system(). it is deprecated.
Instead try
import subprocess
variable = 'thevariable'
subprocess.call("echo "+variable +" | thecommand", shell=True)
the shell=True means that the command will be run in a bash process so that echo and the pipe would work.

os.system(<command>) execution through Python :: Limitations?

I'm writing a python (ver 2.7) script to automate the set of commands in this Getting Started example for INOTOOL.
Problem: When I run this entire script, I repeatedly encounter these errors:
Current Directory is not empty
No project is found in this directory
No project is found in this directory
But, when I run a first script only up till the code line marked, and manually type in the next three lines, or when I run these last three lines (starting from the "ino init -t blink" line) after manually accessing the beep folder, then I am able to successfully execute the same code.
Is there a limitation with os.system() that I'm encountering?
My code:
import os,sys
def upload()
os.system("cd /home/pi/Downloads")
os.system("mkdir beep")
os.system("cd beep") #will refer to this code junction in question description
os.system("ino init -t blink")
os.system("ino build")
os.system("ino upload")
sys.exit(0)

Yes, when os.system() commands are run for cd , it does not actually change the current directory for the python process' context. From documentation -
os.system(command)
Execute the command (a string) in a subshell. This is implemented by calling the Standard C function system(), and has the same limitations. Changes to sys.stdin, etc. are not reflected in the environment of the executed command.
So even though you are changing directory in os.system() call, the next os.system call still occurs in same directory. Which could be causing your issue.
You shoud try using os.chdir() to change the directory instead of os.system() calls.
The Best would be to use subprocess module as #PadraicCunningham explains in his answer.

You can use the subprocess module and os.mkdir to make the directory, you can pass the current working directory cwd to check_callso you actually execute the command in the directory:
from subprocess import check_call
import os
def upload():
d = "/home/pi/Downloads/beep"
os.mkdir(d)
check_call(["ino", "init", "-t", "blink"],cwd=d)
check_call(["ino", "build"],cwd=d)
check_call(["ino", "upload"],cwd=d)
A non-zero exit status will raise CalledProcessError which you may want to catch but once successful you know the commands all returned a 0 exit status.