Develop Reference

How to run HIVE and IMPALA shell commands using python?

I have to run HIVE queries using python script eg,
For this $ bdscc -e "show databases"
I want to do the same in python, but I am getting error-
>>> import subprocess
>>> subprocess.call(['bdscc', '-e', 'show databases'])
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib64/python2.7/subprocess.py", line 524, in call
return Popen(*popenargs, **kwargs).wait()
File "/usr/lib64/python2.7/subprocess.py", line 711, in __init__
errread, errwrite)
File "/usr/lib64/python2.7/subprocess.py", line 1327, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
Is there any way to do this in python?
PS- The command $ bdscc -e "show databases" from terminal is running fine.
UPDATE-- bdscc is an alias set by my company for HIVE shell

Python: I want to execute shell commands using python script

I want to execute shell command "objdump" using python for my research work
I have used subprocess.call("command") to execute linux command but its not working.
Sample code which i have tried is
import subprocess
subprocess.call("date")
after the execution
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Python27\lib\subprocess.py", line 168, in call
return Popen(*popenargs, **kwargs).wait()
File "C:\Python27\lib\subprocess.py", line 390, in __init__
errread, errwrite)
File "C:\Python27\lib\subprocess.py", line 640, in _execute_child
startupinfo)
WindowsError: [Error 2] The system cannot find the file specified

You must do this:
subprocess.call('date', shell=True)
Actually, Shell does allow you to access to global variable and programs that are in $PATH and your shell.

Have you tried ?
import subprocess
subprocess.call("date", shell = True)

Python script executes from terminal, but not when using subprocess?

I'm using subprocess to execute a Python script called trace.py that is located in a different folder. The script trace.py then uses subprocess to run a traceroute command and then prints the output. When I go to the folder that trace.py is located in and type this in the terminal:
python trace.py
or
./trace.py
or from any location:
python /home/.../cgi-bin/trace.py
it works fine and the traceroute is printed to the terminal. However, when I try to execute trace.py from main.py by using subprocess, it doesn't seem to work. I've tested this by creating test.py and using subprocess to execute it from main.py and this works. I do this with the following:
output = subprocess.check_output([sys.executable, script_path])
Where script_path is the absolute path to trace.py.
The full error I get is this (paths are shortened):
Traceback (most recent call last):
File "/home/.../cgi-bin/trace.py", line 11, in <module>
traceroute = subprocess.check_output(["traceroute", "www.google.com"])
File "/usr/lib/python2.7/subprocess.py", line 566, in check_output
process = Popen(stdout=PIPE, *popenargs, **kwargs)
File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1327, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "main.py", line 97, in <module>
serve(args.port, public_html, cgibin)
File "main.py", line 55, in serve
process = subprocess.check_output(["/usr/bin/python", script_path])
File "/usr/lib/python2.7/subprocess.py", line 573, in check_output
raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command '['/usr/bin/python', '/home/.../cgi-bin/trace.py']' returned non-zero exit status 1
Why does this not work, but executing it from the terminal does?

The child can't find traceroute executable.
Compare os.environ['PATH'] in your shell with the value within runnning trace.py
Check file permissions -- whether it is readable and executable by the user that runs trace.py.

UNIX Command within Python Script receiving an error. Script executes command does not

I have a python script. On top of the script "from subprocess import call". If I write "call(["whoami"])" in the script and I execute the Python script from the unix command line it works, my username returns. But if I run a proprietary command with arguments, "call(["mi_xx", "-s", "20141215","-e","20150121","-p",'TX%_XX%',"-f","test","-i","-x","-d"])"
I get the error below. I run the same command directly in unix and it works. I have searched up and down. The python environment in unix is fine.
Error
Traceback (most recent call last):
File "Mixx.py", line 44, in <module>
call(["mi_xx", "-s", "20141215","-e","20150121","-p",'TX%_XX%',"-f","test","-i","-x","-d"])
File "/bb/util/common/ActivePythonEE_2.6.2_32bit/lib/python2.6/subprocess.py", line 444, in call
return Popen(*popenargs, **kwargs).wait()
File "/bb/util/common/ActivePythonEE_2.6.2_32bit/lib/python2.6/subprocess.py", line 595, in __init__
errread, errwrite)
File "/bb/util/common/ActivePythonEE_2.6.2_32bit/lib/python2.6/subprocess.py", line 1092, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory"

Call shell script from python

Attempting to call a shell script with options from a python script.
The line ./dropbox_uploader.sh -s download /test/pictures pictures/ runs fine via SSH but errors when called from a python script:
import subprocess
subprocess.call(['./dropbox_uploader.sh -s download /test/pictures pictures/'])
Here is the error message:
Traceback (most recent call last):
File "sync.py", line 2, in <module>
subprocess.call(['./dropbox_uploader.sh -s download /test/pictures pictures/'])
File "/usr/lib/python2.7/subprocess.py", line 493, in call
return Popen(*popenargs, **kwargs).wait()
File "/usr/lib/python2.7/subprocess.py", line 679, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1259, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory

In case if the first argument of subprocess.call is a list, it must contain the executable and arguments as separate items:
subprocess.call(['./dropbox_uploader.sh', '-s',
'download', '/test/pictures', 'pictures/'])
or, maybe, more convenient:
import shlex
cmd = './dropbox_uploader.sh -s download /test/pictures pictures/'
subprocess.call(shlex.split(cmd))
There is also an option to delegate parsing and execution to the shell:
cmd = './dropbox_uploader.sh -s download /test/pictures pictures/'
subprocess.call(cmd, shell=True)
(But please note the security warning)