Develop Reference

How to run HIVE and IMPALA shell commands using python?

I have to run HIVE queries using python script eg,
For this $ bdscc -e "show databases"
I want to do the same in python, but I am getting error-
>>> import subprocess
>>> subprocess.call(['bdscc', '-e', 'show databases'])
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib64/python2.7/subprocess.py", line 524, in call
return Popen(*popenargs, **kwargs).wait()
File "/usr/lib64/python2.7/subprocess.py", line 711, in __init__
errread, errwrite)
File "/usr/lib64/python2.7/subprocess.py", line 1327, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
Is there any way to do this in python?
PS- The command $ bdscc -e "show databases" from terminal is running fine.
UPDATE-- bdscc is an alias set by my company for HIVE shell

Python: I want to execute shell commands using python script

I want to execute shell command "objdump" using python for my research work
I have used subprocess.call("command") to execute linux command but its not working.
Sample code which i have tried is
import subprocess
subprocess.call("date")
after the execution
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Python27\lib\subprocess.py", line 168, in call
return Popen(*popenargs, **kwargs).wait()
File "C:\Python27\lib\subprocess.py", line 390, in __init__
errread, errwrite)
File "C:\Python27\lib\subprocess.py", line 640, in _execute_child
startupinfo)
WindowsError: [Error 2] The system cannot find the file specified

You must do this:
subprocess.call('date', shell=True)
Actually, Shell does allow you to access to global variable and programs that are in $PATH and your shell.

Have you tried ?
import subprocess
subprocess.call("date", shell = True)

calling hive -e from a python script

I would like to run a very simple hive command from within my python script. I am trying to use hive -e, but I am getting an error
def hive():
cmd = "hive -e \"msck repair table dashboard_report\""
print(cmd)
check_call(cmd)
This is the error I am getting
hive -e "msck repair table dashboard_report"
Traceback (most recent call last):
File "/home/yosi/work/source/slg/tiger/src/main/resources/python/tiger.py", line 59, in <module>
hive()
File "/home/yosi/work/source/slg/tiger/src/main/resources/python/tiger.py", line 57, in hive
check_call(cmd)
File "/usr/lib/python2.7/subprocess.py", line 535, in check_call
retcode = call(*popenargs, **kwargs)
File "/usr/lib/python2.7/subprocess.py", line 522, in call
return Popen(*popenargs, **kwargs).wait()
File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1327, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory

Your check_call function is calling subprocess.Popen. If you want to pass arguments to that function you have to pass them in a list.
Probably:
cmd = ["hive", "-e", "\"msck repair table dashboard_report\""]
check_call(cmd)
Will work. Maybe some refactoring will be needed down the call stack to accept a list instead of a string.

If you are using python2.7 then the below code snippet will work.
import subprocess
command = [""" hive -e "msck repair table dashboard_report" """]
print subprocess.check_output(command,shell=True)

UNIX Command within Python Script receiving an error. Script executes command does not

I have a python script. On top of the script "from subprocess import call". If I write "call(["whoami"])" in the script and I execute the Python script from the unix command line it works, my username returns. But if I run a proprietary command with arguments, "call(["mi_xx", "-s", "20141215","-e","20150121","-p",'TX%_XX%',"-f","test","-i","-x","-d"])"
I get the error below. I run the same command directly in unix and it works. I have searched up and down. The python environment in unix is fine.
Error
Traceback (most recent call last):
File "Mixx.py", line 44, in <module>
call(["mi_xx", "-s", "20141215","-e","20150121","-p",'TX%_XX%',"-f","test","-i","-x","-d"])
File "/bb/util/common/ActivePythonEE_2.6.2_32bit/lib/python2.6/subprocess.py", line 444, in call
return Popen(*popenargs, **kwargs).wait()
File "/bb/util/common/ActivePythonEE_2.6.2_32bit/lib/python2.6/subprocess.py", line 595, in __init__
errread, errwrite)
File "/bb/util/common/ActivePythonEE_2.6.2_32bit/lib/python2.6/subprocess.py", line 1092, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory"

Call shell script from python

Attempting to call a shell script with options from a python script.
The line ./dropbox_uploader.sh -s download /test/pictures pictures/ runs fine via SSH but errors when called from a python script:
import subprocess
subprocess.call(['./dropbox_uploader.sh -s download /test/pictures pictures/'])
Here is the error message:
Traceback (most recent call last):
File "sync.py", line 2, in <module>
subprocess.call(['./dropbox_uploader.sh -s download /test/pictures pictures/'])
File "/usr/lib/python2.7/subprocess.py", line 493, in call
return Popen(*popenargs, **kwargs).wait()
File "/usr/lib/python2.7/subprocess.py", line 679, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1259, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory

In case if the first argument of subprocess.call is a list, it must contain the executable and arguments as separate items:
subprocess.call(['./dropbox_uploader.sh', '-s',
'download', '/test/pictures', 'pictures/'])
or, maybe, more convenient:
import shlex
cmd = './dropbox_uploader.sh -s download /test/pictures pictures/'
subprocess.call(shlex.split(cmd))
There is also an option to delegate parsing and execution to the shell:
cmd = './dropbox_uploader.sh -s download /test/pictures pictures/'
subprocess.call(cmd, shell=True)
(But please note the security warning)