calling hive -e from a python script - python

I would like to run a very simple hive command from within my python script. I am trying to use hive -e, but I am getting an error
def hive():
cmd = "hive -e \"msck repair table dashboard_report\""
print(cmd)
check_call(cmd)
This is the error I am getting
hive -e "msck repair table dashboard_report"
Traceback (most recent call last):
File "/home/yosi/work/source/slg/tiger/src/main/resources/python/tiger.py", line 59, in <module>
hive()
File "/home/yosi/work/source/slg/tiger/src/main/resources/python/tiger.py", line 57, in hive
check_call(cmd)
File "/usr/lib/python2.7/subprocess.py", line 535, in check_call
retcode = call(*popenargs, **kwargs)
File "/usr/lib/python2.7/subprocess.py", line 522, in call
return Popen(*popenargs, **kwargs).wait()
File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1327, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory


Your check_call function is calling subprocess.Popen. If you want to pass arguments to that function you have to pass them in a list.
Probably:
cmd = ["hive", "-e", "\"msck repair table dashboard_report\""]
check_call(cmd)
Will work. Maybe some refactoring will be needed down the call stack to accept a list instead of a string.


If you are using python2.7 then the below code snippet will work.
import subprocess
command = [""" hive -e "msck repair table dashboard_report" """]
print subprocess.check_output(command,shell=True)

Related

How to run HIVE and IMPALA shell commands using python?

I have to run HIVE queries using python script eg,
For this $ bdscc -e "show databases"
I want to do the same in python, but I am getting error-
>>> import subprocess
>>> subprocess.call(['bdscc', '-e', 'show databases'])
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib64/python2.7/subprocess.py", line 524, in call
return Popen(*popenargs, **kwargs).wait()
File "/usr/lib64/python2.7/subprocess.py", line 711, in __init__
errread, errwrite)
File "/usr/lib64/python2.7/subprocess.py", line 1327, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
Is there any way to do this in python?
PS- The command $ bdscc -e "show databases" from terminal is running fine.
UPDATE-- bdscc is an alias set by my company for HIVE shell

Python: I want to execute shell commands using python script

I want to execute shell command "objdump" using python for my research work
I have used subprocess.call("command") to execute linux command but its not working.
Sample code which i have tried is
import subprocess
subprocess.call("date")
after the execution
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Python27\lib\subprocess.py", line 168, in call
return Popen(*popenargs, **kwargs).wait()
File "C:\Python27\lib\subprocess.py", line 390, in __init__
errread, errwrite)
File "C:\Python27\lib\subprocess.py", line 640, in _execute_child
startupinfo)
WindowsError: [Error 2] The system cannot find the file specified

You must do this:
subprocess.call('date', shell=True)
Actually, Shell does allow you to access to global variable and programs that are in $PATH and your shell.

Have you tried ?
import subprocess
subprocess.call("date", shell = True)

UNIX Command within Python Script receiving an error. Script executes command does not

I have a python script. On top of the script "from subprocess import call". If I write "call(["whoami"])" in the script and I execute the Python script from the unix command line it works, my username returns. But if I run a proprietary command with arguments, "call(["mi_xx", "-s", "20141215","-e","20150121","-p",'TX%_XX%',"-f","test","-i","-x","-d"])"
I get the error below. I run the same command directly in unix and it works. I have searched up and down. The python environment in unix is fine.
Error
Traceback (most recent call last):
File "Mixx.py", line 44, in <module>
call(["mi_xx", "-s", "20141215","-e","20150121","-p",'TX%_XX%',"-f","test","-i","-x","-d"])
File "/bb/util/common/ActivePythonEE_2.6.2_32bit/lib/python2.6/subprocess.py", line 444, in call
return Popen(*popenargs, **kwargs).wait()
File "/bb/util/common/ActivePythonEE_2.6.2_32bit/lib/python2.6/subprocess.py", line 595, in __init__
errread, errwrite)
File "/bb/util/common/ActivePythonEE_2.6.2_32bit/lib/python2.6/subprocess.py", line 1092, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory"

Call shell script from python

Attempting to call a shell script with options from a python script.
The line ./dropbox_uploader.sh -s download /test/pictures pictures/ runs fine via SSH but errors when called from a python script:
import subprocess
subprocess.call(['./dropbox_uploader.sh -s download /test/pictures pictures/'])
Here is the error message:
Traceback (most recent call last):
File "sync.py", line 2, in <module>
subprocess.call(['./dropbox_uploader.sh -s download /test/pictures pictures/'])
File "/usr/lib/python2.7/subprocess.py", line 493, in call
return Popen(*popenargs, **kwargs).wait()
File "/usr/lib/python2.7/subprocess.py", line 679, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1259, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory

In case if the first argument of subprocess.call is a list, it must contain the executable and arguments as separate items:
subprocess.call(['./dropbox_uploader.sh', '-s',
'download', '/test/pictures', 'pictures/'])
or, maybe, more convenient:
import shlex
cmd = './dropbox_uploader.sh -s download /test/pictures pictures/'
subprocess.call(shlex.split(cmd))
There is also an option to delegate parsing and execution to the shell:
cmd = './dropbox_uploader.sh -s download /test/pictures pictures/'
subprocess.call(cmd, shell=True)
(But please note the security warning)

Run shell command in mongo aggregation using python file

I need to run a mongo aggregation command in the shell using python script. How can I achieve this? This is my try
import subprocess
printj = 'aggregation_query'
cmd ='mongo --quiet mydb --eval',"'",printj,"'",' > output_traffic.json'
subprocess.call(cmd)
It gives an error
File "shellcmd.py", line 15, in <module>
subprocess.call(cmd)
File "/usr/lib64/python2.7/subprocess.py", line 524, in call
return Popen(*popenargs, **kwargs).wait()
File "/usr/lib64/python2.7/subprocess.py", line 711, in __init__
errread, errwrite)
File "/usr/lib64/python2.7/subprocess.py", line 1308, in _execute_child
raise child_exception

Don't run the mongo shell using subprocess, just import pymongo and connect to the MongoDB server directly. Examples of running aggregation queries from PyMongo are in the documentation.

Categories

Resources