I am trying to use a hive operator in airflow. I have all the dependencies installed (pyhs2, pyhive, and ran pip install airflow[hive]).
However when I use the code below
t1 = HiveOperator(
task_id='simple_query',
hql='select * from cities',
dag=dag)
I get this error. I am not sure what it means
[2016-01-06 03:26:39,500] {models.py:1017} ERROR - [Errno 2] No such file or directory
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 977, in run
result = task_copy.execute(context=context)
File "/usr/local/lib/python2.7/dist-packages/airflow/operators/hive_operator.py", line 65, in execute
self.hook.run_cli(hql=self.hql, schema=self.schema)
File "/usr/local/lib/python2.7/dist-packages/airflow/hooks/hive_hooks.py", line 110, in run_cli
cwd=tmp_dir)
File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1327, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
[2016-01-06 03:26:39,512] {models.py:1053} ERROR - [Errno 2] No such file or directory
This error occurs when the hive command line interface (CLI) isn't available in your system path.
Related
I am able to execute this command successfully from command line and it copies the desired csv to the s3 bucket aws s3 cp /Users/kaswani/tips.csv s3://dplearn/
But, when I try to run the same from within python using subprocess it throws error:
subprocess.check_output(['aws','s3','cp','/Users/kaswani/tips.csv','s3://dplearn/'])
Error:
Traceback (most recent call last):
File "<input>", line 1, in <module>
File "/Users/kaswani/anaconda/envs/aws/lib/python3.5/subprocess.py", line 626, in check_output
**kwargs).stdout
File "/Users/kaswani/anaconda/envs/aws/lib/python3.5/subprocess.py", line 693, in run
with Popen(*popenargs, **kwargs) as process:
File "/Users/kaswani/anaconda/envs/aws/lib/python3.5/subprocess.py", line 947, in __init__
restore_signals, start_new_session)
File "/Users/kaswani/anaconda/envs/aws/lib/python3.5/subprocess.py", line 1551, in _execute_child
raise child_exception_type(errno_num, err_msg)
FileNotFoundError: [Errno 2] No such file or directory: 'aws'
I have not been able to debug it. Subprocess works just fine for basic commands like ls etc
Using, shell=True makes is work, i.e. subprocess.check_output(['aws','s3','cp','/Users/kaswani/tips.csv','s3://dplearn/'], shell=True)
I'm using a program called SPAdes-3.50-Darwin and while running it on a server (one which I do not have root access to), I get the following error even though on my local computer it runs fine:
== Running read error correction tool: /export/home/james/Tools/SPAdes-3.5.0-Darwin/bin/hammer /export/home/james/package/output/pg/assembly_files/NA12877_S1_chr14_105900000-107300000/spades_files/3-49/corrected/configs/config.info
[Errno 13] Permission denied
Traceback (most recent call last):
File "/export/home/james/Tools/SPAdes-3.5.0-Darwin/bin/spades.py", line 586, in main
ext_python_modules_home, log)
File "/export/home/james/Tools/SPAdes-3.5.0-Darwin/share/spades/spades_pipeline/hammer_logic.py", line 128, in run_hammer
support.sys_call(command, log)
File "/export/home/james/Tools/SPAdes-3.5.0-Darwin/share/spades/spades_pipeline/support.py", line 181, in sys_call
proc = subprocess.Popen(cmd_list, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, cwd=cwd)
File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1327, in _execute_child
raise child_exception
OSError: [Errno 13] Permission denied
Does this have to do with not having root permission?
When i execute .\install.bat on C:\Install\IBM_Docs\DocsConversion\DocsConversion\installer where the "install" file is.
Traceback(most recent call last):
File "conversion/install.py", line 219, in <module>
if not pi.do();
File "C:\Install\IBM_Docs\DocsConversion\DocsConversion\installer\conversion\prepare_install.py", line 100 in do
if not self.verify_was();
File "C:\Install\IBM_Docs\DocsConversion\DocsConversion\installer\conversion\prepare_install.py", line 78, in verify_was
succ, ws_out = call_wsadmin(args)
File "C:\Install\IBM_Docs\DocsConversion\DocsConversion\installer\util\common.py", line 24, in call_wsadmin
stdout=ws_log, stderr=ws_log
File "C:\Python27\lib\subprocess.py", line 711, in ___init___
errread, errwrite)
File "C:\Python27\lib\subprocess.py", line 948. in _execute_child
startupinfo)
WindowsError: [Error 2] The System cannot find the file specified
I've already double check if the "install.bat" is on the directory and it is their.
What seems to be the problem?
Thanks!
You can try adding the path to python.exe to your windows system variables.
I'm trying to set up django-nonrel into a new virtualenv, but am getting an error saying: OSError: [Errno 2] No such file or directory.
I can't tell from the error log exactly which file or directory is missing. Please could anyone suggest what might be going wrong?
The bitbucket file seems to be present:
(nonrel)delirium:nonrel anna$ pip install hg+https://bitbucket.org/wkornewald/django-nonrel
Downloading/unpacking hg+https://bitbucket.org/wkornewald/django-nonrel
Cloning hg https://bitbucket.org/wkornewald/django-nonrel to /var/folders/q5/_0_8qh59147_0vmr8tm1_mq80000gn/T/pip-3Trv56-build
Error [Errno 2] No such file or directory while executing command /usr/local/bin/hg clone --noupdate -q https://bitbucket.org/wkornewald/django-nonrel /var/folders/q5/_0_8qh59147_0vmr8tm1_mq80000gn/T/pip-3Trv56-build
Exception:
Traceback (most recent call last):
File "/Users/anna/Dropbox/nonrel/lib/python2.7/site-packages/pip-1.1-py2.7.egg/pip/basecommand.py", line 104, in main
status = self.run(options, args)
File "/Users/anna/Dropbox/nonrel/lib/python2.7/site-packages/pip-1.1-py2.7.egg/pip/commands/install.py", line 245, in run
requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
File "/Users/anna/Dropbox/nonrel/lib/python2.7/site-packages/pip-1.1-py2.7.egg/pip/req.py", line 985, in prepare_files
self.unpack_url(url, location, self.is_download)
File "/Users/anna/Dropbox/nonrel/lib/python2.7/site-packages/pip-1.1-py2.7.egg/pip/req.py", line 1103, in unpack_url
return unpack_vcs_link(link, loc, only_download)
File "/Users/anna/Dropbox/nonrel/lib/python2.7/site-packages/pip-1.1-py2.7.egg/pip/download.py", line 294, in unpack_vcs_link
vcs_backend.unpack(location)
File "/Users/anna/Dropbox/nonrel/lib/python2.7/site-packages/pip-1.1-py2.7.egg/pip/vcs/__init__.py", line 233, in unpack
self.obtain(location)
File "/Users/anna/Dropbox/nonrel/lib/python2.7/site-packages/pip-1.1-py2.7.egg/pip/vcs/mercurial.py", line 81, in obtain
call_subprocess([self.cmd, 'clone', '--noupdate', '-q', url, dest])
File "/Users/anna/Dropbox/nonrel/lib/python2.7/site-packages/pip-1.1-py2.7.egg/pip/__init__.py", line 221, in call_subprocess
cwd=cwd, env=env)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 672, in __init__
errread, errwrite)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1202, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
Storing complete log in /Users/anna/.pip/pip.log
What is going wrong?
I'd recommend getting the latest version from github
https://github.com/django-nonrel/django-nonrel
You'd wanna use git to download it rather than pip.
s=subprocess.Popen(['/home/karthik/Downloads/stanford-parser-2011-06-08/lexparser.csh','-'],stdin=subprocess.PIPE,stdout=subprocess.PIPE,stderr=subprocess.PIPE)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/subprocess.py", line 672, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1213, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
I'm sure this file exists and open() with this filename works.Why am i getting this error ? i use python 2.7
Make sure csh is installed and is in /bin/csh (otherwise edit the command after the shebang in lexparser.sh).