Develop Reference

getting error while running mrjob python scripting in hadoop cluster

hi i want to sort movie ratings by a python script but i am getting error
`[root#sandbox-hdp maria_dev]# python RatingsBreakdown.py -r hadoop --hadoop-streaming-jar /usr/hdp/current/hadoop-mapreduce-client/hadoop-streaming.jar u.data
No configs found; falling back on auto-configuration
No configs specified for hadoop runner
Looking for hadoop binary in $PATH...
Found hadoop binary: /usr/bin/hadoop
Using Hadoop version 3.1.1.3.0.1.0
Creating temp directory /tmp/RatingsBreakdown.maria_dev.20190830.233300.332634
STDERR: mkdir: Permission denied: user=root, access=WRITE, inode="/user/maria_dev" :maria_dev:hdfs:drwxr-xr-x
Traceback (most recent call last):
File "RatingsBreakdown.py", line 19, in <module>
RatingsBreakdown.run()
File "/usr/lib/python2.7/site-packages/mrjob/job.py", line 446, in run
mr_job.execute()
File "/usr/lib/python2.7/site-packages/mrjob/job.py", line 473, in execute
super(MRJob, self).execute()
File "/usr/lib/python2.7/site-packages/mrjob/launch.py", line 202, in execute
self.run_job()
File "/usr/lib/python2.7/site-packages/mrjob/launch.py", line 247, in run_job
return self._handle(name, path, path)
File "/usr/lib/python2.7/site-packages/mrjob/fs/composite.py", line 118, in _han dle
return getattr(fs, name)(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/mrjob/fs/hadoop.py", line 298, in mkdir
raise IOError("Could not mkdir %s" % path)
IOError: Could not mkdir hdfs:///user/maria_dev/tmp/mrjob/RatingsBreakdown.maria_d ev.20190830.233300.332634/files/wd`
can you plese describe what is the problem here

Please take a look at these 2 references:
Permission denied at hdfs
Permission denied: user=root, access=WRITE, inode="/user":hdfs:supergroup:dr

i found that hortonworks take a lot of time to boot up
when i booted up correctly it worked fine
it took about 1 hour to boot

Django API raises an error "No such file or directory: 'manage.py'" when requested

This might seem like an already asked question but I have searched for an answer for a week now and got nothing.
The problem is I have developed an API using Django which is hosted on a server. Now when I run the following command to initiate the server :
python manage.py runserver 0.0.0.0:9000
The server starts as usual. Its only when I send request to the server via "Postman" that I see the following error:
FileNotFoundError: [Errno 2] No such file or directory: 'manage.py'
The strange thing is there is no error in running the server but only when I send a request to it. Also I have many more Django APIs running on the same server with same python version(Python 3.4.3) and same virtual environment (but different port) that are running just fine.
Full error traceback :
Traceback (most recent call last):
File "manage.py", line 15, in <module>
execute_from_command_line(sys.argv)
File "/home/ubuntu/py3env/lib/python3.4/site-packages/django/core/management/__init__.py", line 371, in execute_from_command_line
utility.execute()
File "/home/ubuntu/py3env/lib/python3.4/site-packages/django/core/management/__init__.py", line 365, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/ubuntu/py3env/lib/python3.4/site-packages/django/core/management/base.py", line 288, in run_from_argv
self.execute(*args, **cmd_options)
File "/home/ubuntu/py3env/lib/python3.4/site-packages/django/core/management/commands/runserver.py", line 61, in execute
super().execute(*args, **options)
File "/home/ubuntu/py3env/lib/python3.4/site-packages/django/core/management/base.py", line 335, in execute
output = self.handle(*args, **options)
File "/home/ubuntu/py3env/lib/python3.4/site-packages/django/core/management/commands/runserver.py", line 98, in handle
self.run(**options)
File "/home/ubuntu/py3env/lib/python3.4/site-packages/django/core/management/commands/runserver.py", line 105, in run
autoreload.main(self.inner_run, None, options)
File "/home/ubuntu/py3env/lib/python3.4/site-packages/django/utils/autoreload.py", line 317, in main
python_reloader(wrapped_main_func, args, kwargs)
File "/home/ubuntu/py3env/lib/python3.4/site-packages/django/utils/autoreload.py", line 296, in python_reloader
reloader_thread()
File "/home/ubuntu/py3env/lib/python3.4/site-packages/django/utils/autoreload.py", line 274, in reloader_thread
change = fn()
File "/home/ubuntu/py3env/lib/python3.4/site-packages/django/utils/autoreload.py", line 204, in code_changed
stat = os.stat(filename)
FileNotFoundError: [Errno 2] No such file or directory: 'manage.py'
Things I have tried:
I have tried changing the !# as suggested on various posts.
I have tried using dos2unix to convert the file to unix format(server on which my API is hosted is linux based).
I even have tried to create a new Django project.
And yes I'm running manage.py from the correct directory.
I have also tried making manage.py executable by :
chmod +x manage.py
Nothing worked for me so far. Am I missing something?

`/bin/sh` not found in tox within Jenkins inside docker

I am trying to setup a Jenkins pipeline which runs tox inside a docker container. There is a known issue that shebang lines get very long inside Jenkins, and two solutions are proposed. The first is to use --workdir to select a shorter path. This option works in principle, but I loose the automatic unique path-names per project from Jenkins. I would thus prefer to use the second option, TOX_LIMITED_SHEBANG. Unfortunately, that seems to fail with the following error when the package under test is supposed to be installed: FileNotFoundError: [Errno 2] No such file or directory: "b'/bin/sh'": "b'/bin/sh'". I have verified that /bin/sh is in fact available in the docker container. The Jenkinsfile looks as follows:
node("docker") {
// burnpanck/tox-base contains tox and many python versions
docker.image('burnpanck/tox-base').inside {
checkout scm
stage('Matrix-test using Tox') {
// verify that /bin/sh exists
sh 'ls -al /bin'
// the following does not work
sh 'TOX_LIMITED_SHEBANG=1 tox -vv'
// the following works
// sh 'tox --workdir=/var/jenkins_home/tox'
}
}
}
Tox is version 3.1.2 and runs under python 3.6 (the docker image is generated from this Dockerfile). What surprises me a little is the "b'/bin/sh'" coming from str-ing a bytes instance. Could it be that tox is in fact trying to run a program by the name sh' in the path b'/bin?
The tox.ini in use simply calls pytest:
[tox]
envlist = py36
[testenv]
recreate = True
commands =
pytest
The full backtrace from tox (Jenkins console output) is the following:
py36 create: /var/jenkins_home/workspace/_debug_jenkins-long-shebang-L7UHBNCVPSOBSTKZ7COPFJBJLWR5XZXFIAD7TBGC4WQLVDLZYVQQ/.tox/py36
py36 inst: /var/jenkins_home/workspace/_debug_jenkins-long-shebang-L7UHBNCVPSOBSTKZ7COPFJBJLWR5XZXFIAD7TBGC4WQLVDLZYVQQ/.tox/dist/test_model-0.dev20180717.zip
ERROR: invocation failed (errno 2), args: [b'/bin/sh', '/var/jenkins_home/workspace/_debug_jenkins-long-shebang-L7UHBNCVPSOBSTKZ7COPFJBJLWR5XZXFIAD7TBGC4WQLVDLZYVQQ/.tox/py36/bin/pip', 'install', '/var/jenkins_home/workspace/_debug_jenkins-long-shebang-L7UHBNCVPSOBSTKZ7COPFJBJLWR5XZXFIAD7TBGC4WQLVDLZYVQQ/.tox/dist/test_model-0.dev20180717.zip'], cwd: /var/jenkins_home/workspace/_debug_jenkins-long-shebang-L7UHBNCVPSOBSTKZ7COPFJBJLWR5XZXFIAD7TBGC4WQLVDLZYVQQ
Traceback (most recent call last):
File "/.pyenv/versions/3.6.6/bin/tox", line 11, in <module>
sys.exit(cmdline())
File "/.pyenv/versions/3.6.6/lib/python3.6/site-packages/tox/session.py", line 39, in cmdline
main(args)
File "/.pyenv/versions/3.6.6/lib/python3.6/site-packages/tox/session.py", line 45, in main
retcode = Session(config).runcommand()
File "/.pyenv/versions/3.6.6/lib/python3.6/site-packages/tox/session.py", line 422, in runcommand
return self.subcommand_test()
File "/.pyenv/versions/3.6.6/lib/python3.6/site-packages/tox/session.py", line 620, in subcommand_test
self.installpkg(venv, path)
File "/.pyenv/versions/3.6.6/lib/python3.6/site-packages/tox/session.py", line 561, in installpkg
venv.installpkg(path, action)
File "/.pyenv/versions/3.6.6/lib/python3.6/site-packages/tox/venv.py", line 277, in installpkg
self._install([sdistpath], extraopts=extraopts, action=action)
File "/.pyenv/versions/3.6.6/lib/python3.6/site-packages/tox/venv.py", line 342, in _install
self.run_install_command(packages=packages, options=options, action=action)
File "/.pyenv/versions/3.6.6/lib/python3.6/site-packages/tox/venv.py", line 314, in run_install_command
redirect=self.session.report.verbosity < 2,
File "/.pyenv/versions/3.6.6/lib/python3.6/site-packages/tox/venv.py", line 427, in _pcall
return action.popen(args, cwd=cwd, env=env, redirect=redirect, ignore_ret=ignore_ret)
File "/.pyenv/versions/3.6.6/lib/python3.6/site-packages/tox/session.py", line 153, in popen
popen = self._popen(args, cwd, env=env, stdout=stdout, stderr=subprocess.STDOUT)
File "/.pyenv/versions/3.6.6/lib/python3.6/site-packages/tox/session.py", line 248, in _popen
env=env,
File "/.pyenv/versions/3.6.6/lib/python3.6/subprocess.py", line 709, in __init__
restore_signals, start_new_session)
File "/.pyenv/versions/3.6.6/lib/python3.6/subprocess.py", line 1344, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: "b'/bin/sh'": "b'/bin/sh'"

We currently have an open bug for this. TOX_LIMITED_SHEBANG is broken when tox is running under python3.
The root cause is there's a .decode() missing and a list ends up with mixed bytes and str instances.
Workarounds until I fix it are to use tox with a python2 interpreter or don't use TOX_LIMITED_SHEBANG.

DjangoQ run manage.py qcluster results in PermissionError: [WinError5] Access is denied

I've just recently started played around with web application development using Django and now got to Django-Q for async tasks.
I followed this tutorial up to slide 13. Unfortunately then, when I try to run python manage.py qcluster I get these error messages:
(djangoq_tut) D:\Code\Python\DjangoQ_Tutorial\djangoq_demo>python manage.py qcluster
Traceback (most recent call last):
File "manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "D:\Code\Python\DjangoQ_Tutorial\djangoq_tut\lib\site-packages\django\core\management\__init__.py", line 350, in execute_from_command_line
utility.execute()
File "D:\Code\Python\DjangoQ_Tutorial\djangoq_tut\lib\site-packages\django\core\management\__init__.py", line 342, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "D:\Code\Python\DjangoQ_Tutorial\djangoq_tut\lib\site-packages\django\core\management\base.py", line 348, in run_from_argv
self.execute(*args, **cmd_options)
File "D:\Code\Python\DjangoQ_Tutorial\djangoq_tut\lib\site-packages\django\core\management\base.py", line 399, in execute
output = self.handle(*args, **options)
File "D:\Code\Python\DjangoQ_Tutorial\djangoq_tut\lib\site-packages\django_q\management\commands\qcluster.py", line 22, in handle
q.start()
File "D:\Code\Python\DjangoQ_Tutorial\djangoq_tut\lib\site-packages\django_q\cluster.py", line 57, in start
self.sentinel.start()
File "c:\program files (x86)\python35-32\Lib\multiprocessing\process.py", line 105, in start
self._popen = self._Popen(self)
File "c:\program files (x86)\python35-32\Lib\multiprocessing\context.py", line 212, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "c:\program files (x86)\python35-32\Lib\multiprocessing\context.py", line 313, in _Popen
return Popen(process_obj)
File "c:\program files (x86)\python35-32\Lib\multiprocessing\popen_spawn_win32.py", line 66, in __init__
reduction.dump(process_obj, to_child)
File "c:\program files (x86)\python35-32\Lib\multiprocessing\reduction.py", line 59, in dump
ForkingPickler(file, protocol).dump(obj)
TypeError: can't pickle _thread.lock objects
(djangoq_tut) D:\Code\Python\DjangoQ_Tutorial\djangoq_demo>Traceback (most recent call last):
File "<string>", line 1, in <module>
File "c:\program files (x86)\python35-32\Lib\multiprocessing\spawn.py", line 100, in spawn_main
new_handle = steal_handle(parent_pid, pipe_handle)
File "c:\program files (x86)\python35-32\Lib\multiprocessing\reduction.py", line 86, in steal_handle
_winapi.DUPLICATE_SAME_ACCESS | _winapi.DUPLICATE_CLOSE_SOURCE)
PermissionError: [WinError 5] Access is denied
I'm using a Windows 7 machine and Python 3.5.1 (32 bit) with following requirements:
appdirs==1.4.3
arrow==0.10.0
blessed==1.14.1
Django==1.9
django-picklefield==0.3.2
django-q==0.7.18
future==0.16.0
packaging==16.8
pyparsing==2.2.0
python-dateutil==2.6.0
six==1.10.0
wcwidth==0.1.7
I tried looking up the source of the error, it looks to me as it might not be exclusively related to Django or Django-Q. Other than that I don't really understand what it means.
Also tried with Python 2.7.12, outside of a virtualenv, there I get a TypeError: can't pickle _thread.lock objects, which I don't really get either.
Any ideas how to solve this?
[Edit]
After experimenting with a DjangoQ alternative - Celery - I as able to recreate the problem:
The same error occurs if using Celery 4.0.2, but not with Celery 3.1.25
I still haven't figured out how to fix this, but decided to stick to Celery for the time being, since it kinda works...

Install the latest version with pip:
$ pip install django-q
Add django_q to INSTALLED_APPS in your projects settings.py:
INSTALLED_APPS = (
# other apps
'django_q',
)
Run Django migrations to create the database tables:
$ python manage.py migrate
Choose a message broker , configure it and install the appropriate client library.
Run Django Q cluster in order to handle tasks async:
$ python manage.py qcluster

Azure Batch Apps install python packages on startup

We are using Azure Batch Apps which will create multiple VMs which can be used to run our tasks parallelly. We are using python for data fetching tasks.
We have mentioned in the batch apps to install anaconda on the VMs when they start up. Anaconda is installed properly. We have listed out the packages(requirements.txt) we need to install to run the tasks.
pip install -r requirements.txt
Some packages get installed correclty, but some packages result in the following error,
Error [Error 6] The handle is invalid while executing command python setup.py egg_info
Exception:
Traceback (most recent call last):
File "C:\user\tasks\shared\anaconda2\lib\site-packages\pip\basecommand.py", line 209, in main
status = self.run(options, args)
File "C:\user\tasks\shared\anaconda2\lib\site-packages\pip\commands\install.py", line 310, in run
wb.build(autobuilding=True)
File "C:\user\tasks\shared\anaconda2\lib\site-packages\pip\wheel.py", line 748, in build
self.requirement_set.prepare_files(self.finder)
File "C:\user\tasks\shared\anaconda2\lib\site-packages\pip\req\req_set.py", line 360, in prepare_files
ignore_dependencies=self.ignore_dependencies))
File "C:\user\tasks\shared\anaconda2\lib\site-packages\pip\req\req_set.py", line 591, in _prepare_file
abstract_dist.prep_for_dist()
File "C:\user\tasks\shared\anaconda2\lib\site-packages\pip\req\req_set.py", line 127, in prep_for_dist
self.req_to_install.run_egg_info()
File "C:\user\tasks\shared\anaconda2\lib\site-packages\pip\req\req_install.py", line 430, in run_egg_info
command_desc='python setup.py egg_info')
File "C:\user\tasks\shared\anaconda2\lib\site-packages\pip\utils\__init__.py", line 678, in call_subprocess
cwd=cwd, env=env)
File "C:\user\tasks\shared\anaconda2\lib\subprocess.py", line 702, in __init__
errread, errwrite), to_close = self._get_handles(stdin, stdout, stderr)
File "C:\user\tasks\shared\anaconda2\lib\subprocess.py", line 823, in _get_handles
p2cread = _subprocess.GetStdHandle(_subprocess.STD_INPUT_HANDLE)
WindowsError: [Error 6] The handle is invalid
When we open the VM and give the same command, all packages get installed correctly.
I just wonder where the issue is.

It seems that the issue was caused by some limits for Azure Batch service, you can see these limits here.
According to the error information, it seems that the installation process needs to fork the subprocess, but the maximum number of tasks per computer node is 4, please see below.