Develop Reference

Unable to install packages in AWS EC2 instance target directory using pip install -t

I am following this video on Building a Serverless Machine Learning Inference API with AWS Lambda
Inside my AWS logged in system, I made the directories, copy pasted the requirements.txt file and used the following commands:
pip3 install -t /mnt/efs/fs1/ml/lib --no-cache-dir -r /mnt/efs/fs1/ml/requirements.txt
doing so , gave me error:
ERROR: Exception:
Traceback (most recent call last):
File "/usr/lib64/python3.7/shutil.py", line 566, in move
os.rename(src, real_dst)
OSError: [Errno 18] Invalid cross-device link: '/tmp/pip-target-41qwbfdl/lib/python/zipp.py' -> '/mnt/efs/fs1/ml/lib/zipp.py'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 216, in _main
status = self.run(options, args)
File "/usr/lib/python3.7/site-packages/pip/_internal/cli/req_command.py", line 182, in wrapper
return func(self, options, args)
File "/usr/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 490, in run
options.target_dir, target_temp_dir, options.upgrade
File "/usr/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 548, in _handle_target_dir
target_item_dir
File "/usr/lib64/python3.7/shutil.py", line 580, in move
copy_function(src, real_dst)
File "/usr/lib64/python3.7/shutil.py", line 266, in copy2
copyfile(src, dst, follow_symlinks=follow_symlinks)
File "/usr/lib64/python3.7/shutil.py", line 121, in copyfile
with open(dst, 'wb') as fdst:
PermissionError: [Errno 13] Permission denied: '/mnt/efs/fs1/ml/lib/zipp.py'
then I tried using the command and while installing torch it just prints *Killed.
I tried using
sudo chown ec2-user:ec2-user /mnt/efs/fs1/ml
but same issue. I also tried with sudo pip3 install but then could not install.

Seems your EC2 EFS target destiny is write-protected by default. You can undo that by following this aws tutorial: https://aws.amazon.com/pt/premiumsupport/knowledge-center/efs-enable-read-write-access/

conda: Could not find conda environment (linux, bash script)

I have several bash scripts that I define in the crontab planner to run. This is what the script looks like:
#!/bin/bash
source /home/administrator/anaconda3/bin/activate
python /share/****/VK_RU_YAR.py
conda deactivate
As you can see, anaconda is installed under the administrator. But I myself work under a different user, but I have sudo rights. How can I apply them correctly in the script so that everything works.
Error:
Could not find conda environment:
You can list all discoverable environments with `conda info --envs`.
Traceback (most recent call last):
File "/share/RosAtom-V/Work/VK_RU_YAR.py", line 17, in <module>
from check_size import checking
File "/share/RosAtom-V/Work/check_size.py", line 6, in <module>
set_log(log_conf)
File "/share/RosAtom-V/Work/clogs.py", line 4, in set_log
fileh = logging.FileHandler(file, 'a')
File "/home/administrator/anaconda3/lib/python3.8/logging/__init__.py", line 1143, in __init__
StreamHandler.__init__(self, self._open())
File "/home/administrator/anaconda3/lib/python3.8/logging/__init__.py", line 1172, in _open
return open(self.baseFilename, self.mode, encoding=self.encoding)
PermissionError: [Errno 13] Permission denied: '/share/RosAtom-V/Logs/log_script.txt'
Separate execution of the python script does not cause errors.

Conda Update fails with PermissionError

I'm on Mac OS 10.9 and running Python 3.5.2
I tried updating conda
conda update conda
but it gave me this error
Traceback (most recent call last):
File "/Users/cd/anaconda3/bin/conda", line 6, in <module>
sys.exit(main())
File "/Users/cd/anaconda3/lib/python3.5/site-packages/conda/cli/main.py", line 139, in main
File "/Users/cd/anaconda3/lib/python3.5/site-packages/conda/cli/main.py", line 146, in args_func
File "/Users/cd/anaconda3/lib/python3.5/site-packages/conda/cli/main_update.py", line 64, in execute
File "/Users/cd/anaconda3/lib/python3.5/site-packages/conda/cli/install.py", line 424, in install
File "/Users/cd/anaconda3/lib/python3.5/site-packages/conda/plan.py", line 539, in execute_actions
File "/Users/cd/anaconda3/lib/python3.5/site-packages/conda/instructions.py", line 149, in execute_instructions
File "/Users/cd/anaconda3/lib/python3.5/site-packages/conda/instructions.py", line 92, in LINK_CMD
File "/Users/cd/anaconda3/lib/python3.5/site-packages/conda/instructions.py", line 88, in link
File "/Users/cd/anaconda3/lib/python3.5/site-packages/conda/install.py", line 634, in link
File "/Users/cd/anaconda3/lib/python3.5/os.py", line 241, in makedirs
mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/Users/cd/anaconda3/etc/fish/conf.d’
I searched around and thought the problem was that I needed to chown the folder with my user (which is 'cd' in my case):
sudo chown -R cd anaconda3
Then I tried conda update conda, but now it gives a different error:
zsh: /Users/cd/anaconda3/bin/conda: bad interpreter: /opt/anaconda1anaconda2anaconda3/bin/python3: no such file or directory
I tried putting chownback to root:
sudo chown -R root anaconda3
but when I tried conda update conda it gave me the same "bad interpreter" error.
where conda gives /Users/cd/anaconda3/bin/conda
I don't know if the problem was the chowncommand or not, but please let me know if you need any more details!

Your shebang line in /Users/USER/anaconda/bin/conda is broken:
#!/opt/anaconda1anaconda2anaconda3/bin/python
just switch it back to point to the link of your install of Anaconda:
#!/Users/USER/anaconda/bin/python

Error when trying to setting up the CKAN filestore with local storage: Permission Denied

I set up the local file storage according to these steps
http://docs.ckan.org/en/latest/filestore.html
Create location
sudo mkdir -p /var/lib/ckan/default
I confirm the location exists and is in the right location
I uncomment the lines
ofs.impl = pairtree
ofs.storage_dir = /var/lib/ckan/default
I confirmed I have pairtree and argparse installed
I am using jetty as the web server not Apache so I do
sudo chown jetty /var/lib/ckan/default
sudo chmod u+rwx /var/lib/ckan/default
sudo service jetty restart
Then I run this command to start my site
paster serve /etc/ckan/default/development.ini
I then get this stack trace back, I am doing all of these commands inside my virtual environment
2013-12-04 17:39:46,369 WARNI [ckan.lib.uploader] Please use config option ckan.storage_path instaed of
ofs.storage_path
Traceback (most recent call last):
File "/usr/lib/ckan/default/bin/paster", line 9, in <module>
load_entry_point('PasteScript==1.7.5', 'console_scripts', 'paster')()
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 104, in run
invoke(command, command_name, options, args[1:])
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 143, in invoke
exit_code = runner.run(args)
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 238, in run
result = self.command()
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/serve.py", line 284, in command
relative_to=base, global_conf=vars)
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/serve.py", line 321, in loadapp
**kw)
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 247, in loadapp
return loadobj(APP, uri, name=name, **kw)
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 272, in loadobj
return context.create()
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create
return self.object_type.invoke(self)
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 146, in invoke
return fix_call(context.object, context.global_conf, **context.local_conf)
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/deploy/util.py", line 56, in fix_call
val = callable(*args, **kw)
File "/usr/lib/ckan/default/src/ckan/ckan/config/middleware.py", line 156, in make_app
os.makedirs(path)
File "/usr/lib/ckan/default/lib/python2.7/os.py", line 157, in makedirs
mkdir(name, mode)
OSError: [Errno 13] Permission denied: '/var/lib/ckan/default/storage'

Permission denied: '/var/lib/ckan/default/storage'
It looks like you don't have permission to read and write to this directory. Try running these commands in a terminal:
sudo chown -R `whoami` /var/lib/ckan/default
sudo chmod -R u+rwx /var/lib/ckan/default

For those stumbling on this years later...
I ran into this when trying to setup datastore and run the paster --plugin=ckan datastore set-permissions -c /etc/ckan/default/production.ini command.
I had already setup FileStore as per the docs. What I ended up having to do was upload a resource in the web UI then re-run the command above and the error no longer appeared and the output worked as expected. After uploading a resource the /var/lib/ckan/default/resources and /var/lib/ckan/default/storage directories were created with the proper permissions and the paster command didn't need to try and create them (or so I'm assuming).
Here's some additional resources in case you have a slightly different issue that's causing this error:
https://github.com/ckan/ckan/issues/3676
https://github.com/ckan/ckan/issues/4548
https://lists.okfn.org/pipermail/ckan-dev/2018-November/022974.html

Sample 500 error in ckan 2.9
File "/usr/lib/python3.8/os.py", line 223, in makedirs
mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/var/lib/ckan/default/webassets/.webassets-cache'
To remove the permission error;
Set the permission and ownership. The docs on filestore misses out on the ownership and recursive mode
sudo chown www-data:www-data -R /var/lib/ckan/default
sudo chmod u+rwx -R /var/lib/ckan/default
Finally restart your server:
sudo supervisorctl restart ckan-uwsgi:*

Why does virtualenv throw an error when I try to define the python version as 2.7?

I tried to create a new virtualenv directory with sudo virtualenv curdir -p /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7 and it threw the following error:
Running virtualenv with interpreter /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7
Traceback (most recent call last):
File "/usr/local/bin/virtualenv", line 9, in <module>
load_entry_point('virtualenv==1.6.4', 'console_scripts', 'virtualenv')()
File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/site-packages/virtualenv.py", line 785, in main
popen = subprocess.Popen([interpreter, file] + sys.argv[1:], env=env)
File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/subprocess.py", line 741, in __init__
restore_signals, start_new_session)
File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/subprocess.py", line 1356, in _execute_child
raise child_exception_type(errno_num, err_msg)
OSError: [Errno 13] Permission denied
I understand that I was not allowed to do that, but why? python 2.7 is located there and I want to use it. Is there any way to use it in my virtualenv?
I hope that this isn't too basic of a question. I am still pretty new to Unix command line.

You have to point to the python executable, which you are not doing here. Its located at /Library/Frameworks/Python.framework/Versions/2.7/bin/python. Run this
sudo virtualenv curdir -p /Library/Frameworks/Python.framework/Versions/2.7/bin/python