Develop Reference

Server Error while saving panda dataframe to csv

part of my python script is as below
panda_dataframe.to_csv("myfile.csv")
The code is working fine in my Dev System.
I have deployed it in EC2 - Ubuntu. The above statement return Server Error (500)
I have also tried with
panda_dataframe.to_csv("/usr/share/myfile.csv")
But the same error.
Error.log has the below error.
PermissionError: [Errno 13] Permission denied: '/usr/share/myfile.csv'
I have tried with setting the below permisions
sudo setfacl -m u:www-data:rw /var/www/sitefolder/myfile.csv
sudo setfacl -m g:www-data:rw /var/www/sitefolder/myfile.csv
Guide me what is the permission I have to assign to write the dataframe into a csv file

Your working directory is /usr/share, and saving/deleting/copy/move actions in /user directory requires sudo privileges. so first check you user directory by executing echo $HOME. say the result is /home/amir. now save you data frame as:
panda_dataframe.to_csv("/home/amir/myfile.csv")

pysftp can't create log file - permission denied

I have a small service written in Python 3 which uses pysftp:
with pysftp.Connection(
host=host,
username=connection_data["user"],
port=connection_data["port"], log=log_file, cnopts=cnopts
) as srv:
…
and when I run it (python3 pythonprog.py) I get the following error:
PermissionError: [Errno 13] Permission denied: '/mydisk/folder/logs/pysftp-20181127-231208.log'
Obviously, I don't get this error if I run it with sudo python3 pythonprog.py.
I checked the permissions for this folder:
ls -l
drwxrwxrwx+ 2 myuser myuser 4096 Nov 27 22:38 logs
I also changed ACL with setfacl. Basically, whatever I do the error is still there. How can I grant this permission?

The service is running as someone -- that someone needs permission to write to that file or folder. This is not a code problem, its a permission problem. sudo is not the solution to running the code. I'm taking it that it is running as you? Can you write to that file/folder?

scikit-learn joblib: Permission error importing, run in Serial mode

The following permission error occurs when I try importing joblib from script or python -c 'import joblib':
/usr/local/lib/python2.7/dist-packages/joblib//joblib_multiprocessing_helpers.py:29: UserWarning: [Errno 13] Permission denied. joblib will operate in serial mode
warnings.warn('%s. joblib will operate in serial mode' % (e,))
joblib is installed and forcefully reinstalled via pip
importing works as superuser
I set full permissions on the joblib folder chmod -R 777 /usr/local/lib/python2.7/dist-packages/joblib; to no avail: the permission error remains
So even though every user and group has full rwx permissions on the joblib directory it gives me a permission error. How do I figure out on which directory joblib does the write permission check when importing? Why does it even do the check before I specified a write operation?

Found it by looking a in the joblib source code:
The problem was that semaphoring was not enabled on my system: Joblib checks for multiprocessing.Semaphore() and it turns out that only root had read/write permission on shared memory in /dev/shm. Fixed it as per this answer by permanently setting the correct permissions (even after a reboot) by adding the following to your /etc/fstab:
none /dev/shm tmpfs rw,nosuid,nodev,noexec 0 0 and then remount mount /dev/shm -o remount

Ansible with Github: Permission denied (Publickey)

I'm trying to understand the GitHub ssh configuration with Ansible (I'm working on the Ansible: Up & Running book). I'm running into two issues.
Permission denied (publickey) -
When I first ran the ansible-playbook mezzanine.yml playbook, I got a permission denied:
failed: [web] => {"cmd": "/usr/bin/git ls-remote '' -h refs/heads/HEAD", "failed": true, "rc": 128}
stderr: Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
msg: Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
FATAL: all hosts have already failed -- aborting
Ok, fair enough, I see several people have had this problem. So I jumped to appendix A on running Git with SSH and it said to run the ssh-agent and add the id_rsa public key:
eval `ssh-agent -s`
ssh-add ~/.ssh/id_rsa
Output: Identity AddedI ran ssh-agent -l to check and got the long string: 2048 e3:fb:... But I got the same output. So I checked the Github docs on ssh key generations and troubleshooting which recommended updating the ssh config file on my host machine:
Host github.com
User git
Port 22
Hostname github.com
IdentityFile ~/.ssh/id_rsa
TCPKeepAlive yes
IdentitiesOnly yes
But this still provides the same error. So at this point, I start thinking it's my rsa file, which leads me to my second problem.
Key Generation Issues - I tried to generate an additional cert to use, because the Github test threw another "Permission denied (publickey)" error.
Warning: Permanently added the RSA host key for IP address '192.30.252.131' to the list of known hosts.
Permission denied (publickey).
I followed the Github instructions from scratch and generated a new key with a different name.
ssh-keygen -t rsa -b 4096 -C "me#example.com"
I didn't enter a passphrase and saved it to the .ssh folder with the name git_rsa.pub. I ran the same test and got the following:
$ ssh -i ~/.ssh/git_rsa.pub -T git#github.com
###########################################################
# WARNING: UNPROTECTED PRIVATE KEY FILE! #
###########################################################
Permissions 0644 for '/Users/antonioalaniz1/.ssh/git_rsa.pub' are too open.
It is required that your private key files are NOT accessible by others.
This private key will be ignored.
bad permissions: ignore key: ~/.ssh/github_rsa.pub
Permission denied (publickey).
I checked on the permissions and did a chmod 700 on the file and I still get Permission denied (publickey). I even attempted to enter the key into my Github account, but first got a message that the key file needs to start with ssh-rsa. So I started researching and hacking. Started with just entering the long string in the file (it started with --BEGIN PRIVATE KEY--, but I omitted that part after it failed); however, Github's not accepting it, saying it's invalid.
This is my Ansible command in the YAML file:
- name: check out the repository on the host
git: repo={{ repo_url }} dest={{ proj_path }} accept_hostkey=yes
vars:
repo_url: git#github.com:lorin/mezzanine-example.git
This is my ansible.cfg file with ForwardAgent configured:
[defaults]
hostfile = hosts
remote_user = vagrant
private_key_file = .vagrant/machines/default/virtualbox/private_key
host_key_checking = False
[ssh_connection]
ssh_args = -o ControlMaster=auto -o ControlPersist=60s -o ForwardAgent=yes
The box is an Ubuntu Trusty64 using Mac OS. If anyone could clue me into the file permissions and/or Github key generation, I would appreciate it.

I suspect the key permissions issue is because you are passing the public key instead of the private key as the arugment to "ssh -i". Try this instead:
ssh -i ~/.ssh/git_rsa -T git#github.com
(Note that it's git_rsa and not git_rsa.pub).
If that works, then make sure it's in your ssh-agent. To add:
ssh-add ~/.ssh/git_rsa
To verify:
ssh-add -l
Then check that Ansible respects agent forwarding by doing:
ansible web -a "ssh-add -l"
Finally, check that you can reach GitHub via ssh by doing:
ansible web -a "ssh -T git#github.com"
You should see something like:
web | FAILED | rc=1 >>
Hi lorin! You've successfully authenticated, but GitHub does not provide shell access.

I had the same problem, it took me some time, but I have found the solution.
The problem is the URL is incorrect.
Just try to change it to:
repo_url: git://github.com/lorin/mezzanine-example.git

I ran into this issue and discovered it by turning verbosity up on the ansible commands (very very useful for debugging).
Unfortunately, ssh often throws error messages that don't quite lead you in the right direction (aka permission denied is very generic...though to be fair that is often thrown when there is a file permission issue so perhaps not quite so generic). Anyways, running the ansible test command with verbose on helps recreate the issue as well as verify when it is solved.
ansible -vvv all -a "ssh -T git#github.com"
Again, the setup I use (and a typical one) is to load your ssh key into the agent on the control machine and enable forwarding.
steps are found here Github's helpful ssh docs
it also stuck out to me that when I ssh'd to the box itself via the vagrant command and ran the test, it succeeded. So I had narrowed it down to how ansible was forwarding the connection. For me what eventually worked was setting
[paramiko_connection]
record_host_keys = False
In addition to the other config that controls host keys verification
host_key_checking = False
which essentially adds
-o StrictHostKeyChecking=no
to the ssh args for you, and
-o UserKnownHostsFile=/dev/null
was added to the ssh args as well
found here:
Ansible issue 9442
Again, this was on vagrant VMs, more careful consideration around host key verification should be taken on actual servers.
Hope this helps

Twisted FTP server does not show files after login

I'm running twistd -n ftp --root=/tmp --password-file=/tmp/pass.dat
I can connect with ftp ftp://localhost:2121 and run ls. However if I login, and try the same ls then I get the following error:
550 []: No such file or directory.
Any ideas?
Thanks, Miki

I had the same problem until I found this in bugtracer: http://twistedmatrix.com/trac/ticket/4494
This code
avatar = FTPShell(filepath.FilePath("/home/" + avatarId))
means that your FTP login should be the same as your Linux login, so that "/home/login" directory would exist.