I need to run a bash script periodically on a Jetson Nano (so, Ubuntu 18.04). The script should run system updates, pull some Python code from a repository, and run it as a specified user.
So, I created this script:
#! /bin/bash
## system updates
sudo apt update
sudo apt upgrade
## stop previous instances of the Python code
pkill python3
## move to python script folder
cd /home/user_name/projects/my_folder
## pull updates from repo
git stash
git pull
## create dummy folder to check bash script execution to this point
sudo -u user_name mkdir /home/user_name/projects/dummy_folder_00
## launch python script
sudo -u user_name /usr/bin/python3 python_script.py --arg01 --arg02
## create dummy folder to check bash script execution to this point
sudo -u user_name mkdir /home/user_name/projects/dummy_folder_01
I created a cron job running this script as root, by using
sudo crontab -e
and adding the entry
00 13 * * * /home/user_name/projects/my_folder/script.sh
Now, I can see that at the configured time, both the dummy folders are created, and they actually belong to user_name. However, the Python script isn't launched.
I tried creating the cron job as non root user (crontab -e), but at this point even if the Python script gets exectured, I guess I wouldn't be able to run apt update/upgrade.
How can I fix this?
Well, if the dummy folders did get created, that means the sudo statements work, so i'd say theres a 99%+ chance that python was infact started.
I'm guessing the problem is that you havent specified the path for the python file, and your working directory likely isn't what you're expecting it to be.
change:
sudo -u user_name /usr/bin/python3 python_script.py --arg01 --arg02
to something like
sudo -u user_name /usr/bin/python3 /path/to/your/python_script.py --arg01 --arg02
then test.
If that didn't solve the problem , then enable some logging, change the line to:
sudo -u user_name /usr/bin/python3 /path/to/your/python_script.py --arg01 --arg02 \
1> /home/user_name/projects/dummy_folder_00/log.txt 2>&1 ;
and test again, it should log STDOUT and STDERR to that file then.
My request is simple:
ssh to a remote server with user0
switch user to user1 using: 'sudo su user1'
list all items in current folder
My expected code:
def startRedis():
run('sudo su - user1')
print(run('ls'))
However, it ends with out: user1#server:~$
And waiting for my interactive command forever, never executing the second line. It seems sudo su opened a new shell.
Can anyone help solving this simple task?
You can set sudo_user property in env. this way fabric will switch user to the desired user.
Official doc: http://docs.fabfile.org/
Password for switching user can be specified in the env. itself to avoid getting a prompt when the method is invoked.
fabfile.py
from fabric.api import env, sudo
env.sudo_user='user1'
env.password = '***'
def list_items():
sudo('ls')
Run below command & specify the hosts after -H
fab -H host1 list_items
I have the most simple script called update.sh
#!/bin/sh
cd /home/pi/circulation_of_circuits
git pull
When I call this from the terminal with ./update.sh I get a Already up-to-date or it updates the files like expected.
I also have a python script, inside that scipt is:
subprocess.call(['./update.sh'])
When that calls the same script I get:
Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
(I use SSH).
----------------- update --------------------
Someone else had a look for me:
OK so some progress. When I boot your image I can't run git pull in
your repo directory and the bash script also fails. It seems to be
because the bitbucket repository is private and needs authentication
for pull (the one I was using was public so that's why I had no
issues). Presumably git remembers this after you type it in the first
time, bash somehow tricks git into thinking it's you typing the
command subsequently but running it from python isn't the same.
I'm not a git expert but there must be some way of setting this up so
python can provide the authentication.
sounds like you need to give your ssh command a public or private key it can access perhaps:
ssh -i /backup/home/user/.ssh/id_dsa user#unixserver1.nixcraft.com
-i tells it where to look for the key
This problem is caused by the git repo authentication failing. You say you are using SSH, and git is complaining about publickey auth failing. Normally you can use git commands on a private repo without inputting a password. All this would imply that git is using ssh, but in the latter case it cannot find the correct private key.
Since the problem only manifests itself when run through another script, it is very likely caused by something messing with the environment variables. Subprocess.call should pass the environment as is, so there are a couple of usual suspects:
sudo.
if you are using sudo, it will pass a mostly empty environment to the process
the python script itself
if the python script changes its env, those changes will get propagated to the subprocess too.
sh -lor su -
these commands set up a login shell, which means their environment gets reset to defaults.
Any of these reasons could hide the environment variables ssh-agent (or some other key management tool) might need to work.
Steps to diagnose and fix:
Isolate the problem.
Create a minimal python script that does nothing else than runs subprocess.call(['./update.sh']). Run both update.sh and the new script.
Diagnose the problem and fix accordingly:
a) If update.sh works, and the new script doesn't, you are probably experiencing some weird corner case of system misconfiguration. Try upgrading your system and python; if the problem persists, it probably requires additional debugging on the affected system itself.
b) If both update.sh and the new script work, then the problem lies within the outer python script calling the shell script. Look for occurrences of sudo, su -, sh -l, env and os.environ, one of those is the most likely culprit.
c) If neither the update.sh nor the new script work, your problem is likely to be with ssh client configuration; a typical cause would be that you are using a non-default identity, did not configure it in ~/.ssh/config but used ssh-add instead, and after that, ssh-agent's cache expired. In this case, run ssh-add identityfile for the identity you used to authenticate to that git repo, and try again.
I believe this answer will help you: https://serverfault.com/questions/497217/automate-git-pull-stuck-with-keychain?answertab=votes#tab-top
I didn't use ssh-agent and it worked: Change your script to the one that follows and try.
#!/bin/bash
cd /home/pi/circulation_of_circuits
ssh-add /home/yourHomefolderName/.ssh/id_rsa
ssh-add -l
git pull
This assumes that you have configured correctly your ssh key.
It seems like your version control system, need the authentication for the pull so can build the python with use of pexpect,
import pexpect
child = pexpect.spawn('./update.sh')
child.expect('Password:')
child.sendline('SuperSecretPassword')
Try using the sh package instead of using the subprocess call. https://pypi.python.org/pypi/sh
I tried this snippet and it worked for me.
#!/usr/local/bin/python
import sh
sh.cd("/Users/siyer/workspace/scripts")
print sh.git("pull")
Output:
Already up-to-date.
import subprocess
subprocess.call("sh update.sh", shell=True)
With Git 1.7.9 or later, you can just use one of the following credential helpers:
With a timeout
git config --global credential.helper cache
... which tells Git to keep your password cached in memory for (by default) 15 minutes. You can set a longer timeout with:
git config --global credential.helper "cache --timeout=3600"
(That example was suggested in the GitHub help page for Linux.) You can also store your credentials permanently if so desired.
Saving indefinitely
You can use the git-credential-store via
git config credential.helper store
GitHub's help also suggests that if you're on Mac OS X and used Homebrew to install Git, you can use the native Mac OS X keystore with:
git config --global credential.helper osxkeychain
For Windows, there is a helper called Git Credential Manager for Windows or wincred in msysgit.
git config --global credential.helper wincred # obsolete
With Git for Windows 2.7.3+ (March 2016):
git config --global credential.helper manager
For Linux, you can use gnome-keyring(or other keyring implementation such as KWallet).
Finally, after executing one of the suggested command one time manually, you can execute your script without changes in it.
I can reproduce your fault. It has nothing to do with permission, it depends how your ssh are installed on your system. To verify it's the same cause i need the diff output.
Save the following to a file log_shell_env.sh,
#!/bin/bash
log="shell_env"$1
echo "create shell_env"$1
echo "shell_env" > $log
echo "whoami="$(whoami) >> $log
echo "which git="$(which git) >> $log
echo "git status="$(git status 2>&1) >> $log
echo "git pull="$(git pull 2>&1) >> $log
echo "ssh -vT git#github.com="$(ssh -T git#github.com 2>&1) >> $log
echo "ssh -V="$(ssh -V 2>&1) >> $log
echo "ls -al ~/.ssh="$(ls -a ~/.ssh) >> $log
echo "which ssh-askpass="$(which ssh-askpass) >> $log
echo "ps -e | grep [s]sh-agent="$(ps -e | grep [s]sh-agent ) >> $log
echo "ssh-add -l="$(ssh-add -l) >> $log
echo "set=" >> $log
set >> $log
set execute permission and run it twice:
1. From the console without parameter
2. From your python script with parameter '.python'
Please, run it realy from the same python script!
For instance:
try:
output= subprocess.check_output(['./log_shell_env.sh', '.python'], stderr=subprocess.STDOUT)
print(output.decode('utf-8'))
except subprocess.CalledProcessError as cpe:
print('[ERROR] check_output: %s' % cpe)
Do a diff shell_env shell_env.python > shell_env.diff
The resulting shell_env.diff should show not more than the following diffs:
15,16c15,16
< BASH_ARGC=()
< BASH_ARGV=()
---
> BASH_ARGC=([0]="1")
> BASH_ARGV=([0]=".python")
48c48
< PPID=2209
---
> PPID=2220
72c72
< log=shell_env
---
> log=shell_env.python
Come back and comment, if you get more diffs
update your Question with the diff output.
Use the following python code. This will import the os module in python and make a system call with sudo permissions.
#!/bin/python
import os
os.system("sudo ./update.sh")
I have installed Mongodb 3.0 using this tutorial -
https://docs.mongodb.com/v3.0/tutorial/install-mongodb-on-amazon/
It has installed fine. I have also given permissions to 'ec2-user' to all the data and log folders ie var/lib/mongo and var/log/mongodb but and have set conf file as well.
Now thing is that mongodb server always fails to start with command
sudo service mongod start
it just say failed, nothing else.
While if I run command -
mongod --dbpath var/lib/mongo
it starts the mongodb server correctly (though I have mentioned same dbpath in .conf file as well)
What is it I am doing wrong here?
When you run sudo mongod it does not load a config file at all, it literally starts with the compiled in defaults - port 27017, database path of /data/db etc. - that is why you got the error about not being able to find that folder. The "Ubuntu default" is only used when you point it at the config file (if you start using the service command, this is done for you behind the scenes).
Next you ran it like this:
sudo mongod -f /etc/mongodb.conf
If there weren't problems before, then there will be now - you have run the process, with your normal config (pointing at your usual dbpath and log) as the root user. That means that there are going to now be a number of files in that normal MongoDB folder with the user:group of root:root.
This will cause errors when you try to start it as a normal service again, because the mongodb user (which the service will attempt to run as) will not have permission to access those root:root files, and most notably, it will probably not be able to write to the log file to give you any information.
Therefore, to run it as a normal service, we need to fix those permissions. First, make sure MongoDB is not currently running as root, then:
cd /var/log/mongodb
sudo chown -R mongodb:mongodb .
cd /var/lib/mongodb
sudo chown -R mongodb:mongodb .
That should fix it up (assuming the user:group is mongodb:mongodb), though it's probably best to verify with an ls -al or similar to be sure. Once this is done you should be able to get the service to start successfully again.
If you’re starting mongod as a service using:
sudo service mongod start
Make sure the directories defined for logpath, dbpath, and pidfilepath in your mongod.conf exist and are owned by mongod:mongod.
First I created a new group tcpdumpers, added current user to that group, and then I edited /etc/sudoers according to the top answer of this link: Running commands from within python that need root access
import os
os.system("% sudo tcpdump")
os.system("cd /var/www/tbg/media/uploads/")
os.system("mkdir " + str(request.user.id))
os.system("cat .htaccess")
os.system("chown www-data:www-data .htaccess")
myfile = open(".htaccess", "a")
This yields the error Permission denied: .htaccess
This is Apache within Django, so request.user.id is the user object ID in Django.
I also used the touch command instead of cat which yielded identical results. It seems that Python has no rights at all no matter what I do.
EDIT: I'd like to point out that no directory is created either with the mkdir command. So the problem starts there, not with .htaccess. It doesn't even exist.