Develop Reference

Cron, execute bash script as root, but one part (Python script) as user

I need to run a bash script periodically on a Jetson Nano (so, Ubuntu 18.04). The script should run system updates, pull some Python code from a repository, and run it as a specified user.
So, I created this script:
#! /bin/bash
## system updates
sudo apt update
sudo apt upgrade
## stop previous instances of the Python code
pkill python3
## move to python script folder
cd /home/user_name/projects/my_folder
## pull updates from repo
git stash
git pull
## create dummy folder to check bash script execution to this point
sudo -u user_name mkdir /home/user_name/projects/dummy_folder_00
## launch python script
sudo -u user_name /usr/bin/python3 python_script.py --arg01 --arg02
## create dummy folder to check bash script execution to this point
sudo -u user_name mkdir /home/user_name/projects/dummy_folder_01
I created a cron job running this script as root, by using
sudo crontab -e
and adding the entry
00 13 * * * /home/user_name/projects/my_folder/script.sh
Now, I can see that at the configured time, both the dummy folders are created, and they actually belong to user_name. However, the Python script isn't launched.
I tried creating the cron job as non root user (crontab -e), but at this point even if the Python script gets exectured, I guess I wouldn't be able to run apt update/upgrade.
How can I fix this?

Well, if the dummy folders did get created, that means the sudo statements work, so i'd say theres a 99%+ chance that python was infact started.
I'm guessing the problem is that you havent specified the path for the python file, and your working directory likely isn't what you're expecting it to be.
change:
sudo -u user_name /usr/bin/python3 python_script.py --arg01 --arg02
to something like
sudo -u user_name /usr/bin/python3 /path/to/your/python_script.py --arg01 --arg02
then test.
If that didn't solve the problem , then enable some logging, change the line to:
sudo -u user_name /usr/bin/python3 /path/to/your/python_script.py --arg01 --arg02 \
1> /home/user_name/projects/dummy_folder_00/log.txt 2>&1 ;
and test again, it should log STDOUT and STDERR to that file then.

Python script runs on command line but not from .sh file

I'm attempting to create a .sh file to batch a number of runs of a neural network on Python whilst on holidays.
At the moment I have been calling this from the command line:
python neural_network_trainer.py [args]
I now have a .sh script written:
#!/bin/bash
python neural_network_trainer.py [args]
# Repeated with varied args
That I am attempting to call in the same terminal as the original command line was running:
./august_hols.sh
I get the following error:
File "/data/Python-3.6.9/lib/python3.6/site.py", line 177
file=sys.stderr)
^
SyntaxError: invalid syntax
Where the Python install is in /data (for reasons).
Running which on the command line reports the correct Python directory set via an alias in ~/.bashrc:
alias python=/data/Python-3.6.9/bin/python3
But running which between the Bash shebang and the first python call reports /bin/python.
I've attempted to set the alias again at the start of the .sh script to no avail. I'm scratching my head as this is exact process I have used elsewhere, albeit not on this precise PC. I can copy the exact command from the top of the bash file into the terminal and it runs fine, try and call ./august_hols.sh and get the above Python error.
Where is Bash getting that path from, and why is it not using my expected route through ~/.bashrc?

Bash sub-shell does not inherit alias in the main shell
You can source the script (run in the main shell), instead of execute it (run in the sub-shell)
source script.sh
EDIT:
Solution 2:
Run bash as the login shell so ~/.bashrc is executed, so your alias is loaded before your script.
The subshell needs to be interactive to enable alias, because alias is enabled by default only for interactive shell, but script is non-interactive by default.
bash --login -i script.sh
Solution 3:
Similar to above, except alias is enabled explicitly
bash --login -O expand_aliases script.sh

Have you tried:
python=/data/Python-3.6.9/bin/python3 ./[your_bash].sh

In your .sh
Do this
#!/usr/bin/env bash
export PATH=/data/Python-3.6.9/bin:$PATH
exec python neural_network_trainer.py "$#"
Aliases are tricky.

A maybe more nasty solution
mapfile < <(declare -p | grep -m 1 BASH_ALIASES) && bash script.sh "${MAPFILE[#]}"
within your script you will need
shopt -s expand_aliases
eval $1
echo ${BASH_ALIASES[python]}
python --version

How about this:
#!/bin/bash
/data/Python-3.6.9/bin/python3 neural_network_trainer.py [args]
# Repeated with varied args

ec2 run scripts every boot

I have followed a few posts on here trying to run either a python or shell script on my ec2 instance after every boot not just the first boot.
I have tried the:
[scripts-user, always] to /etc/cloud/cloud.cfg file
Added script to ./scripts/per-boot folder
and
adding script to /etc/rc.local
Yes the permissions were changed to 755 for /etc/rc.local
I am attempting to pipe the output of the file into a file located in the /home/ubuntu/ directory and the file does not contain anything after boot.
If I run the scripts (.sh or .py) manually they work.
Any suggestions or request for additional info to help?

So the current solution appears to be a method I wrote off in my initial question post as I may have not performed the setup exactly as outline in the link below...
This link -->
How do I make cloud-init startup scripts run every time my EC2 instance boots?
The link shows how to modify the /etc/cloud/cloud.cfg file to update scripts-user to [scripts-user, always]
Also that link says to add your *.sh file to /var/lib/cloud/scripts/per-boot directory.
Once you reboot your system your script should have executed and you can verify this in: sudo cat /var/log/cloud-init.log
if your script still fails to execute try to erase the instance state of your server with the following command: sudo rm -rf /var/lib/cloud/instance/*
--NOTE:--
It appears print commands from a python script do not pipe (>>) as expected but echo commands pipe easily
Fails to pipe
sudo python test.py >> log.txt
Pipes successfully
echo "HI" >> log.txt

Is this something along the lines that you want?
It copies the script to the instance, connects to the instance, and runs the script right away.
ec2 scp ~/path_to_script.py : instance_name -y && ec2 ssh instance_name -yc "python script_name.py" 1>/dev/null

I read that the use of rc.local is getting deprecated. One thing to try is a line in /etc/crontab like this:
#reboot full-path-of-script
If there's a specific user you want to run the script as, you can list it after #reboot.

Linux run shell cmd from python, Failed to load config file

I have installed a backup program called rclone on my raspberry pi which is running Debian, I have successfully ran the cmd in the shell to backup a folder to google drive but I really need to be able to do so each time a take a photo with my python script, I have little experience in Linux compared to others and I thought that if I made a shell script with a basic shebang of
#!/bin/sh
or
#!/bin/bash
then the cmd below
rclone copy /var/www/html/camera_images pictures::folder1
I then made the .sh file executable, and this works if I just click it in the folder and execute but if I try to call that .sh script from python with
os.system('sh /home/pi/py/upload.sh')
or
os.system(' rclone copy /var/www/html/camera_images pictures::folder1 ')
I get an error in the shell saying
Failed to load config file "/root/.rclone.conf" using default - no such directory.
But the .conf is located in /home/pi as it should be. and if i try
os.system(' sh rclone copy /var/www/html/camera_images pictures::folder1 ')
I get
sh: 0: Cant open rclone.
How can I can run the copy cmd or a script to do so from python?
this is how i installed rclone
cd
wget http://downloads.rclone.org/rclone-v1.34-linux-arm.zip
unzip rclone-v1.34-linux-arm.zip
cd rclone-v1.34-linux-arm
sudo cp rclone /usr/sbin/
sudo chown root:root /usr/sbin/rclone
sudo chmod 755 /usr/sbin/rclone
sudo mkdir -p /usr/local/share/man/man1
sudo cp rclone.1 /usr/local/share/man/man1/
sudo mandb
rclone config

Use --config in your rclone command
From docs:
--config string Config file. (default /home/ncw/.rclone.conf")
Your command should looks like:
os.system(' sh rclone copy --config /home/pi/.rclone.conf /var/www/html/camera_images pictures::folder1 ')

You should be using subprocess module instead of os.system.
You can use subprocess.Popen to create a process and give it a working directory.
subprocess.Popen(your_command, cwd=path_to_your_executable_dir, shell=True)
(Use shell=True to pass a simple string command among other conveniences).
The shell argument (which defaults to False) specifies whether to use
the shell as the program to execute. If shell is True, it is
recommended to pass args as a string rather than as a sequence.
On Unix with shell=True, the shell defaults to /bin/sh. If args is a
string, the string specifies the command to execute through the shell.
This means that the string must be formatted exactly as it would be
when typed at the shell prompt. This includes, for example, quoting or
backslash escaping filenames with spaces in them. If args is a
sequence, the first item specifies the command string, and any
additional items will be treated as additional arguments to the shell
itself. That is to say, Popen does the equivalent of: ....

Thank you every one :)
I have it working now with
os.system(' rclone copy --config /home/pi/.rclone.conf /var/www/html/camera_images pictures::folder1 ')
Note that if i put sh at the start i got the error sh: 0: Can't open rclone though i read yesterday about putting something like ,:0 at the end as a return value ? either way it works without the sh.
and the subprocess works too which i shall use instead.
subprocess.Popen('rclone copy --config /home/pi/.rclone.conf /var/www/html/camera_images pictures::folder1', shell=True)

Run Python script at startup in Ubuntu

I have a short Python script that needs to run at startup - Ubuntu 13.10. I have tried everything I can think of but can't get it to run. The script:
#!/usr/bin/python
import time
with open("/home/username/Desktop/startup.txt", 'a') as f:
f.write(str(time.time()) + " It worked!")
(The actual script is a bit different, as I'm just using this for testing purposes, but you get the idea.)
I've tried all of the following, with no luck:
Put the command python startuptest.py in crontab, as #reboot
python /home/username/Documents/startuptest.py, both as the regular user and as sudo
Put the command python /home/username/Documents/startuptest.py in /etc/rc.local
Opened Ubuntu's Startup Applications and put the command there
Done all of the preceding, putting the command into a shell script
and calling that shell script instead
Nothing works. I get the feeling I'm missing something simple. Any ideas? (The script runs fine if I just run the command from a terminal.)

Instructions
Copy the python file to /bin:
sudo cp -i /path/to/your_script.py /bin
Add A New Cron Job:
sudo crontab -e
Scroll to the bottom and add the following line (after all the #'s):
#reboot python /bin/your_script.py &
The “&” at the end of the line means the command is run in the background and it won’t stop the system booting up.
Test it:
sudo reboot
Practical example:
Add this file to your Desktop: test_code.py (run it to check that it works for you)
from os.path import expanduser
import datetime
file = open(expanduser("~") + '/Desktop/HERE.txt', 'w')
file.write("It worked!\n" + str(datetime.datetime.now()))
file.close()
Run the following commands:
sudo cp -i ~/Desktop/test_code.py /bin
sudo crontab -e
Add the following line and save it:
#reboot python /bin/test_code.py &
Now reboot your computer and you should find a new file on your Desktop: HERE.txt

Put this in /etc/init (Use /etc/systemd in Ubuntu 15.x)
mystartupscript.conf
start on runlevel [2345]
stop on runlevel [!2345]
exec /path/to/script.py
By placing this conf file there you hook into ubuntu's upstart service that runs services on startup.
manual starting/stopping is done with
sudo service mystartupscript start
and
sudo service mystartupscript stop

If you are on Ubuntu you don't need to write any other code except your Python file's code , Here are the Steps :-
Open Dash (The First Icon In Sidebar).
Then type Startup Applications and open that app.
Here Click the Add Button on the right.
There fill in the details and in the command area browse for your Python File and click Ok.
Test it by Restarting System . Done . Enjoy !!

Create file ~/.config/autostart/MyScript.desktop
with
[Desktop Entry]
Encoding=UTF-8
Name=MyScript
Comment=MyScript
Icon=gnome-info
Exec=python /home/your_path/script.py
Terminal=false
Type=Application
Categories=
X-GNOME-Autostart-enabled=true
X-GNOME-Autostart-Delay=0
It helps me!

In similar situations, I've done well by putting something like the following into /etc/rc.local:
cd /path/to/my/script
./my_script.py &
cd -
echo `date +%Y-%b-%d_%H:%M:%S` > /tmp/ran_rc_local # check that rc.local ran
This has worked on multiple versions of Fedora and on Ubuntu 14.04 LTS, for both python and perl scripts.

nano /etc/rc.local
and edit in
python ~/path-to-script.py
worked for me