Develop Reference

Python script does not restart properly with systemd

I am learning how to restart a Python script in case of error (following this tutorial, however with some small tweaks regarding filenames and alike). First things first, the needed files:
/home/myuser/Desktop/test/test.py:
from datetime import datetime
from time import sleep
path = "/home/dec13666/Desktop/test/log.txt"
while True:
with open(path, "a") as f:
now = datetime.now().strftime("%d/%m/%Y %H:%M:%S")
f.write(now+"\n")
f.close()
sleep(1)
raise Exception("Error Simulation!!!")
davidcustom.service:
[Unit]
Description=Python Script Made By Me
After=multi-user.target
[Service]
RestartSec=10
Restart=always
ExecStart=python3 /home/myuser/Desktop/test/test.py
[Install]
WantedBy=multi-user.target
Finally, the commands run:
sudo nano /etc/systemd/system/davidcustom.service
sudo systemctl daemon-reload
sudo systemctl enable davidcustom.service
sudo systemctl start davidcustom.service
sudo systemctl status davidcustom.service
The message I am getting:
● davidcustom.service - Python Script Made By Me
Loaded: loaded (/etc/systemd/system/davidcustom.service; enabled; vendor preset: enabled)
Active: activating (auto-restart) (Result: exit-code) since Sun 2022-09-04 14:12:12 EDT; 8s ago
Process: 3341 ExecStart=python3 /home/myuser/Desktop/test/test.py (code=exited, status=1/FAILURE)
Main PID: 3341 (code=exited, status=1/FAILURE)
CPU: 100ms
Notes:
When I run test.py manually, it works OK, but then that python script is run from a service (as seen here), it generates that error.
I have tried to set User=myusername and Type=simple, in davidcustom.service ([Service]), with no difference in the results.
What am I doing wrong?

ExecStart requires an absolute path to the executable, it doesn't search $PATH. So use
ExecStart=/usr/bin/python3 /home/myuser/Desktop/test/test.py
(assuming that's where python3 is installed -- you can use type python3 to get the actual location).

I inadvertently added some wrong indentation in the original code (corrected by now in the original post). That solved my issue; when running sudo systemctl status davidcustom.service, now the service is active:
● davidcustom.service - Python Script Made By Dave
Loaded: loaded (/etc/systemd/system/davidcustom.service; enabled; vendor preset: enabled)
Active: active (running) since Sun 2022-09-04 16:16:02 EDT; 23ms ago
Main PID: 4287 (python3)
Tasks: 1 (limit: 14215)
Memory: 3.0M
CPU: 16ms
CGroup: /system.slice/davidcustom.service
└─4287 python3 /home/dec13666/Desktop/test/test.py
Other suggestions which would worth keeping in mind, but were NOT necessary for my case (thanks #Barmar & #tdelaney):
Use an absolute path at ExecStart (easily obtainable by running which python command)
If you only want to use your User and/or Group, then explicitely write those parameters.
The 2 previous suggestions should be added in [Service], in your service script:
[Service]
RestartSec=10
Restart=always
ExecStart=/usr/bin/python3 /home/myuser/Desktop/test/test.py
User=youruser
Group=yourgroup
Read the comments in the original post, for other options.
Thanks.

ExecStart bash script fails; However, running it manually works

I need help with bash and systemctl service...
My problem is that my company requires me to automate the boot of all virtual machines running on our KVM server (CentOS), we have 5 physical hosts in total; however, I came across an issue on one of them, which I recently updated my VM versions, where my service <vmx.service> located under /etc/systemd/system does not run my ExecStart=/bin/bash -c './vmx.sh -lv --start --cfg config/pod8/vmx1.conf' bash script.
file: vmx.service
[Unit]
Description=Juniper vMX Router
Wants=network-online.target
After=network-online.target
After=libvirtd.service
[Service]
WorkingDirectory=/home/vMX-21.1R1/
Environment="PATH=/opt/rh/python27/root/usr/bin:/usr/lib64/qt3.3/bin:/usr/local/sbin:/usr/local/bin:/$
Type=oneshot
User=root
Group=root
RemainAfterExit=true
## Commands used to stop/start/restart the vMX
ExecStart=/bin/bash -c './path-python.sh'
ExecStart=/bin/bash -c './vmx.sh -lv --start --cfg config/pod8/vmx1.conf' <<<<<<<<<<<<<<<<<<<<<<<
ExecStart=/bin/bash -c './vmx.sh -lv --start --cfg config/pod8/vmx2.conf'
ExecStart=/bin/bash -c './vmx.sh -lv --start --cfg config/pod8/vr-device.conf'
[Install]
WantedBy=multi-user.target
Here is my service status:
Loaded: loaded (/etc/systemd/system/vmx.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Mon 2021-11-29 14:04:37 AEST; 2h 4min ago
Process: 2324 ExecStart=/bin/bash -c ./vmx.sh -lv --start --cfg config/pod8/vmx1.conf (code=exited, status=2)
Process: 1877 ExecStart=/bin/bash -c ./path-python.sh (code=exited, status=0/SUCCESS)
Main PID: 2324 (code=exited, status=2)
Nov 29 14:04:37 Juniper-KVM3 bash[2324]: import netifaces as ni
Nov 29 14:04:37 Juniper-KVM3 bash[2324]: ImportError: No module named netifaces
Nov 29 14:04:37 Juniper-KVM3 bash[2324]: Log file........................................../home....log
Nov 29 14:04:37 Juniper-KVM3 bash[2324]: ==================================================
Nov 29 14:04:37 Juniper-KVM3 bash[2324]: Aborted!. 1 error(s) and 0 warning(s)
Nov 29 14:04:37 Juniper-KVM3 bash[2324]: ==================================================
Nov 29 14:04:37 Juniper-KVM3 systemd[1]: vmx.service: main process exited, code=exited, status=2...MENT
Nov 29 14:04:37 Juniper-KVM3 systemd[1]: Failed to start Juniper vMX Router.
Nov 29 14:04:37 Juniper-KVM3 systemd[1]: Unit vmx.service entered failed state.
Nov 29 14:04:37 Juniper-KVM3 systemd[1]: vmx.service failed.
I thought that the error: ImportError: No module named netifaces would be fixed by running
pip uninstall netifaces && pip install netifaces as read in this article.
However, it did not work either, Regardless, the weirdest thing is that when I run the same script in my terminal, it works:
[root#system]# ./vmx.sh -lv --stop --cfg config/pod8/vmx1.conf
[...]
==================================================
VMX Status Verification Completed.
==================================================
Log file........................................../home/vMX-21.1R1/build/8m1/logs/vmx_1638166233.log
==================================================
Thank you for using VMX
==================================================
I made sure that SElinux is disabled:
SELinux status: disabled
It is worth noting that ./path-python.sh script runs without a problem, this script allows me to change my python27 path, otherwise my VM Installation would fail, but I can certainly say that this problem has nothing to do with python itself, as the script works only when I run it within my terminal. (My Other servers works as expected using the same scripts and bash, I have no clue what is wrong).

The ExecStart commands are run separately, not in the same shell or somesuch.
In other words, if
ExecStart=/bin/bash -c './path-python.sh'
sets up some Python-related path tweaks, they will not be available for the subsequent ExecStart commands.
Instead of a script that "changes your python27 path" I would recommend you to set up a virtualenv for that Python environment with the packages you require installed in it, so you could just use that virtualenv's interpreter in your vmx.sh. If you can't edit vmx.sh, you could set up a script that does something like
#!/bin/bash
source /opt/myvirtualenv/bin/activate
./vmx.sh -lv --start --cfg config/pod8/vmx1.conf
# ...
and run that instead; the activate script ensures that virtualenv's Python is the primary one for that shell session.

So, after playing around with virtualenv I finally made it work! Previously, when running ./vmx.sh script, it would require the system to use Python2.7; however, every time my host would reboot, if I wanted to fully automate the boot of all of my VMs, I would require to change the PATH of Python by running manually the following command onto my terminal:
echo 'export PATH=/opt/rh/python27/root/usr/bin:$PATH' >> /etc/profile
PATH=/opt/rh/python27/root/usr/bin:$PATH
export PATH
cd /opt/rh/python27/ && . enable && pip install netifaces pyyaml
Instead of running the previous commands from a .sh script, which ended up not working for me, It was pointed out that "Python-related path tweaks, will not be available for the subsequent ExecStart commands". For that reason, I did the following:
Instead of running scl enable python27 bash (which would make as default my Python Version until you logout), I decided to keep the changes by creating a script under /etc/profile.d/
#!/bin/bash
source scl_source enable python27
I installed virtualenv pip install virtualenv
Created a virtual environment virtualenv --python=/usr/bin/python2.7 vMX-ENV which I would use to install all my packages and scripts to install my VMs
Activated my Virtual Environment source /home/vMX-ENV/bin/activate
Created a Directory (vMX-*) where ./vmx.sh script will be located.
# ls
bin lib lib64 pyvenv.cfg vMX-21.1R1
Inside of that Directory, I created a bash script that will be initiated by the service when on boot, I called it: pyrun.sh, this script runs the ./vmx.sh script "Credit: AKX"
#!/bin/bash
source /home/vMX-ENV/bin/activate
./vmx.sh -lv --start --cfg config/pod17/vmx1.conf
# ...
Created the vmxd.service and ExecStart=/bin/bash -c './pyrun.sh'
By creating a virtualenv I was able to isolate my Python project by only creating a unique environment with all the needed packages and running only Python2.7 which is required to run ./vmx.sh
After rebooting the host, here is the journalctl -u vmxd.service successful output as a result:
Starting Juniper vMX Router...
==================================================
Welcome to VMX
==================================================
[...]

Adding a python daemon to the systemd

The essence:
I have created a daemon to manage some tasks on a remote platform.
It is written in python and accepts start, stop and restart arguments.
While trying to add it to the systemd (so it would start on system startup and be stopped on shutdown, etc.) I encountered a problem:
It seems to see daemon running, but I am not sure if it actually works, because restarting or requesting status returns with an error:
[user#centos ~]# systemctl restart mydaemon
Failed to restart mydaemon.service: Unit mydaemon.service failed to load: No such file or directory.
[user#centos ~]# systemctl status mydaemon
● mydaemon.service
Loaded: not-found (Reason: No such file or directory)
Active: inactive (dead)
The specifics:
The code itself follows the well-known example by Sander Marechal with very few changes. By itself it works without any problems, and properly reacts to all accepted arguments. The pid is saved in /tmp/my-daemon.pid.
The systemd service file is in the user daemons directory: /usr/lib/systemd/user/mydaemon.service, and the code is as follows:
[Unit]
Description=The user daemon
[Service]
Type=forking
ExecStart=/usr/bin/python /home/frcr/mydaemon_v01.py start
ExecStop=/usr/bin/python /home/frcr/mydaemon_v01.py stop
RestartSec=5
TimeoutSec=60
RuntimeMaxSec=infinity
Restart=always
PIDFile=/tmp/my-daemon.pid
[Install]
WantedBy=multi-user.target
systemctl returns the status of it as active, but only if provided the pid:
[user#centos ~]# systemctl status 9177
● session-481.scope - Session 481 of user user
Loaded: loaded
Drop-In: /run/systemd/system/session-481.scope.d
└─50-After-systemd-logind\x2eservice.conf, 50-After-systemd-user-sessions\x2eservice.conf, 50-Description.conf, 50-SendSIGHUP.conf, 50-Slice.conf
Active: active (running) since Tue 2016-05-17 06:24:51 EDT; 1h 43min ago
CGroup: /user.slice/user-0.slice/session-481.scope
├─8815 sshd: root#pts/0
├─8817 -bash
├─9177 python /home/user/mydaemon_v01.py start
└─9357 systemctl status 9177
I have seen a similar question here on stack overflow, but it doesn't seem to have the solution to my problem.
I assume I am missing something very obvious due to the sheer lack of experience with systemd, and I'd be extremely grateful if somebody could point it out for me or show me the right direction to move. Thanks in advance and please forgive my mad English skillz.

Enabling the daemon with a full path name worked around the issue but there is a better solution.
The issue was the service was in a user directory but was started as a system service. However /usr/lib was not the right place to add new service files anyway. That directory is for files shipped as part of operating system packages. The correct directory to add a new system service is in /etc/systemd/system See related docs about systemd paths.
You still want to enable the service to make sure it gets loaded at boot time.

After some additional googling I found a solution: I actually forgot to add the daemon to systemctl:
[root#centos ~]# systemctl enable /usr/lib/systemd/user/mydaemon.service
Created symlink from /etc/systemd/system/multi-user.target.wants/mydaemon.service to /usr/lib/systemd/user/mydaemon.service.
Created symlink from /etc/systemd/system/mydaemon.service to /usr/lib/systemd/user/mydaemon.service.
It is also worth mentioning that the absolute path is required.
The only thing left is to refresh systemctl:
[root#centos ~]# systemctl daemon-reload
After that the service is added, and
[root#centos ~]# systemctl start mydaemon
[root#centos ~]# systemctl restart mydaemon
[root#centos ~]# systemctl stop mydaemon
all work perfectly.

running web2py scheduler in production

I know that I can run the scheduler manually by using
python web2py.py -K myapp
But where should this be specified in production environment? I am using standard web2py deployment script for apache, on ubuntu.

Just to round the picture. Using Debian or other Linux distributions after 2015, the way to go is systemd. For systemd the following steps have to be taken:
Create the file /etc/systemd/system/web2py-sched.service
Containing the following
[Unit]
Description=Web2Py scheduler service
[Service]
ExecStart=/usr/bin/python /home/www-data/web2py/web2py.py -K <yourapp>
Type=simple
[Install]
WantedBy=multi-user.target
Then install the service calling:
sudo systemctl enable /etc/systemd/system/web2py-sched.service

With Ubuntu 12.04 I make it manually:
in /etc/init directory create web2py-scheduler.conf file:
description "Web2py scheduler"
start on filesystem or runlevel [2345]
stop on runlevel [!2345]
respawn
respawn limit 8 60
exec sudo -u user <path_to_web2py>/web2py.py -K <your_app>
in /etc/init.d exec:
ln -s /lib/init/upstart-job web2py-scheduler
(optional, only if you want manual startup) in /etc/init directory create the web2py-scheduler.override file:
manual

Please see
Web2Py Book
which worked for me running Ubuntu 14:
Start the scheduler as a Linux service (upstart)
To install the scheduler as a permanent daemon on Linux (w/ Upstart), put the following into /etc/init/web2py-scheduler.conf, assuming your web2py instance is installed in user's home directory, running as user, with app myapp, on network interface eth0.
description "web2py task scheduler"
start on (local-filesystems and net-device-up IFACE=eth0)
stop on shutdown
respawn limit 8 60 # Give up if restart occurs 8 times in 60 seconds.
exec sudo -u <user> python /home/<user>/web2py/web2py.py -K <myapp>
respawn
You can then start/stop/restart/check status of the daemon with:
sudo start web2py-scheduler
sudo stop web2py-scheduler
sudo restart web2py-scheduler
sudo status web2py-scheduler

nginx and supervisor setup in Ubuntu

I'm using django-gunicorn-nginx setup by following this tutorial http://ijcdigital.com/blog/django-gunicorn-and-nginx-setup/ Upto nginx setup, it is working. Then I installed supervisor, configured it and then I reboot my server and checked, it shows 502 bad gateway. I'm using Ubuntu 12.04 LTS
/etc/supervisor/conf.d/qlimp.conf
[program: qlimp]
directory = /home/nirmal/project/qlimp/qlimp.sh
user = nirmal
command = /home/nirmal/project/qlimp/qlimp.sh
stdout_logfile = /path/to/supervisor/log/file/logfile.log
stderr_logfile = /path/to/supervisor/log/file/error-logfile.log
Then I restarted supervisor and I run this command $ supervisorctl start qlimp and I'm getting this error
unix:///var/run/supervisor.sock no such file
Is there any problem in my supervisor setup?
Thanks!

That there is no socket file probably means that supervisor isn't running. A reason that it isn't running might be that your qlimp.conf file has some sort of error in it. If you do a
sudo service supervisor start
you can see whether or not this is the case. If supervisor is already running, it will say. And if it is catching an error, it will usually give you a more helpful error message than supervisorctl.

I have met the same issue as you and after several times, here comes the solution:
First remove the apt-get supervisor version:
sudo apt-get remove supervisor
Kill the backend supervisor process:
sudo ps -ef | grep supervisor
Then get the newest version(apt-get version was 3.0a8):
sudo easy_install(pip install) supervisor==3.0b2
Echo the config file(root premission):
echo_supervisord_conf > /etc/supervisord.conf
5.Start supervisord:
sudo supervisord
6.Enter supervisorctl:
sudo supervisorctl
Anything has been done! Have fun!

Try this
cd /etc/supervisor
sudo supervisord
sudo supervisorctl restart all

Are you sure that supervisord is installed and running? Is there a socket file in present at /var/run/supervisor.sock?
The error indicates that supervisorctl, the control CLI, cannot reach the UNIX socket to communicate with supervisord, the daemon.
You could also check /etc/supervisor/supervisord.conf and see if the values for the unix_http_server and supervisorctl sections match.
Note that this is a Ubuntu-level problem, not a problem with Python, Django or nginx and as such this question probably belongs on ServerFault.

On Ubuntu 16+ it seems to been caused by the switch to systemd, this workaround may fix for new servers:
# Make sure Supervisor comes up after a reboot.
$ sudo systemctl enable supervisor
# Bring Supervisor up right now.
$ sudo systemctl start supervisor
and then do check your status of iconic.conf [My example] of supervisor
$ sudo supervisorctl status iconic
PS: Make sure gunicorn should not have any problem while running.

The error may be due to that you don't have the privilege.
Maybe you can fix the error by this way, open your terminal, and input vim /etc/supervisord.conf to edit the file, search the lines
[unix_http_server]
;file=/tmp/supervisor.sock ; (the path to the socket file)
;chmod=0700 ; socket file mode (default 0700)
and delete the Semicolon in the start of the string ;file=/tmp/supervisor.sock and ;chmod=0700, restart your supervisord. I suggest you do it.

Make sure that in /etc/supervisor.conf the following two sections exists
[unix_http_server]
file=/tmp/supervisor.sock ; path to your socket file
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

You can do something like this :-
sudo touch /var/run/supervisor.sock
sudo chmod 777 /var/run/supervisor.sock
sudo service supervisor restart
It's definitely work, try this.

In my case, Supervisor was not running. To spot the issue I run:
sudo systemctl status supervisor.service
The problem was that I had my logs pointing to a non-existing directory, so I just had to create it.
I hope it helps :)

touch /var/run/supervisor.sock
sudo supervisord -c /etc/supervisor/supervisord.conf
and after
supervisorctl restart all
if you want to listen the supervisor port
ps -ef | grep supervisord
if you want kill the process
kill -s SIGTERM 2503

Create a conf file and below add lines
Remember that in order to work with Nginx, you must have to disable autostart on system boot, that you activated while installing Nginx.
https://askubuntu.com/questions/177041/nginx-disable-autostart
Note: All the supervisor processes must be on "daemon off" mode, in order to work with supervisor
[program:nginx]
command=/usr/sbin/nginx -g "daemon off;"
autostart=true
autorestart=true
startretries=5
stopasgroup=true
stopsignal=QUIT
numprocs=1
startsecs=0
process_name=WebServer(Nginx)
stderr_logfile=/var/log/nginx/error.log
stderr_logfile_maxbytes=10MB
stdout_logfile=/var/log/nginx/access.log
stdout_logfile_maxbytes=10MB
sudo supervisorctl reread && sudo supervisorctl update

I have faced this error several times -
If server is newly created instance and facing this issue
Might be because of some wrong config or mistake happened during the process, or supervisor is not enabled.
Try restarting your supervisor and reconnecting ec2
or
Try reinstalling supervisor (#Scen)
or
Try approaches mentioned by #Yuvaraj Loganathan, #Dinesh Sunny. But mostly you might end up creating a new instance.
If server was running perfectly from long time but then it suddenly stopped
and threw
unix:///var/run/supervisor.sock no such file on sudo supervisorctl status.
It may be due to high memory usage, refer below image usage was 99% of 7.69GB earlier.
You can find the above config after connecting with ec2 via ssh or putty at the top.
You can upgrade your ec2 instance or you can then delete any extra files like logs (/var/logs), zip to free up the space. But careful do not delete any system files.
Restart supervisor
sudo service supervisor restart
Check sudo supervisorctl status