After reading the uWSGI's documentation on reloading, my understanding was that, for an app that uses lazy-apps, writing w to uWSGI's master FIFO should trigger a restart of all workers (and hence activate changes in the Python code).
However, that doesn't seem to work for me. I need to restart the systemd service (systemctl restart myservice) for code changes to take effect. Am I misunderstanding the documentation, or is there an issue with my setup?
My myservice.service file looks like this:
...
ExecStart=/usr/lib/myservice/virtualenv/bin/uwsgi --ini /etc/myservice/uwsgi.ini
ExecReload=/bin/echo 'w' > /run/myservice/masterfifo
ExecStop=/bin/kill -INT $MAINPID
...
In particular, systemctl reload myservice should write w to the master FIFO. I can see from the logs in systemctl status myservice that the reload was executed, but the responses to HTTP requests tell me that the old code is still active.
My /etc/myservice/uwsgi.ini like this:
[uwsgi]
processes = 16
procname-master = myservice
master-fifo = /run/myservice/masterfifo
touch-chain-reload
listen = 128
thunder-lock
reload-on-as = 4096
limit-as = 8192
max-requests = 2000
; termination options
vacuum
die-on-term
; application
chdir = /usr/lib/myservice
virtualenv = /usr/lib/myservice/virtualenv
module = myservice.uwsgi
callable = app
master
need-app
enable-threads
lazy = True
lazy-apps = True
; logging
logto = /var/log/myservice/uwsgi.log
log-maxsize = 5242880
logdate = [%%Y/%%m/%%d %%H:%%M:%%S]
disable-logging
; stats server
stats-server = :8201
memory-report
; unix socket config (nginx->uwsgi)
socket = /run/myservice/myservice.sock
chown-socket = api
chmod-socket = 660
I'm running version 2.0.19.1 of uWSGI.
All I know about uWSGI is that it exists, but I noticed a mistake here:
ExecReload=/bin/echo 'w' > /run/myservice/masterfifo
The man page explains the problem:
This syntax is inspired by shell syntax, but only the meta-characters
and expansions described in the following paragraphs are understood,
and the expansion of variables is different. Specifically, redirection
using "<", "<<", ">", and ">>", pipes using "|", running programs in
the background using "&", and other elements of shell syntax are not
supported.
In other words, no redirection is taking place and echo will simply receive 3 arguments to print: char w, char > and the string /run/myservice/masterfifo.
Try this instead:
ExecReload=/bin/sh -c '/bin/echo w > /run/myservice/masterfifo'
Related
I'm using Fabric 2 and I'm trying to run a shell script on a number of hosts sequentially. In the script, it configures a few settings and reboots that host. When I run my task however it ends after the script has run on the first host (I'm guessing because the SSH connection terminates due to the reboot). I tried looking into setting 'warn_only' to True, but I don't see where to set this value on Fabric 2.
Adding:
with settings(warn_only=True):
throws a "NameError: global name 'settings' is not defined" error.
Is there a correct format to warn_only? If not possible yet in Fabric 2, is there a way to continue running my task regardless of this reboot?
My script:
from fabric import *
import time
import csv
#task
def test(ctx):
hosts = ['1.1.1.1', '2.2.2.2']
for host in hosts:
c = Connection(host=host, user="user", connect_kwargs={"password": "password"})
c.run("./shell_script.sh")
configured = False
while not configured:
result = c.run("cat /etc/hostname")
if result != "default": configured = True
time.sleep(10)
So a workaround has been to run the task with the -w flag which will enable warn_only and give me the desired functionality. It would be more preferable to be able to set this property in the code though.
Looks like you can use the config argument to the connection class instead of the with settings() construction, and warn_only as been renamed to warn;
with Connection(host=host, user="user", connect_kwargs={"password": "password"}) as c:
c.run("./shell_script.sh", warn=True)
More generally, you can get upgrade documentation at https://www.fabfile.org/upgrading.html#run
i created a script which runs on boot which checks if there's an internet connection on my raspberry pi, and at the same time updates the time (care of ntp) - via os.system().
import datetime, os, socket, subprocess
from time import sleep
dir_path = os.path.dirname(os.path.abspath(__file__))
def internet(host="8.8.8.8"):
result = subprocess.call("ping -c 1 "+host, stdout=open(os.devnull,'w'), shell=True)
if result == 0:
return True
else:
return False
timestr = time.strftime("%Y-%m-%d--%H:%M:%S")
netstatus = internet()
while netstatus == False:
sleep(30)
netstatus = internet()
if netstatus == True:
print "successfully connected! updating time . . . "
os.system("sudo bash "+dir_path+"/updatetime.sh")
print "time updated! time check %s"%datetime.datetime.now()
where updatetime.sh contains the following:
service ntp stop
ntpd -q -g
service ntp start
this script runs at reboot/boot and i'm running this in our workplace, 24/7. also, outputs from scripts like these are saved in a log file. it's working fine, but is there a way how NOT to output connect: Network is unreachable
if there's no internet connection? thanks.
edit
i run this script via a shell script i named launch.sh which runs check_net.py (this script's name), and other preliminary scripts, and i placed launch.sh in my crontab to run on boot/reboot:
#reboot sh /home/pi/launch.sh > /home/pi/logs/cronlog 2>&1
from what i've read in this thread: what does '>/dev/null/ 2>&1' mean, 2 handles stderr where as 1 handles stdout.
i am new to this. I wish to see my stdout - but not the stderrs (in this case, the connect: Network is unreachable messages (only)..
/ogs
As per #shellter 's link suggestion in the comments, i restructured my cron to:
#reboot sh /home/pi/launch.sh 2>&1 > /home/pi/logs/cronlog | grep "connect: Network is unreachable"
alternatively, i also came up of an alternative solution, which involves a different way of checking an internet connection with the use urllib2.urlopen():
def internet_init():
try:
urllib2.urlopen('https://www.google.com', timeout=1)
return True
except urllib2.URLError as err:
return False
either of the two methods above omitted any connect: Network is unreachable error output in my logs.
thanks!
/ogs
I am having some weird issues when I run my application on a dev. server using UWSGI+nginx.. It works fine when my request completes within 5-6 mins.. For long deployments and requests taking longer than that, the UWSGI logs repeats the logs after around 5mins.. It's as if it spawns another process and I get two kinds of logs(one for current process and the repeated process). I am not sure why this is happening.. Did not find anything online. I am sure this is not related to my code because the same thing works perfectly fine in the lab env.. where I use the django runserver. Any insight would be appreciated..
The uwsgi.ini:-
# netadc_uwsgi.ini file
#uid = nginx
#gid = nginx
# Django-related settings
env = HTTPS=on
# the base directory (full path)
chdir = /home/netadc/apps/netadc
# Django's wsgi file
module = netadc.wsgi
# the virtualenv (full path)
home = /home/netadc/.venvs/netadc
# process-related settings
# master
master = true
# maximum number of worker processes
processes = 10
buffer-size = 65536
# the socket (use the full path to be safe
socket = /home/netadc/apps/netadc/netadc/uwsgi/tmp/netadc.sock
# ... with appropriate permissions - may be needed
#chmod-socket = 666
# daemonize
daemonize = true
# logging
logger = file:/home/netadc/apps/netadc/netadc/uwsgi/tmp/netadc_uwsgi.log
# clear environment on exit
vacuum = true
I wanted to know if there is a way to find out the status of the ssh server in the system using Python. I just want to know if the server is active or not (just yes/no). It would help even if it is just a linux command so that I can use python's popen from subprocess module and run that command.
Thanks
PS: I'm using openssh-server on linux (ubuntu 12.04)
If you want a generic way of telling if a process is running you could use ps.
def IsThisProcessRunning( ps_name ):
ps = subprocess.Popen("ps axf | grep %s | grep -v grep" % ps_name,
shell=True, stdout=subprocess.PIPE)
output = ps.stdout.read()
ps.stdout.close()
ps.wait()
if re.search(ps_name, output) is None:
return False
else:
return True
IsThisProcessRunning('/usr/sbin/apache2') # True, if Apache2 is running.
If you have a name that is commonly used; you can specify the location e.g. /usr/sbin/apache2
To be safe you can in combination to looking for the process name, also look for the pid file. This is a common technique used in init.d scripts.
try:
pf = file('/var/run/my_program.pid', 'r')
pid = int(pf.read().strip())
pf.close()
except IOError:
pid = None
if pid:
# Process is running.
Run service sshd status (e.g. via Popen()) and read what it tells you.
We are using supervisor to deploy a python web application. On deployment, the web application is installed on the server through buildout, and a script for running supervisor is created using collective.recipe.supervisor . This script is called at the end of the deployment process by a fabric script. The problem is that when the deployment script is finished, a SIGHUP signal is sent to the process, which causes supervisor to restart (as per this line: https://github.com/Supervisor/supervisor/blob/master/supervisor/supervisord.py#L300 ), but due to some reason, the web app is not restarted after it is terminated. There is no log output after the following:
2012-10-24 15:23:51,510 WARN received SIGHUP indicating restart request
2012-10-24 15:23:51,511 INFO waiting for app-server to die
2012-10-24 15:23:54,650 INFO waiting for app-server to die
2012-10-24 15:23:57,653 INFO waiting for app-server to die
2012-10-24 15:24:00,657 INFO waiting for app-server to die
2012-10-24 15:24:01,658 WARN killing 'app-server' (28981) with SIGKILL
2012-10-24 15:24:01,659 INFO stopped: app-server (terminated by SIGKILL)
So I have two questions. The first one is, does anyone know why supervisor restarts on SIGHUP? I couldn't find any explanation for this, and there are no command line options that would turn this behavior off. The second question is, how can we fix the problem we are facing? We tried starting supervisor with a nohup, but the SIGHUP is still received. The weird thing is that this doesn't happen when I log on to the server, start supervisor by hand, and log out.
Here is the supervisor script generated by buildout:
#!/usr/bin/python2.6
import sys
sys.path[0:0] = [
'/home/username/.buildout/eggs/supervisor-3.0b1-py2.6.egg',
'/home/username/.buildout/eggs/meld3-0.6.9-py2.6.egg',
'/home/username/.buildout/eggs/distribute-0.6.30-py2.6.egg',
]
import sys; sys.argv.extend(["-c","/home/username/app_directory/parts/supervisor/supervisord.conf"])
import supervisor.supervisord
if __name__ == '__main__':
sys.exit(supervisor.supervisord.main())
And here is the configuration file for supervisor, also generated by buildout:
[supervisord]
childlogdir = /home/username/app_directory/var/log
logfile = /home/username/app_directory/var/log/supervisord.log
logfile_maxbytes = 50MB
logfile_backups = 10
loglevel = info
pidfile = /home/username/app_directory/var/supervisord.pid
umask = 022
nodaemon = false
nocleanup = false
[unix_http_server]
file = /home/username/app_directory/supervisor.sock
username = username
password = apasswd
chmod = 0700
[supervisorctl]
serverurl = unix:///home/username/app_directory/supervisor.sock
username = username
password = apasswd
[rpcinterface:supervisor]
supervisor.rpcinterface_factory=supervisor.rpcinterface:make_main_rpcinterface
[program:app-server]
command = /home/username/app_directory/bin/gunicorn --bind 0.0.0.0:5000 app:wsgi
process_name = app-server
directory = /home/username/app_directory/bin
priority = 50
redirect_stderr = false
directory = /home/username/app_directory
We don't want to install a patched version of supervisor before really understanding the problem, so any information would be highly appreciated.
Thanks in advance
Restarting or reloading on SIGHUP is common practice in system programming for linux. The question is why you are getting SIGHUP after deployment ends. Since supervisor daemonize itself correctly (because you can start it and log out and it will work), the reload signal may be sent to supervisor by building bot, indicating that webapp need to be restarted, because code has changed.
So supervisor initiates app shutdown, in order to start app with new code. But app would not stop in given timeout, and supervisor decides that app hang and kill it with SIGKILL.
To solve a problem, you need teach the app to shutdown when supervisor ask for it.
The supervisord docs clearly state that sending SIGHUP to a supervisord process will "stop all processes, reload the configuration from the first config file it finds, and restart all processes".
ref - http://supervisord.org/running.html#signal-handlers
Perhaps your process is misbehaving; it looks like supervisor made several attempts to nicely shut it down, but then decided it needed a hard kill:
process.py:560
# kill processes which are taking too long to stop with a final
# sigkill. if this doesn't kill it, the process will be stuck
# in the STOPPING state forever.
self.config.options.logger.warn(
killing %r (%s) with SIGKILL' % (self.config.name, self.pid))
self.kill(signal.SIGKILL)
Maybe the kill call is failing?
You might have run into this bug: https://github.com/Supervisor/supervisor/issues/121
The workaround would be to downgrade supervisord until that is fixed in a released version.
Ran into the exactly same problem, downgrading to 3.0a10 solved it.