I ran an API RestFul with bottle and python, all works fine, the API is a daemon running in the system, if I stop the daemon by command line the service stop very well and closed all the port and connections, but when I go to close the service through the API, the port keep alive in state LISTEN and later in TIME_WAIT, it does not liberate the port. I've read for two days but the problem is because bottle have a socket and it does not close the server well, but I can find he solution
The code to close the API as a service is a subprocess launched by python like this
#get('/v1.0/services/<id_service>/restart')
def restart_service(id_service):
try:
service = __find_a_specific_service(id_service)
if(service == None or len(service) < 1):
logging.warning("RESTful URI: /v1.0/services/<id_service>/restart " + id_service +" , restart a specific service, service does not exists")
response.status = utils.CODE_404
return utils.convert_to_json(utils.FAILURE, utils.create_failed_resource(utils.WARNING, utils.SERVICES_API_SERVICE_NOT_EXIST))
else:
if id_service != "API":
api.ServiceApi().restart(id_service)
else:
import subprocess
args='/var/lib/stackops-head/bin/apirestd stop; sleep 5; /var/lib/stackops-head/bin/apirestd start'
subprocess.Popen(args, shell=True)
logging.info("RESTful URI: /v1.0/services/<id_service>/restart " + id_service +" , restart a specific service, ready to construct json response...")
return utils.convert_to_json(utils.SERVICE, None)
except Exception, e:
logging.error("Services: Error during the process of restart a specific service. %r", e)
raise HTTPError(code=utils.CODE_500, output=e.message, exception=e, traceback=None, head
To terminate a bottle process from the outside, send SIGINT.
If app is exited or killed, all file descriptors/handles also including socket are closed by OS.
You can also use
sudo netstat -anp --tcp
in Linux to make sure if the specified port is owned by some processes. Or use
netstat -a -n -b -p tcp
in Windows to do the same thing.
TIME_WAIT is normal state managed by OS rather than app to keep a connection/port for a while. Sometimes it is annoying. Your can tune OS for how long it will keep, but it is not safe.
Related
I have a follow up question that builds off the question I asked here: Run multiple commands in different SSH servers in parallel using Python Paramiko, which was already answered.
Thanks to the answer on the link above, my python script is as follows:
# SSH.py
import paramiko
import argparse
import os
path = "path"
python_script = "worker.py"
# definitions for ssh connection and cluster
ip_list = ['XXX.XXX.XXX.XXX', 'XXX.XXX.XXX.XXX', 'XXX.XXX.XXX.XXX']
port_list = [':XXXX', ':XXXX', ':XXXX']
user_list = ['user', 'user', 'user']
password_list = ['pass', 'pass', 'pass']
node_list = list(map(lambda x: f'-node{x + 1} ', list(range(len(ip_list)))))
cluster = ' '.join([node + ip + port for node, ip, port in zip(node_list, ip_list, port_list)])
# run script on command line of local machine
os.system(f"cd {path} && python {python_script} {cluster} -type worker -index 0 -batch 64 > {path}/logs/'command output'/{ip_list[0]}.log 2>&1")
# loop for IP and password
stdouts = []
clients = []
for i, (ip, user, password) in enumerate(zip(ip_list[1:], user_list[1:], password_list[1:]), 1):
try:
print("Open session in: " + ip + "...")
client = paramiko.SSHClient()
client.connect(ip, user, password)
except paramiko.SSHException:
print("Connection Failed")
quit()
try:
path = f"C:/Users/{user}/Desktop/temp-ines"
stdin, stdout, stderr = ssh.exec_command(
f"cd {path} && python {python_script} {cluster} -type worker -index {i} -batch 64>"
f"C:/Users/{user}/Desktop/{ip}.log 2>&1 &"
)
clients.append(ssh)
stdouts.append(stdout)
except paramiko.SSHException:
print("Cannot run file. Continue with other IPs in list...")
client.close()
continue
# Wait for commands to complete
for i in range(len(stdouts)):
print("hello")
stdouts[i].read()
print("hello1")
clients[i].close()
print('hello2")
print("\n\n***********************End execution***********************\n\n")
This script, which is run locally, is able to SSH into the servers and run the command (i.e., run a python script called worker.py and log the command output to a log file). I.e., it is able to go through the first for loop with no issues.
My issue is related to the second for loop. Please see the print statements I added in the second for loop to be clear. When I run SSH.py locally, this is what I observe:
As you can see, I ssh into each of the servers and then stay at reading the command output of the first server I ssh over to. The worker.py script can take 30 mins or so to complete and the command output is the same on each server -- so it will take 30 mins to read the command output of the first server, then close the SSH connection of the first server, take a couple seconds to read the command output of the second server (as it is the same as the first one and would already be entirely printed), close its SSH connection, and so on. Please see below some of the command line output, if this helps.
Now, my question is, what if I don't want to wait until the worker.py script finishes, i.e., those entire 30 mins? I cannot/do not know how to raise a KeyboardInterrupt exception. What I have tried is quitting the local SSH.py script. However, as you can see from the print statements, this will not close the SSH connections although the training, and thus the log files, will stop logging info. In addition, after I quit the local SSH.py script, if I try to delete any of the log files, I get an error saying "cannot delete file because it is being used in cmd.exe" -- this only happens sometimes and I believe it is because of not closing the SSH connections?
First run in python console:
It hangs: Local python and log file running and saving but no print statements and no python and log file being run/saved in servers.
I run it again so second process starts:
Now, the first process doesn't hang anymore (python running and log files being saved in server). And can close this second run/process. It is like the second run/process helps with the hang of the first run/process.
If I were to run python SSH.py in the terminal it would just hang.
This was not happening before.
If you know that SSHClient.close cleanly close the connection and abort the remote command, call it on response to KeyboardInterrupt.
For this you cannot use the simple solution with stdout.read, as it blocks and prevents handling of the Ctrl+C on Windows.
Use the waiting code from my answer to Run multiple commands in different SSH servers in parallel using Python Paramiko (the while any(x is not None for x in stdouts): snippet).
And wrap it to try:...except (KeyboardInterrupt):.
try:
while any(x is not None for x in stdouts):
for i in range(len(stdouts)):
stdout = stdouts[i]
if stdout is not None:
channel = stdout.channel
# To prevent losing output at the end, first test for exit,
# then for output
exited = channel.exit_status_ready()
while channel.recv_ready():
s = channel.recv(1024).decode('utf8')
print(f"#{i} stdout: {s}")
while channel.recv_stderr_ready():
s = channel.recv_stderr(1024).decode('utf8')
print(f"#{i} stderr: {s}")
if exited:
print(f"#{i} done")
clients[i].close()
stdouts[i] = None
time.sleep(0.1)
except (KeyboardInterrupt):
print("Aborting")
for i in range(len(clients)):
print(f"#{i} closing")
clients[i].close()
If you do not need to separate the stdout and stderr, you can greatly simplify the code by using Channel.set_combine_stderr. See Paramiko ssh die/hang with big output.
I have a mariadb running in a container. On 'docker run', an import script (from a db dump) is run by mariadb, which creates users, builds schema, etc.
As the size of that dump script grows, the time to do the import increases. At this point it's about 8-10 seconds, but i expect amount of data to increase substantially, and the import time will be more difficult to predict.
I'd like to be able to send a signal from the container to the host, to let it know that the data has been loaded, and that db is ready to be used. So far i have found info on how to send signal from one container to another container, but there's no information on how to send signal from container to the host. Also, i need to be able to do this programmatically, as creating container is part part of a larger pipeline.
Ideally, i'd like to be able to do something like this:
client = docker.from_env()
db_c = client.containers.run('my_db_image', ....)
# check for signal from db_c container
# do other things
Thank you!
AFAIK you cannot send signals from the container to a process running on the host but there are other ways to know when the import has finished. I think the easiest is to start the container in detached mode and wait until a specific line gets logged. The following script for example waits until the line done is logged:
import os
import docker
client = docker.from_env()
container = client.containers.run('ubuntu:latest', 'bash -c "for i in {1..10}; do sleep 1; echo working; done; echo done"', detach=True)
print('container started')
for line in container.logs(stream=True):
print line.strip()
if line.strip() == 'done':
break
print('continue....')
If the output of the import script goes to stdout it could contain a simple print at the end:
select 'The import has finished' AS '';
Wait for this string in the python script.
Another approach is to use some other form of inter-process communication. An example using named pipes:
import os
import docker
import errno
client = docker.from_env()
FIFO = '/tmp/apipe'
# create the pipe
try:
os.mkfifo(FIFO)
except OSError as oe:
if oe.errno != errno.EEXIST:
raise
# start the container sharing the pipe
container = client.containers.run('ubuntu:latest', 'bash -c "sleep 5; echo done > /tmp/apipe"', volumes={FIFO: {'bind': FIFO, 'mode': 'rw'}}, detach=True)
print("container started")
with open(FIFO) as fifo:
print("FIFO opened")
while True:
data = fifo.read()
if len(data) == 0:
print("Writer closed")
break
print('Read: "{0}"'.format(data))
print("continue...")
The host shares the named pipe with the container. In the python script the read call to the FIFO is blocked until some data is available in the pipe.
In the container the import script writes to the pipe notifying the program that the data has been loaded. The mysql system command, \! command to execute an external command might come in handy in this situation. You could simply add to the end of the script:
\! echo done > /tmp/apipe
In a similar way you could use IPC sockets (aka Unix sockets) or shared memory but things get a bit more complicated.
Yet another solution is to add a health-check to the container. The health status can be polled on the host by inspecting the container. See How to wait until docker start is finished?
Edited:
The above approaches assume the container is initialized and accepting connections. If the script is executed as part of the initialization process (Initializing a fresh instance), which seems to be the case here, the database is not ready and accepting connections when the import completes. For the initialization the server is temporarily started with the
--skip_networking (allowing only local clients) and only after the initialization completes it is restarted and becomes available remotely.
you can add this code to check if the db is ready to accept the connections:
import MySQLdb
import time
db = MySQLdb.connect(host='MYHost', user='MYNAME', passwd='PASS', db='MYDB')
if not db:
while True:
db = MySQLdb.connect(host='MYHost', user='MYNAME', passwd='PASS', db='MYDB')
if db:
break
print("Still waiting for the DB")
time.sleep(10)
I'm building a tester for a Python script which performs works with a RethinkDB database. As part of the setUp() method, I'm trying to make the tester start up the RethinkDB server on localhost on port 28016 in case that has not yet been done.
I'm using subprocess to start the server. The problem is that, according to https://docs.python.org/2/library/subprocess.html, subprocess waits for the command to complete. In this case, it seems that as long as the server is up and running, the process is not complete and testing does not continue beyond the setUp() stage.
Here is the script I'm trying:
import unittest
import rethinkdb as r
import subprocess
class TestController(unittest.TestCase):
HOST = "localhost"
PORT_OFFSET = 1
PORT = 28015 + PORT_OFFSET
DB = "ipercron"
TABLE = "sensor_data"
def setUp(self):
try:
self.conn = r.connect(self.HOST, self.PORT)
except r.ReqlDriverError:
print("The RethinkDB server is not yet ready. Starting it up...")
subprocess.call(["rethinkdb", "--port-offset", str(TestController.PORT_OFFSET)])
self.conn = r.connect(self.HOST, self.PORT)
if TestController.DB not in r.db_list().run(self.conn):
r.db_create(TestController.DB).run(self.conn)
self.conn.use(TestController.DB)
if TestController.TABLE not in r.table_list().run(self.conn):
r.table_create(TestController.TABLE).run(self.conn) # Create the table if it does not yet exist
r.table(TestController.TABLE).delete().run(self.conn) # Empty the table to start with a clean slate
def test_upper(self):
self.assertEqual('foo'.upper(), 'FOO')
suite = unittest.TestLoader().loadTestsFromTestCase(TestController)
unittest.TextTestRunner(verbosity=2).run(suite)
The subprocess is meant to perform the rethinkdb --port-offset 1 command at the command line and then continue with the script. However, when I run the script I get the usual message that the server is ready:
kurt#kurt-ThinkPad:~/dev/clones/ipercron-compose/controller$ python unittest_controller.py
test_upper (__main__.TestController) ... The RethinkDB server is not yet ready. Starting it up...
Running rethinkdb 2.3.5~0xenial (GCC 5.3.1)...
Running on Linux 4.4.0-42-generic x86_64
Loading data from directory /home/kurt/dev/clones/ipercron-compose/controller/rethinkdb_data
Listening for intracluster connections on port 29016
Listening for client driver connections on port 28016
Listening for administrative HTTP connections on port 8081
Listening on cluster addresses: 127.0.0.1, 127.0.1.1, ::1
Listening on driver addresses: 127.0.0.1, 127.0.1.1, ::1
Listening on http addresses: 127.0.0.1, 127.0.1.1, ::1
To fully expose RethinkDB on the network, bind to all addresses by running rethinkdb with the `--bind all` command line option.
Server ready, "kurt_ThinkPad_a0k" 07bb35f6-3a33-4e8b-9e9c-a78504457969
Without any further action. How can I make the unittest proceed with the testing?
The problem is as you said, that subprocess.call is blocking and will wait for the command to finish. For scenarios where you need to spawn a child process without waiting for it to finish, you can use subprocess.Popen:
process = subprocess.Popen(["rethinkdb", "--port-offset", str(TestController.PORT_OFFSET)])
This gives you back a Popen object that provides a whole bunch of very useful methods to communicate with the child process. For example, you will probably want to use process.kill() in your unit test's tearDown() function to shut down your database.
Try with the following code
subprocess.call(["rethinkdb", "--port-offset", str(TestController.PORT_OFFSET)], shell = True)
I am using python and pika on linux OS Environment.
Message/Topic Receiver keeps crashing when RabbitMQ is not running.
I am wondering is there a way to keep the Message/Topic Receiver running when RabbitMQ is not because RabbitMQ would not be on the same Virtual Machine as the Message/Topic Receiver.
This cover if RabbitMQ crashes for some reason but the Message/Topic Receiver should keep running. Saving having to start/restart the Message/Topic Receiver again.
As far as I understand "Message/Topic Reciever" in your case is the consumer.
You are responsible to make an application in such a way that it will catch an exception if it is trying to connect to the not running RabbitMQ.
for example:
creds = pika.PlainCredentials(**creds)
params = pika.ConnectionParameters(credentials=creds,
**conn_params)
try:
connection = pika.BlockingConnection(params)
LOG.info("Connection to Rabbit was established")
return connection
except (ProbableAuthenticationError, AuthenticationError):
LOG.error("Authentication Failed", exc_info=True)
except ProbableAccessDeniedError:
LOG.error("The Virtual Host configured wrong!", exc_info=True)
except ChannelClosed:
LOG.error("ChannelClosed error", exc_info=True)
except AMQPConnectionError:
LOG.error("RabbitMQ server is down or Host Unreachable")
LOG.error("Connection attempt timed out!")
LOG.error("Trying to re-connect to RabbitMQ...")
time.sleep(reconnection_interval)
# <here goes your reconnection logic >
And as far as making sure that you Rabbit server is always up and running:
you can create a cluster make you queue durable, HA
install some type of supervision (let say monit or supervisord) and configure it to check rabbit process. for example:
check process rabbitmq with pidfile /var/run/rabbitmq/pid
start program = "/etc/init.d/rabbitmq-server stop"
stop program = "/etc/init.d/rabbitmq-server start"
if 3 restarts within 5 cycles then alert
I am new to Flask and want to make sure the redis server is running and start it if it isn't. Here's what I have:
#app.before_first_request
def initialize():
cmd = 'src/redis-cli ping'
p = subprocess.Popen(cmd,stdout=subprocess.PIPE)
out, err = p.communicate()
#if out.startswith('Could not connect to Redis'): #start redis here
if err is not None: raise Exception(err)
However, I get an error "OSError: [Errno 2] No such file or directory"
Is there an easier way to check if the redis server is running?
Use ping cmd of redis:
import redis
from redis import ConnectionError
import logging
logging.basicConfig()
logger = logging.getLogger('redis')
rs = redis.Redis("localhost")
try:
rs.ping()
except ConnectionError:
logger.error("Redis isn't running. try `/etc/init.d/redis-server restart`")
exit(0)
Sample Output:
ERROR:redis:Redis isn't running. try `/etc/init.d/redis-server restart`
I would suggest you to use some kind of a supervision like supervisord or monit they are designed to check if process, host, file and so on is doing it;s job, and if not,then restart it.
For example here is the config to check redis:
check host redis with address <your_redis_host>
if failed icmp type echo count 3 with timeout 3 seconds then alert
if failed port 6379 with timeout 15 seconds then alert