I've just started using the psycopg2 module to access a local PostgreSQL server and I'll like a way to progammatically check if the server is already started so my program can handle the error when I try to start the server:
psqldir = 'C:/Program Files/PostgreSQL/9.2/bin' # Windows
username = os.environ.get('USERNAME')
dbname = 'mydb'
os.chdir(psqldir)
os.system('pg_ctl start -D C:/mydatadir') # Error here if server already started
conn = psycopg2.connect('dbname=' + dbname + ' user=' + username)
cur = conn.cursor()
I experimented a little and it seems that this returns a "0" or "3", which would solve my problem, but I didn't find any information on the PostgreSQL/psycopg2 manual that confirms if this is a documented behavior:
server_state = os.system('pg_ctl status -D C:/mydatadir')
What's the best way? Thanks!
From the pg_ctl documentation:
-w
Wait for the startup or shutdown to complete. Waiting is the default option for shutdowns, but not startups. When waiting for
startup, pg_ctl repeatedly attempts to connect to the server. When
waiting for shutdown, pg_ctl waits for the server to remove its PID
file. pg_ctl returns an exit code based on the success of the startup
or shutdown.
You will also find pg_ctl status useful:
status mode checks whether a server is running in the specified data
directory. If it is, the PID and the command line options that were
used to invoke it are displayed. If the server is not running, the
process returns an exit status of 3.
Try to use pgrep command.
proc = subprocess.Popen(["pgrep -u postgres -f -- -D"], stdout=subprocess.PIPE, shell=True)
(out, err) = proc.communicate()
try:
if int(out) > 0:
return True
except Exception as e:
return False
Related
I have a follow up question that builds off the question I asked here: Run multiple commands in different SSH servers in parallel using Python Paramiko, which was already answered.
Thanks to the answer on the link above, my python script is as follows:
# SSH.py
import paramiko
import argparse
import os
path = "path"
python_script = "worker.py"
# definitions for ssh connection and cluster
ip_list = ['XXX.XXX.XXX.XXX', 'XXX.XXX.XXX.XXX', 'XXX.XXX.XXX.XXX']
port_list = [':XXXX', ':XXXX', ':XXXX']
user_list = ['user', 'user', 'user']
password_list = ['pass', 'pass', 'pass']
node_list = list(map(lambda x: f'-node{x + 1} ', list(range(len(ip_list)))))
cluster = ' '.join([node + ip + port for node, ip, port in zip(node_list, ip_list, port_list)])
# run script on command line of local machine
os.system(f"cd {path} && python {python_script} {cluster} -type worker -index 0 -batch 64 > {path}/logs/'command output'/{ip_list[0]}.log 2>&1")
# loop for IP and password
stdouts = []
clients = []
for i, (ip, user, password) in enumerate(zip(ip_list[1:], user_list[1:], password_list[1:]), 1):
try:
print("Open session in: " + ip + "...")
client = paramiko.SSHClient()
client.connect(ip, user, password)
except paramiko.SSHException:
print("Connection Failed")
quit()
try:
path = f"C:/Users/{user}/Desktop/temp-ines"
stdin, stdout, stderr = ssh.exec_command(
f"cd {path} && python {python_script} {cluster} -type worker -index {i} -batch 64>"
f"C:/Users/{user}/Desktop/{ip}.log 2>&1 &"
)
clients.append(ssh)
stdouts.append(stdout)
except paramiko.SSHException:
print("Cannot run file. Continue with other IPs in list...")
client.close()
continue
# Wait for commands to complete
for i in range(len(stdouts)):
print("hello")
stdouts[i].read()
print("hello1")
clients[i].close()
print('hello2")
print("\n\n***********************End execution***********************\n\n")
This script, which is run locally, is able to SSH into the servers and run the command (i.e., run a python script called worker.py and log the command output to a log file). I.e., it is able to go through the first for loop with no issues.
My issue is related to the second for loop. Please see the print statements I added in the second for loop to be clear. When I run SSH.py locally, this is what I observe:
As you can see, I ssh into each of the servers and then stay at reading the command output of the first server I ssh over to. The worker.py script can take 30 mins or so to complete and the command output is the same on each server -- so it will take 30 mins to read the command output of the first server, then close the SSH connection of the first server, take a couple seconds to read the command output of the second server (as it is the same as the first one and would already be entirely printed), close its SSH connection, and so on. Please see below some of the command line output, if this helps.
Now, my question is, what if I don't want to wait until the worker.py script finishes, i.e., those entire 30 mins? I cannot/do not know how to raise a KeyboardInterrupt exception. What I have tried is quitting the local SSH.py script. However, as you can see from the print statements, this will not close the SSH connections although the training, and thus the log files, will stop logging info. In addition, after I quit the local SSH.py script, if I try to delete any of the log files, I get an error saying "cannot delete file because it is being used in cmd.exe" -- this only happens sometimes and I believe it is because of not closing the SSH connections?
First run in python console:
It hangs: Local python and log file running and saving but no print statements and no python and log file being run/saved in servers.
I run it again so second process starts:
Now, the first process doesn't hang anymore (python running and log files being saved in server). And can close this second run/process. It is like the second run/process helps with the hang of the first run/process.
If I were to run python SSH.py in the terminal it would just hang.
This was not happening before.
If you know that SSHClient.close cleanly close the connection and abort the remote command, call it on response to KeyboardInterrupt.
For this you cannot use the simple solution with stdout.read, as it blocks and prevents handling of the Ctrl+C on Windows.
Use the waiting code from my answer to Run multiple commands in different SSH servers in parallel using Python Paramiko (the while any(x is not None for x in stdouts): snippet).
And wrap it to try:...except (KeyboardInterrupt):.
try:
while any(x is not None for x in stdouts):
for i in range(len(stdouts)):
stdout = stdouts[i]
if stdout is not None:
channel = stdout.channel
# To prevent losing output at the end, first test for exit,
# then for output
exited = channel.exit_status_ready()
while channel.recv_ready():
s = channel.recv(1024).decode('utf8')
print(f"#{i} stdout: {s}")
while channel.recv_stderr_ready():
s = channel.recv_stderr(1024).decode('utf8')
print(f"#{i} stderr: {s}")
if exited:
print(f"#{i} done")
clients[i].close()
stdouts[i] = None
time.sleep(0.1)
except (KeyboardInterrupt):
print("Aborting")
for i in range(len(clients)):
print(f"#{i} closing")
clients[i].close()
If you do not need to separate the stdout and stderr, you can greatly simplify the code by using Channel.set_combine_stderr. See Paramiko ssh die/hang with big output.
I have an SSH.py with the goal of connecting to many servers over SSH to run a Python script (worker.py). I am using Paramiko, but am very new to it and learning as I go. On each server I ssh over with, I need to keep the Python script running -- this is for training a model parallely and so the script needs to run on all machines as to update model parameters/train jointly. The Python script on the servers need to be running so either all the SSH connections cannot close or I have to figure out a way for the Python script on the servers to keep running even if I close the connection.
From extensive googling, it looks like you can achieve this with nohup or:
client = paramiko.SSHClient()
client.connect(ip_address, username, password)
transport = client.get_transport()
channel = transport.open_session()
channel.exec_command("python worker.py > /logs/'command output' 2>&1")
However, what is unclear to me is how do we close/exit all SSH connections? I am running the SSH.py file on cmd.exe, would closing the cmd.exe be enough for all processes remotely to close?
In addition, is my use of client.close() correct for my purposes?
Please see below what I have for my code.
# SSH.py
import paramiko
import argparse
import os
path = "path"
python_script = "worker.py"
# definitions for ssh connection and cluster
ip_list = ['XXX.XXX.XXX.XXX', XXX.XXX.XXX.XXX', XXX.XXX.XXX.XXX']
port_list = [':XXXX', ':XXXX', ':XXXX']
user_list = ['user', 'user', 'user']
password_list = ['pass', 'pass', 'pass']
node_list = list(map(lambda x: f'-node{x + 1} ', list(range(len(ip_list)))))
cluster = ' '.join([node + ip + port for node, ip, port in zip(node_list, ip_list, port_list)])
# run script on command line of local machine
os.system(f"cd {path} && python {python_script} {cluster} -type worker -index 0 -batch 64 > {path}/logs/'command output'/{ip_list[0]}.log 2>&1")
# loop for IP and password
for i, (ip, user, password) in enumerate(zip(ip_list[1:], user_list[1:], password_list[1:]), 1):
try:
print("Open session in: " + ip + "...")
client = paramiko.SSHClient()
client.connect(ip, user, password)
transport = client.get_transport()
channel = transport.open_session()
except paramiko.SSHException:
print("Connection Failed")
quit()
try:
channel.exec_command(f"cd {path} && python {python_script} {cluster} -type worker -index {i} -batch 64 > {path}/logs/'command output'/{ip_list[i]}.log 2>&1", timeout=30)
client.close() # here I am closing connection but above command should be running, my question is can I safely close cmd.exe on which I am running SSH.py?
except paramiko.SSHException:
print("Cannot run file. Continue with other IPs in list...")
client.close()
continue
The code is based on Running process of remote SSH server in the background using Python Paramiko
Edit: It seems like the channel.exec_command() is not executing the command
f"cd {path} && python {python_script} {cluster} -type worker -index {i} -batch 64 > {path}/logs/'command output'/{ip_list[i]}.log 2>&1"
So I wonder if it is because of client.close()? What would happen if I comment out all the lines with client.close()? Would this help? Is this dangerous? When I quit my local Python script, would this close all my SSH connections and hence, no need for client.close()?
Also all my machines have Windows OS.
Indeed, the problem is that you close the SSH connection. As the remote process is not detached from the terminal, closing the terminal terminates the process. On Linux servers, you can use nohup. I do not know what is (if there is) a Windows equivalent.
Anyway, it seems that you do not need to close the connection. I understood, that you are ok with waiting for all the commands to complete.
stdouts = []
clients = []
# Start the commands
commands = zip(ip_list[1:], user_list[1:], password_list[1:])
for i, (ip, user, password) in enumerate(commands, 1):
print("Open session in: " + ip + "...")
client = paramiko.SSHClient()
client.connect(ip, user, password)
command = \
f"cd {path} && " + \
f"python {python_script} {cluster} -type worker -index {i} -batch 64 " + \
f"> {path}/logs/'command output'/{ip_list[i]}.log 2>&1"
stdin, stdout, stderr = client.exec_command(command)
clients.append(client)
stdouts.append(stdout)
# Wait for commands to complete
for i in range(len(stdouts)):
stdouts[i].read()
clients[i].close()
Note that the above simple solution with stdout.read() is working only because you redirect the commands output to a remote file. Were you not, the commands might deadlock.
Without that (or if you want to see the command output locally) you will need a code like this:
while any(x is not None for x in stdouts):
for i in range(len(stdouts)):
stdout = stdouts[i]
if stdout is not None:
channel = stdout.channel
# To prevent losing output at the end, first test for exit,
# then for output
exited = channel.exit_status_ready()
while channel.recv_ready():
s = channel.recv(1024).decode('utf8')
print(f"#{i} stdout: {s}")
while channel.recv_stderr_ready():
s = channel.recv_stderr(1024).decode('utf8')
print(f"#{i} stderr: {s}")
if exited:
print(f"#{i} done")
clients[i].close()
stdouts[i] = None
time.sleep(0.1)
If you do not need to separate the stdout and stderr, you can greatly simplify the code by using Channel.set_combine_stderr. See Paramiko ssh die/hang with big output.
Regarding your question about SSHClient.close: If you do not call it, the connection will be closed implicitly, when the script finishes, when Python garbage collector cleans up the pending objects. It's a bad practice. And even if Python won't do it, the local OS will terminate all connections of the local Python process. That's a bad practice too. In any case, that will terminate the remote processes along.
i created a script which runs on boot which checks if there's an internet connection on my raspberry pi, and at the same time updates the time (care of ntp) - via os.system().
import datetime, os, socket, subprocess
from time import sleep
dir_path = os.path.dirname(os.path.abspath(__file__))
def internet(host="8.8.8.8"):
result = subprocess.call("ping -c 1 "+host, stdout=open(os.devnull,'w'), shell=True)
if result == 0:
return True
else:
return False
timestr = time.strftime("%Y-%m-%d--%H:%M:%S")
netstatus = internet()
while netstatus == False:
sleep(30)
netstatus = internet()
if netstatus == True:
print "successfully connected! updating time . . . "
os.system("sudo bash "+dir_path+"/updatetime.sh")
print "time updated! time check %s"%datetime.datetime.now()
where updatetime.sh contains the following:
service ntp stop
ntpd -q -g
service ntp start
this script runs at reboot/boot and i'm running this in our workplace, 24/7. also, outputs from scripts like these are saved in a log file. it's working fine, but is there a way how NOT to output connect: Network is unreachable
if there's no internet connection? thanks.
edit
i run this script via a shell script i named launch.sh which runs check_net.py (this script's name), and other preliminary scripts, and i placed launch.sh in my crontab to run on boot/reboot:
#reboot sh /home/pi/launch.sh > /home/pi/logs/cronlog 2>&1
from what i've read in this thread: what does '>/dev/null/ 2>&1' mean, 2 handles stderr where as 1 handles stdout.
i am new to this. I wish to see my stdout - but not the stderrs (in this case, the connect: Network is unreachable messages (only)..
/ogs
As per #shellter 's link suggestion in the comments, i restructured my cron to:
#reboot sh /home/pi/launch.sh 2>&1 > /home/pi/logs/cronlog | grep "connect: Network is unreachable"
alternatively, i also came up of an alternative solution, which involves a different way of checking an internet connection with the use urllib2.urlopen():
def internet_init():
try:
urllib2.urlopen('https://www.google.com', timeout=1)
return True
except urllib2.URLError as err:
return False
either of the two methods above omitted any connect: Network is unreachable error output in my logs.
thanks!
/ogs
I am try to use Python to set up a reverse SSH tunnel. Some software that starts with the system is going to manage it and kill it or start it based on commands it receives.
I have written a class to manage the reverse tunnel as follows:
# imports omitted for brevity
class SshProcess():
def __init__(self):
self.process = None
def start(self, port):
if self.process is not None:
return None
command = [
# 'sudo',
'ssh',
'-R {port}:127.0.0.1:22'.format(port=port),
'{username}#{host}'.format(username=config.USERNAME, host=config.HOST),
'-o StrictHostKeyChecking=no'
]
def threaded_popen():
self.process = subprocess.Popen(
(' '.join(command)), # command, # shlex.split(command),
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
shell=True
)
self.process.wait()
logger.info('Reverse SSH to {username}#{host} has exited'.format(username=config.USERNAME, host=config.HOST))
logger.debug('command raw: {command}'.format(command=command))
logger.debug('command joined: {command}'.format(command=(' '.join(command))))
self.thread = Thread(target=threaded_popen)
self.thread.start()
def stop(self):
if self.process is not None:
try:
self.process.communicate(input="exit\n")
self.process.terminate()
except (ValueError, OSError) as e:
logger.warning('Closing reverse SSH raised {error}'.format(error=e.__class__.__name__))
logger.warning(e)
self.process = None
if self.thread is not None:
self.thread.join()
Now whenever I call start I receive the following log statements:
2017-06-28 14:32:46,343 - module - DEBUG - command raw: ['ssh', '-R 4000:127.0.0.1:22', 'tich#192.168.0.88', '-o StrictHostKeyChecking=no']
2017-06-28 14:32:46,344 - module - DEBUG - command joined: ssh -R 4000:127.0.0.1:22 tich#192.168.0.88 -o StrictHostKeyChecking=no
2017-06-28 14:32:46,797 - module - INFO - Reverse SSH to tich#192.168.0.88 has exited
The issue is the ssh tunnel exits nearly instantly after starting. performing a simple pidof ssh in Linux gives no output as if the process does not even exist.
I have also tried using communicate() after starting the process and you can see it establishes the connection and receives output. However shortly after the function exits, the subprocess exits as well.
I have set up RSA keypairs for both the root and the regular user. Copying and pasting the command into a terminal does not produce the instant exit bug.
The purpose is setting up a reverse SSH session so a remote user can log in. But I currently have not found an existing packaged solution that offers this functionality.
You done some weird ssh connection.My advice is to use paramiko a great ssh package.
on the other hand, you are sub-processioning only for a linux commamd so if u like it like that use:
install sshpass (yum install or apt-get)
sshpass -p your_password ssh user#hostname
and change this setting instead of the flag u sent:
change ssh_config
vi /etc/ssh/ssh_config
change the below key from "ask" to "no"
StrictHostKeyChecking no
My goal is to connect to SSH with python and authenticate which i can do with Paramiko or Fabric. But i would like to keep the session open after each execution and read the input/output. With paramiko i can only run 1 command before the session is closed and i am asked to authenticate again and the session hangs. And since fabric is using the paramiko library its giving me the same issue. For example if my directory structure is like this
-home
--myfolder1
--myfolder2
I would like to execute the below commands without having to re-authenticate because the sessions closes.
(make connection)
run cmd: 'pwd'
output: /home
run cmd: 'cd myfolder2'
run cmd: 'pwd'
output: /home/myfolder2
Is this possible with any module that is out there right now? Could it be made from scratch with native python? And also is this just not possible...?
Edit Added code. Without the new open_session it closes and i cannot run any command. After running the first command with this i will be prompted again to authenticate and it creates an infinite loop.
Edit2 If it closes after each command then there is no way this will work at all correct?
edit3 If i run this on a different server and exec_command with the paramikio.SSHClient it wont ask me to reauthenticate but if i 'cd somedir' and then 'pwd' it will output that i am back in the root directory of where i created.
class connect:
newconnection = ''
def __init__(self,username,password):
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
try:
ssh.connect('someserver', username=username,password=password,port=22,timeout=5)
except:
print "Count not connect"
sys.exit()
self.newconnection = ssh
def con(self):
return self.newconnection
#This will create the connection
sshconnection = connect('someuser','somepassword').con()
while True:
cmd = raw_input("Command to run: ")
if cmd == "":
break
try:
transport = sshconnection.get_transport()
transport.set_keepalive(999999)
chan = transport.open_session()
chan.settimeout(3)
chan.setblocking(0)
except:
print "Failed to open a channel"
chan.get_exception()
sys.exit()
print "running '%s'" % cmd
stdout_data = []
stderr_data = []
pprint.pprint(chan)
nbytes = 4096
chan.settimeout(5)
chan.get_pty()
chan.exec_command(cmd)
while True:
print "Inside loop " , chan.exit_status_ready()
time.sleep(1.2)
if chan.recv_ready():
print "First if"
stdout_data.append(chan.recv(nbytes))
if chan.recv_stderr_ready():
print "Recv Ready"
stderr_data.append(chan.recv_stderr(nbytes))
if chan.exit_status_ready():
print "Breaking"
break
print 'exit status: ', chan.recv_exit_status()
print ''.join(stdout_data)
This is possible by using the normal modules when you can concatenate the commands into one. Try
pwd ; cd myfolder2 ; pwd
as command. This should work but quickly becomes tedious when you have more complex commands which need arguments and horrible when the arguments contain spaces. The next step then is to copy a script with all the commands to the remote side and tell ssh to execute said script.
Another problem of this approach is that SSH doesn't return until all commands have executed.
Alternatively, you could build a "command server", i.e. a simple TCP server that listens for incoming connections and executes commands sent to it. It's pretty simple to write but also pretty insecure. Again, the solution is to turn the server into a (Python) script which reads commands from stdin and start that script remotely via SSH and then send commands.