I am using the aws s3 sync command to sync files from s3 to local and perform some operations using python. Since there is no boto3 implementation of s3 sync I am running the cli command using subprocess.Popen(). I also tried .run but it still doesn't wait for all the files to be downloaded and continues to the next line of code with only a few files downloaded.
My sync function looks like this:
def sync_s3(self, aws_cli_path, s3_path, local_directory, aws_profile):
print("Running sync command from s3")
try:
process = subprocess.Popen([aws_cli_path, "s3", "sync", s3_path, local_directory, "--profile", aws_profile], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
while process.poll() is None:
pass
stdout, stderr = process.communicate()
if process.returncode != 0:
raise Exception(stderr.decode())
else:
print("S3 sync command completed successfully:", stdout.decode())
logging.info(f'S3 sync completed successfully: {stdout.decode()}')
except Exception as e:
print("Command failed with error:", e)
logging.critical(f'S3 sync failed: {e}')
sys.exit(701)
Also tried using a subprocess.run() but it still doesn't wait for the files to be downloaded.
Related
I run an aws batch jobs. For that i use a python script. This script also runs bash subprocesses.
I used the following function to put message on stdout:
def printf(msg):
print(msg, file=sys.stdout)
sys.stdout.flush()
And for exemple :
printf('Downloading misc inputs')
With this i can send some messages to aws cloudwatch.
But i would like to know if there is a way to send the log of one specific Bash command to a specific cloudwatch stream ?
To exec my Bash command i use the function:
def exec_cmd(cmd, shell=True):
printf("Executing %s" % cmd.strip())
if not shell:
p = subprocess.Popen(cmd.split())
else:
p = subprocess.Popen(cmd, shell=True, executable='/bin/bash')
err = p.wait()
return err
And then :
my_command1 = "/usr/local/bin/mytool arg1 arg2 > /ephemaral/output/log_command1.log 2>&1"
Exit_code1 = exec_cmd(my_command1)
my_command2 = "/usr/local/bin/mytool arg1 arg2 > /ephemaral/output/log_command2.log 2>&1"
Exit_code2 = exec_cmd(my_command2)
I want to know if i can send in Real time the content of log_command1.log to one cloudwatch stream, and the content of log_command2 to another cloudwatch stream.
I have a follow up question that builds off the question I asked here: Run multiple commands in different SSH servers in parallel using Python Paramiko, which was already answered.
Thanks to the answer on the link above, my python script is as follows:
# SSH.py
import paramiko
import argparse
import os
path = "path"
python_script = "worker.py"
# definitions for ssh connection and cluster
ip_list = ['XXX.XXX.XXX.XXX', 'XXX.XXX.XXX.XXX', 'XXX.XXX.XXX.XXX']
port_list = [':XXXX', ':XXXX', ':XXXX']
user_list = ['user', 'user', 'user']
password_list = ['pass', 'pass', 'pass']
node_list = list(map(lambda x: f'-node{x + 1} ', list(range(len(ip_list)))))
cluster = ' '.join([node + ip + port for node, ip, port in zip(node_list, ip_list, port_list)])
# run script on command line of local machine
os.system(f"cd {path} && python {python_script} {cluster} -type worker -index 0 -batch 64 > {path}/logs/'command output'/{ip_list[0]}.log 2>&1")
# loop for IP and password
stdouts = []
clients = []
for i, (ip, user, password) in enumerate(zip(ip_list[1:], user_list[1:], password_list[1:]), 1):
try:
print("Open session in: " + ip + "...")
client = paramiko.SSHClient()
client.connect(ip, user, password)
except paramiko.SSHException:
print("Connection Failed")
quit()
try:
path = f"C:/Users/{user}/Desktop/temp-ines"
stdin, stdout, stderr = ssh.exec_command(
f"cd {path} && python {python_script} {cluster} -type worker -index {i} -batch 64>"
f"C:/Users/{user}/Desktop/{ip}.log 2>&1 &"
)
clients.append(ssh)
stdouts.append(stdout)
except paramiko.SSHException:
print("Cannot run file. Continue with other IPs in list...")
client.close()
continue
# Wait for commands to complete
for i in range(len(stdouts)):
print("hello")
stdouts[i].read()
print("hello1")
clients[i].close()
print('hello2")
print("\n\n***********************End execution***********************\n\n")
This script, which is run locally, is able to SSH into the servers and run the command (i.e., run a python script called worker.py and log the command output to a log file). I.e., it is able to go through the first for loop with no issues.
My issue is related to the second for loop. Please see the print statements I added in the second for loop to be clear. When I run SSH.py locally, this is what I observe:
As you can see, I ssh into each of the servers and then stay at reading the command output of the first server I ssh over to. The worker.py script can take 30 mins or so to complete and the command output is the same on each server -- so it will take 30 mins to read the command output of the first server, then close the SSH connection of the first server, take a couple seconds to read the command output of the second server (as it is the same as the first one and would already be entirely printed), close its SSH connection, and so on. Please see below some of the command line output, if this helps.
Now, my question is, what if I don't want to wait until the worker.py script finishes, i.e., those entire 30 mins? I cannot/do not know how to raise a KeyboardInterrupt exception. What I have tried is quitting the local SSH.py script. However, as you can see from the print statements, this will not close the SSH connections although the training, and thus the log files, will stop logging info. In addition, after I quit the local SSH.py script, if I try to delete any of the log files, I get an error saying "cannot delete file because it is being used in cmd.exe" -- this only happens sometimes and I believe it is because of not closing the SSH connections?
First run in python console:
It hangs: Local python and log file running and saving but no print statements and no python and log file being run/saved in servers.
I run it again so second process starts:
Now, the first process doesn't hang anymore (python running and log files being saved in server). And can close this second run/process. It is like the second run/process helps with the hang of the first run/process.
If I were to run python SSH.py in the terminal it would just hang.
This was not happening before.
If you know that SSHClient.close cleanly close the connection and abort the remote command, call it on response to KeyboardInterrupt.
For this you cannot use the simple solution with stdout.read, as it blocks and prevents handling of the Ctrl+C on Windows.
Use the waiting code from my answer to Run multiple commands in different SSH servers in parallel using Python Paramiko (the while any(x is not None for x in stdouts): snippet).
And wrap it to try:...except (KeyboardInterrupt):.
try:
while any(x is not None for x in stdouts):
for i in range(len(stdouts)):
stdout = stdouts[i]
if stdout is not None:
channel = stdout.channel
# To prevent losing output at the end, first test for exit,
# then for output
exited = channel.exit_status_ready()
while channel.recv_ready():
s = channel.recv(1024).decode('utf8')
print(f"#{i} stdout: {s}")
while channel.recv_stderr_ready():
s = channel.recv_stderr(1024).decode('utf8')
print(f"#{i} stderr: {s}")
if exited:
print(f"#{i} done")
clients[i].close()
stdouts[i] = None
time.sleep(0.1)
except (KeyboardInterrupt):
print("Aborting")
for i in range(len(clients)):
print(f"#{i} closing")
clients[i].close()
If you do not need to separate the stdout and stderr, you can greatly simplify the code by using Channel.set_combine_stderr. See Paramiko ssh die/hang with big output.
Command i'm trying to run using paramiko in python 3.7:
Windows:
type file.ext4.gz | ssh user#address sudo update.sh
Mac:
cat file.ext4.gz | ssh user#address sudo update.sh
From the cmd / terminals and from .bat / .sh this works, after entering the password. I've been working on a simple python gui (PysimpleGui) to allow the user to fo this, but without the need to enter the password (this is saved from initial connection).
I've tried:
client = paramiko.SSHClient()
client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
client.connect(config["IP_ADDRESS"], username=config["USERNAME"], password=config["PASSWORD"], timeout=5)
a = client.open_sftp()
a.put(file_location, "sh update.sh", callback=sent)
While this works to send the file, it doesn't run it and gives the error:
OSError: Failure
I don't want to do this in subprocess, as this tool is to prevent the use of terminal for the "end user"
I've been beating my head against this for 2 days now. Thank you.
EDIT:
Here is the STDIO Code:
def send_ssh(value, input=None):
if input:
transport = client.get_transport()
channel = transport.open_session()
channel.exec_command(value)
with open(input, "rb") as file:
for chunk in iter(functools.partial(file.read, read_size), b''):
if channel.send_ready():
channel.sendall(chunk)
if channel.recv_ready():
print(channel.recv(1024).decode().strip())
if channel.recv_stderr_ready():
print(channel.recv_stderr(1024).decode().strip())
while not channel.exit_status_ready():
if channel.recv_ready():
print(channel.recv(1024).decode().strip())
if channel.recv_stderr_ready():
print(channel.recv_stderr(1024).decode().strip())
else:
w, r, e = client.exec_command(value, get_pty=True)
error = e.read().strip().decode()
if error != "":
return error
else:
return r.read().strip().decode()
Once the file is cat to the script it's the verified by the script. I worked around this by just using SFTP to send the file and running my
cat file | sudo script.sh
this works, but does require that i transfer a 600mb file (thankfully always over a local connection (LAN)) each time. The above code does transfer the file, but it doesn't complete. If i just try sending it via for line in file: i'll corrupt.
Keeping things simpler, below we're using threading to allow synchronous APIs to be used rather than needing to write explicit asynchronous code:
import shutil
client = SSHClient()
client.load_system_host_keys()
client.connect('user#address')
# here's the important part: we're using the file handles returned by exec_command()
update_stdin, update_stdout, update_stderr = client.exec_command('sudo update.sh')
# copy stdout and stderr from the remote thread to our own process's stdout and stderr
t_out = Thread(target=shutil.copyfileobj, args=[update_stdout, sys.stdout]); t_out.start()
t_err = Thread(target=shutil.copyfileobj, args=[update_stderr, sys.stderr]); t_err.start()
# write your local file to the remote stdin, in the foreground: we don't exit until done.
shutil.copyfileobj(open('file.ext4.gz', 'r'), update_stdin)
update_stdin.close()
# optional, but let's be graceful: wait for the threads to exit, and collect exit status
t_out.join(); t_err.join()
result = stdout.channel.recv_exit_status()
print(f"Remote process exited with status {result}")
I have the following task with runs a python scripts using Popen (I already tried with check_output, call with the same result)
#app.task
def TaskName(filename):
try:
proc = subprocess.Popen(['python', filename],stdout=subprocess.PIPE,
stderr=subprocess.PIPE):
taskoutp, taskerror = proc.communicate()
print('Output: ', taskoutp)
print('Error: ', taskerror)
Except:
raise Exception('Task error: ', taskerror)
If I generate an error in the subprocess it is displayed in the worker cmd window as if it were a normal output/print, the task remains indefinetly with the status 'STARTED' even when I manually close the window opened by the Popen subprocess.
What can I do so the tasks not only prints the error but actually stops and changes its status to FAILURE?
If you want to store results of your task , you could use this parameter result_backend or CELERY_RESULT_BACKEND depending on the version of celery you're using.
This parameter can be used for acknowledging a task only after it is processed :
task_acks_late
Default: Disabled.
Late ack means the task messages will be acknowledged after the task has been executed, not just before (the default behavior).
Complete list of Configuration options can be found here => https://docs.celeryproject.org/en/stable/userguide/configuration.html
I've just started using the psycopg2 module to access a local PostgreSQL server and I'll like a way to progammatically check if the server is already started so my program can handle the error when I try to start the server:
psqldir = 'C:/Program Files/PostgreSQL/9.2/bin' # Windows
username = os.environ.get('USERNAME')
dbname = 'mydb'
os.chdir(psqldir)
os.system('pg_ctl start -D C:/mydatadir') # Error here if server already started
conn = psycopg2.connect('dbname=' + dbname + ' user=' + username)
cur = conn.cursor()
I experimented a little and it seems that this returns a "0" or "3", which would solve my problem, but I didn't find any information on the PostgreSQL/psycopg2 manual that confirms if this is a documented behavior:
server_state = os.system('pg_ctl status -D C:/mydatadir')
What's the best way? Thanks!
From the pg_ctl documentation:
-w
Wait for the startup or shutdown to complete. Waiting is the default option for shutdowns, but not startups. When waiting for
startup, pg_ctl repeatedly attempts to connect to the server. When
waiting for shutdown, pg_ctl waits for the server to remove its PID
file. pg_ctl returns an exit code based on the success of the startup
or shutdown.
You will also find pg_ctl status useful:
status mode checks whether a server is running in the specified data
directory. If it is, the PID and the command line options that were
used to invoke it are displayed. If the server is not running, the
process returns an exit status of 3.
Try to use pgrep command.
proc = subprocess.Popen(["pgrep -u postgres -f -- -D"], stdout=subprocess.PIPE, shell=True)
(out, err) = proc.communicate()
try:
if int(out) > 0:
return True
except Exception as e:
return False