I'm calling snowsql client from shell script. I'm importing properties file by doing source. And invoking snowsql client. How can I do the same in Python? Any help would be highly appreciated.
Shell script to call snowsql client:
source /opt/data/airflow/config/cyrus_de/snowflake_config.txt
sudo /home/user/snowsql -c $connection --config=/home/user/.snowsql/config -w $warehouse --variable database_name=$dbname --variable stage=$stagename --variable env=$env -o exit_on_error=true -o variable_substitution=True -q /data/snowsql/queries.sql
Assuming you're switching to using Python purely for improved control flow and would still like to continue using shell features, a straight-forward translation would require writing a function that acts as the source command to import environment variables, then using them in a subprocess call that executes with a shell to allow environment variable substitution:
import os, shlex, subprocess
def source_file_into_env():
command = shlex.split("env -i bash -c 'source /opt/data/airflow/config/cyrus_de/snowflake_config.txt && env'")
proc = subprocess.Popen(command, stdout = subprocess.PIPE)
for line in proc.stdout:
(key, _, value) = line.partition("=")
os.environ[key] = value
proc.communicate()
def run():
source_file_into_env()
subprocess.run("""sudo /home/user/snowsql \
-c $connection \
--config=/home/user/.snowsql/config \
-w $warehouse \
--variable database_name=$dbname \
--variable stage=$stagename \
--variable env=$env \
-o exit_on_error=true \
-o variable_substitution=True \
-q /data/snowsql/queries.sql""", \
shell=True, \
env=os.environ)
if __name__ == '__main__':
run()
If you are instead looking to go purely Python without any shell calls, then the more native connector offered by Snowflake can be used instead of snowsql. This would be a far more invasive change but the connection examples will help you get started.
Related
I am trying to execute a bash command from python script that is wrapped in a docker exec call as the command needs to be executed inside a container.
This script is being executing on the host machine:
command_line_string = f"java -cp {omnisci_utility_path}:{driver_path} com.mapd.utility.SQLImporter" \
f" -u {omni_user} -p {omni_pass} -db {database_name} --port {omni_port}" \
f" -t {self.table_name} -su {denodo_user} -sp {denodo_pass}" \
f" -c {self.reader.connection_string}"\
f" -ss \"{read_data_query}\""
# in prod we have docker so we wrap it in docker exec:
if(args.env_type == "prod"):
command_line_string = f"docker exec -t {args.container_id} /bin/bash -c \"{command_line_string}\""
command_line_args = shlex.split(command_line)
command_line_process = subprocess.Popen(
command_line_args,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
)
process_output, _ = command_line_process.communicate()
However, when I execute the command, supplying the arguments I get a "Java Usage" response suggesting that the java command I am invoking did not have the correct parameters:
2021-09-01:09:19:09 [default_omnisci_ingestion.py:64] INFO - docker exec -t 5d874bffcdf8 /bin/bash -c "java -cp /omnisci/bin/omnisci-utility-5.6.5.jar
:/root/denodo-8-vdp-jdbcdriver.jar com.mapd.utility.SQLImporter -u admin -p mypass -db omnisci --port 6274 -t MyTable -su sourceDBuser -sp sourceDBpass -c jdbc:vdb://sourceDBURL -ss "SELECT
basin as Basin,
reservoir as Reservoir, cast(case when wkt like '%M%' Then wkt Else replace(wkt, 'POLYGON ', 'MULTIPOLYGON (') || ')' End as varchar(999999)) as wkt
FROM
schema.myTable;""
2021-09-01:09:19:10 [command_executor.py:10] INFO - Usage: java [options] <mainclass> [args...]
2021-09-01:09:19:10 [command_executor.py:10] INFO - (to execute a class)2021-09-01:09:19:10 [command_executor.py:10] INFO - or java [options] -jar <jarfile> [args...]
2021-09-01:09:19:10 [command_executor.py:10] INFO - (to execute a jar file)
...
I know that the problem is due to the use of quotes but I just don't understand how to go about them.
For example, the java command I am nesting inside bin/bash -c needs to be wrapped with quotes like sl
bin/bash -c "java -cp ..."
Note: the command works fine if I execute it in our dev env where we do not have the "docker setup" and we execute the command as it is but on Stage we have the system running in a container thus the reason why I need to use docker exec` to invoke the same command in the contaner
What is the best way to execute the below command in Python in a single line?
echo $(readlink /sys/dev/block/$(mountpoint -d /))
Tried using individual os.system(cmd) by separating - "mountpoint -d /" first and taking the output and appending to "readlink /sys/dev/block/${0}".format(out.strip()) and doing an echo works. Tried using subprocess and subprocess.Popen and subprocess.check_output but it raises raise CalledProcessError
cmd = "echo $(readlink /sys/dev/block/$(mountpoint -d /))"
You have to call the subcommand separately. And you can use python methods to read the link:
import subprocess
import os
path = "/"
device = subprocess.run(["mountpoint", "-d", path], stdout=subprocess.PIPE, encoding="utf8").stdout.strip()
link = os.readlink("/sys/dev/block/" + device)
print(link)
You probably want to use something like the following:
cmd = "bash -c 'echo $(readlink /sys/dev/block/$(mountpoint -d /))'"
echo doesn't substitute $() blocks, that's what your shell does, so you have to call the shell. os.system(cmd) should work then.
I am trying to port:
https://coderwall.com/p/ewk0mq/stop-remove-all-docker-containers
to a python script. So far I have:
def remove_all_containers():
subprocess.call(['docker', 'stop','$(docker ps -a -q)'])
subprocess.call(['docker', 'rm','$(docker ps -a -q)'])
return;
But get:
Error response from daemon: No such container: $(docker ps -a -q)
I have also tried:
def remove_all_containers():
subprocess.call(['docker', 'stop',$(docker ps -a -q)])
subprocess.call(['docker', 'rm',$(docker ps -a -q)])
return;
But that gives:
subprocess.call(['docker', 'stop',$(docker ps -a -q)])
SyntaxError: invalid syntax
it seems I need to nest another subprocess call into the parent subprocess call. Or is there a simpler way to do this?
TL;DR: Command substitution $(...) is a shell feature, therefore you must run your commands on a shell:
subprocess.call('docker stop $(docker ps -a -q)', shell=True)
subprocess.call('docker rm $(docker ps -a -q)', shell=True)
Additional improvements:
It's not required, but I would suggest using check_call (or run(..., check=True), see below) instead of call(), so that if an error occurs it doesn't go unnoticed:
subprocess.check_call('docker stop $(docker ps -a -q)', shell=True)
subprocess.check_call('docker rm $(docker ps -a -q)', shell=True)
You can also go another route: parse the output of docker ps -a -q and then pass to stop and rm:
container_ids = subprocess.check_output(['docker', 'ps', '-aq'], encoding='ascii')
container_ids = container_ids.strip().split()
if container_ids:
subprocess.check_call(['docker', 'stop'] + container_ids])
subprocess.check_call(['docker', 'rm'] + container_ids])
If you're using Python 3.5+, you can also use the newer run() function:
# With shell
subprocess.run('docker stop $(docker ps -a -q)', shell=True, check=True)
subprocess.run('docker rm $(docker ps -a -q)', shell=True, check=True)
# Without shell
proc = subprocess.run(['docker', 'ps', '-aq'], check=True, stdout=PIPE, encoding='ascii')
container_ids = proc.stdout.strip().split()
if container_ids:
subprocess.run(['docker', 'stop'] + container_ids], check=True)
subprocess.run(['docker', 'rm'] + container_ids], check=True)
There is nice official library for python, that helps with Docker.
https://docker-py.readthedocs.io/en/stable/index.html
import docker
client = docker.DockerClient(Config.DOCKER_BASE_URL)
docker_containers = client.containers.list(all=True)
for dc in docker_containers:
dc.remove(force=True)
We've received all containers and remove them all doesn't matter container status is 'started' or not.
The library could be useful if you can import it into code.
I have a ssh command which I was using in a system() statement but I want to replace it with a subprocess.call() statement. My ssh command is:
cmd ="ssh -i pem-file.pem user#" + hostname + " 'cd /user/home/ && java -cp jar-file.jar com.packg.class -a opt1 -f text-file_" + ts + ".txt'"
system(cmd)
I want to replace above with a subprocess.call() statement as it is giving me some performance issues and I read that subprocess.call() is a much better option to use. I formulated this query but it is not executing:
result = subprocess.call(["ssh","-i", "pem-file.pem","user#" + hostname + " 'cd /user/home/ && java -cp jar-file.jar com.packg.class -a opt1 -f text-file_" + ts + ".txt'"])
What is the mistake I am doing and what is the correct syntax?
The function shlex.split() is useful for parsing command line arguments into the proper format. This should resolve your syntax error:
import shlex
import subprocess
cmd ="ssh -i pem-file.pem user#" + hostname + " 'cd /user/home/ && java -cp jar-file.jar com.packg.class -a opt1 -f text-file_" + ts + ".txt'"
result = subprocess.call(shlex.split(cmd))
If that doesn't fix your error, then you can pass subprocess.call the shell=True argument:
import subprocess
cmd ="ssh -i pem-file.pem user#" + hostname + " 'cd /user/home/ && java -cp jar-file.jar com.packg.class -a opt1 -f text-file_" + ts + ".txt'"
result = subprocess.call(cmd, shell=True)
Using the shell argument will cause your command to be executed through a shell, rather than having the interpreter parse it. However, don't use the shell option if cmd can ever come from an untrusted source. Take at look at the warning in the Python docs.
One more note:
subprocess.system() is newer and more flexible than os.system(), but don't worry too much about "upgrading" to the new function. The advantages of subprocess.call() are in the more flexible options for communicating with your subprocess. If all you're doing is executing a single command and getting the return code, os.system() is probably fine. If you're finding that your command is being flaky and unreliable, switching to subprocess.call() probably isn't going to help much.
Assuming there are no shell meta-characters in hostname (likely), your command could look like this: each command-line argument is a separate list item:
#!/usr/bin/env python
import subprocess
cmd = ["ssh", "-i", "pem-file.pem", "user#" + hostname,
"cd /user/home/ && java -cp jar-file.jar com.packg.class -a opt1 "
"-f text-file_" + ts + ".txt"]
subprocess.check_call(cmd)
Unlike os.system(); it doesn't run the (local) shell.
You could get the argument list using shlex.split(your_original_system_command) (mentioned by #skrrgwasme) but shlex.split() can be fooled and therefore it is mostly useful as a hint on how the result should look like.
When I run the following commands in Python interpreter, all is good, but fail when run as a script. Somehow the opened tunnel is not recognized. Any pointers will be appreciated.
Interpreter:
gateway = subprocess.Popen("ssh -N sshgw", shell=True)
scp_prod = "scp -r server:/NFS/{0} .".format(filePath)
subprocess.call(scp_prod, shell=True)
gateway.kill()
Script:
try:
gateway = subprocess.Popen("ssh -N sshgw", shell=True)
scp_prod = "scp -r server:/NFS/{0} .".format(filePath)
subprocess.call(scp_prod, shell=True)
finally:
gateway.kill()