Develop Reference

How to use subprocess.run to run sql from a file

I'm trying to get a Python script to work that downloads a file and then loads the data into a mariadb. My best option seems to be using subprocess.run, as I cannot update the Python version to a version that includes the new mariadb connector.
I've tried different options:
subprocess.run(["mysql", "-u" + mdb_usr, "-p" + mdb_pwd, "database", " < " + file.sql])
subprocess.run(["mysql", "-u" + mdb_usr, "-p" + mdb_pwd, "database < " + file.sql])
subprocess.run(["mysql -u" + mdb_usr + " -p" + mdb_pwd + "database < " + file.sql], shell = True)
But none of them seems to work.
Without the < file.sql, I can get the script to work, but I can't seem to get the sql to execute. Can anyone point me in the right direction?

How can I start Tensorboard dev from within Python as a subprocess parallel to the Python session?

I want to monitor training progress of a CNN which is trained via a slurm process on a server (i.e., the Python script is executed through a bash script whenever the server has resources available; the session is not interactive. Hence, I cannot simply open a terminal and run Tensorboard dev).
So far, I have tried the following without finding a new experiment on my Tensorboard dev site:
mod = "SomeModelType"
logdir = "/some/directory/used/in/Tensorboard/callback"
PARAMETERS = "Some line of text describing the training settings"
subprocess.Popen(["tensorboard", "dev upload --logdir '" + logdir + \
"' --name Myname_" + mod + " --description '" + \
PARAMETERS + "'"])
If I insert the text string "tensorboard dev upload --logdir 'some/directory..." in a terminal, Tensorboard will start as expected.
If I include the code showed above, no new Tensorboard experiment will be started.
I also tried this:
subprocess.run(["/pfs/data5/home/kit/ifgg/mp3890/.local/bin/tensorboard", \
"dev", "upload", "--logdir", "'" + logdir + \
"'", "--name", "LeleNet" + mod#, "--description" + "'" + \
#PARAMETERS + "'"
], \
capture_output = False, text = False)
which starts Tensorboard, but it will not continue the Python script. Hence, Tensorboard, will be listening to output that never comes, because the Python session is listening to its own output instead of training the CNN.
Edit
This:
subprocess.Popen(["/pfs/data5/home/kit/ifgg/mp3890/.local/bin/tensorboard", \
"dev", "upload", "--logdir", "'" + logdir + \
"'", "--name", "LeleNet" + mod#, "--description" + "'" + \
#PARAMETERS + "'"
])
led to some message "Listening for new data in the log dir..." popping up all the time in interactive mode and led to cancellation of the slurm job (job disappeared). Moreover, Tensorboard does not work correcty this way. The experiment is created, but never receives any data.

I got it to work as follows:
logdir = "/some/directory"
tbn = "some_name"
DESCRIPTION = "some description of the experiment"
subprocess.call("tensorboard dev upload --logdir '" + logdir + \
"' --name " + tbn + " --description '" + \
DESCRIPTION + "' &", shell = True)

Cannot use gcloud compute ssh command in python subprocess

I have a compute engine already set up, I can use ssh command in command prompt, like
gcloud compute ssh test-instance#cloud-engine-noapi --zone us-central1-f --command "rm -f test.txt"
and successfully delete the test.txt in server.
However, when I call these in python.
subprocess.call(['gcloud', 'compute', '--project', 'test-project','ssh', temp_instance, '--zone', zone, '--command', '"cd /home"'], shell=True)
subprocess.call(['gcloud', 'compute', '--project', 'test-project','ssh', temp_instance, '--zone', zone, '--command', '"ls -l"'], shell=True)
subprocess.call(['gcloud', 'compute', '--project', 'test-project','ssh', temp_instance, '--zone', zone, '--command', '"rm -f /home/test.txt"'], shell=True)
The return is always like
bash: /command : No such file or directory
and for command like ls
bash: /command : command not found
Is there any process I have to do first?

Though, probably no one has this problem... but I finally come up with a method to solve it.
As the problem only occurs when using subprocess, I bypass it by writing a (bat/sh) file to temporarily save the commands.
Like,
with open(os.path.join(__location__, 'gcloud_command.bat'), 'w') as bat:
command_arr = []
for instance in all_instances_name:
temp_instance = user_name + "#" + instance
temp_file_path = '/home/' + user_name + '/'
command_arr.append('call gcloud compute ssh ' + temp_instance + ' --zone ' + zone + ' --command "cd ' + temp_file_path + '; rm -rf ' + temp_file_path + projectname.split('.')[0] + '; rm -f ' + temp_file_path + projectname.split('.')[0] + '*"\n')
command_arr.append('call gcloud compute scp "' + fullpath_projectname + '" ' + instance + ':' + temp_file_path + '\n')
if is_zip:
command_arr.append('call gcloud compute ssh ' + temp_instance + ' --zone ' + zone + ' ' + ' --command "cd ' + temp_file_path + '; unzip ' + temp_file_path + projectname + '"' + '\n')
bat.writelines(command_arr)
And execute with
subprocess.Popen(os.path.join(__location__, 'gcloud_command.bat'))

I know that this is quite old but it took me some time to figure it out.
In my case helped avoiding double quotes (despite it's recommended by gcloud cli help).
Example:
subprocess.call(['gcloud', 'compute', f'--project={test-project}','ssh', temp_instance, f'--zone={zone}, '--command=cd /home'], shell=True)

Is there any difference in using subprocess.check_output() in Windows and OS X?

I want to use subprocess.check_output(cmd, shell=True) to execute cmd in Windows. It turns out that there is no output after executing this statement, but it works in OS X. I want to know if there is some problem when using shell=True. Here's my original source.
paper_name = sheet[location].value
name = '"' + paper_name + '"'
cmd = py + options + name + ' -t'
out_str = subprocess.check_output(cmd,shell=True)
pdb.set_trace()
#a = out_str.split('\n')
fp_str = to_str(out_str)
a = fp_str.split('\n')
cmd is like below
cmd

Error: function Input_stream::Input_stream(const string&, bool) line 63. Error opening file #HWI-M02942_file1.fasta

I am writing a Python script to perform a BLAST by using the BLAST program DIAMOND automatically. The script executes commands in the terminal of Ubuntu 14.04.
My Python script is:
import subprocess
data_location = "/home/markschuurman/Desktop/Onderzoek_BioCentre/data_course_4/"
input_fasta_file = "#HWI-M02942_file1.fasta"
diamond_temp_dir = "/home/markschuurman/Desktop/DIAMOND_temp_dir/"
diamond_blast_database_location = "/home/markschuurman/Desktop/Onderzoek_BioCentre/BLAST_with_DIAMOND/DIAMOND_BLAST_databases/"
diamond_blast_output_file_directory = "/home/markschuurman/Desktop/Onderzoek_BioCentre/BLAST_with_DIAMOND/output_files/"
diamond_blast_output_filemame_daa = "matches.daa"
diamond_blast_output_filemame_tsv = "matches.tsv"
max_hits_per_read = "5"
max_evalue = "10"
commands = ["cd " + data_location,
"diamond blastx -d " + diamond_blast_database_location + "tcdb -q " + input_fasta_file + " -a " + diamond_blast_output_file_directory + diamond_blast_output_filemame_daa + " -t " + diamond_temp_dir + " -k " + max_hits_per_read + " -e " + max_evalue,
"diamond view -a " + diamond_blast_output_file_directory + diamond_blast_output_filemame_daa + " -o " + diamond_blast_output_file_directory + diamond_blast_output_filemame_tsv]
for command in commands:
print "Command : " + command
p = subprocess.Popen(command, stdout=subprocess.PIPE, shell=True)
p_status = p.wait()
print "Command finished"
The script creates the commands to execute after assigning the correct file paths and file names to the variables.
When I try to run this script I get the following error:
/usr/bin/python2.7 /home/markschuurman/Desktop/Onderzoek_BioCentre/BLAST_with_DIAMOND/scripts_to_parse_DIAMOND_output/execute_DIAMOND_BLAST.py
Command : cd /home/markschuurman/Desktop/Onderzoek_BioCentre/data_course_4/
Command finished
Command : diamond blastx -d /home/markschuurman/Desktop/Onderzoek_BioCentre/BLAST_with_DIAMOND/DIAMOND_BLAST_databases/tcdb -q #HWI-M02942_file1.fasta -a /home/markschuurman/Desktop/Onderzoek_BioCentre/BLAST_with_DIAMOND/output_files/matches.daa -t /home/markschuurman/Desktop/DIAMOND_temp_dir/ -k 5 -e 10
Error: function Input_stream::Input_stream(const string&, bool) line 63. Error opening file #HWI-M02942_file1.fasta
Command finished
Command : diamond view -a /home/markschuurman/Desktop/Onderzoek_BioCentre/BLAST_with_DIAMOND/output_files/matches.daa -o /home/markschuurman/Desktop/Onderzoek_BioCentre/BLAST_with_DIAMOND/output_files/matches.tsv
Error: function Input_stream::Input_stream(const string&, bool) line 75. Error opening file /home/markschuurman/Desktop/Onderzoek_BioCentre/BLAST_with_DIAMOND/output_files/matches.daa
Command finished
I am sure that the commands are correct because, when I execute the commands printed in line 20 separately in terminal there are no errors and the output of the BLAST application is correct.
Why does this error occur while executing the commands in this Python script and not separately in terminal and how to solve this error?

The problem here is that subprocess.Popen() runs the command in the separate process that exits when the command has completed running. The cd command and diamond commands are run in separate processes.
This means that diamond is looking for #HWI-M02942_file1.fasta in the directory where you run the command from.
Your solution to simply use the absolute path here is probably the simplest.