import sys
import os
from subprocess import Popen, PIPE, STDOUT
# Transfer Database
print ('Transferring from ' + mysql_source_database)
mysql = Popen(f"mysql -h {mysql_dest_host} -P 3306 -u {mysql_dest_username} -p{mysql_dest_pw} {mysql_dest_database}".split(), stdin=PIPE, stdout=PIPE)
dbnamerewrite = Popen(f"sed s/{mysql_source_database}/{mysql_dest_database}/g".split(), stdin=PIPE, stdout=mysql.stdin)
mysqldump = Popen(f"mysqldump --set-gtid-purged=OFF --column-statistics=0 -h {mysql_source_host} -P 3306 -u {mysql_source_username} -p{mysql_source_pw} {mysql_source_database}".split(), stdout=dbnamerewrite.stdin)
mysql_stdout = mysql.communicate()[0]
mysqldump.wait()
The above code does what I want it to but never stops waiting. Does anyone know how to fix the wait. If I ctrl-c it after the SQL work has finished this is the given error:
^CERROR 1064 (42000) at line 3829: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '' at line 3
Traceback (most recent call last):
File "test.py", line 19, in <module>
mysql_stdout = mysql.communicate()[0]
File "/usr/lib/python3.8/subprocess.py", line 1028, in communicate
stdout, stderr = self._communicate(input, endtime, timeout)
File "/usr/lib/python3.8/subprocess.py", line 1868, in _communicate
ready = selector.select(timeout)
File "/usr/lib/python3.8/selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
KeyboardInterrupt
One thing is that you should drop the explicit call to mysqldump.wait(). According to the docs:
Note: This will deadlock when using stdout=PIPE or stderr=PIPE and the child process generates enough output to a pipe such that it blocks waiting for the OS pipe buffer to accept more data. Use Popen.communicate() when using pipes to avoid that.
mysql.communicate is sufficient in this case, because it will not receive an EOF until all the elements up the pipeline send one. So mysql.communicate() returning directly implies that the other two processes are done.
Another problem is that with the ordering of processes that you have, you will have to call communicate on all of them in reverse order to get data flowing through the pipeline. One solution is to do just that:
db_param = ['-h', mysql_dest_host, '-P', '3306', '-u', mysql_dest_username, f'p{mysql_dest_pw}', mysql_dest_database]
mysql = Popen(['mysql'] + db_param,
stdin=PIPE, stdout=PIPE)
dbnamerewrite = Popen(['sed', f's/{mysql_source_database}/{mysql_dest_database}/g'],
stdin=PIPE, stdout=mysql.stdin)
mysqldump = Popen(['mysqldump', '--set-gtid-purged=OFF', '--column-statistics=0'] + db_param,
stdout=dbnamerewrite.stdin)
mysqldump.communicate()
dbnamerewrite.communicate()
mysql_stdout = mysql.communicate()[0]
The other alternative is to set up your pipe in the opposite order, in which case you only need to communicate with the last process:
db_param = ['-h', mysql_dest_host, '-P', '3306', '-u', mysql_dest_username, f'p{mysql_dest_pw}', mysql_dest_database]
mysqldump = Popen(['mysqldump', '--set-gtid-purged=OFF', '--column-statistics=0'] + db_param,
stdout=PIPE)
dbnamerewrite = Popen(['sed', f's/{mysql_source_database}/{mysql_dest_database}/g'],
stdin=mysqldump.stdout, stdout=PIPE)
mysql = Popen(['mysql'] + db_param, stdin=dbnamerewrite.stdout, stdout=PIPE)
mysql_stdout = mysql.communicate()[0]
Related
bash_fc = rf"mysql -u {source_user_name} -h {source_ipv4_addr} --password={source_db_password}"
When I use the following functions for the above command,
from subprocess import PIPE,run
def cons_r(cmd):
response = run(cmd, stdout=PIPE, stderr=PIPE, universal_newlines=True, shell=True)
return response.stdout
response = os.popen(bash_fc)
mysql: [Warning] Using a password on the command line interface can be insecure.
mysql: Unknown OS character set 'cp857'.
mysql: Switching to the default character set 'utf8mb4'.
ERROR 1045 (28000): Access denied for user 'root'#'pc.mshome.net' (using password: YES)
I can't read the output, is there a method you know of so I can read it?
You can read errors from standard error
import subprocess as sp
ret = sp.run(['ls', '*.bad'], stderr=sp.PIPE,
stdout=sp.PIPE, shell=True, encoding="cp857")
if ret.returncode == 0:
print(ret.stdout)
else:
print(ret.stderr)
This change fixed the problem
from subprocess import PIPE, run, STDOUT
def cons_r(cmd):
response = run(cmd, stdout=PIPE, stderr=STDOUT, universal_newlines=True, shell=True)
return response.stdout
What I want to do is call mysqldump to make a backup of a database table.
I have assembled a list of command line arguments, where db and tbl are the database and table names, respectively, and cnf is a properly formatted mysql configuration file (containing host, user and password),
args = ["mysqldump","--defaults-extra-file="+cnf,db,tbl]
print " ".join(args)
The output of the print is,
mysqldump --defaults-extra-file=dbserver.cnf test mytable
When I copy/paste the above line into the (bash) shell, it works.
When I use subprocess.Popen, the command fails,
import subprocess as sp
...
proc = sp.Popen(args, shell=False, stdin=sp.PIPE, stdout=sp.PIPE)
stdout = proc.communicate()
retcode = proc.wait()
if retcode>0: print "error",retcode
But when I join the args together, and call subprocess.Popen with shell=True,
the command works as desired,
cmd = " ".join(args)
proc = sp.Popen(cmd, shell=True, stdin=sp.PIPE, stdout=sp.PIPE)
stdout = proc.communicate()
retcode = proc.wait()
if retcode>0: print "error",retcode
Further investigation reveals that the first command does not seem to use the user/password credentials from the config file, but the second variant does.
Responses to other questions about subprocess.Popen make it clear that one should avoid the shell=True, but the first approach above fails.
Can anyone identify what I am doing wrong above?
I want to emulate command line call of my script (TensorFlow neaural chatbot model) in Django view and get output from console to variable.
When i do manually in terminal of my server:
python3 var/www/engine/chatbot/udc_predict.py --model_dir=var/www/engine/chatbot/runs/1486057482/
the output is good and printed out.
So in my Django view i do:
import subprocess
answer = subprocess.check_output(['python3', 'var/www/engine/chatbot/udc_predict.py','--model_dir=var/www/engine/chatbot/runs/1486057482/'], shell=True, stderr=subprocess.STDOUT, timeout=None)
print('answer', answer)
And answer variable is printed as b'' in Apache error log.
I cannot figure out what's wrong in my call.
The answer is to use .communicate() and PIPE:
from subprocess import Popen, PIPE
proc = Popen(
"python3 var/www/engine/chatbot/udc_predict.py --model_dir=var/www/engine/chatbot/runs/1486057482/",
shell=True,
stdout=PIPE, stderr=PIPE
)
proc.wait()
res = proc.communicate()
if proc.returncode:
print(res[1])
print('result:', res[0])
answer = res[0]
I have a simple function that calls the same subprocess.popen command on multiple items from a list. It executes the first 2 items in the list no problem, however the action on the third hangs.
It seems that the process_null.communicate() never executes or doesnt finish at least as I never get the output from the 3rd item in the list. I have tried changing the list, but get the same results. Any ideas to whats going on here?
def check_list(server_list):
null_list = []
for target in server_list:
command_null="rpcclient -N -U '' {}". format (str(target))
process_null = subprocess.Popen(command_null, shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT)
output_null = process_null.communicate()[0].replace('\n','')
if "NT_STATUS_ACCESS_DENIED" in output_null or "NT_STATUS_CONNECTION_REFUSED" in output_null or "NT_STATUS_UNSUCCESSFUL" in output_null:
print '[-] ' + target + '\tDenied NULL Session'
else:
print '[+] ' + target + '\tAccepted NULL Session'
null_list.append(target)
return null_list
Output
[-] 192.168.1.3 Denied NULL Session
[-] 192.168.1.65 Denied NULL Session
When rpcclient successfully establishes a connection it starts a shell and waits for commands to be entered on stdin if none are given using the -c flag, which is what happens here. You don't see the prompt because you redirect all output (stdout + stderr) to a pipe, but you don't do the same for stdin, which means the input will be read from the same tty the python interpreter is running on.
Also, you shouldn't use shell=True with a string argument if not absolutely necessary, use a list of arguments instead:
command_null = ['rpcclient', '-N', '-U', '', str(target)]
To fix your problem you have two options:
supply a command to execute when successfully connected:
command_null = ['rpcclient', '-N', '-U', '', '-c', 'exit', str(target)]
use stdin=PIPE when opening the process, which causes rcpclient to exit when stdin is closed by communicate():
process_null = subprocess.Popen(command_null, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, stdin=subprocess.PIPE)
I am writing a simple Python script to copy a MySQL database. I am attempting to copy the database based on the following SO questions and their answers: "Copy/duplicate database without using mysqldump", "python subprocess and mysqldump" and "Python subprocess, mysqldump and pipes". However, my script does not work for some reason I cannot see as the tables and the data do not appear in my new database.
I can see from my output that the mysqldump works correctly (I see a "Dump completed on..." in my output), so I think that something is wrong with my pipeline.
Here is my script:
#!/usr/bin/env python
import pymysql
from subprocess import Popen, PIPE, STDOUT
conn = pymysql.connect(host='127.0.0.1', port=3306, user='root', passwd='', db='mydb')
cur = conn.cursor()
print("Attempting to create new database...")
try:
cur.execute("CREATE DATABASE mydb2")
print("Creating new database")
except Exception:
print("Database already exists")
print()
# close connection just to be sure
cur.close()
conn.close()
print("Trying to copy old database to new database...")
args1 = ["mysqldump", "-h", "localhost", "-P", "3306", "-u", "root", "-p", "mydb"]
args2 = ["mysql", "-h", "localhost", "-P", "3306", "-u", "root", "-p", "mydb2"]
p1 = Popen(args1, stdout=PIPE, stderr=STDOUT)
p2 = Popen(args1, stdin=p1.stdout, stdout=PIPE, stderr=STDOUT)
output = p2.communicate()
print("output:")
print(output)
print()
As you can see I took the copy database pipeline from this answer. And at first I had the error mysqldump: Couldn't find table: "|" just as in that other question. So now I use two subprocess.Popen calls as suggested, which solved that error message.
The output variable shows that a mysqldump is performed, but I see nothing being mentioned about the mysql command.
I have tried to use p2.wait() and p1.wait() instead of p2.communicate() as suggested in one answer, but that just makes my Python script become unresponsive.
I have also tried the following:
output1 = p1.communicate()
output2 = p2.communicate()
But then both output1 and output2 show the same mysqldump output. So that was just a silly thing to do I guess..
I have also tried to use subprocess.call instead of subprocess.Popen, but that also makes my script become unresponsive.
Also including shell=True in either Popen or call also results in the script being just unresponsive.
However, it does work to type in the command in the command prompt (I use Windows 8.1) as follows:
mysqldump -h localhost -P 3306 -u root -p mydb | mysql -h localhost -P 3306 -u root -p mydb2
It copies my small test database in less than three seconds.
I wish I could also get it to work in Python.
I don't know the degree of pure Python you want to use for the copy, but you can just delegate the entire pipe operation to the shell.
subprocess.Popen('mysqldump -h localhost -P 3306 -u -root mydb | mysql -h localhost -P 3306 -u root mydb2', shell=True)
This should work the same way it works when you run it on the shell.
One problem that I saw is on this line:
p2 = Popen(args1, stdin=p1.stdout, stdout=PIPE, stderr=STDOUT)
It should read:
p2 = Popen(args2, stdin=p1.stdout, stdout=PIPE, stderr=STDOUT)
(args1 were being passed to the second proc, so that the program did two dumps and zero restores)
I keep coming back to this post as I try to carry out the same task, and it occurs to me that the reason for the unresponsiveness here is the "-p" switch in your mysql and mysqldump commands. "-p" by itself means "prompt for password," so the subprocesses are unresponsive because they're waiting for a password input.
Just in case anybody else comes across this ancient thread and tries to make it work for themselves, this was a trip-up for me.
Here's how you could run mysqldump .. | mysql pipeline without the shell:
#!/usr/bin/env python
from subprocess import Popen, PIPE
mysql = Popen("mysql -h localhost -P 3306 -u root -p mydb2".split(),
stdin=PIPE, stdout=PIPE)
mysqldump = Popen("mysqldump -h localhost -P 3306 -u root -p mydb".split(),
stdout=mysql.stdin)
mysql_stdout = mysql.communicate()[0]
mysqldump.wait()
See How do I use subprocess.Popen to connect multiple processes by pipes?
If you don't need to pass command-line parameters that require complex (possibly non-portable) escaping, capture the exit statuses, stdout then it is simpler to use the shell here.