I am writing a simple Python script to copy a MySQL database. I am attempting to copy the database based on the following SO questions and their answers: "Copy/duplicate database without using mysqldump", "python subprocess and mysqldump" and "Python subprocess, mysqldump and pipes". However, my script does not work for some reason I cannot see as the tables and the data do not appear in my new database.
I can see from my output that the mysqldump works correctly (I see a "Dump completed on..." in my output), so I think that something is wrong with my pipeline.
Here is my script:
#!/usr/bin/env python
import pymysql
from subprocess import Popen, PIPE, STDOUT
conn = pymysql.connect(host='127.0.0.1', port=3306, user='root', passwd='', db='mydb')
cur = conn.cursor()
print("Attempting to create new database...")
try:
cur.execute("CREATE DATABASE mydb2")
print("Creating new database")
except Exception:
print("Database already exists")
print()
# close connection just to be sure
cur.close()
conn.close()
print("Trying to copy old database to new database...")
args1 = ["mysqldump", "-h", "localhost", "-P", "3306", "-u", "root", "-p", "mydb"]
args2 = ["mysql", "-h", "localhost", "-P", "3306", "-u", "root", "-p", "mydb2"]
p1 = Popen(args1, stdout=PIPE, stderr=STDOUT)
p2 = Popen(args1, stdin=p1.stdout, stdout=PIPE, stderr=STDOUT)
output = p2.communicate()
print("output:")
print(output)
print()
As you can see I took the copy database pipeline from this answer. And at first I had the error mysqldump: Couldn't find table: "|" just as in that other question. So now I use two subprocess.Popen calls as suggested, which solved that error message.
The output variable shows that a mysqldump is performed, but I see nothing being mentioned about the mysql command.
I have tried to use p2.wait() and p1.wait() instead of p2.communicate() as suggested in one answer, but that just makes my Python script become unresponsive.
I have also tried the following:
output1 = p1.communicate()
output2 = p2.communicate()
But then both output1 and output2 show the same mysqldump output. So that was just a silly thing to do I guess..
I have also tried to use subprocess.call instead of subprocess.Popen, but that also makes my script become unresponsive.
Also including shell=True in either Popen or call also results in the script being just unresponsive.
However, it does work to type in the command in the command prompt (I use Windows 8.1) as follows:
mysqldump -h localhost -P 3306 -u root -p mydb | mysql -h localhost -P 3306 -u root -p mydb2
It copies my small test database in less than three seconds.
I wish I could also get it to work in Python.
I don't know the degree of pure Python you want to use for the copy, but you can just delegate the entire pipe operation to the shell.
subprocess.Popen('mysqldump -h localhost -P 3306 -u -root mydb | mysql -h localhost -P 3306 -u root mydb2', shell=True)
This should work the same way it works when you run it on the shell.
One problem that I saw is on this line:
p2 = Popen(args1, stdin=p1.stdout, stdout=PIPE, stderr=STDOUT)
It should read:
p2 = Popen(args2, stdin=p1.stdout, stdout=PIPE, stderr=STDOUT)
(args1 were being passed to the second proc, so that the program did two dumps and zero restores)
I keep coming back to this post as I try to carry out the same task, and it occurs to me that the reason for the unresponsiveness here is the "-p" switch in your mysql and mysqldump commands. "-p" by itself means "prompt for password," so the subprocesses are unresponsive because they're waiting for a password input.
Just in case anybody else comes across this ancient thread and tries to make it work for themselves, this was a trip-up for me.
Here's how you could run mysqldump .. | mysql pipeline without the shell:
#!/usr/bin/env python
from subprocess import Popen, PIPE
mysql = Popen("mysql -h localhost -P 3306 -u root -p mydb2".split(),
stdin=PIPE, stdout=PIPE)
mysqldump = Popen("mysqldump -h localhost -P 3306 -u root -p mydb".split(),
stdout=mysql.stdin)
mysql_stdout = mysql.communicate()[0]
mysqldump.wait()
See How do I use subprocess.Popen to connect multiple processes by pipes?
If you don't need to pass command-line parameters that require complex (possibly non-portable) escaping, capture the exit statuses, stdout then it is simpler to use the shell here.
Related
I am using subprocess in python to execute a custom vpn command which excepts password to execute.
below is the command which asks password:
./vpn -u <user_id> -d "description" /var/tmp/1.txt
password: XXXX
below is vpn shell command which works perfectly for above command.
/usr/bin/expect -c 'spawn ./vpn -u <user_id> -d "description" /var/tmp/1.txt; expect "Password"; send "<Password here>\r"; interact'
In python I am trying to achieve the same with below subprocess module, where below script executes below is the output.
"(b"\x1b[31m\xe2\x9c\x97 Can't read 'user id' password from the console.\x1b[0m""
import subprocess
args = ["./vpn", "-u", "<user_id>", "-d", "description", "/var/tmp/1.txt"]
# args = ['sudo','cat', '/var/tmp/1.txt']
proc = subprocess.Popen(args,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
proc.stdin.write('<passowrd>'.encode())
stdout, stderr = proc.communicate()
print(stdout)
Note: if I use time.sleep(2) before writing password, it prompts me for password in console without any error and my goal is to give password in the script.
Is there any better way in python to pass the shell command as this is pretty lengthy OR provide password in subprocess module execute seamlessly.
You are using Expect for Shell/Bash CLI command. When you use Python, just use pexpect to get the same functionality in Python.
Explore pexpect at https://github.com/pexpect/pexpect
Example passing password at executed program is here: https://github.com/pexpect/pexpect/blob/master/examples/passmass.py
Example:
import pexpect
child = pexpect.spawn('ftp ftp.openbsd.org')
child.expect('Name .*: ')
child.sendline('anonymous')
child.expect('Password:')
child.sendline('noah#example.com')
I tried in shell script and it works well: Just for anyone if it will be help full:
/usr/bin/expect<<vpn
spawn ./vpn -u <user_id> -d "description" /var/tmp/1.txt
expect "Password"
send "<password here>\r"
interact
sleep 5
vpn
echo "It's done"
I'm trying to create a user in postgreSQL using a python(2.7) script. I'm using subprocess.Popen.
When I call the function I see stdout for the OS.system code, but I don't get any type of output for subprocess.Popen so I have not idea why it's not working. What I do know is the user is not being created. I've also tried print with no luck.
Here is a working bash command that I'm trying to mimic. When this command is entered I'm prompted twice for the password.
createuser -PSdRU postgres -w csdashboard
My python function:
def setup_postgresql():
print('\n'*10)
print('Configuring PostgreSQL')
time.sleep(3)
#initialize the db
os.system('/etc/init.d/postgresql initdb')
#boot at start
os.system('chkconfig postgresql --add')
os.system('chkconfig postgresql on')
#copy the configs
shutil.copyfile('configs/pg_hba.conf', '/var/lib/pgsql/data/pg_hba.conf')
shutil.copyfile('configs/postgresql.conf', '/var/lib/pgsql/data/postgresql.conf')
#start postgresql
os.system('/etc/init.d/postgresql start')
os.system('createdb -U postgres csdashboard')
#add my user
pguseradd_csdashboard_user = subprocess.Popen(['createuser', ' -PSdRU', 'postgres', '-w',
'csdashboard'],
shell=False,
stdout=subprocess.PIPE,
stdin=subprocess.PIPE,
stderr=subprocess.PIPE)
pguseradd_csdashboard_user.stdin.write(csdashboard_password + '\n')
pguseradd_csdashboard_user.stdin.write(csdashboard_password + '\n')
pguseradd_csdashboard_user.stdin.flush()
os.system('/etc/init.d/postgresql restart')
exit()
I need to ssh into the server and execute few commands and process the response using subprocess. Here's my code
command = 'ssh -t -t buildMachine.X.lan; sudo su - buildbot ; build-set sets/set123'
print "submitting command"
result = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE, stdin=subprocess.PIPE)
print "got response"
response,err = result.communicate()
print response
This is getting stuck. I have seen other threads talking about passing a list instead of string to subprocess and removing shell=True.. I did that too but didn't work.
Ultimately I need the result of last command i.e. build-set in order to extract some information out of it.. help?
I figured the solution by using univerio's comment
The command needs to be
command = 'ssh -t -t buildMachine.X.lan \'sudo su - buildbot \'build-set sets/set123\'\''
Individual commands are like argument to previous command. This works.
I'm using the subprocess module of python to run an ssh command over my servers for collecting their disk usage. The one thing on which i'm stuck is if the ssh is not configured in any server then subprocess prompt for the password input which makes my whole script stuck and then i have to voluntarily kill the script itself. I just want it to let go all the servers which asks for password prompt(where ssh is not configured) and continue processing the rest.
def MyFunction(server):
msg=""
ps = subprocess.Popen("ssh -l mygroup %s 'df -k /some/directory'" % server,stdout=subprocess.PIPE,shell=True)
out, err = ps.communicate()
if err != None:
msg += "\n"+err
else:
msg = out
return msg
server_list= ['server A','server B','server C','server D']
for server in server_list:
Final_msg+=MyFunction(server)
Any help would be appreciated! :)
If it is just the thing that you want to avoid ssh ask you for anything, then you can forbid it to do so.
You can use the SSH option
BatchMode
If set to “yes”, passphrase/password querying will be disabled.
This option is useful in scripts and other batch jobs where no user is present to supply the password.
The argument must be “yes” or “no”. The default is “no”.
So just add -o BatchMode=yes:
ps = subprocess.Popen("ssh -o BatchMode=yes -l mygroup %s 'df -k /some/directory'" % server, stdout=subprocess.PIPE, shell=True)
BTW, why do you need shell=True here? Better do
ps = subprocess.Popen(["ssh", "-o", "BatchMode=yes", "-l", "mygroup", server, "df -k /some/directory"], stdout=subprocess.PIPE)
as it is cleaner, safer and internally simpler.
I have a python script that I want to use to make remote calls on a server, connect to Cassandra CLI, and execute commands to create keyspaces. One of the attempts that I made was something to this effect:
connect="cassandra-cli -host localhost -port 1960;"
create_keyspace="CREATE KEYSPACE someguy;"
exit="exit;"
final = Popen("{}; {}; {}".format(connect, create_keyspace, exit), shell=True, stdin=PIPE, stdout=PIPE, stderr=STDOUT, close_fds=True)
stdout, nothing = final.communicate()
Looking through various solutions, I'm not finding what I need. For example, the above code is throwing a "/bin/sh: 1: CREATE: not found", which I think means that it's not executing the CREATE statement on the CLI command line.
Any/all help would be GREATLY appreciated! Thank you!
try this out. I don't have cassandra-cli installed on my machine, so I couldn't test it myself.
from subprocess import check_output
from tempfile import NamedTemporaryFile
CASSANDRA_CMD = 'cassandra-cli -host localhost -port 1960 -f '
def cassandra(commands):
with NamedTemporaryFile() as f:
f.write(';\n'.join(commands))
f.flush()
return check_output(CASSANDRA_CMD + f.name, shell=True)
cassandra(['CREATE KEYSPACE someguy', 'exit'])
As you mentioned in the comment below pycassa a Python client for Cassandra cannot be used since it doesn't seem to support create statements.