I would like to execute a specific version of mysqld through Python for unit testing. The idea is to execute the server on a thread, test, and kill the server when it's done. (Similar to testing.mysqld, which sadly doesn't work on Windows.). This is the current code:
#Create a temporary folder.
base_path = tempfile.mkdtemp()
#Extract the default files
zipfile.ZipFile(r"O:\Tools\mysql\min_mysql.zip").extractall(base_path)
#Setup my_ini file
my_ini_path = os.path.join(base_path, "my.ini").replace("\\", "/")
unix_base_path = posixpath.normpath(base_path).replace("\\", "/")
with open(my_ini_path, 'r') as my_ini:
filedata = my_ini.read()
filedata = filedata.replace("{{basedir}}", unix_base_path)
with open(my_ini_path, 'w', 0) as my_ini:
my_ini.write(filedata)
#Open mysqld
args = r"O:/Tools/mysql/bin/mysqld.exe --defaults-file=\"%s\"" % (my_ini_path)
args = shlex.split(args)
mysqld_process = subprocess.Popen(args, shell=True)
mysqld_process.wait()
But if I execute it through Python, I get this error:
Could not open required defaults file:
"c:\users\pelfeli1\appdata\local\temp\tmp2vct38\my.ini"
Fatal error in defaults handling. Program aborted
So far I have verified that the file exists before starting the process. If I print the command verbatim and execute it, the server runs fine.
There seems to be a difference between Popen and just executing in shell. What am I missing?
I'll copy my comment here, if you want to accept it as an answer:
I don't think this is the problem, but the args string shouldn't be
defined as raw (with the r). Instead, do this:
'O:/Tools/mysql/bin/mysqld.exe --defaults-file="%s"' (ie. use single
quotes). Unless you intend to pass the backslashes to the command line
Now, take into account that the following two strings
"foo\"bar\""
r"foo\"bar\""
Are not the same. The first one renders foo"bar", while the second gives you foo\"bar\".
So, what was happening is that the shell sees this as the file name: "c:\users\pelfeli1\appdata\local\temp\tmp2vct38\my.ini", including the quotes, because of the backquotes (\). You could have just written this:
args = 'O:/Tools/mysql/bin/mysqld.exe --defaults-file="%s"' % (my_ini_path)
just in case of spaces in my_ini_path, without problems.
Well, the problem was in the quotes. Just changed this line:
args = r"O:/Tools/mysql/bin/mysqld.exe --defaults-file=\"%s\"" % (my_ini_path)
to this line
args = "O:/Tools/mysql/bin/mysqld.exe --defaults-file=%s" % (my_ini_path)
I still have no idea why this changes anything, because printing args gives a valid (and working) command in both cases.
Related
I want to call a command like
scp username#hostname:/dir/to/files/\{a,b,c\} /target/dir
from Python to copy many files in one command.
The command works perfectly if entered directly into the shell.
But if I use
import subprocess
p = subprocess.Popen(['scp', 'username#hostname:/dir/to/files/\{a,b,c\}',
'/target/dir'])
sts = os.waitpid(p.pid, 0)
I get the error
scp: /dir/to/files/{a,b,c}: No such file or directory
Obviously, the backslashes are missing. And if I use double backslashes in the Popen arguments like
\\{a,b,c,d\\}
I get the error
scp: /dir/to/target/\a: No such file or directory
scp: /dir/to/target/\b: No such file or directory
scp: /dir/to/target/\c\: No such file or directory
Nothing changes if I use raw strings like r'\{' + r'\}'
How can I call the scp command from Python with the correctly escaped curly braces '\\{a,b,c\\}'?
cannot test, but I would remove all blackslashes altogether since they're just here to protect the expansion from the shell on the local machine:
import subprocess
p = subprocess.Popen(['scp', 'username#hostname:/dir/to/files/{a,b,c}',
'/target/dir'])
sts = p.wait()
also note that p.wait() is way better than the wait command you performed (more portable!)
Jean-Francois Fabre got me on the right track:
import subprocess
p = subprocess.Popen('scp username#hostname:/dir/to/files/\{a,b,c\} /target/dir',
shell=True)
sts = p.wait()
The
shell=True
argument was the missing bit. It is a solution that is not recommended, but at least it's working.
Don't use a shell feature unnecessarily in a script; you have your text editor to make typing easier. Just pass the three file names individually:
p = subprocess.Popen(['scp',
'username#hostname:/dir/to/files/a',
'username#hostname:/dir/to/files/b',
'username#hostname:/dir/to/files/c',
'/target/dir'])
Alternatively, let Python build the list of files for you.
file_list = ['username#hostname:/dir/to/files/%s' % (s,)
for f in ['a', 'b', 'c']]
p = subprocess.Popen(['scp'] + file_list + ['/target/dir'])
If I'm getting it correctly, you need to have the "\" and the "{". Since you need to escape both of them, what about 'username#hostname:/dir/to/files/\\\{a,b,c\\\}'
I am wanting to run an executable that would normally be run directly on the command line but ultimately via a Python script.
I used subprocess.Popen after reading through here and multiple Google results to achieve some limited success.
>>>import subprocess
>>>exe_path = sys.argv[1]
>>>dir_path_in = sys.argv[2]
>>>dir_path_out = sys.argv[3]
>>>subprocess.Popen([exe_path])
It then displays
<subprocess.Popen object at 0x021B7B30>
Followed by
>>>usage: <path to exe> [options] <dir_path> <dir_path_out>
But if I enter what you would normally expect to on the command line if used exclusively it returns:
>>>SyntaxError: invalid token
I have tested what is entered exclusively on the command line with the exe and it works fine just not via Python
I have had a look through StackOverFlow and the best kind of comparison I found was here How to handle an executable requiring interactive responses?
Ultimately the "usage" part will not even be required in the end as the declared sys.argvs will provide all the information the executable requires to run automatically.
The subprocess.call() achieved the desired result by declaring the argv variables and then concatenating the variables and using that final variable in a subprocess.call() as opposed to using shlex.split() which I first tried but it struggled with paths even with the '\' escaped for Windows
import subprocess
exe_path = sys.argv[1]
dir_path_in = sys.argv[2]
dir_path_out = sys.argv[3]
command = exe_path, dir_path_in, dir_path_out
p = subprocess.call(command)
I am running a python program I have coded with a subprocess WolfPsort Program.
It is a bioinformatics tool for protein localization detection program.
However, the python subprocess does not excute my input file.
This is the code
#!/usr/bin/python
# secref.py is for secretome refining
import os
import sys
import subprocess
if len(sys.argv) != 2:
print >> sys.stderr, 'Usage: python secref.py [*.fasta]'
exit(1)
if sys.argv[1].endswith('.fasta'):
filename = sys.argv[1]
else:
print >> sys.stderr, 'Input correct file... [*.fasta]'
exit(1)
filehandle = open(filename,'r')
progWolf = subprocess.Popen(['runWolfPsortSummary','fungi','<',filename,'>','tmpWolfResult'])
progWolf.wait()
If I run the code it gives error message like this:
[karyo#hostname secref.1.0]$ python secref.py A.carbonarius.fasta
Command Line Parsing Error; Do not know what to do with argument "<"
Usage:
runWolfPsortSummary [*OPTIONS*] *organismType*
runWolfPsortSummary (--usage|--help|--man)
Pipe sequences in from standard in.
The subprocess dose not recognise "<" symbol, but WolfPsort Program needs "<" to recognise the input fasta file and ">" is required to write a temporary result file.
How can I make the subprocess to understand the argument "<"?
Please, help me out!
I'm guessing you're trying to use shell magic to read from filename and write to tmpWolfResult. In order to make that happen, you need:
progWolf = subprocess.Popen('runWolfPsortSummary fungi < %s > tmpWolfResult'%filename, shell=True)
I feel obligated to mention that since this input is coming from a commandline argument, it's technically not safe/trusted and a malicious user with access to run this script on your system could do some nasty things.
However, it's probably more likely that you're distributing this script (or just using it yourself) and you and your users probably aren't interested in messing up your own system...
<, > are usually interpreted by the shell that Popen() doesn't spawn unnecessarily by default. You could use stdin, stdout parameters instead to redirect input/output from/to files:
from subprocess import check_call
with open(filename) as file, open('tmpWolfResult', 'w') as output_file:
check_call(['runWolfPsortSummary', 'fungi'], stdin=file, stdout=output_file)
Note: check_call() raises an exception if runWolfPsortSummary exits with non-zero status.
The Popen function takes an argument list separated by commas. The way you wrote it,
'<'
filename
'>'
is sent as three separate arguments. I'm assuming you want to concatenate that into one argument.
progWolf = subprocess.Popen(['runWolfPsortSummary','fungi','<' + filename + '>','tmpWolfResult'])
I'm trying to get the filename thats given in the command line. For example:
python3 ritwc.py < DarkAndStormyNight.txt
I'm trying to get DarkAndStormyNight.txt
When I try fileinput.filename() I get back same with sys.stdin. Is this possible? I'm not looking for sys.argv[0] which returns the current script name.
Thanks!
In general it is not possible to obtain the filename in a platform-agnostic way. The other answers cover sensible alternatives like passing the name on the command-line.
On Linux, and some related systems, you can obtain the name of the file through the following trick:
import os
print(os.readlink('/proc/self/fd/0'))
/proc/ is a special filesystem on Linux that gives information about processes on the machine. self means the current running process (the one that opens the file). fd is a directory containing symbolic links for each open file descriptor in the process. 0 is the file descriptor number for stdin.
You can use ArgumentParser, which automattically gives you interface with commandline arguments, and even provides help, etc
from argparse import ArgumentParser
parser = ArgumentParser()
parser.add_argument('fname', metavar='FILE', help='file to process')
args = parser.parse_args()
with open(args.fname) as f:
#do stuff with f
Now you call python2 ritwc.py DarkAndStormyNight.txt. If you call python3 ritwc.py with no argument, it'll give an error saying it expected argument for FILE. You can also now call python3 ritwc.py -h and it will explain that a file to process is required.
PS here's a great intro in how to use it: http://docs.python.org/3.3/howto/argparse.html
In fact, as it seams that python cannot see that filename when the stdin is redirected from the console, you have an alternative:
Call your program like this:
python3 ritwc.py -i your_file.txt
and then add the following code to redirect the stdin from inside python, so that you have access to the filename through the variable "filename_in":
import sys
flag=0
for arg in sys.argv:
if flag:
filename_in = arg
break
if arg=="-i":
flag=1
sys.stdin = open(filename_in, 'r')
#the rest of your code...
If now you use the command:
print(sys.stdin.name)
you get your filename; however, when you do the same print command after redirecting stdin from the console you would got the result: <stdin>, which shall be an evidence that python can't see the filename in that way.
I don't think it's possible. As far as your python script is concerned it's writing to stdout. The fact that you are capturing what is written to stdout and writing it to file in your shell has nothing to do with the python script.
I'm pretty new to writing python for windows (linux is no problem), and am having problems getting python to recognize files when running scripts, though it behaves fine in the command line
What am I doing wrong here?
def verifyFile(x):
#
return os.path.isfile(x)
This will return true (with a valid file, of course) when called from the python command line, but when I run the script from eclipse, or launch it from windows, it ALWAYS returns false. Any thoughts on why this is?
I've tried passing pathnames like this:
D:\Documents and Settings\BDE\Desktop\cdburn.jpg
and like this:
D:/Documents and Settings/BDE/Desktop/cdburn.jpg
I've changed sys,argv[0] to ''
I've tried this:
def verifyFile(x):
#
try:
f = open(x, 'r')
f.close()
return True
except:
return False
and am getting no love!
Any help would be appreciated.
Thanks
Blake
There is not really enough information here to debug your issue, but I have a suspicion.
Try adding the line
print sys.argv
to the start of your code, and see what the actual arguments that are being passed in to your program. I have a feeling that you will find the the filename D:\Documents and Settings\BDE\Desktop\cdburn.jpg is being split into 3 separate arguments, D:\Documents, and, Settings\BDE\Desktop\cdburn.jpg. If so, you need to quote any filename that has spaces in it.