When I'm using Python 3 to launch a program via subprocess.call(), why do I need 4 backslashes in paths?
This is my code:
cmd = 'C:\\\\Windows\\\\System32\\\\cmd.exe'
cmd = shlex.split(cmd)
subprocess.call(cmd)
When I examine the command line of the launched cmd.exe instance with Task Manager, it shows the path correctly with only one backslash separating each path.
Because of this, I need this on Windows to make the paths work:
if platform.platform().startswith('Windows'):
cmd = cmd.replace(os.sep, os.sep + os.sep)
is there a more elegant solution?
Part of the problem is that you're using shlex, which implements escaping rules used by Unix-ish shells. But you're running on Windows, whose command shells use different rules. That accounts for one level of needing to double backslashes (i.e., to worm around something shlex does that you didn't need to begin with).
That you're using a regular string instead of a raw string (r"...") accounts for the other level of needing to double backslashes, and 2*2 = 4. QED ;-)
This works fine on Windows:
cmd = subprocess.call(r"C:\Windows\System32\cmd.exe")
By the way, read the docs for subprocess.Popen() carefully: the Windows CreateProcess() API call requires a string for an argument. When you pass a sequence instead, Python tries to turn that sequence into a string, via rules explained in the docs. When feasible, it's better - on Windows - to pass the string you want directly.
When you are creating the string, you need to double each backslash for escaping, and then when the string is passed to your shell, you need to double each backslash again. You can cute the backslashes in half by using a raw string:
cmd = r'C:\\Windows\\System32\\cmd.exe'
\ has special meaning - you're using it as part of an escape sequence. Double up the backslashes, and you have a literal backslash \.
The caveat is that, with only one pair of escaped backslashes, you still have only one literal backslash. You need to escape that backslash, too.
Alternatively, why not just use os.sep instead? You'll be able to ensure your code is more portable (since it'll use the system-specific separator), and you won't have to deal [directly] with escaping backslashes.
As John points out 4 slashes isn't necessary when accessing files locally.
One place where 4 slashes is necessary is when connecting to (generally windows) servers over SMB or CIFS.
Normally you would just use \servername\share\
But each one of those slashes needs to be escaped. So thus the 4 slashes before servernames.
you could also use subprocess.call()
import subprocess as sp
sp.call(['c:\\program files\\<path>'])
Related
This question already has answers here:
How do you activate an Anaconda environment within a Python Script?
(5 answers)
Closed 2 years ago.
I'm trying to trigger the execution of a python script via conda.
I would then capture the output and report it to command prompt where this is executed.
This is basically the concept in the easiest way
wrap.py - wrapper inted to execute multiple times the following script
import subprocess
def wrap():
while True:
cmd1=r"call C:\\Users\\my_user\\anaconda3\\Scripts\\activate.bat"
cmd2=r"cd C:\\myfolder\\mysubfolder"
cmd3=r"C:\\Users\\my_user\\anaconda3\\python.exe C:\\myfolder\\mysubfolder\\test.py"
proc = subprocess.run([cmd1,cmd2,cmd3])
if __name__ == '__main__':
wrap()
test.py - script that has to be executed
def mytest():
print("success")
if __name__ == '__main__':
mytest()
since mytest prints success once, I would like the output of the wrapper (run on anaconda) to be
(base) C:\myfolder\mysubfolder> python wrap.py
success
success
success
...
I tried with
1 - subprocess.Popen
2 - using shell=True or not
3 - using a list ["first command","second command","third command"] or a single string "first;second;third"
4 - using or removing "r" in front of the string, here the blanks are breaking the game
5 - using single or double ""
6- in my_user the underscore is also resulting in an encoding error
I actually tried to replicate at least 20 different stackoverflow "solutions" but none of them really worked for me. I also read properly the subprocessing page of python documentation, but this didn't help.
Any hint is appreciated, I'm lost.
The syntax subprocess.run([cmd1, cmd2, cmd3]) means run cmd1 with cmd2 and cmd3 as command-line arguments to cmd1. You instead want to execute a single sequence of shell commands; several of the things you are trying to do here require the shell, so you do want shell=True, which dictates the use of a single string as input, rather than a list consisting of a command and its arguments.
(Windows has some finicky processing behind the scenes which makes it not completely impossible to use a list of strings as the first argument with shell=True; but this really isn't portable or obvious. Just don't.)
Regarding the requirement for shell=True here, commands like call and cd (and source or . in Bourne-family shells) are shell built-ins which do not exist as separate binaries; if you don't have shell=True you will simply get "command not found" or your local equivalent. (Under other circumstances, you should generally avoid shell=True when you can. But this is not one of those cases; here, it really is unavoidable without major code changes.)
If your shell is cmd I guess the command might look like
subprocess.run(
r"call C:\Users\my_user\anaconda3\Scripts\activate.bat & C:\Users\my_user\anaconda3\python.exe C:\myfolder\mysubfolder\test.py",
shell=True)
or equivalently the same without r before the string and with all backslashes doubled; the only difference between an r"..." string and a regular "..." string is how the former allows you to put in literal backslashes, whereas the latter requires you to escape them; in the former case, everything in the string is literal, whereas in the latter case, you can use symbolic notations like \n for a newline character, \t for tab, etc.
In Python, it doesn't really matter whether you use single or double quotes; you can switch between them freely, obviously as long as you use the same opening and closing quotes. If you need literal single quotes in the string, use double quotes so you don't have to backslash-escape the literal quote, and vice versa. There's also the triple-quoted string which accepts either quoting character, but is allowed to span multiple lines, i.e. contain literal newlines without quoting them.
If your preferred shell is sh or bash, the same syntax would look like
subprocess.run(r"""
source C:\Users\my_user\anaconda3\Scripts\activate.bat &&
C:\Users\my_user\anaconda3\python.exe C:\myfolder\mysubfolder\test.py""",
shell=True)
I left out the cd in both cases because nothing in your code seems to require the subprocess to run in a particular directory. If you do actually have that requirement, you can add cwd=r'C:\myfolder\mysubfolder' after shell=True to run the entire subprocess in a separate directory.
There are situations where the facilities of subprocess.run() are insufficient, and you need to drop down to bare subprocess.Popen() and do the surrounding plumbing yourself; but this emphatically is not one of those scenarios. You should stay far away from Popen() if you can, especially if your understanding of subprocesses is not very sophisticated.
I am using python v3.6 on Windows 10. When specifying a string to represent a directory location, what is the difference between the 2 approaches below?
folder_location = 'C:\\Users\\username\\Dropbox\\Inv'
folder_location = 'C:/Users/username/Dropbox/Inv'
This is a follow-up question to another question I just posted. My problem was solved when I used \\ instead of /.
What is wrong with this selenium firefox profile to download file into customized folder?
On Unix systems, the folder separator is /, while on Windows systems, the separator is \. Unfortunately this \ is also an escape character in most programming languages and text based formats (including C, Python and many others). Strangely enough a / character is not allowed in windows paths.
So Python on windows is designed to accept both / and \ as folder separator when dealing with the filesystem, for convenience. But the \ must be escaped by another \ (unless of course you use raw strings like r'backslashes are now normal characters \\\ !')
Selenium, on the other hand, will write values into Firefox preferences, which, unlike Python, expects the appropriate kind of separator. That's why using forward slashes does not work in your example.
Windows uses by default backslashes as file/folder seperator the \\ is an escaped \. The POSIX compliant file/folder seperator / is also supported by the windows api. But the library you use (which is not recognizable in your example) need also support it.
The standard Windows path separator is backslash \. But it is used in string formatting so for example \n is end of line.
For the above reason you rather don't want to use backslash in you path as if the name of the folder will start with a letter corresponding to special characters you will run into troubles.
To use native backslash separator in windows you have two ways. Yo can use raw string and then all special characters are read literary. path = r"C:\user\myFolder" or escape backslach with escape character with turns out to be the backslash too path = "C:\\user\\myFolder".
But coming back to DOS it accepted forward slash in path string too
Python is able to accept both separators. It is advised to use native way of formatting on your system
If you want you script working on both systems try:
import os
if os.name == 'posix':
path = '/net/myFolder/'
else:
path = r'C:\Users\myFolder'
Windows inherited backslashes as a path separator from Microsoft DOS. DOS initially didn't support subdirectories and opted to use the (on US keyboards) easily typed slash / character for command line switches.
When they did introduce subdirectories in DOS 2, either slash / or backslash \ worked as a path separator, but to use slashes on the command line you had to reconfigure the switch character, a feature they later removed entirely.
Thus the command line for certain commands that look for switches without space in front (like dir/w) is the one place you can't use forward slashes (this has to do with the command line being passed as a single string, unlike POSIX which passes distinct arguments in a list). That, and poorly written code that tries things like splitting on backslash, not knowing that slash is also a path separator.
It's also sometimes complicated by either character having other meanings, such as \ being the escape character in string literals; that's why you use \\ unless you use a raw string r'foo\bar'.
The other path separator I know of is classic Mac OS, which uses colon :. Python handles these differences by including reasonable routines in os.path or pathlib.
Windows and Linux/macOS use different path separators - UNIX uses forward slashes (/) while Windows use back slashes (\).
You should never type your own separators, always use os.path.join or os.sep, which handle this for you based on the platform you're running on. Example:
import os
folder_location = os.path.join('C:\\', 'Users', 'username', 'Dropbox', 'Inv')
# or
folder_location = os.sep.join(['C:\\', 'Users', 'username', 'Dropbox', 'Inv']);
Also, you will need to manually escape the drive letter's trailing slash manually, as specified on the Python docs:
Note that on Windows, since there is a current directory for each drive, os.path.join("c:", "foo") represents a path relative to the current directory on drive C: (c:foo), not c:\foo.
Hard-coding a full path like this is usually useless, as C: will only work on Windows anyway. You will most likely want to use this later on using relative paths or paths that were fetched elsewhere and need to have segments added to them.
When running the following script:
import os
import sys
import subprocess
if len(sys.argv) > 1:
print sys.argv[1]
sys.exit(0)
commandline = [sys.executable]
commandline.append(os.path.realpath(__file__))
commandline.append('"test"')
p = subprocess.Popen(commandline)
p.wait()
p = subprocess.Popen(" ".join(commandline))
p.wait()
It returns the following output
"test"
test
Why is there a difference between providing a list of arguments or one string?
This is run on a windows machine and you will see backslashes before the quotes on the command in the task manager.
I expected the same result in both runs.
Edit:
The problem is not so much in the automatic escaping of spaces (I find that is the programmers responsibility), but more about my quotes being escaped or not in the process commandline.
These are the two subprocesses taken from the windows task manager:
A different non-python process parses the first commandline with the backslashes, which brings unexpected behaviour. How can I have it so that I can use a list and not have the quotes escaped on the commandline?
Edit2:
The quotes are definitely added by python. If you run the following:
import subprocess
commandline = ['echo']
commandline.append('"test"')
commandline.append('>')
commandline.append(r'D:\test1.txt')
p = subprocess.Popen(commandline, shell=True)
p.wait()
commandline = 'echo "test" > D:\\test2.txt'
p = subprocess.Popen(commandline, shell=True)
p.wait()
Then you will see that the outputs are
D:\test1.txt:
\"test\"
D:\test2.txt:
"test"
The string API is dangerous since it might change the meaning of arguments. Example: You want to execute C:\Program Files\App\app.exe. If you use the string version of Popen(), you get an error:
C:\Program: Error 13
What happens is that with the string API, Python will split the input by spaces and try to start the command C:\Program with the single argument Files\App\app.exe. To fix this, you need to quote properly. Which gets you in quote hell when you have quotes in your arguments (i.e. when you really want to pass "test" as an argument with the quotes).
To solve this (and other subtle) bugs, there is the list API where each element of the list will become a single item passed to the OS without any modifications. With list API, you get what you see. If you quote an argument with the list API, it will be passed on with the quotes. If there are spaces, they won't split your argument. If there are arbitrary other special characters (like * or %), they will all be passed on.
[EDIT] As usual, things are much more complex on Windows. From the Python documentation for the subprocess module:
17.1.5.1. Converting an argument sequence to a string on Windows
On Windows, an args sequence is converted to a string that can be parsed using the following rules (which correspond to the rules used by the MS C runtime):
Arguments are delimited by white space, which is either a space or a tab.
A string surrounded by double quotation marks is interpreted as a single argument, regardless of white space contained within. A quoted string can be embedded in an argument.
A double quotation mark preceded by a backslash is interpreted as a literal double quotation mark.
Backslashes are interpreted literally, unless they immediately precede a double quotation mark.
If backslashes immediately precede a double quotation mark, every pair of backslashes is interpreted as a literal backslash. If the number of backslashes is odd, the last backslash escapes the next double quotation mark as described in rule 3.
So the backslashes are there because MS C runtime wants it that way.
I have two strings:
C:\Data
and another folder
Foo1
I need, the windows output to be
C:\Data\Foo1
and the Linux output to be
/data/foo1
assuming /data is in linux. Is there any constant separator that can be used in Python, that makes it easy to use irrespective of underlying OS?
Yes, python provides os.sep, which is that character, but for your purpose, the function os.path.join() is what you are looking for.
>>> os.path.join("data", "foo1")
"data/foo1"
os.path.normpath() will normalize a path correctly for Linux and Windows. FYI, Windows OS calls can use either slash, but should be displayed to the user normalized.
The os.path.join() is always better. As Mark Tolonen wrote (my +1 to him), you can use a normal slash also for Windows, and you should prefer this way if you have to write the path explicitly. You should avoid using the backslash for paths in Python at all. Or you would have to double them in strings or you would have to use r'raw strings' to suppress the backslash interpretation. Otherwise, 'c:\for\a_path\like\this' actually contains \f, \a, and \t escape sequences that you may not notice in the time of writing... and they may be source of headaches in future.
I am trying to run a program from the command prompt in windows. I am having some issues. The code is below:
commandString = "'C:\Program Files\WebShot\webshotcmd.exe' //url '" + columns[3] + "' //out '"+columns[1]+"~"+columns[2]+".jpg'"
os.system(commandString)
time.sleep(10)
So with the single quotes I get "The filename, directory name, or volume label syntax is incorrect." If I replace the single quotes with \" then it says something to the effect of "'C:\Program' is not a valid executable."
I realize it is a syntax error, but I am not quite sure how to fix this....
column[3] contains a full url copy pasted from a web browser (so it should be url encoded). column[1] will only contain numbers and periods. column[2] contains some text, double quotes and colons are replaced. Mentioning just in case...
Thanks!
Windows requires double quotes in this situation, and you used single quotes.
Use the subprocess module rather than os.system, which is more robust and avoids calling the shell directly, making you not have to worry about confusing escaping issues.
Dont use + to put together long strings. Use string formatting (string %s" % (formatting,)), which is more readable, efficient, and idiomatic.
In this case, don't form a long string as a shell command anyhow, make a list and pass it to subprocess.call.
As best as I can tell you are escaping your forward slash but not your backslashes, which is backwards. A string literal with // has both slashes in the string it makes. In any event, rather than either you should use the os.path module which avoids any confusion from parsing escapes and often makes scripts more portable.
Use the subprocess module for calling system commands. Also ,try removing the single quotes and use double quotes.