Develop Reference

In Python, with subprocess.Popen, is it possible to pass literal quotes to the command to be run, when Popen's command line parameter is in list form?

In Python, with subprocess.Popen, is it possible to pass literal quotes as an argument, when the command and its parameters are in list form?
I'll explain further what I mean. Some commands can have literal quotes in their arguments e.g. I'm trying "C:\Program Files (x86)\Google\Chrome\Application\chrome.exe" --profile-directory="Profile 1" Some might even require them.
Note that one answer points out that technically it is possible to get Chrome from the command line to launch whatever profile, without passing a literal quote C:\Users\User>"C:\Program Files....\chrome.exe" "--profile-directory=Profile 2"
Nevertheless, i'm asking about passing literal quotes so "C:\Program Files (x86)\Google\Chrome\Application\chrome.exe" --profile-directory="Profile 1"
For simplicity's sake i'll use calc.exe since it's in the path.
import time
import subprocess
proc=subprocess.Popen("calc.exe"+" "+'--profile-directory="Profile 3"')
proc2=subprocess.Popen(["calc.exe",'--profile-directory="Profile 4"'])
time.sleep(3)
proc.wait()
proc2.wait()
Now look at the difference in the command line as visible in task manager or via wmic.
C:\Users\User>wmic process where caption="calc.exe" get commandline | findstr calc
c:\windows\system32\calc.exe --profile-directory="Profile 3"
c:\windows\system32\calc.exe "--profile-directory=\"Profile 4\""
C:\Users\User>
You can see this from the python interpreter
>>> subprocess.Popen(["c:/windows/system32/calc.exe","abc"+'"'+"def"])
...
>>>
>>> subprocess.run("C:\Windows\System32\wbem\WMIC.exe process where caption=\"calc.exe\" get commandline")
...
c:/windows/system32/calc.exe abc\"def
....
>>>
You see it's sticking a backslash in there.
Some comments regarding some suggestions given.
One suggestion assumes that --profile-directory="Profile 1" is the same as --profile-directory "Profile 1" i.e. the assumption that you can replace the = with a space and chrome will work the same. But that isn't the case. So writing subprocess.Popen(["C:\...\chrome.exe", "--profile-directory", "Profile 3"]) will indeed produce "C:\....\chrome.exe" --profile-directory "Profile 1" but that won't work.. it leads chrome to either not open at all, or to open a browser window that offers profiles to click on. The equals sign is necessary.
Another suggestion does
subprocess.Popen(
" ".join(
[
"C:\\Program Files\\Google\\Chrome\\Application\\chrome.exe",
'--profile-directory="Person 1"',
]
)
That's not passing a list to Popen, that's passing a list to join, and join is converting it to a string.
Another suggestion is
subprocess.Popen('C:\\Program Files (x86)\\Google\\Chrome\\Application\\chrome.exe --profile-directory="Profile 3"')
That's using a string. But as you see from my question, I managed it using a string. I'm asking about using a list.
Another suggestion suggested "--profile-directory='Profile 1'"
If I run chrome with --profile-directory="Profile 1" I get a particular profile that I use sometimes. But if running chrome with "--profile-directory='Profile 1'" Then it doesn't load up that profile. It loads up a blank profile. And going to chrome://version shows "'profile 1'" rather than "profile 1" It's like a different profile, like you may as well have said chrome.exe --profile-directory="profile A". And it also creates directories starting with ' like C:\Users\User\AppData\Local\Google\Chrome\User Data\'Profile 1234' that should be removed.
Another suggestion suggested
subprocess.Popen(
[
"C:\\Program Files (x86)\\Google\\Chrome\\Application\\chrome.exe",
"--profile-directory=Profile 1",
]
That is interesting because it does "C:\...chrome.exe" "--profile-directory=Profile 1"
And it does infact load chrome with the specified profile. Though it doesn't try to pass literal quotes!
My question asks about when passing literal quotes. It's as if maybe it assumes it's a linux shell and inserts a backslash before it, which in a linux would ensure the quote makes it past the shell and to the program being run. Though i'm not sure it'd even go to the linux shell on linux. e.g. on Windows if I stick a cmd escape character in there like ^ so "--pro^file-directory=Profile 1" then the ^ just gets passed literally. So the cmd shell doesn't intervene.
Why is it that on Windows, subprocess.Popen calls list2cmdline when passed a list, which(and here's the big 'why'), then adds a backslash to any literal double quote within a string, meaning that when using the 'method' of passing a list to to Popen rather than passing a string to it, there is this problem, that you can't pass a literal double quote! So, why does it add that backslash!
I did here a suggestion that looking at argsv in windows vs linux might show a difference. I'm not sure that they would since both implement C.
I don't see why POpen in any situation should behave like Windows needs a backslash inserted more than Linux does.
$ cat ./testargs.py
#!/usr/bin/env python3
import sys
print(sys.argv)
C:\blah>type .\testargsw.py
import sys
print(sys.argv)
in both cases
C:\blah>.\testargsw.py abc\^"def
['C:\\Users\\User\\testargsw.py', 'abc"def']
>.\testargsw.py abc\"def
['C:\\Users\\User\\testargsw.py', 'abc"def']
C:\blah>
$ ./testargs.py abc\"def
['./testargs.py', 'abc"def']
Maybe Windows , specifically the MS C Runtime.. The Code responsible for sending a program's arguments received from the shell, to the main method into argv, is requiring an extra backslash, in a sense because after escaping the double quote, a backslash is then required. (And [here] is put in by the user).
That said, I have heard though that looking at what a shell does on Linux is basically misleading, because a major part of the purpose of the subprocess module is to ensure that you can avoid using a shell entirely.
The script example is perhaps not that relevant(it was just something somebody suggested I check), but my issue is that POpen when passed a list is adding in a backslash as shown by WMIC output(also visible in task manager in the command line column).
added
I spoke to a person that has used python for a long time. They said subprocess was added somewhere in 2.x They still use os.popen(). That takes a string not a list. There have been moves to shift people from os.popen to subprocess.Popen https://docs.python.org/3/library/subprocess.html#replacing-os-popen-os-popen2-os-popen3
An issue with subprocess.Popen in Windows, is it has this list feature, that I think behaves funny.
The easy workaround to that is to not use the list feature of it. To not pass it a list. It's a new feature and not necessary. You can pass it a string.
The question includes an example from the python interpreter and shows how (on windows at least), python adds a backslash to the literal quote.
The person I spoke to pointed out to me two documents that relate to that.
A string is a sequence. A sequence could be a string or list or tuple, though in this document they use the term sequence to just mean list or tuple, and they don't mean string when they say sequence.
https://peps.python.org/pep-0324/
"class Popen(args........."
"args should be a string, or a sequence of program arguments"
It mentions about on unix, shell=True and shell=False
And then it says
"On Windows: the Popen class uses CreateProcess() to execute the child program, which operates on strings. If args is a sequence, it will be converted to a string using the list2cmdline method. Please note that not all MS Windows applications interpret the command line the same way: The list2cmdline is designed for applications using the same rules as the MS C runtime."
Technically a string is a sequence, but that document uses the term sequence in a funny way. But what it means is it's that on Windows, if args is not given a string, but is given a list or tuple, then it uses the list2cmdline method.
Be sure to use print otherwise it uses repr() of the string
>>> print(subprocess.list2cmdline(['a', '"b c"']))
a "\"b c\""
>>>
so that's the function that it's using behind the scenes, on windows, that is inserting a backslash in there.
The guy I spoke to pointed me to this document too
https://bugs.python.org/issue11827
a technical user comments, "list2cmdline() in subprocess is publicly accessible (doesn't begin with underscores) but it isn't documented."
And the point is made there that, let's say they made list2cmdline() private, the fact is that what Popen is doing to the list, in Windows, to get the command line, is undocumented.
So the question then becomes, what is the design trying to do, what is the justification for the insertion of backslash. If a programmer wanted to insert a backslash they could do so. It seeems to me to make more sense then to avoid passing a list to subprocess.POpen.
Windows cmd doesn't even use backslash as an escape character!!!! It uses caret.
C:\Users\User>echo \\
\\
C:\Users\User>echo ^\
\
C:\Users\User>
it's linux eg bash, that uses backslash as an escape character
$ echo \\
\
$
Some executables in windows might want a quote escaped and with a backslash, but then the technical user can do that just as a technical linux user does.
So given that they haven't even documented the "feature" (or bug), how they would justify it, I don't know, but they could start by documenting it!
So I don't understand why passing a list to subprocess.ppopen is adding a backslash?
I could take the list join it with space and pass it as a string to popen, so it won't add a backslash, but as mentioned, that'd be avoiding the question.

In Python, with subprocess.Popen, is it possible to pass literal
quotes as an argument, when the command and its parameters are in list
form?
I think yes, but to elements of argsv, not to the command line..
I'll explain further what I mean. Some commands can have literal
quotes in their arguments e.g. I'm trying "C:\Program Files (x86)\Google\Chrome\Application\chrome.exe" --profile-directory="Profile 1" Some might even require them.
Note that one answer points out that technically it is possible to get
Chrome from the command line to launch whatever profile, without
passing a literal quote C:\Users\User>"C:\Program Files....\chrome.exe" "--profile-directory=Profile 2"
Nevertheless, i'm asking about passing literal quotes so
"C:\Program Files (x86)\Google\Chrome\Application\chrome.exe" --profile-directory="Profile 1"
I think there was some confusion, that I might have resolved now.
When running a binary executable in windows(so, an exe, not a bat file) and giving it arguments
People often speak of how linux and windows behave differently..
Windows doesn't separate arguments out, it just dumps stuff to the command line. And there's no concept of space separating out arguments.
Apparently in linux, the command line is parsed by the shell and split into arguments and the arguments are passed by the shell to a POSIX function called execv, which then get passed to main's argsv.
Whereas in windows, there's an aspect of C pertaining to Windows called the MS C Runtime, that parses the command line and splits it into arguments.
So when looking in task manager or from WMIC, at the command line, it's just a dump when it comes to quotes. The quotes aren't some special some literal. But when the MS C Runtime looks at it then the quotes will have that meaning of some special some literal, a special one will not go to argsv, and a literal one will. Usually they are all special, none are literal.
>calc """"""""""^G
>wmic process where caption="calc.exe" get commandline | findstr calc
calc """"""""""G
One we see what's in the command line then one can consider what the arguments are.. what would would go to argsv.
In the case of the two Chrome examples, there's actually no literal quote, in the sense of a literal quote that'd go to argsv. in either example
Example 1
"C:\Program Files(x86)\Google\Chrome\Application\chrome.exe" --profile-directory="Profile 1"
Example 2
"C:\Program Files....\chrome.exe" "--profile-directory=Profile 2"
Looking at "Example 1"
It's not that there's an argument --profile-directory="Profile 1" that has two literal quotes.
The quotes when read by MS CRT(MS C runtime), will just preserve space keeping that "1" as part of the same argument.
So a program that displays argv will show
C:\blah>type w.c
#include <stdio.h>
int main(int argc, char *argv[]) {
int i = 0;
while (argv[i]) {
printf("argv[%d] = %s\n", i, argv[i]);
i++;
}
return 0;
}
C:\blah>w.exe --profile-directory="Profile 1"
argv[0] = w.exe
argv[1] = --profile-directory=Profile 1
C:\blah>
See, no literal quotes there
Infact both different command lines, will produce the same thing going to argv
>w.exe "--profile-directory=Profile 1"
argv[0] = w.exe
argv[1] = --profile-directory=Profile 1
>w.exe --profile-directory="Profile 1"
argv[0] = w.exe
argv[1] = --profile-directory=Profile 1
>
So, how would you even get a literal quote there, well, this link mentions it https://learn.microsoft.com/en-us/cpp/c-language/parsing-c-command-line-arguments?view=msvc-170
the find command might have funny parsing, but if you are using find or a windows implementation of grep.. and wanted to look for a quote.. Then, you'd want one of the argsv parameters to contain a quote.
And that's when you'd put a backslash in there
>w.exe \"
argv[0] = w.exe
argv[1] = "
>
The command line would have a backslash.. So that the argv would have a literal quote.
And that's what passing a list to subprocess.Popen is all about..
You are putting in what you want to be in the argv.
It is then producing the command line that would result in those things being in the argv.
That's why it's inserting a backslash before the double quotes!
So..
When I trying to use list, I was under the mistaken impression that I needed to get literal quotes into the arguments. And I thought I was forming the command line. Both of those premises were wrong.
I suppose if a python script has two options.. one to run an executable for linux users, and one to run an executable for windows users. The command line might be different, because on the one hand, the windows command line (either what's typed or it, what windows dumps as it), and on the other hand, what's typed on the linux command line.
And with using a list where each element is an element of argsv, it lets the library do the work, in the case of windows, using the list2cmd function, to produce whatever the command line would be in order for the MS C Runtime to produce the result you want in argsv.
side note
In the testargs.py file used in the question, it'd not give expected output https://learn.microsoft.com/en-us/cpp/c-language/parsing-c-command-line-arguments?view=msvc-170 for some strings e.g. `"ab\"c" "\\" d` because it is giving the repl representation Adjust it as follows.
user#comp:~# cat ./testargs.py
#!/usr/bin/env python3
import sys
print(sys.argv)
user#comp:~# cat ./testargs_corrected.py
#!/usr/bin/env python3
import sys
# print(sys.argv) would behaves unintended for "ab\"c" "\\" d
# (it's escaping strings for output in an repr)
# so not using print(sys.argv), using below line instead.
# '//' that it shows, means / 'cos '//' is a single character in python
# but much clearer to show it not in the repl form.
for n, a in enumerate(sys.argv): print(f"argv[{n}] = {a}")
user#comp:~# ./testargs.py "ab\"c" "\\" d
['./testargs.py', 'ab"c', '\\', 'd']
user#comp:~#
user#comp:~# ./testargs_corrected.py "ab\"c" "\\" d
argv[0] = ./testargs_corrected.py
argv[1] = ab"c
argv[2] = \
argv[3] = d
user#comp:~#

Should do the trick (might wanna pass shell=True to Popen if it doesn't):
subprocess.Popen(["C:\Program Files (x86)\Google\Chrome\Application\chrome.exe", "--profile-directory", "Profile 3"]);
This is possible because --some-flag="some value" is the same as --some-flag "some value"
ChRomE solution (working, omg):
import subprocess
subprocess.Popen(
[
"C:\\Program Files (x86)\\Google\\Chrome\\Application\\chrome.exe",
"--profile-directory=Profile 1",
]
)

Calling batch commands from python

I am using python with some external program API in order to automate some stuff.
Have in mind that these "extpro" and "--file" and "--dont-breakaway-from-job" are built in commands that I have to use, my code looks like this:
send = os.system('extpro --file '(os.path.join(base_dir, extpr_path))' --dont-breakaway-from-job')
Error that I am getting is on the --dont-breakaway-from-job, saying Expected ")" pylance.
But when I try this there is no error:
send = os.system('extpro --file "C:/user/program/run.exe" --dont-breakaway-from-job')
Anyone have idea what might be behind this behavior?
Is there a way maybe to split whole command into two or three?
Any advice is welcome, thanks!

os.path.join returns str object, thus your
'extpro --file '(os.path.join(base_dir, extpr_path))' --dont-breakaway-from-job'
expands into:
'extpro --file '"<whatever that path is>"' --dont-breakaway-from-job'
^
In the position marked with ^ the string ends. And, since os.system expects only one single parameter, pylance (and interpreter too) supposes, that this parameter is already passed and throws an error, that closing bracket is expected.
Interpreter doesn't concatenate os.path.join result with string before it, because it doesn't know, that os.path.join is a string. Function call will become string only in runtime.
However, your second variant contains 'some str "inner str" some more'. Python interpreter sees string, starting with ' (single qotation mark) and looks for a matching pair, that will mean end of the string. All " (double quotation marks) between single ones are considered part of the string.
Solution is simple. You can do any of:
# Concatenating strings with +
send = os.system('extpro --file "'+ os.path.join(base_dir, extpr_path) + '" --dont-breakaway-from-job')
# Using format (or f-strings, ifyou're using python 3.6+)
send = os.system('extpro --file "{}" --dont-breakaway-from-job'.format(os.path.join(base_dir, extpr_path)))
# or
send = os.system(f'extpro --file "{os.path.join(base_dir, extpr_path)}" --dont-breakaway-from-job')
In any of 3 variations it's worth wrapping os.path.join results with " (double quotation mark) in case it contains spaces or other undesired symbols, that may e parsed incorrectly

Your line send = os.system('extpro --file '(os.path.join(base_dir, extpr_path))' --dont-breakaway-from-job') doesn't create the string that you expect/need and doesn't concatenate 'extpro --file ' with the filepath (within "!) and the ' --dont-breakaway-from-job').
The easiest solution might be to use a F-string:
send = os.system(f'extpro --file "{os.path.join(base_dir, extpr_path)}" --dont-breakaway-from-job')
Important here is also to add " to the string, since os.path.join(...) doesn't create a string with the ".

If i understood it right, the issue is that you need to concatenate the string with quotation marks? If that is the case, you can use the following
send = os.system("extpro --file \""+os.path.join(base_dir, extpr_path)+"\" --dont-breakaway-from-job")
The os.path.join returns a string which needs to be concatenated using +. Furthermore, you need to include quotation marks around os.path.join. This can be achieved in many ways, but the one i used in this answer is \".

re.escape returns unusable directory

using re.escape() on this directory:
C:\Users\admin\code
Should theoratically return this, right?
C:\\Users\\admin\\code
However, what I actually get is this:
C\:\\Users\\admin\\code
Notice the backslash immediately after C. This makes the string unusable, and trying to use directory.replace('\', '') just bugs out Python because it can't deal with a single backslash string, and treats everything after it as string.
Any ideas?
Update
This was a dumb question :p

No it should not. It's help says "Escape all the characters in pattern except ASCII letters, numbers and '_'"
What you are reporting you are getting is after calling the print function on the resulting string. In console, if you type directory and press enter, it would give something like: C\\:\\\\Users\\\\admin\\\\code. When using directory.replace('\\','') it would replace all backslashes. For example: directory.replace('\\','x') gives Cx:xxUsersxxadminxxcode. What might work in this case is replacing both the backslash and colon with ':' i.e. directory.replace('\\:',':'). This will work.
However, I will suggest doing something else. A neat way to work with Windows directories in Python is to use forward slash. Python and the OS will work out a way to understand your paths with forward slashes. Further, if you aren't using absolute paths, as far as the paths are concerned, your code will be portable to Unix-style OSes.
It also seems to me that you are calling re.escape unnecessarily. If the printing the directory is giving you C:\Users\admin\code then it's a perfectly fine directory to use already. And you don't need to escape it. It's already done. If it wasn't escaped print('C:\Users\admin\code') would give something like C:\Usersdmin\code since \a has special meaning (beep).

Why do I need 4 backslashes in a Python path?

When I'm using Python 3 to launch a program via subprocess.call(), why do I need 4 backslashes in paths?
This is my code:
cmd = 'C:\\\\Windows\\\\System32\\\\cmd.exe'
cmd = shlex.split(cmd)
subprocess.call(cmd)
When I examine the command line of the launched cmd.exe instance with Task Manager, it shows the path correctly with only one backslash separating each path.
Because of this, I need this on Windows to make the paths work:
if platform.platform().startswith('Windows'):
cmd = cmd.replace(os.sep, os.sep + os.sep)
is there a more elegant solution?

Part of the problem is that you're using shlex, which implements escaping rules used by Unix-ish shells. But you're running on Windows, whose command shells use different rules. That accounts for one level of needing to double backslashes (i.e., to worm around something shlex does that you didn't need to begin with).
That you're using a regular string instead of a raw string (r"...") accounts for the other level of needing to double backslashes, and 2*2 = 4. QED ;-)
This works fine on Windows:
cmd = subprocess.call(r"C:\Windows\System32\cmd.exe")
By the way, read the docs for subprocess.Popen() carefully: the Windows CreateProcess() API call requires a string for an argument. When you pass a sequence instead, Python tries to turn that sequence into a string, via rules explained in the docs. When feasible, it's better - on Windows - to pass the string you want directly.

When you are creating the string, you need to double each backslash for escaping, and then when the string is passed to your shell, you need to double each backslash again. You can cute the backslashes in half by using a raw string:
cmd = r'C:\\Windows\\System32\\cmd.exe'

\ has special meaning - you're using it as part of an escape sequence. Double up the backslashes, and you have a literal backslash \.
The caveat is that, with only one pair of escaped backslashes, you still have only one literal backslash. You need to escape that backslash, too.
Alternatively, why not just use os.sep instead? You'll be able to ensure your code is more portable (since it'll use the system-specific separator), and you won't have to deal [directly] with escaping backslashes.

As John points out 4 slashes isn't necessary when accessing files locally.
One place where 4 slashes is necessary is when connecting to (generally windows) servers over SMB or CIFS.
Normally you would just use \servername\share\
But each one of those slashes needs to be escaped. So thus the 4 slashes before servernames.
you could also use subprocess.call()
import subprocess as sp
sp.call(['c:\\program files\\<path>'])

Quote POSIX shell special characters in Python output

There are times that I automagically create small shell scripts from Python, and I want to make sure that the filename arguments do not contain non-escaped special characters. I've rolled my own solution, that I will provide as an answer, but I am almost certain I've seen such a function lost somewhere in the standard library. By “lost” I mean I didn't find it in an obvious module like shlex, cmd or subprocess.
Do you know of such a function in the stdlib? If yes, where is it?
Even a negative (but definite and correct :) answer will be accepted.

pipes.quote():
>>> from pipes import quote
>>> quote("""some'horrible"string\with lots of junk!$$!""")
'"some\'horrible\\"string\\\\with lots of junk!\\$\\$!"'
Although note that it's arguably got a bug where a zero-length arg will return nothing:
>>> quote("")
''
Probably it would be better if it returned '""'.

The function I use is:
def quote_filename(filename):
return '"%s"' % (
filename
.replace('\\', '\\\\')
.replace('"', '\"')
.replace('$', '\$')
.replace('`', '\`')
)
that is: I always enclose the filename in double quotes, and then quote the only characters special inside double quotes.