I want to set a parameter for a python script by using the parameter field in PyCharm.
My config:
But the command in the Run console is:
python3 path_to_script.py '{app_id: picoballoon_network, dev_id: ferdinand_8c ... and so on
and not:
python3 path_to_script.py '{"app_id": "picoballoon_network", "dev_id": "ferdinand_8c" ... and so on
Basically, it deletes all " in the parameter.
Does anyone know how to turn this off?
My PyCharm version is:
PyCharm 2020.3.1 (Professional Edition)
Build #PY-203.6682.86, built on January 4, 2021
Runtime version: 11.0.9.1+11-b1145.37 amd64
VM: OpenJDK 64-Bit Server VM by JetBrains s.r.o.
Windows 10 10.0
To avoid the quotation marks being deleted notice the rules to writing parameters that contain quotation marks.
Run/Debug Configuration: Python
Configuration tab
When specifying the script parameters, follow these rules:
Use spaces to separate individual script parameters.
Script parameters containing spaces should be delimited with double quotes, for example, some" "param or "some param".
If script parameter includes double quotes, escape the double quotes with backslashes, for example:
-s"main.snap_source_dirs=[\"pcomponents/src/main/python\"]" -s"http.cc_port=8189"
-s"backdoor.port=9189"
-s"main.metadata={\"location\": \"B\", \"language\": \"python\", \"platform\": \"unix\"}"
The case in the question would be a single parameter, lets apply the rules to the example:
'{"app_id": "picoballoon_network", "dev_id": "ferdinand_8c"'
Because it's a single parameter containing spaces it has to be surounded by quotation marks.
Since the content of the parameter also contains quotation marks they must be escaped using a backslash \. So applying the parameter formatting rules gives:
"'{\"app_id\": \"picoballoon_network\", \"dev_id\": \"ferdinand_8c\"}'"
(Side note): In the example the parameter was surrounded by Apostrophes, this may be unnecessary and will probably have to be stripped later in your Python code (the below example uses the strip method).
You can test it with this simple script:
import sys
import ast
your_dictionary = ast.literal_eval(sys.argv[1].strip("'"))
(Side note): Your example parameter is a string containing a Python dictionary, there are several ways to convert it, in the example I included the highest voted answer from this question: "Convert a String representation of a Dictionary to a dictionary?"
A screenshot showing the parameter and test code in use:
Related
I need to transform a string containing single and double quotes and newline characters for use in a system call. Consider the following input string:
"""one'two\nthree"four"""
This should be transformed to the following output string:
"$'one\'two\nthree\"four'"
So that it can be submitted as a message argument in a git commit command:
git commit --message=$'one\'two\nthree\"four'
The odd syntax with the leading $ and surrounding single quotes ' is a bash construct described in the bash manpage in the section on quoting (search for QUOTING). I have tried many python functions including str.replace, re.sub, json.dumps, repr, and str.encode('unicode-escape'). But none have yielded the required transformation. It seems that, in this case, python is too high-level for its own good. Suggestions on how to proceed will be very gratefully received.
The system call itself will be made using code like this (omitting the try block for clarity):
import subprocess
call = ["git", "commit", "--message", "'one\'two\nthree\"four'"]
cpi = subprocess.run(call)
I may also use a git library of some description, but I have not done my homework yet for that.
Note: the unnecessary $ character in the last item in the above call list has been removed.
Your wanted command is erroneous at the moment, it is not a valid Python string since a right " is missing, it should be:
"$'one\'two\nthree\"four'"
This is easily constructed by a simple .format call:
>>> "$'{}'".format("""one'two\nthree"four""") == "$'one\'two\nthree\"four'"
True
When running the following script:
import os
import sys
import subprocess
if len(sys.argv) > 1:
print sys.argv[1]
sys.exit(0)
commandline = [sys.executable]
commandline.append(os.path.realpath(__file__))
commandline.append('"test"')
p = subprocess.Popen(commandline)
p.wait()
p = subprocess.Popen(" ".join(commandline))
p.wait()
It returns the following output
"test"
test
Why is there a difference between providing a list of arguments or one string?
This is run on a windows machine and you will see backslashes before the quotes on the command in the task manager.
I expected the same result in both runs.
Edit:
The problem is not so much in the automatic escaping of spaces (I find that is the programmers responsibility), but more about my quotes being escaped or not in the process commandline.
These are the two subprocesses taken from the windows task manager:
A different non-python process parses the first commandline with the backslashes, which brings unexpected behaviour. How can I have it so that I can use a list and not have the quotes escaped on the commandline?
Edit2:
The quotes are definitely added by python. If you run the following:
import subprocess
commandline = ['echo']
commandline.append('"test"')
commandline.append('>')
commandline.append(r'D:\test1.txt')
p = subprocess.Popen(commandline, shell=True)
p.wait()
commandline = 'echo "test" > D:\\test2.txt'
p = subprocess.Popen(commandline, shell=True)
p.wait()
Then you will see that the outputs are
D:\test1.txt:
\"test\"
D:\test2.txt:
"test"
The string API is dangerous since it might change the meaning of arguments. Example: You want to execute C:\Program Files\App\app.exe. If you use the string version of Popen(), you get an error:
C:\Program: Error 13
What happens is that with the string API, Python will split the input by spaces and try to start the command C:\Program with the single argument Files\App\app.exe. To fix this, you need to quote properly. Which gets you in quote hell when you have quotes in your arguments (i.e. when you really want to pass "test" as an argument with the quotes).
To solve this (and other subtle) bugs, there is the list API where each element of the list will become a single item passed to the OS without any modifications. With list API, you get what you see. If you quote an argument with the list API, it will be passed on with the quotes. If there are spaces, they won't split your argument. If there are arbitrary other special characters (like * or %), they will all be passed on.
[EDIT] As usual, things are much more complex on Windows. From the Python documentation for the subprocess module:
17.1.5.1. Converting an argument sequence to a string on Windows
On Windows, an args sequence is converted to a string that can be parsed using the following rules (which correspond to the rules used by the MS C runtime):
Arguments are delimited by white space, which is either a space or a tab.
A string surrounded by double quotation marks is interpreted as a single argument, regardless of white space contained within. A quoted string can be embedded in an argument.
A double quotation mark preceded by a backslash is interpreted as a literal double quotation mark.
Backslashes are interpreted literally, unless they immediately precede a double quotation mark.
If backslashes immediately precede a double quotation mark, every pair of backslashes is interpreted as a literal backslash. If the number of backslashes is odd, the last backslash escapes the next double quotation mark as described in rule 3.
So the backslashes are there because MS C runtime wants it that way.
Given this variables:
cardIP="00.00.00.00"
dir="D:\\TestingScript"
mainScriptPath='"\\\\XX\\XX\\XX\\Testing\\SNMP Tests\\Python Script\\MainScript.py"'
When using subprocess.call("cmd /c "+mainScriptPath+" "+dir+" "+cardIP) and print(mainScriptPath+" "+dir+" "+cardIP) I get this:
"\\XX\XX\XX\Testing\SNMP Tests\Python Script\MainScript.py" D:\TestingScript 00.00.00.00
which is what I wanted, OK.
But now, I want the 'dir' variable to be also inside "" because I am going to use dir names with spaces.
So, I do the same thing I did with 'mainScriptPath':
cardIP="00.00.00.00"
dir='"D:\\Testing Script"'
mainScriptPath='"\\XX\\XX\\XX\\Testing\\SNMP Tests\\Python Script\\MainScript.py"'
But now, when I'm doing print(mainScriptPath+" "+dir+" "+cardIP) I get:
"\\XX\XX\XX\Testing\SNMP Tests\Python Script\MainScript.py" "D:\Testing Script" 00.00.00.00
Which is great, but when executed in subprocess.call("cmd /c "+mainScriptPath+" "+dir+" "+cardIP) there is a failure with 'mainScriptPath' variable:
'\\XX\XX\XX\Testing\SNMP' is not recognized as an internal or external command...
It doesn't make sense to me.
Why does it fail?
In addition, I tried also:
dir="\""+"D:\\Testing Script"+"\""
Which in 'print' acts well but in 'subprocess.call' raise the same problem.
(Windows XP, Python3.3)
Use proper string formatting, use single quotes for the formatting string and simply include the quotes:
subprocess.call('cmd /c "{}" "{}" "{}"'.format(mainScriptPath, dir, cardIP))
The alternative is to pass in a list of arguments and have Python take care of quoting for you:
subprocess.call(['cmd', '/c', mainScriptPath, dir, cardIP])
When the first argument to .call() is a list, Python uses the process described under the section Converting an argument sequence to a string on Windows.
On Windows, an args sequence is converted to a string that can be
parsed using the following rules (which correspond to the rules used
by the MS C runtime):
Arguments are delimited by white space, which is either a space or a tab.
A string surrounded by double quotation marks is interpreted as a single argument, regardless of white space contained within. A quoted
string can be embedded in an argument.
A double quotation mark preceded by a backslash is interpreted as a literal double quotation mark.
Backslashes are interpreted literally, unless they immediately precede a double quotation mark.
If backslashes immediately precede a double quotation mark, every pair of backslashes is interpreted as a literal backslash. If the
number of backslashes is odd, the last backslash escapes the next
double quotation mark as described in rule 3.
This means that passing in your arguments as a sequence makes Python worry about all the nitty gritty details of escaping your arguments properly, including handling embedded backslashes and double quotes.
I am writing documentation for a project and I would like to make sure I did not miss any method. The code is written in Python and I am using PyCharm as an IDE.
Basically, I would need a REGEX to match something like:
def method_name(with, parameters):
someVar = something()
...
but it should NOT match:
def method_name(with, parameters):
""" The doc string """
...
I tried using PyCharm's search with REGEX feature with the pattern ):\s*[^"'] so it would match any line after : that doesn't start with " or ' after whitespace, but it doesn't work. Any idea why?
You mentioned you were using PyCharm: there is an inspection "Missing, empty, or incorrect docstring" that you can enable and will do that for you.
Note that you can then change the severity for it to show up more or less prominently.
There is a tool called pydocstyle which checks if all classes, functions, etc. have properly formatted docstrings.
Example from the README:
$ pydocstyle test.py
test.py:18 in private nested class `meta`:
D101: Docstring missing
test.py:27 in public function `get_user`:
D300: Use """triple double quotes""" (found '''-quotes)
test:75 in public function `init_database`:
D201: No blank lines allowed before function docstring (found 1)
I don't know about PyCharm, but pydocstyle can, for example, be integrated in Vim using the Syntastic plugin.
I don't know python, but I do know my regex.
And your regex has issues. First of all, as comments have mentioned, you may have to escape the closing parenthesis. Secondly, you don't match the new line following the function declaration. Finally, you look for single or double quotations at the START of a line, yet the start of a line contains whitespace.
I was able to match your sample file with \):\s*\n\s*["']. This is a multiline regex. Not all programs are able to match multiline regex. With grep, for example, you'd have to use this method.
A quick explanation of what this regex matches: it looks for a closing parenthesis followed by a semicolon. Any number of optional whitespace may follow that. Then there should be a new line followed by any number of whitespace (indentation, in this case). Finally, there must be a single or double quote. Note that this matches functions that do have comments. You'd want to invert this to find those without.
In case PyCharm is not available, there is a little tool called ckdoc written in Python 3.5.
Given one or more files, it finds modules, classes and functions without a docstring. It doesn't search in imported built-in or external libraries – it only considers objects defined in files residing in the same folder as the given file, or subfolders of that folder.
Example usage (after removing some docstrings)
> ckdoc/ckdoc.py "ckdoc/ckdoc.py"
ckdoc/ckdoc.py
module
ckdoc
function
Check.documentable
anykey_defaultdict.__getitem__
group_by
namegetter
type
Check
There are cases when it doesn't work. One such case is when using Anaconda with modules. A possible workaround in that case is to use ckdoc from Python shell. Import necessary modules and then call the check function.
> import ckdoc, main
> ckdoc.check(main)
/tmp/main.py
module
main
function
main
/tmp/custom_exception.py
type
CustomException
function
CustomException.__str__
False
The check function returns True if there are no missing docstrings.
Is there any way I can tell argparse to not eat quotation marks?
For example, When I give an argument with quotes, argparse only takes what's inside of the quotes as the argument. I want to capture the quotation marks as well (without having to escape them on the command line.)
pbsnodes -x | xmlparse -t "interactive-00"
produces
interactive-00
I want
"interactive-00"
I think it is the shell that eats them, so python will actually never see them. Escaping them on the command line may be your only option.
If it's the \"backslash\" style escaping you don't like for some reason, then this way should work instead:
pbsnodes -x | xmlparse -t '"interactive-00"'
Command line is parsed into argument vector by python process itself. Depending on how python is built, that would be done by some sort of run-time library. For Windows build, that would be most likely MS Visual C++ runtime library. More details about how it parses command line can be found in Visual C++ documentation: Parsing C++ command-Line arguments.
In particular:
A string surrounded by double quotation marks ("string") is interpreted as a single argument, regardless of white space contained within. A quoted string can be embedded in an argument.
A double quotation mark preceded by a backslash (\") is interpreted as a literal double quotation mark character (").
If you want to see unprocessed command line, on Windows you can do this:
import win32api
print(win32api.GetCommandLine())