Develop Reference

Windows cmd python subprocess.call

I have a command which I can run manually from the windows command prompt, and it works as expected:
> gridcoinresearchd sendmany "Default" {"""x9Bqs6dTm17tsiEyZ9m4j5Yd8N2dpqw""":1.00000000} 2 "Hello World"
So I build my subprocess.call() as follows from a set of variables, some of which are derived from other variables, where;
call_insert = [val for pair in zip(quotes, quotes, quotes, address, quotes, quotes, quotes, colon, call_amount, comma) for val in pair]
call_insert = str("{"+(''.join(call_insert))+"}")
account_label = str('"'+(raw_input("Choose Account Label: "))+'"')
message = str('"'+(raw_input("Enter if you wish to send a message: "))+'"')
then:
subprocess.call(['gridcoinresearchd', 'sendmany', account_label, call_insert, "2", message], shell=True)
when using the subprocess.call I get an error from the target program:
error: Error parsing JSON:{x9Bqs6dTm17tsiEyZ9m4j5Yd8N2dpqw:1.00000000}
if I manually make the variable:
call_insert = str("""{"x9Bqs6dTm17tsiEyZ9m4j5Yd8N2dpqw":1.00000000}""")
then my subprocess.call() works.
So this I think is my confusion with how the windows command prompt is interpreting the quotes I need in my subprocess.call() strings.
I hope that makes sense.

If we inspect the given error message
{x9Bqs6dTm17tsiEyZ9m4j5Yd8N2dpqw:1.00000000}
and compare it to your desired JSON input
{"""x9Bqs6dTm17tsiEyZ9m4j5Yd8N2dpqw""":1.00000000}
we can see that double quotes are missing in the first statement, and it is probably the main reason why JSON is not parsed.
So basically I may suspect that call_insert variable is not formed as you expected, i.e. does not contain double quotes, and is equal to:
str({x9Bqs6dTm17tsiEyZ9m4j5Yd8N2dpqw:1.00000000})
I suggest you to look closer into the first 2 lines of your example and into the code above it to check its output.

OK by trial and error I found that the string must be in this form:
{"x9Bqs6dTm17tsiEyZ9m4j5Yd8N2dpqw":1.00000000}

Python encoding issue in script if string not hard-coded

I have an encoding issue with strings I get from an external source.
This source sends the strings encoded to me and I can decode them only if they are part of the script's code.
I've looked at several threads here and even some recommended tutorials (such as this one) but came up empty.
For example, if I run this:
python -c 'print "gro\303\237e"'
I get:
große
Which is the correct result.
But If I use it in a script, such as:
import sys
print sys.argv[1]
and call it like test.py "gro\303\237e", I get:
gro\303\237e
I intend to write the correct string to syslog, but I can't seem to get this to work.
Some data on my system:
- Python 2.7.10
- CentOS Linux
- LANG=en_US.UTF-8
- LC_CTYPE=UTF-8
I will appreciate any help, please let me know if you need more information.
Thanks!

If you really have the chars gro\303\237e which is something else as "gro\303\237e" (the first one are the chars g r o \ 3 0 3 \ 2 3 7, the second one is the chars g r o ß e) you can use decode("escape_string") as described in this SO answer
Note that this is probably an encoding error whoever produced the data. So it may contain other errors that you can not fix with this method.

This will work:
import sys
import ast
print ast.literal_eval('b"%s"' % sys.argv[1]).decode("utf-8")
But please read about literal_eval first to make sure it suits your needs (I think it should be safe to use but you should read and make sure).

Ignore evaluation of '\n' character in sys.argv

Oddly enough, when running this program with the arguments of
program.py "(lp0\nS'cat'\np1\naI5\na."
With program.py being:
import sys,pickle
print sys.argv[1]=="(lp0\nS'cat'\np1\naI5\na."
False is printed... I have narrowed the difference in evaluation to the \n character however I can find no way of ignoring such.
Why is this and how can I fix it?

You need to use raw string literal like this:
sys.argv[1] == r"(lp0\nS'cat'\np1\naI5\na."
Also, you can use a string in the parameters without quotes.

It is because the syntax of strings in Python and in the shell (presumably Bash) is different.
You may want to run the program as
echo $'"(lp0\nS\'cat\'\np1\naI5\na.'
program.py $'"(lp0\nS\'cat\'\np1\naI5\na.'

Input a Unix directory address in Python and not triggering str error

I was trying to write a Python 2.7 script that processes a Unix directory input and use the input as a parameter to start another program.
However, I run into a problem that python's str() function doesn't like slashes in the input. When I am trying to str() a input with slashes, just like:
inputData = str(input('Put directory here:')) // OS is Mac OS X
> Put directory here: /User/username/abc.file
...
SyntaxError: invalid syntax
I think this is due to str() cannot naturally process a string with slashes, because if I manually add quotes on each site during the input (keyboard input "/User/username/abc.file"), this error will not be triggered.
As this script needs to process user input, I hope it can add quotes automatically. I tried the following walk-around:
inputDataRaw = input('Put directory here:')
if (not inputDataRaw.startswith('"')) and (not inputDataRaw.startswith("'")):
inputDataRaw = '"' + inputDataRaw
if (not inputDataRaw.endswith("'")) and (not inputDataRaw.endswith('"')):
inputDataRaw = inputDataRaw + '"'
inputData = str(inputDataRaw)
But apparently the input value cannot be stored in inputDataRaw without str()ing it, and the first line directly triggered the same error. It looks like all functions that escape a slash in python can't work without forming a string first.
Would anyone please point out the correct way to walk-around this problem? Thanks.

input attempts to eval the given input. That is, it's expecting something that would be a valid Python literal - including quotes for a string value. Use raw_input instead - that always returns the string the user entered, leaving conversions to your code.
From the docs:
Equivalent to eval(raw_input(prompt)).
This function does not catch user errors. If the input is not syntactically valid, a SyntaxError will be raised. Other exceptions may be raised if there is an error during evaluation.
If the readline module was loaded, then input() will use it to provide elaborate line editing and history features.
Consider using the raw_input() function for general input from users.

The problem is that you are not using python comments ;-)
Change:
inputData = str(input('Put directory here:')) // OS is Mac OS X
to:
inputData = str(input('Put directory here:')) # OS is Mac OS X

Working with unicode encoded Strings from Active Directory via python-ldap

I already came up with this problem, but after some testing I decided to create a new question with some more specific Infos:
I am reading user accounts with python-ldap (and Python 2.7) from our Active Directory. This does work well, but I have problems with special chars. They do look like UTF-8 encoded strings when printed on the console. The goal is to write them into a MySQL DB, but I don't get those strings into proper UTF-8 from the beginning.
Example (fullentries is my array with all the AD entries):
fullentries[23][1].decode('utf-8', 'ignore')
print fullentries[23][1].encode('utf-8', 'ignore')
print fullentries[23][1].encode('latin1', 'ignore')
print repr(fullentries[23][1])
A second test with a string inserted by hand as follows:
testentry = "M\xc3\xbcller"
testentry.decode('utf-8', 'ignore')
print testentry.encode('utf-8', 'ignore')
print testentry.encode('latin1', 'ignore')
print repr(testentry)
The output of the first example ist:
M\xc3\xbcller
M\xc3\xbcller
u'M\\xc3\\xbcller'
Edit: If I try to replace the double backslashes with .replace('\\\\','\\) the output remains the same.
The output of the second example:
Müller
M�ller
'M\xc3\xbcller'
Is there any way to get the AD output properly encoded? I already read a lot of documentation, but it all states that LDAPv3 gives you strictly UTF-8 encoded strings. Active Directory uses LDAPv3.
My older question this topic is here: Writing UTF-8 String to MySQL with Python
Edit: Added repr(s) infos

First, know that printing to a Windows console is often the step that garbles data, so for your tests, you should print repr(s) to see the precise bytes you have in your string.
You need to find out how the data from AD is encoded. Again, print repr(s) will let you see the content of the data.
UPDATED:
OK, it looks like you're getting strange strings somehow. There might be a way to get them better, but you can adapt in any case, though it isn't pretty:
u.decode('unicode_escape').encode('iso8859-1').decode('utf8')
You might want to look into whether you can get the data in a more natural format.