Develop Reference

Python encoding issue in script if string not hard-coded

I have an encoding issue with strings I get from an external source.
This source sends the strings encoded to me and I can decode them only if they are part of the script's code.
I've looked at several threads here and even some recommended tutorials (such as this one) but came up empty.
For example, if I run this:
python -c 'print "gro\303\237e"'
I get:
große
Which is the correct result.
But If I use it in a script, such as:
import sys
print sys.argv[1]
and call it like test.py "gro\303\237e", I get:
gro\303\237e
I intend to write the correct string to syslog, but I can't seem to get this to work.
Some data on my system:
- Python 2.7.10
- CentOS Linux
- LANG=en_US.UTF-8
- LC_CTYPE=UTF-8
I will appreciate any help, please let me know if you need more information.
Thanks!

If you really have the chars gro\303\237e which is something else as "gro\303\237e" (the first one are the chars g r o \ 3 0 3 \ 2 3 7, the second one is the chars g r o ß e) you can use decode("escape_string") as described in this SO answer
Note that this is probably an encoding error whoever produced the data. So it may contain other errors that you can not fix with this method.

This will work:
import sys
import ast
print ast.literal_eval('b"%s"' % sys.argv[1]).decode("utf-8")
But please read about literal_eval first to make sure it suits your needs (I think it should be safe to use but you should read and make sure).

Python Printing in Terminal

I am sorry to post what I think may be a very basic question, but my attempts at solving this have been futile, and I can't find a useful solution that has already been suggested to similar questions on this site.
My basic issue is this: I am attempting to run a file (coding UTF-8) as a program in Mac terminal (running Python 2.7.5). This works fine when I print the results of a mathematical operations, but for some reason I cannot print a simple string of characters.
I have tried running both:
# coding: utf-8
print "Hello, World."
exit()
and
# coding: utf-8
print("Hello, World.")
exit()
Both return an invalid syntax error, with the caret pointing at first set of quotation marks that I've used. What am I missing here?
Thank you for your help!

It turned out that I needed to disable smart quotes in TextEdit.

Python: Output prefixed with b

I'm pretty new to Python so please bear with me here!
I've taken some code from ActiveState (and then butchered it around a bit) to open a DBF file and then output to CSV.
This worked perfectly well on Python 2.5 but I've now moved it to Python 3.3 and ran into a number of issues, most of which I've resolved.
The final issue I have is that in order to run the code, I've had to prefix some items with b (because I was getting TypeError: expected bytes, bytearray or buffer compatible object errors)
The code now works, and outputs correctly, except that every field is displayed as b'DATAHERE' (where DATAHERE is the actual data of course!)
So... does anyone know how I can stop it from outputting the b character? I can post code if required but it's fairly lengthy so I was hoping someone would be able to spot what I expect to be something simple that I've done wrong!
Thanks!

You are seeing the code output byte values; if you expected unicode strings instead, simply decode:
yourdata.decode('ascii')
where ascii should be replaced by the encoding your data uses.

Repeated errors while working with Portable Python [duplicate]

This question already has answers here:
Syntax error on print with Python 3 [duplicate]
(3 answers)
Closed 9 years ago.
Note: This was not answered by the question that was marked as the original. This is more than just a Python v2 vs v3 problem, which I explain in the comments below.
Original post:
I am trying to learn Python at work, so I am currently using Portable Python 3.2.1.1 (which will henceforth be referred to as PP). (I mention this because this problem doesn't happen at home when I use my Mac and regular Python.)
I am working through exercise 16 of Learning Python the Hard Way (http://learnpythonthehardway.org/book/ex16.html). I've heard this isn't the best learning tool, but I am a complete programming n00b and I'm a hands-on learner. If you have any better suggestions, I'm open!
The first few lines of the exercise read:
from sys import argv
script, filename = argv
print "We're going to erase %r." % filename
print "If you don't want that, hit CTRL-C (^C)."
My script is titled Ex16.py and the file I am using is Python.txt, and both of these are in the same folder as the PP .exes. I don't think that's necessary, but hoped maybe it would fix the problem... negative. When I press "Run" in PP, it doesn't work because argv requires you provide an argument when you start the script: python Ex16.py Python.txt
When I launch Python.exe (which, in PP is Portable-Python.exe), I get the standard Python prompt, >>>, but whatever I enter I get the same error message:
File "<stdin>", line 1
with whatever I've just tried repeated back to me with the marker to
indicate where the problem is. (has not been helpful so far)
SyntaxError: invalid syntax
I have tried typing the following at the >>> prompt:
python Ex16.py Python.txt,,
Ex16.py Python.txt,,
"%PATH&\Ex16.py" "%PATH%\Python.txt" (with the actual filepaths),,
print 'hello world'
I just keep getting the same invalid syntax error over and over. Even a basic print command returned an invalid syntax error. The only one that triggered a different error was the one where I tried whole filepaths. That one returned:
File "<stdin>", line 1
SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in
position 2-3: truncated \UXXXXXXXX escape
Yes, I have Googled the crap outta both errors. I read that sometimes the problem is not doubling the backspaces, so I tried that, too, putting two \ where just one had been before in both filepaths. I even tried putting — # -*- coding: utf-8 -*- at the beginning of the script thinking maybe there was some unicode error. That, with the full filepaths, resulted in the same unicode error mentioned earlier.
Yes, I have checked that my code is matching that in the exercise.
Yes, this works at home on non-PP.
All this leads me to believe that the problem is probably in the way I'm trying to run the scripts in PP (but why won't print work?), but I haven't a clue what I'm doing wrong.
Thanks!

print is a function in Python 3:
print('my string with content and the like')
It is no longer supported as being a 'statement'. You might want to check out a list of things that changed from python2.x to python3.x (there's a number of incompatibilities). Also, you might be better off finding a tutorial using Python3.

You have to type:
Portable-Python.exe Ex16.py Python.txt
at your command prompt. To get a command prompt, press WindowsKey-R, then type "cmd" and press enter. You should now be looking at something like c:\>. Navigate to your portable python installation by using the cd command.

Copy-paste into Python interactive interpreter and indentation

This piece of code, file test.py,
if 1:
print "foo"
print "bar"
can be successfully executed with execfile("test.py") or python test.py, but when one tries to copy-paste it into a Python interpreter:
File "<stdin>", line 3
print "bar"
^
SyntaxError: invalid syntax
Why is it so? Can the interpreter by configured in such a way that it would read copy-pasted text successfully?
I guess that may affect typing in the interpreter, but that's OK for me.

Indentation is probably lost or broken.
Have a look at IPython -- it's an enhanced Python interpreter with many convenient features. One of them is a magic function %paste that allows you to paste multiple lines of code.
It also has tab-completion, auto-indentation... and many more. Have a look at their site.
Using %paste in IPython:
And copy-and-paste stuff is one of the things fixed in the Qt console. Here's using a plain old copy-and-paste of your code block that "just works" in the new IPython qtconsole:

I don't know any trick for the standard command prompt, but I can suggest you a more advanced interpreter like IPython that has a special syntax for multi-line paste:
In [1]: %cpaste
Pasting code; enter '--' alone on the line to stop.
:for c in range(3):
: print c
:
:--
0
1
2
Another option is the bpython interpreter that has an automatic paste mode (if you are typing too fast to be an human):
>>> for c in range(3):
... print c
...
0
1
2
>>>
<C-r> Rewind <C-s> Save <F8> Pastebin <F9> Pager <F2> Show Source

Do %autoindent to make automatic indentation off. After that, you can paste your code in IPython.

Continuation lines are needed when entering a multi-line construct.
--Interactive mode, The Python Tutorial (v2) (v3)
So you need to enter:
if 1:
print "foo"
print "bar"
I've yet to find a suitable explanation as to why it's different to a non-interactive session, alas.

All of the current answers suggest you change to IPython. For a Python-only solution, you can use textwrap to remove leading whitespace from lines.
For example,
>>> code=""" x='your pasted code'
y='with common indentation'"""
>>> formatted=textwrap.dedent(code)
>>> exec(formatted)

If you are like me and use Notepad++ (to copy and paste from), try to replace tabs by spaces by going to menu Settings → Preferences → Language and check the replace by spaces.
I had this problem myself for so long and I found out that python.exe recognizes spaces.

If the pasted content has any empty lines, the interpreter triggers evaluation when encountering them. If any line after an empty line has indentation, it will cause an IndentationError since any previous context has been closed.
Solutions:
Delete any empty lines before copying to clipboard.
Add any amount of indentation to empty lines (doesn't need to match code) before copying to clipboard.
Note that spaces vs. tabs does not seem to matter.

exec(pyperclip.paste())
provided you don't mind using exec. Any other third party clipboard package than pyperclip will do I guess.

One other solution I recently found for a similar problem:
$ python << EOF
if 1:
print "foo"
print "bar"
EOF