Develop Reference

Non-ASCII character '\x97' in file, but no encoding declared

I'm trying to encode the contents of a Python script in Linux. I just started off with a simple script called test.py -
# !/app/logs/Python/bin/python3
# -*- coding: ascii -*-
print ("hi")
Once I have the script, I execute the vim -x test.py and enter the encryption key twice. Then save the file as normal and then execute the script using python test.py
I tried almost all the examples provided in the link here but still i end up getting the below error -
SyntaxError: Non-ASCII character '\x97' in file test.py on line 1, but no encoding declared; see http://www.python.org/peps/pep-0263.html for details
I checked the default encoding using print sys.getdefaultencoding() and it is acsii.
What am i missing here. kindly clarify. I know this question is a duplicate but none of the solutions helped.

Python knows how to execute clear text Python source code. If you encrypt the source file, it no longer contains valid Python source and cannot be directly executed.
There are 2 possible ways here. First is to only obfuscate your source. You should be aware that obfuscation is not security, and a user could recover some Python source with some work (not necessarily the original source, because comments and doc strings could have been stripped off and variable names could have been changed). You could read How do I protect Python code? and google for python obfuscate to find some possible ways of obfuscating Python source and their trade-offs.
The good news with obfuscated source is that it can be used by anybody without any password.
Second way is to encrypt the source. If you use a decent tool, you can assume that it will be impossible to read or execute the file without knowing the key. In that sense, vim crypto has not the highest possible reputation. In the simplest way (for example with your example vim -x) you will have to decrypt the file to execute it. Unfortunately, good crypto modules are not shipped in a standard Python installation and must be downloaded from pypi. Well known crypto modules include pycrypto and cryptography.
You can then encrypt the most part of the code, and then at run time ask for the key, decrypt it and execute it. Still a serious job but feasible.
Alternatively, you could build in another language (C/C++) a decryptor that decrypts the remaining of the file and feed it into a python interpretor, but functionally, this is only a variant of the above method.
As per your comment I assume that you want to encrypt the source code and decrypt it (with the password) at run time. The princips are:
build a Python script that will take another arbitrary Python script, encode it with a secure crypto module and prepend it with some decrypting code.
at run time, the prepended code will ask for the password, decrypt the encrypted code exec it
The builder could be (this code uses the cryptography module):
import cryptography.fernet
import cryptography.hazmat.primitives.kdf.pbkdf2
import cryptography.hazmat.primitives.hashes
import cryptography.hazmat.backends
import base64
import os
import sys
# the encryption function
def encrypt_source(infile, outfile, passwd):
with open(infile, 'rb') as fdin: # read original file
plain_data = fdin.read()
salt, key = gen_key(passwd) # derive a key from the password
f = cryptography.fernet.Fernet(key)
crypted_data = f.encrypt(plain_data) # encrypt the original code
with open(outfile, "w") as fdout: # prepend a decoding block
fdout.write("""#! /usr/bin/env python
import cryptography.fernet
import cryptography.hazmat.primitives.kdf.pbkdf2
import cryptography.hazmat.primitives.hashes
import cryptography.hazmat.backends
import base64
import os
def gen_key(passwd, salt): # key derivation
kdf = cryptography.hazmat.primitives.kdf.pbkdf2.PBKDF2HMAC(
algorithm = cryptography.hazmat.primitives.hashes.SHA256(),
length = 32,
salt = salt,
iterations = 100000,
backend = cryptography.hazmat.backends.default_backend()
)
return base64.urlsafe_b64encode(kdf.derive(passwd))
passwd = input("Password:") # ask for the password
salt = base64.decodebytes({})
key = gen_key(passwd.encode(), salt) # derive the key from the password and the original salt
crypted_source = base64.decodebytes( # decode (base64) the crypted source
b'''{}'''
)
f = cryptography.fernet.Fernet(key)
plain_source = f.decrypt(crypted_source) # decrypt it
exec(plain_source) # and exec it
""".format(base64.encodebytes(salt),
base64.encodebytes(crypted_data).decode()))
# derive a key from a password and a random salt
def gen_key(passwd, salt=None):
if salt is None: salt = os.urandom(16)
kdf = cryptography.hazmat.primitives.kdf.pbkdf2.PBKDF2HMAC(
algorithm = cryptography.hazmat.primitives.hashes.SHA256(),
length = 32,
salt = salt,
iterations = 100000,
backend = cryptography.hazmat.backends.default_backend()
)
return salt, base64.urlsafe_b64encode(kdf.derive(passwd))
if __name__ == '__main__':
if len(sys.argv) != 3:
print("Usage {} infile outfile".format(sys.argv[0]))
sys.exit(1)
passwd = input("Password:").encode() # ask for a password
encrypt_source(sys.argv[1], sys.argv[2], passwd) # and generates an encrypted Python script

Compiled Extension File error Report in LiClipse Python

doc_holder_str = ''
sample_H_value = open("C:\testGupixwin\BX-N-H.HED", "r")
standard_conc_value = open("C:\testGupixwin\gupixwin_H_stdConc.txt", "r")
sample_H_value_str = sample_H_value.readline()
while sample_H_value_str is not '' :
stripper_sample_H = float(sample_H_value_str[5:].lstrip(' '))
I'm trying to write a piece of code (as shown above) which reads some values, do some calculations on it and returns the said values. I am using the LiClipse IDE, for python.
I have tested my code and it works, but when I tried to run it with real data, ( for which I created a new folder to put in all the files I will be working with) I received an OS error suggesting I inserted an invalid argument.
The error report says ;
Traceback (most recent call last):
File "C:\Python34\workspace\Gupixwin_Project.py", line 11, in <module>
sample_H_value = open("C:\testGupixwin\BX-N-H.HED", "r")
OSError: [Errno 22] Invalid argument: 'C:\testGupixwin\\BX-N-H.HED'
on clicking on the C:\testGupixwin\\BX-N-H.HED it bring up a message box suggesting, and I quote,
The definition was found at C:\testGupixwin\BX-N-H.HED, (which
cannot be opened because it is a compiled extension)
I must point out that I feel the error is that the system sees ...\\BX-N.... Instead of ..\BX-N... Which I expect.
Some one suggested I do this
[Open Window -> Preferences, goto PyDev -> Editor -> Code Style ->
File Types, look for "Valid source files (comma-separated)" and append
", log".]
I HAVE DONE THIS BUT I STILL GET THE OSERROR REPORT.
Thanks for any assistance

I think the problem is the escaping with \
alternate the string in: open("C:\testGupixwin\BX-N-H.HED", "r")
with:
open( r"C:\testGupixwin\BX-N-H.HED", "r" ) #rawstring
# or
open( "C:\\testGupixwin\\BX-N-H.HED", "r" ) #escaping the '\' with '\'
(also do this in the following line)

Python 2.4.4 incessant type error

Im using python 2.4.4 and for some reason it keeps throwing a type error when i try to open a log file for writing... Here is the function in question...
import os
def write(inlog, outlog):
# parse the logs and save them to our own...
parsed = NTttcpParse(inlog)
debug("PARSED")
debug(parsed)
if os.path.exists(outlog):
fh = os.open(outlog, os.O_WRONLY | os.O_APPEND)
debug("Opened '%s' for writing (%d)" % (outlog, fh))
else:
fh = os.open(outlog, os.O_WRONLY | os.O_CREAT)
debug("Created '%s' for writing (%d)" % (outlog, fh))
debug("type(fh) = %s" % type(fh))
os.write(fh, LOGFORMAT % parsed)
os.close(fh)
And here is the maddening error...
TypeError: int argument required
Please hel... and thanks in advance :P

You are doing file I/O in a strange way. Here is the way to do it:
f = open(outlog, "w")
f.write("some data written to file\n")
f.close()
If you want to append, use open(outlog, "a") instead. If you want to read, use open(outlog, "r"). Also read the Python tutorial, which explains basic file I/O operations like this.
Note that in Python 2.5 and up, you can use the with statement:
with open(outlog, "w") as f:
f.write("some data written to file\n")
(I originally posted this as the main answer before I noticed you said you were using 2.4.)

Yikes, i made a simple error-- i wasnt entering enough items in the parsed tuple ^_^ Thanks for your answers though!

Python print statement--wrinting into a file

I saw this piece of code in a book and when I try to implement it I get a invalid syntax error.
This code basically reads a dictionary and writes into a txt file..
main.py
from Basics import data
dbfilename = 'people-file'
ENDDB = 'enddb.'
ENDREC = 'endrec.'
RECSEP = '=>'
def storelist(db,dbfilename):
print('In storelist function')
dbfile = open(dbfilename, 'w')
for key in db:
print(key, file=dbfile)
dbfile.close()
if __name__ == '__main__':
print('In Main list-items=',data.people)
storelist(data.people,dbfilename)
#for key in data.people:
# print('Values are', key['name'])
data.py
bob={'name':'bobs mith','age':42,'salary':5000,'job':'software'}
sue={'name':'sue more','age':30,'salary':3000,'job':'hardware'}
people={}
people['bob'] = bob
people['sue'] = sue
Error:
Syntax error:Invalid syntax.
Is it possible to write a file using a print statement.

I'm guessing you're really using python from the 2.x family. Print is a builtin function in python 3 and a statement in python 2. What happens if you try to print to a file using the 2.x syntax?
print >>dbFile, key
To check your version, open an interactive python shell and do
sys.version_info
I have 2.7, so I get
sys.version_info(major=2, minor=7, micro=2, releaselevel='final', serial=0)

You could just change it from using print to dbfile.write(key + "\n"). It is easier to understand what you are trying to accomplish.

If you're on python 2.6 or newer, you can try adding
from __future__ import print_function

Please ignore it..Its just a print statement..Its not a write statetement..I will delete this thread so that folks are not confused.

Setting the encoding for sax parser in Python

When I feed a utf-8 encoded xml to an ExpatParser instance:
def test(filename):
parser = xml.sax.make_parser()
with codecs.open(filename, 'r', encoding='utf-8') as f:
for line in f:
parser.feed(line)
...I get the following:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "test.py", line 72, in search_test
parser.feed(line)
File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/xml/sax/expatreader.py", line 207, in feed
self._parser.Parse(data, isFinal)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xb4' in position 29: ordinal not in range(128)
I'm probably missing something obvious here. How do I change the parser's encoding from 'ascii' to 'utf-8'?

Your code fails in Python 2.6, but works in 3.0.
This does work in 2.6, presumably because it allows the parser itself to figure out the encoding (perhaps by reading the encoding optionally specified on the first line of the XML file, and otherwise defaulting to utf-8):
def test(filename):
parser = xml.sax.make_parser()
parser.parse(open(filename))

Jarret Hardie already explained the issue. But those of you who are coding for the command line, and don't seem to have the "sys.setdefaultencoding" visible, the quick work around this bug (or "feature") is:
import sys
reload(sys)
sys.setdefaultencoding('utf-8')
Hopefully reload(sys) won't break anything else.
More details in this old blog:
The Illusive setdefaultencoding

The SAX parser in Python 2.6 should be able to parse utf-8 without mangling it. Although you've left out the ContentHandler you're using with the parser, if that content handler attempts to print any non-ascii characters to your console, that will cause a crash.
For example, say I have this XML doc:
<?xml version="1.0" encoding="utf-8"?>
<test>
<name>Champs-Élysées</name>
</test>
And this parsing apparatus:
import xml.sax
class MyHandler(xml.sax.handler.ContentHandler):
def startElement(self, name, attrs):
print "StartElement: %s" % name
def endElement(self, name):
print "EndElement: %s" % name
def characters(self, ch):
#print "Characters: '%s'" % ch
pass
parser = xml.sax.make_parser()
parser.setContentHandler(MyHandler())
for line in open('text.xml', 'r'):
parser.feed(line)
This will parse just fine, and the content will indeed preserve the accented characters in the XML. The only issue is that line in def characters() that I've commented out. Running in the console in Python 2.6, this will produce the exception you're seeing because the print function must convert the characters to ascii for output.
You have 3 possible solutions:
One: Make sure your terminal supports unicode, then create a sitecustomize.py entry in your site-packages and set the default character set to utf-8:
import sys
sys.setdefaultencoding('utf-8')
Two: Don't print the output to the terminal (tongue-in-cheek)
Three: Normalize the output using unicodedata.normalize to convert non-ascii chars to ascii equivalents, or encode the chars to ascii for text output: ch.encode('ascii', 'replace'). Of course, using this method you won't be able to properly evaluate the text.
Using option one above, your code worked just fine for my in Python 2.5.

To set an arbitrary file encoding for a SAX parser, one can use InputSource as follows:
def test(filename, encoding):
parser = xml.sax.make_parser()
with open(filename, "rb") as f:
input_source = xml.sax.xmlreader.InputSource()
input_source.setByteStream(f)
input_source.setEncoding(encoding)
parser.parse(input_source)
This allows parsing an XML file that has a non-ASCII, non-UTF8 encoding. For example, one can parse an extended ASCII file encoded with LATIN1 like: test(filename, "latin1")
(Added this answer to directly address the title of this question, as it tends to rank highly in search engines.)

Commenting on janpf's answer (sorry, I don't have enough reputation to put it there), note that Janpf's version will break IDLE which requires its own stdout etc. that is different from sys's default. So I'd suggest modifying the code to be something like:
import sys
currentStdOut = sys.stdout
currentStdIn = sys.stdin
currentStdErr = sys.stderr
reload(sys)
sys.setdefaultencoding('utf-8')
sys.stdout = currentStdOut
sys.stdin = currentStdIn
sys.stderr = currentStdErr
There may be other variables to preserve, but these seem like the most important.