I read the "Unicdoe Pain" article days ago. And I keep the "Unicode Sandwich" in mind.
Now I have to handle some Chinese and I've got a list
chinese = [u'中文', u'你好']
Do i need to proceed encoding before writing to file?
add_line_break = [word + u'\n' for word in chinese]
encoded_chinese = [word.encode('utf-8') for word in add_line_break]
with open('filename', 'wb') as f:
f.writelines(encoded_chinese)
Somehow I find out that in python2. I can do this:
chinese = ['中文', '你好']
with open('filename', 'wb') as f:
f.writelines(chinese)
no unicode matter involed. :D
You don't have to do that, you could use io or codecs to open the file with encoding.
import io
with io.open('file.txt', 'w', encoding='utf-8') as f:
f.write(u'你好')
codecs.open has the same syntax.
In python3;
with open('file.txt, 'w', encoding='utf-8') as f:
f.write('你好')
will do just fine.
Related
I was trying to decode a json file that has escaped unicode text /uHHH .. the original text is Arabic
my research lead me to the following code using python.
s = '\u00d8\u00b5\u00d9\u0088\u00d8\u00b1 \u00d8\u00a7\u00d9\u0084\u00d9\u008a\u00d9\u0088\u00d9\u0085\u00d9\u008a\u00d8\u00a7\u00d8\u00aa'
ouy= s.encode('utf-8').decode('unicode-escape').encode('latin1').decode('utf-8')
print(ouy)
the result text will be: صÙر اÙÙÙÙÙات
which still needs some fix using online tool to become the original text: صور اليوميات
Is there any way to perform that fix using the above code?
Would appreciate your help guys, thanks in advance
you can use this script to update all JSON files
import json
filename = 'YourFile.json' # file name we want to compress
newname = filename.replace('.json', '.min.json') # Output file name
with open(filename, encoding="utf8") as fp:
print("Compressing file: " + filename)
print('Compressing...')
jload = json.load(fp)
newfile = json.dumps(jload, indent=None, separators=(',', ':'), ensure_ascii=False)
newfile = newfile.encode('latin1').decode('utf-8') # remove this
#print(newfile)
with open(newname, 'w', encoding="utf8") as f: # add encoding="utf8"
f.write(newfile)
print('Compression complete!')
DecodeJsonToOrigin
I want to read some cyrilic text from a txt file in Python 3.
This is what the text file contains.
абцдефгчийклмнопярстувшхыз
I used:
with open('text.txt', 'r') as myfile:
text=myfile.read()
print (text)
But this is the ouput in the python shell:
ÿþ01F45D3G89:;<=>?O#ABC2HEK7
Can someone explain why this is the output?
Python supports utf-8 for this sort of thing.
You should be able to do:
with open('text.txt', encoding = 'utf-8', mode = 'r') as my_file:
...
Also, be sure that your text file is saved with utf-8 encoding. I tested this in my shell and without proper encoding my output was:
?????????????????????
With proper encoding:
file = open('text.txt', encoding='utf-8', mode='r')
text = file.read()
print(text)
абцдефгчийклмнопярстувшхы
Try working on the file using codecs, you need to
import codecs
and then do
text = codecs.open('text.txt', 'r', 'utf-8')
Basically you need utf8
I'm currently have an issue with my python 3 code.
replace_line('Products.txt', line, tenminus_str)
Is the line I'm trying to turn into utf-8, however when I try to do this like I would with others, I get errors such as no attribute ones and when I try to add, for example...
.decode("utf8")
...to the end of it, I still get errors that it is using ascii. I also tried other methods that worked with other lines such as adding io. infront and adding a comma with
encoding = 'utf8'
The function that I am using for replace_line is:
def replace_line(file_name, line_num, text):
lines = open(file_name, 'r').readlines()
lines[line_num] = text
out = open(file_name, 'w')
out.writelines(lines)
out.close()
How would I fix this issue? Please note that I'm very new to Python and not advanced enough to do debugging well.
EDIT: Different fix to this question than 'duplicate'
EDIT 2:I have another error with the function now.
File "FILELOCATION", line 45, in refill replace_line('Products.txt', str(line), tenminus_str)
File "FILELOCATION", line 6, in replace_line lines[line_num] = text
TypeError: list indices must be integers, not str
What does this mean and how do I fix it?
Change your function to:
def replace_line(file_name, line_num, text):
with open(file_name, 'r', encoding='utf8') as f:
lines = f.readlines()
lines[line_num] = text
with open(file_name, 'w', encoding='utf8') as out:
out.writelines(lines)
encoding='utf8' will decode your UTF-8 file correctly.
with automatically closes the file when its block is exited.
Since your file started with \xef it likely has a UTF-8-encoding byte order mark (BOM) character at the beginning. The above code will maintain that on output, but if you don't want it use utf-8-sig for the input encoding. Then it will be automatically removed.
codecs module is just what you need. detail here
import codecs
def replace_line(file_name, line_num, text):
f = codecs.open(file_name, 'r', encoding='utf-8')
lines = f.readlines()
lines[line_num] = text
f.close()
w = codecs.open(file_name, 'w', encoding='utf-8')
w.writelines(lines)
w.close()
Handling coding problems You can try adding the following settings to your head
import sys
reload(sys)
sys.setdefaultencoding('utf-8')
Type = sys.getfilesystemencoding()
Try adding encoding='utf8' if you are reading a file
with open("../file_path", encoding='utf8'):
# your code
I was wondering, is there any chance to add encoding right into tkFileDialog.askopenfile body?
Right now i use askopenfilename instead and then use codecs to encode file like this:
def open():
filename = askopenfilename(filetypes=[("Text files","*.xml")])
if filename == '':
return
with codecs.open(filename, encoding='utf-8') as f:
txt = f.read()
delete(xml)
xml.insert("1.0", txt)
What i'm asking is, how to do the same but using askopenfile, something like this:
with askopenfile(filetypes=[("Text files","*.xml")], encoding='utf-8') as f:
Any suggestions or other approaches to do the same are strongly appreciated.
Just merge the askopenfilename() into the with():
with codecs.open(askopenfilename(filetypes=[("Text files","*.xml")]), encoding='utf-8') as f:
f = open("go.txt", "w")
f.write(title)
f.close()
What if "title" is in japanese/utf-8? How do I modify this code to be able to write "title" without having the ascii error?
Edit: Then, how do I read this file in UTF-8?
How to use UTF-8:
import codecs
# ...
# title is a unicode string
# ...
f = codecs.open("go.txt", "w", "utf-8")
f.write(title)
# ...
fileObj = codecs.open("go.txt", "r", "utf-8")
u = fileObj.read() # Returns a Unicode string from the UTF-8 bytes in the file
It depends on whether you want to insert a Unicode UTF-8 byte order mark, of which the only way I know of is to open a normal file and write:
import codecs
f = open('go.txt', 'wb')
f.write(codecs.BOM_UTF8)
f.write(title.encode('utf-8')
f.close()
Generally though, I don't want to add a UTF-8 BOM and the following will suffice though:
import codecs
f = codecs.open('go.txt', 'w', 'utf-8')
f.write(title)
f.close()