I am trying to grab some text written in Arabic from Youtube, writting it into a file and reading it again.
The source file to grab the text has:
#!/usr/bin/python
#encoding: utf-8
in the beginning of the file.
Writing the text are done like this:
f.write(comment + '\n' )
The file contents is readable Arabic, so I assume the previous steps were correct.
But the problem appears when trying to read the contents from the file (and writing them for example into another file) like this:
in = open('data_Pass1/EG', 'rb')
out.write(in.read())
Which results in output file like this:
\xd8\xa7\xd9\x8a\xd9\x87
What is causing this?
In python 3.x
in = open('data_Pass1/EG', 'r', encoding='utf-8')
out = open('_file_name_', 'w', encoding='utf-8')
In python 2.x.
import codecs
in = codecs.open('data_Pass1/EG', 'r', encoding='utf-8')
out = codecs.open('_file_name_', 'w', encoding='utf-8')
You're opening the input file in binary ('rb') mode. Open the file to read as text ('r'). I tend to use Python 3 so the source files are UTF-8 by default, so I don't know what effect setting the encoding for .py files inside the files has on text I/O, but if necessary you may also want to use encoding='utf8' inside the calls to open() for all your file I/O, unless that doesn't work in 2.7 in which case I'm not sure what the best way to handle that in Python 2.7 would be...
As Lee Daniel Crocker suggests, you'd probably be better off just opening both input and output files in binary mode ('rb' for the input file, 'wb' for the output) if you're passing the input directly to the output without doing any textual manipulation of it. (Though going by Andy's comment, in Python 2 it's better to open text files in binary mode and do explicit encoding/decoding anyway.)
Related
I'm looking to edit a Minecraft Windows 10 level.dat file in python. I've tried using the package nbt and pyanvil but get the error OSError: Not a gzipped file. If I print open("level.dat", "rb").read() I get a lot of nonsensical data. It seems like it needs to be decoded somehow, but I don't know what decoding it needs. How can I open (and ideally edit) one of these files?
To read data just do :
from nbt import nbt
nbtfile = nbt.NBTFile("level.dat", 'rb')
print(nbtfile) # Here you should get a TAG_Compound('Data')
print(nbtfile["Data"].tag_info()) # Data came from the line above
for tag in nbtfile["Data"].tags: # This loop will show us each entry
print(tag.tag_info())
As for editing :
# Writing data (changing the difficulty value
nbtfile["Data"]["Difficulty"].value = 2
print(nbtfile["Data"]["Difficulty"].tag_info())
nbtfile.write_file("level.dat")
EDIT:
It looks like Mojang doesn't use the same formatting for Java and bedrock, as bedrock's level.dat file is stored in little endian format and uses non-compressed UTF-8.
As an alternative, Amulet-Nbt is supposed to be a Python library written in Cython for reading and editing NBT files (supposedly works with Bedrock too).
Nbtlib also seems to work, as long as you set byteorder="little when loading the file.
Let me know if u need more help...
You'll have to give the path either relative to the current working directory
path/to/file.dat
Or you can use the absolute path to the file
C:user/dir/path/to/file.dat
Read the data,replace the values and then write it
# Read in the file
with open('file.dat', 'r') as file :
filedata = file.read()
# Replace the target string
filedata = filedata.replace('yuor replacement or edit')
# Write the file out again
with open('file.dat', 'w') as file:
file.write(filedata)
I have the byte-code of a png-file in a string variable. How do I write it to .png file without python trying to encode it? The string is '\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x00\n\x00\x00\x00\x07\x08\x02\x00\x00\x00\xbe\xceK4\x00\x00\x00\x01sRGB\x00\xae\xce\x1c\xe9\x00\x00\x00\x04gAMA\x00\x00\xb1\x8f\x0b\xfca\x05\x00\x00\x00\tpHYs\x00\x00\x0e\xc3\x00\x00\x0e\xc3\x01\xc7o\xa8d\x00\x00\x00DIDAT\x18Wc\xf8\xff\xff\xff\xaf\xfd\x07\xdf[:\xbc\x95Q\x81 \xfb\xc7\xaa\xb5#q \x00I#\xcb\xc1\x11D\x11H\xfa\xdb\x94\x19hr\x10\xf4NY\x1b$\x8d\x0c\x90\x95~\xad\xacE\x97F\x03\x94H\xff\xff\x0f\x00\x1f]\xa2\x03U|Z\xa3\x00\x00\x00\x00IEND\xaeB`\x82'
edit: I feel like you might need more info on my situation: I am trying to make a little encryption program, and although it works on strings, I want to make it work for any file too. I am reading a .png file in byte-mode(which gives the string mentioned above), and after it is done being encrypted and decrypted, I have a string with the exact same content, but no way to put it back into a file.
For python3, you have to open the file in binary write mode and encode the string to bytes:
with open('filename', 'wb') as f:
f.write(the_string.encode())
You could try using PyPNG, looks like a possible solution:
http://pythonhosted.org/pypng/ex.html#writing
This will let you write binary to a file in python.
with open('filename', 'wb') as f:
f.write(bytecode)
A friend of mine has written simple poetry using C's fprintf function. It was written using the 'wb' option so the generated file is in binary. I'd like to use Python to show the poetry in plain text.
What I'm currently getting are lots of strings like this: ��������
The code I am using:
with open("read-me-if-you-can.bin", "rb") as f:
print f.read()
f.close()
The thing is, when dealing with text written to a file, you have to know (or correctly guess) the character encoding used when writing said file. If the program reading the file is assuming the wrong encoding here, you will end up with strange characters in the text if you're lucky and with utter garbage if you're unlucky.
Don't try to guess, try to know: you need to ask your friend in what character encoding he or she wrote the poetry text to the file. You then have to open the file in Python specifying that character encoding. Let's say his/her answer is "UTF-16-LE" (for sake of example), you then write:
with open("poetry.bin", encoding="utf-16-le") as f:
print(f.read())
It seems you're on Python 2 still though, so there you write:
import io
with io.open("poetry.bin", encoding="utf-16-le") as f:
print f.read()
You could start by trying UTF-8 first though, that is an often used encoding.
I want to read a file and write it back out. Here's my code:
file = open( zipname , 'r' )
content = file.read()
file.close()
alt = open('x.zip', 'w')
alt.write(content )
alt.close()
This doesn't work, why?????
Edit:
The rewritten file is corrupt
(python 2.7.1 on windows)
Read and write in the binary mode, 'rb' and 'wb':
f = open(zipname , 'rb')
content = f.read()
f.close()
alt = open('x.zip', 'wb')
alt.write(content )
alt.close()
The reason the text mode didn't work on Windows is that the newline translation from '\r\n' to '\r' mangled the binary data in the zip file.
From this bit of the manual:
On Windows, 'b' appended to the mode opens the file in binary mode, so
there are also modes like 'rb', 'wb', and 'r+b'. Python on Windows
makes a distinction between text and binary files; the end-of-line
characters in text files are automatically altered slightly when data
is read or written. This behind-the-scenes modification to file data
is fine for ASCII text files, but it’ll corrupt binary data like that
in JPEG or EXE files. Be very careful to use binary mode when reading
and writing such files. On Unix, it doesn’t hurt to append a 'b' to
the mode, so you can use it platform-independently for all binary
files.
If I run this program on my OS X or Linux box, it works exactly as you would expect. The file x.zip has exactly the same checksum as the original zip file and is not corrupt. I believe that Windows is one of the platforms where you need to explicitly open files in binary mode; try:
file = open(zipname, 'rb')
Whenever I try to open a .csv file with the python command
fread = open('input.csv', 'r')
it always opens the file with spaces between every single character. I'm guessing it's something wrong with the text file because I can open other text files with the same command and they are loaded correctly. Does anyone know why a text file would load like this in python?
Thanks.
Update
Ok, I got it with the help of Jarret Hardie's post
this is the code that I used to convert the file to ascii
fread = open('input.csv', 'rb').read()
mytext = fread.decode('utf-16')
mytext = mytext.encode('ascii', 'ignore')
fwrite = open('input-ascii.csv', 'wb')
fwrite.write(mytext)
Thanks!
The post by recursive is probably right... the contents of the file are likely encoded with a multi-byte charset. If this is, in fact, the case you can likely read the file in python itself without having to convert it first outside of python.
Try something like:
fread = open('input.csv', 'rb').read()
mytext = fread.decode('utf-16')
The 'b' flag ensures the file is read as binary data. You'll need to know (or guess) the original encoding... in this example, I've used utf-16, but YMMV. This will convert the file to unicode. If you truly have a file with multi-byte chars, I don't recommend converting it to ascii as you may end up losing a lot of the characters in the process.
EDIT: Thanks for uploading the file. There are two bytes at the front of the file which indicates that it does, indeed, use a wide charset. If you're curious, open the file in a hex editor as some have suggested... you'll see something in the text version like 'I.D.|.' (etc). The dot is the extra byte for each char.
The code snippet above seems to work on my machine with that file.
The file is encoded in some unicode encoding, but you are reading it as ascii. Try to convert the file to ascii before using it in python.
Isn't csv a simple txt file with values separated with comma.
Just try to open it with a text editor to see if the file is correctly formed.
To read an encoded file, you can simply replace open with codecs.open.
fread = codecs.open('input.csv', 'r', 'utf-16')
It did never ocurred to me, but as truppo said, it must be something wrong with the file.
Try to open the file in Excel/BrOffice Calc and Save As the file as Csv again.
If the problem persists, try a subset of the data: fist 10/last 10/intermediate 10 lines of the file.
Ok, I got it with the help of Jarret Hardie's post
this is the code that I used to convert the file to ascii
fread = open('input.csv', 'rb').read()
mytext = fread.decode('utf-16')
mytext = mytext.encode('ascii', 'ignore')
fwrite = open('input-ascii.csv', 'wb')
fwrite.write(mytext)
Thanks!
Open the file in binary mode, 'rb'. Check it in a HEX Editor and check for null padding '00'. Open the file in something like Scintilla Text Editor to check the characters present in the file.
Here's the quick and easy way, esp if python won't parse the input correctly
sed 's/ \(.\)/\1/g'