Develop Reference

Python Memory error when writing large string to file (source is XML/minidom)

I have been working on some scripts to convert a large amount of XML data from format 1 to format 2 to allow data to migrate betweens systems. I'm using Python 3.8 on Windows 10.
This is a once off job. There has been a huge amount of data incompatibility that I've have to reverse engineer on both systems to make the data compatible and manually translate most of the XML fields. Learning XLST was too big a curve for a single job and I don't have SQL experience to do it.
All was going well until the output string reached I think about 86MB (the limit may be quite a bit less than this but it was the first file to fail).
I have built the XML using xml.etree.Elemtree.
I need the XML output pretty printed and have borrowed a prettify function I found on stackoverflow that uses minidom: Use xml.etree.elementtree to print nicely formatted xml files - copied here:
def prettify(elem):
"""Return a pretty-printed XML string for the Element.
"""
rough_string = tostring(elem, 'ISO-8859-1')
reparsed = minidom.parseString(rough_string)
return reparsed.toprettyxml(indent="\t")
My write to file function:
def write_to_file(root_xml,filenumber):
# Simply write the XML to the output folder
file = open(outputxml + "\\" + filenumber + ".xml","w",encoding="UTF-8")
file.write(prettify(root_xml))
file.close
return
My error:
File "C:\\mycode.py", line 501, in write_to_file
file.write(prettify(root_xml))
MemoryError
I've read that minidom isn't a great way of handling my data and that I probably should not be creating my entire XML in memory. My upcoming biggest XML will probably be about 250MB, maybe even higher and I'm failing to write a string of 86MB. It seems to be a simple issue, however I'm stuck.
Is there a good workaround for this? I'm really hoping to not have to re-engineer a lot of code to write the XML output in chunks. Is there an easy way to break up the string into smaller pieces and then write to file? Other ideas?
Thanks!

In case others have a similar issue, I stumbled into an answer that works for me which only occured due to me getting hung up on using "prettify" to write to a file. I stopped using prettify and recast the XML data into ElementTree and can write large XML files no problem (well at least a few hundred MB):
tree = ET.ElementTree(root_xml)
tree.write("myxmlfile.xml")
I didn't need my pretty printing in the end.
Happy for any code geniuses to give feedback/criticisms or suggestions.

python3's "pdfkit"-import keeps destroying german umlaute

Hey people and codingfriends.
Im currently writing a little tool in python3 which i want to write some userinput from a .txt-file into a pdf-file. For this i am using "pdfkit". Im first writing my input into a 2nd. txt file, because i need the output in a special form and with "pdfkit.from_string(source, destination.pdf)" it doesnt seem to work properly. So im Writing it first into a 2nd txt file how i already said and then converting it with "pdfkit.from_file("source.txt", destination.pdf)". I already sorted out the problem with the different unicodestuff but somehow the "pdfcreator" keeps deleting/changing all "umlaute". I guess i can figure out why it keeps doing that, but i cant cant find a solution due its a 3rd party import. The createt txt file btw is in exactly the form i need it, just the pdf stuff is not working.
In example:
Does anyone have a solution for me?
For me would also work:
writing from python into a different txt format which is easy to convert into pdf (with umlaute!)
OR
using a 3rd party-software which i am calling from python to convert my txt file
So far i got this code(including some readLines methods which are also working pretty well - i guess):
def dok2pdf(fileName):
dokument=open(fileName, "r", encoding="latin-1").read()
for line in dokument:
if line==" ":
dokument=dokument.replace(line, " ")
if line==";":
dokument=dokument.replace(line, "\n")
file = open("tryhard.txt","w")
file.write(dokument)
file.close()
pdfkit.from_file("tryhard.txt", 'tryhard.pdf')
thanks a lot.

Converting binary data to hex in python properly

I'm working on a program that uses a BMP and a separate file for the transparency layer. I need to convert them into a PNG from that so I'm using PIL in python to do so. However, I need the data from the transparency file in hex so it can be added to the image. I am using the binascii.hexlify function to do that.
Now, the problem I'm having is for some reason the data, after going through the hexlify function (I've systematically narrowed it down by going through my code piece by piece), looks different than it does in my hex editor and is causing slight distortions in the images. I can't seem to figure out where I am going wrong.
Data before processing in Hex editor
Data after processing in Hex editor
Here is the problematic part off my code:
filename = askopenfilename(parent=root)
with open(filename, 'rb') as f:
content = f.read()
f.close()
hexContent = binascii.hexlify(content).decode("utf-8")
My input
My output (This is hexcontent written to a file. Since I know that it is not going wrong in the writing of the file, and it is also irrelevant to my actual program I did not add that part to the code snippet)
Before anyone asks I tried codecs.encode(content, 'hex') and binascii.b2a_hex(content).
As for how I know that it is this part that is messing up, I printed out binascii.hexlify(content) and found the same part as in the hex editor and it looked identical to what I had got in the end.
Another possibility for where it is going wrong is in the "open(filename, 'rb')" step. I haven't yet thought of a way to test that. So any help or suggestions would be appreciated. If you need one of the files I'm using for testing purposes, I'll gladly add one here.

If I understand your question correctly then your desired output should match Data before processing in Hex editor. I can obtain this with the following code:
with open('Input.alp', 'rb') as f:
i = 0
for i, chunk in enumerate(iter(lambda: f.read(16), b'')):
if 688 <= i * 16 <= 736:
print i * 16, chunk.encode('hex')
Outputs:
688 ffffffffffffffffffffffffffffffff
704 ffffffffffffffffffffffe000000000
720 000000000000000001dfffffffffffff
736 ffffffffffffffffffffffffffffffff
See this answer for a more detailed explanation.

Is it possible to read from a specific csv line? [duplicate]

This question already has answers here:
How to read specific lines from a file (by line number)?
(30 answers)
Closed 8 years ago.
I just found out it's not possible to write to a specific line in a csv file (only the end).
I have just come across another obstacle that I'm having trouble tackling, which is reading from a specific line in a csv file.
One way I have found to accomplish this is:
with open('file.csv',newline = '') as csvfile:
spamreader = csv.reader(csvfile,delimiter=',',quotechar = '"')
lines = []
for row in spamreader:
lines.append(row)
print('What line do you want to read from?')
line = lines[int(input())-1] #I think the -1 is right. since lists start at 0
However, I believe that this might be a slightly inefficient way to do this, since the more rows in the list "lines", the more RAM the program would be using.
Could someone tell me if this is actually an efficient way of doing this? Otherwise, I will just go with this.
Is there any way that I can do something like this?
spamreader.readRow(5) #I just made this up, but is there a similar function?
This is the page that I've been using, it's possible I skipped over it. https://docs.python.org/3/library/csv.html
Also, I'm not very advanced in programming, so if there is an advanced answer, can you try to keep the explanations fairly simple?

If you want to read starting from line 123:
for _ in range(122):
spamreader.next()
for row in spamreader:
...
With Python 3 it seems to be
next(spamreader)
One can also navigate in the file by moving the cursor to a specific byte using find and seek.

Easiest way to save game data for a mastermind game. [duplicate]

This question already has answers here:
Saving an Object (Data persistence)
(5 answers)
Closed 9 years ago.
I'm looking for the easiest way to save and load gamedata for a mastermind game. At the moment the things that need to be saved are, the number of colours being played with, the number of pegs being played with, how many guesses there have been and the actual answer. At the moment I'm saving all this information to a text file. I am having difficulty loading this information back in to the game, and have come across something called pickling. I've done some reading on it but I don't fully understand how it works and how it's different from the way I'm trying to do.
Thanks!

Pickling is a method of serialization - persisting data on disk.
You can handle manual pickle/unpickling if you want. But you asked for the easiest way - python has that. Just use shelve:
import shelve
d = shelve.open('my_mastermind_shelf')
That's.. it. Now just treat d as you would treat any other dict; shelve handles all the pickling behind the scenes. The only caveat: remember to call its .close() method when you're done with it.

As I said in my comment, you can use a simple text file to save the current state of your game. You can use open() to open a (new) file. Usually it's like this:
fname = raw_input("Put in the filename: ")
f = open(fname, 'w') # This opens the text file for write operations.
# If the file already exists, it will truncate it.
# If not, then it will creaate a new file.
f.write("Hey, I'm a hippie coder, ho.") # Or, whatever string you want.
# You don't have to be a hippie coder.
f.close()
Oh, and don't mind my poor example. It's 4 in the morning here, and I'm supposed to be in bed.