Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 8 years ago.
Improve this question
I've got a python script that is slowly consuming all of my memory (48GB). If I recall, python will perform garbage collection so there is no need for me to cleanup after myself?
for example:
class data_store:
dat1={}
dat2={}
dat3={}
class myclass ():
def mem_func(self):
self.x = data_store()
self.x.dat1 = (lots of data)
self.x.dat2 = (lots of data)
y = x.dat1 + 1
...
Most of my data is stored in data_store() temporarily before it is written out to files. I would think that this would be the source of the leak. Everytime mem_func() is called a new data_store() object is created and assigned to self.x. I assume that the old data_store() object would now be a candidate for the GC to delete. In addition, I would assume that y also be able to be deleted after mem_func completes.
The only other thing I can think of is that I am creating figures with matplotlib and saving them to a file. That is all done in one function but perhaps I need to delete the figure properly. Also, I have a sqlite db that is open the whole time where I am writing data but that is not alot of data. The image is much bigger.
You need to remember that GC only collects data that no pointer (variable) is pointing at it. In other words, as long as the memory is accessible via your variables, it won't be collected/freed.
So you need to assing None to the variables you don't need any more, or assign new data to the same variable names, if you don't need them any more.
Related
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 10 months ago.
Improve this question
I have an C++ game which sends a Python-SocketIO request to a server, which loads the requested JSON data into memory for reference, and then sends portions of it to the client as necessary. Most of the previous answers here detail that the server has to repeatedly search the database, when in this case, all of the data is stored in memory after the first time, and is released after the client disconnects.
I don't want to have a large influx of memory usage whenever a new client joins, however most of what I have seen points away from using small files (50-100kB absolute maximum), and instead use large files, which would cause the large memory usage I'm trying to avoid.
My question is this: would it still be beneficial to use one large file, or should I use the smaller files; both from an organization standpoint and from a performance one?
Is it better to have one large file or many smaller files for data storage?
Both can potentially be better. Each have their advantage and disadvantage. Which is better depends on the details of the use case. It's quite possible that best way may be something in between such as a few medium sized files.
Regarding performance, the most accurate way to verify what is best is to try out each of them and measure.
You should separate it into multiple files for less memory if you're only accessing small parts of it. For example, if you're only accessing let's say a player, then your folder structure would look like this:
players
- 0.json
- 1.json
other
- 0.json
Then you could write a function that just gets the player with a certain id (0, 1, etc.).
If you're planning on accessing all of the players, other objects, and more at once, then have the same folder structure and just concatenate the parts you need into one object in memory.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 5 years ago.
Improve this question
Currently I am working on a project in which I need to save n number of images (to be used in the program's scope). Since the number of images to be saved is dynamic, it may end up exhausting the whole space which i have for my project.
I wanted to know that can there be something added to my code so that after 100% completion of my code the images get automatically deleted as I do not need them after the code's execution.
How can this be done?
I need to save images as they are passed as an argument to one of my functions inside my code. If you know how can I pass image without saving it to my function then please comment here
might be an idea to delete the files immediately after you've done the code you need to do i.e
import os
# Open image
# Manipulate image
os.remove(path_to_image)
Keep track of all the image files you're creating, then delete them in a finally block to ensure they'll be deleted even if an exception is raised.
import os
temp_images = []
try:
# ...do stuff
# ...create image at path_to_file
temp_images.append(path_to_file) # called multiple times
# ...other stuff
finally:
for image in temp_images:
os.remove(image)
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 7 years ago.
Improve this question
I am new to python, been coding in school for about a year now, but I like to code when I get bored. I have made two programs but both are useless as I have to input each value of the variables every time I start it up. Is there anyway I can save the value of a variable externally so when it loads it will open up the file and assign each variable?
You should use the pickle module for that purpose:
l =[1,2,3,4,5]
import pickle
pickle.dump(l,open("mydata","wb"))
and for getting your variable back:
import pickle
l = pickle.load(open("mydata","rb"))
If you have many variables to save, consider embedding them in a dictionary for instance.
Yo can use the shelve module its pretty simple it puts all variables into a dictionary then when your file reopens you can make shelve set the variables back. Here is a good example of using the shelve module.
To save data to a file, you could use
filehandle = open(filename, 'w')
filehandle.write(string)
filehandle.close()
Preferred in Python is
with open(filename, 'w') as filehandle:
filehandle.write(string)
because the file will be closed upon exiting the with block even if the block exits with an error, and without requiring the programmer to remember to close the file.
Load the values back in with filehandle.readline() or readlines().
You can also use the Python libraries json or csv to facilitate moving data into and out of files. If you have no need to inspect or modify the data in the file using another program (e.g. Notepad++ or MS Excel), you might prefer pickle or shelve.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 7 years ago.
Improve this question
As the title suggests, I'm interested in the best (perhaps the most Pythonic way) to structure a program which uses many global variables.
First of all, by "many", I mean some 30 variables (which may be dictionaries, floats or strings) which every module of my program needs to access. Now, there seem to be two ways to do this:
define the "global" variables in seperate modules
use an object oriented approach
The advantage of using an object oriented approach is that I can have many instances of some main class initialized, and perhaps compare different values (results of some analysis, for example) later on.
I already have a program written, but basically it breaks down to one class with some 30 or so attributes. Although it works fine, I'm aware this is a pretty messy way to do this.
So, basically, is I use OOP approach, I would perhaps need to break my main class down to a few subclasses, every one of which stores specific logically related variables.
Any suggestions are welcome.
P.S. Just to be concrete about what I'm trying to do: I have a FEM-solver which needs to store structure info, element and node data, analysis result data, etc. So, I'm dealing with a lot of data types most of which are connected in some way.
Unfortunately, as was hinted at in the comments, there is no "Pythonic" way to do this. Having a large number of global constants is just fine - many programs and libraries do this. But in the comments, you've specified that all of your globals are being modified.
You need to take your program's architecture back to the drawing board. Rethink the relationships between your program's entities (functions, classes, modules, etc). There has to be a better way to organize it.
And by the way, it also sounds like you're getting close to using the God Object Antipattern. Use some of the advice in this SO question to refactor your massive class that has it's fingers all over your program.
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 9 years ago.
Improve this question
I have python file that has a user interface, that mainly create objects of some class.
That file will be used by my colleagues, on their own computers.
In another file, from my computer, I'm willing to reach those objects that the first file generated.
What will be the best way to "save" the objects of the class, and then reach them from
my computer?
Thanks
What you want to do is have the script serialize the objects, and send them to your computer over the network.
As inspectorG4dget has said, you can use the pickle module to serialize your objects, and the requests library should be good for sending the objects from the client side.
On your machine, you would need a web-server/socket-listener, listening for the sent messages. You would deserialize them, and use them in some way after that.
Pickle or cPickle nicely handles saving object instances (as well as anything else); documentation here.
Two notes from when I fumbled through the a similar problem:
When you load a pickled object instance, you must have the object's class definition present in the namespace of the script/environment where you load.
Not everything can be pickled; I ran into this when saving objects that contained scipy spline instances. In your class definition, you can override the default behavior when pickling and unpickling in order to safely save and restore such attributes.