Develop Reference

make function memory efficent or store data somewhere else to avoid memory error

I currently have a for loop which is finding and storing combinations in a list. The possible combinations are very large and I need to be able to access the combos.
can I use an empty relational db like SQLite to store my list on a disk instead of using list = []?
Essentially what I am asking is whether there is a db equivalent to list = [] that I can use to store the combinations generated via my script?
Edit:
SQLlite is not a must. Any will work if it can accomplish my task.
Here is the exact function that is causing me so much trouble. Maybe there is a better solution in general.
Idea - Could I insert the list into the database on each loop and then empty the list? Basically, create a list on each loop, send that list to PostgreSQL and then empty the list in the python to keep the RAM usage down?
def permute(set1, set2):
set1_combos = list(combinations(set1, 2))
set2_combos = list(combinations(set2, 8))
full_sets = []
for i in set1_combos:
for j in set2_combos:
full_sets.append(i + j)
return full_sets

Ok, a few ideas
My first thought was, why do you explode the combinations objects in lists? But of course, since we have two nested for loops, the iterator in the inner loop is consumed at the first iteration of the outer loop if it is not converted to a list.
However, you don't need to explode both objects: you can explode just the smaller one. For instance, if both our sets are made of 50 elements, the combinations of 2 elements are 1225 with a memsize (if the items are integers) of about 120 bytes each, i.e. 147KB, while the combinations of 8 elements are 5.36e+08 with a memsize of about 336 bytes, i.e. 180GB. So the first thing is, keep the larger combo set as a combinations object and iterate over it in the outer loop. By the way, this will also be really faster.
Now the database part. I assume a relational DBMS, be it SQLite or anything.
You want to create a table with a single column defined. Each row of your table will contain one final combination. Instead of appending each combination to a list, you will insert it in the table.
Now the question is, how do you need to access the data you created? Do you just need to iterate over the final combos sequentially, or do you need to query them, for instance finding all the combos which contain one specific value?
In the latter case, you'll want to define your column as the Primay Key, so your queries will be efficient; otherwise, you will save space on disk using an auto incrementing integer as the PK (SQLite will create it for you if you don't explicitly define a PK, and so will do a few other DMBS as well).
One final note: the insert phase may be painfully slow if you don't take some specific measures: check this very interesting SO post for details. In short, with a few optimizations they were able to pass from 85 to over 96K insert per second.
EDIT: iterating over the saved data
Once we have the data in the DB, iterating over them could be as simple as:
mycursor.execute('SELECT * FROM <table> WHERE <conditions>')
for combo in mycursor.fetchall():
print(combo) #or do what you need
But if your conditions don't filter away most of the rows you will meet the same memory issue we started with. A first step could be using fetchmany() or even fetchone() instead of fetchall() but still you may have a problem with the size of the query result set.
So you will probably need to read from the DB a chunk of data at a time, exploiting the LIMIT and OFFSET parameters in your SELECT. The final result may be something like:
chunck_size = 1000 #or whatever number fits your case
chunk_count = 0
chunk = mycursor.execute(f'SELECT * from <table> WHERE <conditions> LIMIT {chunk_size} ORDER BY <primarykey>'}
while chunk:
for combo in mycursor.fetchall():
print(combo) #or do what you need
chunk_count += 1
chunk = mycursor.execute(f'SELECT * from <table> WHERE <conditions> ORDER BY <primarykey>' OFFSET {chunk_size * chunk_count} LIMIT {chunk_size}}
Note that you will usually need the ORDER BY clause to ensure rows are returned as you expect them, and not in a random manner.

I don't believe SQLite has a built in array data type. Other DBMSs, such as PostgreSQL, do.
For SQLite, a good recommendation by another user on this site to obtain an array in SQLite can be found here: How to store array in one column in Sqlite3?
Another solution can be found: https://sqlite.org/forum/info/99a33767e8a07e59
In either case, yes it is possible to have a DBMS like SQLite store an array (list) type. However, it may require a little setup depending on the DBMS.
Edit: If you're having memory issues, have you thought about storing your data as a string and accessing the portions of the string you need when you need it?

postgres is not correctly incrementing my id which is of serial data-type

there are 500 records in my database and when I delete the id =5 and then try to re-insert the same record it is not updating the sequence correctly. It inserts the new record where id = 505 however my largest id was 500.
I am also updating sequencing before and after transaction with the following query:
SELECT setval(pg_get_serial_sequence('table_name', 'id'), max(id)) FROM table_name;
Still, it is incrementing the sequence not correctly.
id bigserial primary key NOT NULL
Any help will be appreciated.
Thanks

Deleting a row for which as sequence has been used to populate a column does NOT update the sequence.
If you need a gapless ordering number you cannot use a sequence.
See https://www.postgresql.org/docs/9.5/sql-createsequence.html which says:
Because nextval and setval calls are never rolled back, sequence
objects cannot be used if "gapless" assignment of sequence numbers is
needed. It is possible to build gapless assignment by using exclusive
locking of a table containing a counter; but this solution is much
more expensive than sequence objects, especially if many transactions
need sequence numbers concurrently.

Peewee incrementing an integer field without the use of primary key during migration

I have a table I need to add columns to it, one of them is a column that dictates business logic. So think of it as a "priority" column, and it has to be unique and a integer field. It cannot be the primary key but it is unique for business logic purposes.
I've searched the docs but I can't find a way to add the column and add default (say starting from 1) values and auto increment them without setting this as a primarykey..
Thus creating the field like
example_column = IntegerField(null=False, db_column='PriorityQueue',default=1)
This will fail because of the unique constraint. I should also mention this is happening when I'm migrating the table (existing data will all receive a value of '1')
So, is it possible to do the above somehow and get the column to auto increment?

It should definitely be possible, especially outside of peewee. You can definitely make a counter that starts at 1 and increments to the stop and at the interval of your choice with range(). You can then write each incremented variable to the desired field in each row as you iterate through.

Depends on your database, but postgres uses sequences to handle this kind of thing. Peewee fields accept a sequence name as an initialization parameter, so you could pass it in that manner.

Storing Python Lists in SQL Database

I have a SQL database that I store python lists in. Currently I convert the list to a string and then insert it into the database (using sqlite3) i.e.
foo = [1,2,3]
foo = str(foo)
#Establish connection with database code here and get cursor 'cur'
cur.execute("INSERT INTO Table VALUES(?, ?)", (uniqueKey, foo,))
It seems strange to convert my list to a string first, is there a better way to do this?

Replace your (key, listdata) table with (key, index, listitem). The unique key for the table becomes (key, index) instead of just key, and you'll want to ensure as a consistency condition that the set of indexes in the table for any given key is contiguous starting from 0.
You may or may not also need to distinguish between a key whose list is empty and a key that doesn't exist at all. One way is to have two tables (one of lists, and one of their elements), so that an empty but existing list is naturally represented as a row in the lists table with no corresponding rows in the elements table. Another way is just to fudge it and say that a row with index=null implies that the list for that key is empty.
Note that this is worthwhile if (and probably only if) you want to act on the elements of the list using SQL (for example writing a query to pull the last element of every list in the table). If you don't need to do that, then it's not completely unreasonable to treat your lists as opaque data in the DB. You're just losing the ability for the DB to "understand" it.
The remaining question then is how best to serialize/deserialize the list. str/eval does the job, but is a little worrying. You might consider json.dumps / json.loads, which for a list of integers is the same string format but with more safety restrictions in the parser. Or you could use a more compact binary representation if space is an issue.

2 ways.
Normalize tables, to you need setup new table for list value. so you get something like "TABLE list(id)" and "TABLE list_values(list_id, value)".
You can serialize the list and put in a column. Ex. Json, XML and so on (its not a very good practice in SQL).

Hidden id on tkInter Listbox

I'm wondering if it's possible to somehow store a hidden id along with each entry on a Listbox. The reason for this is that I've got a table which contains a unique id which is from a database (not visible to the user but used to uniquely identify each record) I'm caching the table in memory and using a dictionary keyed on the id
I'd like to create a Listbox which allows me to select one of the records - the displayed text would not be the unique id but a descriptive field (such as 'Name') which is probably unique but this is not enforced and there is no index on it. So for example, If I have:
Id Name
-- ----
2 Rod
5 Jane
15 Freddy
Then selecting Jane, I would somehow be able to easily access the id 5
My problem is that I can't find a way to associate the unique id (5) with the selection (Jane) so that I can easily identify the cached record. I know that I can use control variables but this just gives me a list of all the strings in the list - not what I want. Also, the index (for example on insert) does not seem to be reliable for this purpose.
The only way that I've managed to do this is to have another dictionary mapping the name to the id. For a number of reasons, this is sub-optimal.
Am I missing something here? Is there an easier way of doing this?

Keep your ids in a list, then use the .curselection() index to map these back to the row ids, as long as you keep the ordering the same.
In your example, Jane is the second choice in your list, so if selected .curselection() returns 1. If you have a rowids list in the same order, rowids[1] will be 5:
>>> rowids = [2, 5, 15]
>>> rowids[listbox.curselection()]
5
Slightly more efficient than mapping names to rowids in a dictionary.

If you use the ttk.Treeview widget instead of a Listbox, you can store the id in an invisible column.