This question has a little history — Is there a way to make a query respect the order of the inputted parameters?
I'm new to building "specialized" queries, so I assumed that if I supply an IN clause as part of a SELECT query, it'll return results in the same order. Unfortunately that's not the case.
SELECT * FROM artists WHERE id IN (8, 1, 2, 15, 14, 3, 13, 31, 16, 5, 4, 7, 32, 9, 37)
>>> [7, 32, 3, 8, 4, 2, 31, 9, 37, 13, 16, 1, 5, 15, 14]
(Didn't include the step where I used Python to loop through the result and append the IDs to a list.)
So the question is, is there a way to make Postgres respect the ordering of the parameters given in an IN clause by returning results the same order?
Query results will be returned in non-deterministic order unless you specify an ORDER BY clause.
If you really want to do the query in the manner you are requesting, then you could construct such a clause. Here's an example using part of your data.
create table artists (
id integer not null primary key,
name char(1) not null);
insert into artists
values
(8, 'a'),
(1, 'b'),
(2, 'c'),
(15, 'd'),
(14, 'e'),
(3, 'f'),
(13, 'g');
select *
from artists
where id in (8, 1, 2, 15, 14, 3, 13)
order by
id = 8 desc,
id = 1 desc,
id = 2 desc,
id = 15 desc,
id = 14 desc,
id = 3 desc,
id = 13 desc;
Based on this and on your other question, I think there is something wrong with your model or the way you are trying to do this. Perhaps you should post a more generic question about how to do what you are trying to do.
If you do have artists and ranking tables, you should be able to do something like this (or the equivalent through your ORM).
select
a.*
from
artists a,
rankings r
where
a.id = r.artist_id
order by
r.score desc;
I suggest you let PostGreSQL return the set in any arbitrary order (especially since it's difficult to do fine-grained SQL-level control from a Django interface), then sort it in the way you wish in Python -- theresultset.sort(key=yourlistofids.index) should do fine (when theresultset is the arbitrary-order list resulting from the database and yourlistofids is the list whose order you want to preserve).
Another way:
SELECT *
FROM artists
WHERE id IN (8, 1, 2, 15, 14, 3, 13, 31, 16, 5, 4, 7, 32, 9, 37)
ORDER BY POSITION(id::text in '(8, 1, 2, 15, 14, 3, 13, 31, 16, 5, 4, 7, 32, 9, 37)');
Related
I have large dict which has 3-tuples of integers as keys. I would like to save it to disk so I can read it in quickly. Sadly it seems I can't save it as a JSON file (which would let me use a fast JSON module such as orjson). What are my options other than pickle?
A tiny example would be:
my_dict = {
(1, 2, 3): [4, 5, 6],
(4, 5, 6): [7, 8, 9],
(7, 8, 9): [10, 11, 12]
}
I have about 500,000 keys and each value list is of length 500.
I will make this data once and it will not be modified after it is made. my_dict will only ever be used as a lookup table
You can try with the package pprint. This is a code saving the file as a Python module, which you can import either as module or just the dictionary object. This is the code.
import pprint
my_dict = {
(1, 2, 3): [4, 5, 6],
(4, 5, 6): [7, 8, 9],
(7, 8, 9): [10, 11, 12]
}
obj_str = pprint.pformat(my_dict, indent=4, compact=False)
message = f'my_dict = {obj_str}\n'
with open('data.py', 'w') as f:
f.write(message)
Of course, you don't have to save it as a Python module, you can just save it as text/binary data and read it into your program as an object; maybe with eval in case you save it as text.
EDIT
Just saw you edited the question. This might be enough for 500,000 keys with 500 items each.
I have created a generic function to process consecutive pairings of n length from a given list of integers and give them to a function. It works but I very much dislike the eval in the function but don't know how to change that and still use the zip function.
def consecutive_element_pairing(data: list[int], consecutive_element=3, map_to_func=sum) -> list[int]:
"""
Return a list with consecutively paired items given to a function that can handle an iterable
:param data: the list of integers to process
:param consecutive_element: how many to group consecutively
:param map_to_func: the function to give the groups to
:return: the new list of consecutive grouped functioned items
"""
if len(data) < consecutive_element:
return []
return list(map(map_to_func, eval("zip(%s)" % "".join((map(lambda x: "data[%d:], " % x, range(consecutive_element)))))))
given a list of e.g.:
values = [1, 2, 3, 4, 5, 6, 7, 8, 9]
and I call it like this:
print("result:", consecutive_element_pairing(values))
[6, 9, 12, 15, 18, 21, 24]
This is correct as it correctly groups ((1,2,3),(2,3,4),(3,4,5)...) them by consecutive groups of 3 and then sums those.
The trouble I have with my code is the eval statement on the generated string of zip(data[0:], data[1:], data[2:], ).
I have no idea how to do this a different way as zip with a list inside does something completely different.
Can this be done differently while still using zip?
Any help is appreciated.
I know how to do this in many different ways but the challenge for myself was the usage of zip here :-) and making it a "generic" function.
You can simply use zip(*(values[i:] for i in range(N))):
Example
values = [1, 2, 3, 4, 5, 6, 7, 8, 9]
N = 3
list(zip(*(values[i:] for i in range(N))))
# [(1, 2, 3), (2, 3, 4), (3, 4, 5), (4, 5, 6), (5, 6, 7), (6, 7, 8), (7, 8, 9)]
A slightly improved variant for long lists and large N might be:
zip(*(values[i:len(values)-(N-i)+1] for i in range(N)))
function
def consecutive_element_pairing(data: list[int], consecutive_element=3, map_to_func=sum) -> list[int]:
N = consecutive_element
return list(map(map_to_func, zip(*(data[i:len(data)-(N-i)+1] for i in range(N)))))
consecutive_element_pairing(values)
# [6, 9, 12, 15, 18, 21, 24]
When printing PostgreSQL query results I only see the result values, I would like to see the column name along with the result values
postgreSQL_select_Query = "SELECT epic,timestamp FROM market_data_historic s1 WHERE timestamp = (SELECT MAX(timestamp) FROM market_data_historic s2 WHERE s1.epic = s2.epic)"
cursor.execute(postgreSQL_select_Query)
# Close the connection
result=(cursor.fetchall())
for row in result:
print (row)
This is the result I get:
('CC.D.LCO.USS.IP', datetime.datetime(2019, 11, 13, 22, 0))
('IX.D.DAX.DAILY.IP', datetime.datetime(2019, 7, 23, 4, 0))
('KB.D.ELECTY.DAILY.IP', datetime.datetime(2020, 1, 24, 16, 0))
('CS.D.ALUMINIUM.TODAY.IP', datetime.datetime(2019, 7, 23, 1, 0))
('CS.D.NZDCAD.TODAY.IP', datetime.datetime(2020, 1, 24, 21, 0))
('CS.D.CADCNH.TODAY.IP', datetime.datetime(2020, 1, 16, 8, 0))
How can I get it to be like this:
(epic:'CC.D.LCO.USS.IP',timestamp: datetime.datetime(2019, 11, 13, 22, 0))
(epic:'IX.D.DAX.DAILY.IP',timestamp: datetime.datetime(2019, 7, 23, 4, 0))
(epic:'KB.D.ELECTY.DAILY.IP',timestamp: datetime.datetime(2020, 1, 24, 16, 0))
(epic:'CS.D.ALUMINIUM.TODAY.IP',timestamp: datetime.datetime(2019, 7, 23, 1, 0))
(epic:'CS.D.NZDCAD.TODAY.IP',timestamp: datetime.datetime(2020, 1, 24, 21, 0))
(epic:'CS.D.CADCNH.TODAY.IP',timestamp: datetime.datetime(2020, 1, 16, 8, 0))
Use the cursor.description attribute to retrieve column names and convert results to dictionaries:
result = cursor.fetchall()
columns = [desc[0] for desc in cursor.description]
for row in result:
print (dict(zip(columns, row)))
Alternatively, you can use Real dictionary cursor or Named tuple cursor.
See also DictCursor vs RealDictCursor.
Try the description attribute of cursor:
Read-only attribute describing the result of a query. It is a sequence of Column instances, each one describing one result column in order. The attribute is None for operations that do not return rows or if the cursor has not had an operation invoked via the execute*() methods yet.
For compatibility with the DB-API, every object can be unpacked as a 7-items sequence: the attributes retuned this way are the following. For further details and other attributes available check the Column documentation.
name: the name of the column returned.
type_code: the PostgreSQL OID of the column.
display_size: the actual length of the column in bytes.
internal_size: the size in bytes of the column associated to this column on the server.
precision: total number of significant digits in columns of type NUMERIC. None for other types.
scale: count of decimal digits in the fractional part in columns of type NUMERIC. None for other types.
null_ok: always None as not easy to retrieve from the libpq.
I have a list of lists with tuples. I want to get the length of a tuple using:
item1=(4, 8, 16, 30)
list6=[[(4, 8, 16, 29)], [(4, 8, 16, 30)], [(4, 8, 16, 32)]]
print("list6.index((4, 8, 16, 29)):",list6.index([item1]))
print("len(list6[1]):"), len(list6[1])
Output:
list6.index((4, 8, 16, 29)): 1
len(list6[1]):
There is no value for len(list6[1]). Can someone show me the correct syntax for this?
The code works fine in Python 2. If you are using Python 3, there is an issue with last line, because print is a function. So, because of where you've put the close parenthesis, only the first part is actually passed to print. Try this instead
print("len(list6[1]):", len(list6[1]))
I have two separate query sets A and B, I want to append B to A without any constraint.
Let's assume two queryset A and B.
A = (1, 5, 7, 15, 20 )
B = (4, 6, 10,14, 19, 21)
Now What I really want:
Final_queryset = (1, 5, 7, 15, 20 ,4, 6, 10,14, 19, 21)
I just want to append B to A without giving any order_by and without disturbing any order.
Here I can not put any order_by constraint because it will disturb the order. I do not want to use list because list because it loads whole objects into memory and I have 50000- 60000 objects so I can not use list.
Any idea on how I can achieve this using only querysets in python
You want itertools.chain(A, B).