cx_Oracle ignores order by clause - python

I've created complex query builder in my project, and during tests stumbled upon strange issue: same query with the same plan produces different results on different clients: cx_Oracle ignores order by clause, while Oracle SQLDeveloper Studio process query correctly, however in both cases order by present in both plans.
Query in question is:
select *
from
(
select
a.*,
ROWNUM tmp__rnum
from
(
select base.*
from
(
select id
from
(
(
select
profile_id as id,
surname as sort__col
from names
)
/* here usually are several other subqueries chained by unions */
)
group by id
order by min(sort__col) asc
) tmp
left join (profiles) base
on tmp.id = base.id
where exists
(
select t.object_id
from object_rights t
where
t.object_id = base.id
and t.subject_id = :a__subject_id
and t.rights in ('r','w')
)
) a
where ROWNUM < :rows_to
)
where tmp__rnum >= :rows_from
and plan from cx_Oracle in case I missed anything:
{'operation': 'SELECT STATEMENT', 'position': 9225, 'cardinality': 2164, 'time': 1, 'cost': 9225, 'depth': 0, 'bytes': 84396, 'optimizer': 'ALL_ROWS', 'id': 0, 'cpu_cost': 1983805801},
{'operation': 'VIEW', 'position': 1, 'filter_predicates': '"TMP__RNUM">=TO_NUMBER(:ROWS_FROM)', 'parent_id': 0, 'object_instance': 1, 'cardinality': 2164SEL$1', 'projection': '"from$_subquery$_001"."ID"[NUMBER,22], "from$_subquery$_001"."CREATION_TIME"[TIMESTAMP,11], "TMP__RNUM"[NUMBER,22]', 'time': 1, 'cost': 9225, 'depth': 1, 'bytes': 84396, 'id': 1, 'cpu_cost': 1983805801},
{'operation': 'COUNT', 'position': 1, 'filter_predicates': 'ROWNUM<TO_NUMBER(:ROWS_TO)', 'parent_id': 1, 'projection': '"BASE"."ID"[NUMBER,22], "BASE"."CREATION_TIME"[TIMESTAMP,11], ROWNUM[8]', 'options': 'STOPKEY', 'depth': 2, 'id': 2,
{'operation': 'HASH JOIN', 'position': 1, 'parent_id': 2, 'access_predicates': '"TMP"."ID"="BASE"."ID"', 'cardinality': 2164, 'projection': '(#keys=1) "BASE"."ID"[NUMBER,22], "BASE"."CREATION_TIME"[TIMESTAMP,11]', 'time': 1, 'cost': 9225, 'depth': 3, 'bytes': 86560, 'id': 3, 'cpu_cost': 1983805801},
{'operation': 'JOIN FILTER', 'position': 1, 'parent_id': 3, 'object_owner': 'SYS', 'cardinality': 2219, 'projection': '"BASE"."ID"[NUMBER,22], "BASE"."CREATION_TIME"[TIMESTAMP,11]', 'object_name': ':BF0000', 'time': 1, 'cost': 662, 'options': 'CREATE', 'depth': 4, 'bytes': 59913, 'id': 4, 'cpu_cost': 223290732},
{'operation': 'HASH JOIN', 'position': 1, 'parent_id': 4, 'access_predicates': '"T"."OBJECT_ID"="BASE"."ID"', 'cardinality': 2219, 'projection': '(#keys=1) "BASE"."ID"[NUMBER,22], "BASE"."CREATION_TIME"[TIMESTAMP,11]', 'time': 1, 'cost': 662, 'options': 'RIGHT SEMI', 'depth': 5, 'bytes': 59913, 'id': 5, 'cpu_cost': 223290732},
{'operation': 'TABLE ACCESS', 'position': 1, 'filter_predicates': '"T"."SUBJECT_ID"=TO_NUMBER(:A__SUBJECT_ID) AND ("T"."RIGHTS"=\'r\' OR "T"."RIGHTS"=\'w\')', 'parent_id': 5, 'object_type': 'TABLE', 'object_instance': 8, 'cardinality': 2219, 'projection': '"T"."OBJECT_ID"[NUMBER,22]', 'object_name': 'OBJECT_RIGHTS', 'time': 1, 'cost': 5, 'options': 'FULL', 'depth': 6, 'bytes': 24409, 'optimizer': 'ANALYZED', 'id': 6, 'cpu_cost': 1823386},
{'operation': 'TABLE ACCESS', 'position': 2, 'parent_id': 5, 'object_type': 'TABLE', 'object_instance': 6, 'cardinality': 753862, 'projection': '"BASE"."ID"[NUMBER,22], "BASE"."CREATION_TIME"[TIMESTAMP,11]', 'object_name': 'PROFILES', 'time': 1, 'cost': 654, 'options': 'FULL', 'depth': 6, 'bytes': 12061792, 'optimizer': 'ANALYZED', 'id': 7, 'cpu_cost': 145148296},
{'operation': 'VIEW', 'position': 2, 'parent_id': 3, 'object_instance': 3, 'cardinality': 735296, 'projection': '"TMP"."ID"[NUMBER,22]', 'time': 1, 'cost': 8559, 'depth': 4, 'bytes': 9558848, 'id': 8, 'cpu_cost': 1686052619},
{'operation': 'SORT', 'position': 1, 'parent_id': 8, 'cardinality': 735296, 'projection': '(#keys=1) MIN("SURNAME")[50], "PROFILE_ID"[NUMBER,22]', 'time': 1, 'cost': 8559, 'options': 'ORDER BY', 'temp_space': 18244000, 'depth': 5, 'bytes': 10294144, 'id': 9, 'cpu_cost': 1686052619},
{'operation': 'HASH', 'position': 1, 'parent_id': 9, 'cardinality': 735296, 'projection': '(#keys=1; rowset=200) "PROFILE_ID"[NUMBER,22], MIN("SURNAME")[50]', 'time': 1, 'cost': 8559, 'options': 'GROUP BY', 'temp_space': 18244000, 'depth': 6, 'bytes': 10294144, 'id': 10, 'cpu_cost': 1686052619},
{'operation': 'JOIN FILTER', 'position': 1, 'parent_id': 10, 'object_owner': 'SYS', 'cardinality': 756586, 'projection': '(rowset=200) "PROFILE_ID"[NUMBER,22], "SURNAME"[VARCHAR2,50]', 'object_name': ':BF0000', 'time': 1, 'cost': 1202, 'options': 'USE', 'depth': 7, 'bytes': 10592204, 'id': 11, 'cpu_cost': 190231639},
{'operation': 'TABLE ACCESS', 'position': 1, 'filter_predicates': 'SYS_OP_BLOOM_FILTER(:BF0000,"PROFILE_ID")', 'parent_id': 11, 'object_type': 'TABLE', 'object_instance': 5, 'cardinality': 756586, 'projection': '(rowset=200) "PROFILE_ID"[NUMBER,22], "SURNAME"[VARCHAR2,50]', 'object_name': 'NAMES', 'time': 1, 'cost': 1202, 'options': 'FULL', 'depth': 8, 'bytes': 10592204, 'optimizer': 'ANALYZED', 'id': 12, 'cpu_cost': 190231639}
cx_Oracle output (appears to be ordered by id):
ID, Created, rownum
(1829, 2016-08-24, 1)
(2438, 2016-08-24, 2)
SQLDeveloper Output (ordered by surname, as expected):
ID, Created, rownum
(518926, 2016-08-28, 1)
(565556, 2016-08-29, 2)

I don't see an ORDER BY clause that would affect the ordering of the results of the query. In SQL, the only way to guarantee the ordering of a result set is to have an ORDER BY clause for the outer-most SELECT.
In almost all cases, an ORDER BY in a subquery is not necessarily respected (Oracle makes an exception when there are rownum comparisons in the next level of the query -- and even that is now out of date with the support of FETCH FIRST <n> ROWS).
So, there is no reason to expect that an ORDER BY in the innermost subquery would have any effect, particularly with the JOIN that then happens.
Suggestions:
Move the ORDER BY to the outermost query.
Use FETCH FIRST syntax, if you are using Oracle 12c+.
Move the ORDER BY after the JOIN.
Use ROW_NUMBER() instead of rownum.

Related

How to do error-handling of JSON Parser Loop

I found some elegant code that builds a list by iterating through each element of another JSON list:
results = [
(
t["vintage"]["wine"]["winery"]["name"],
t["vintage"]["year"],
t["vintage"]["wine"]["id"],
f'{t["vintage"]["wine"]["name"]} {t["vintage"]["year"]}',
t["vintage"]["wine"]["statistics"]["ratings_average"],
t["vintage"]["wine"]["statistics"]["ratings_count"],
t["price"]["amount"],
t["vintage"]["wine"]["region"]["name"],
t["vintage"]["wine"]["style"]["name"], #<--------------issue here
)
for t in r.json()["explore_vintage"]["matches"]
]
The problem is that sometimes the JSON doesn't have a "name" element because the "style" is null (or None in JSON world). See the second-last line below for the JSON sample.
Is there a simple way to handle this error?
Error:
matches[23]["vintage"]["wine"]["style"]["name"]
Traceback (most recent call last):
File "<ipython-input-94-59447d0d4859>", line 1, in <module>
matches[23]["vintage"]["wine"]["style"]["name"]
TypeError: 'NoneType' object is not subscriptable
Perhaps something like:
iferror(t["vintage"]["wine"]["style"]["name"], "DoesNotExist")
JSON:
{'id': 4026076,
'name': 'Shiraz - Petit Verdot',
'seo_name': 'shiraz-petit-verdot',
'type_id': 1,
'vintage_type': 0,
'is_natural': False,
'region': {'id': 685,
'name': 'South Eastern Australia',
'name_en': '',
'seo_name': 'south-eastern',
'country': {'code': 'au',
'name': 'Australia',
'native_name': 'Australia',
'seo_name': 'australia',
'sponsored': False,
'currency': {'code': 'AUD',
'name': 'Australian Dollars',
'prefix': '$',
'suffix': None},
'regions_count': 120,
'users_count': 867353,
'wines_count': 108099,
'wineries_count': 13375,
'most_used_grapes': [{'id': 1,
'name': 'Shiraz/Syrah',
'seo_name': 'shiraz-syrah',
'has_detailed_info': True,
'wines_count': 536370},
{'id': 2,
'name': 'Cabernet Sauvignon',
'seo_name': 'cabernet-sauvignon',
'has_detailed_info': True,
'wines_count': 780931},
{'id': 5,
'name': 'Chardonnay',
'seo_name': 'chardonnay',
'has_detailed_info': True,
'wines_count': 586874}],
'background_video': None},
'class': {'typecast_map': {'background_image': {}, 'class': {}}},
'background_image': {'location': '//images.vivino.com/regions/backgrounds/0iT8wuQXRWaAmEGpPjZckg.jpg',
'variations': {'large': '//thumbs.vivino.com/region_backgrounds/0iT8wuQXRWaAmEGpPjZckg_1280x760.jpg',
'medium': '//thumbs.vivino.com/region_backgrounds/0iT8wuQXRWaAmEGpPjZckg_600x356.jpg'}}},
'winery': {'id': 74363,
'name': 'Barramundi',
'seo_name': 'barramundi',
'status': 0,
'background_image': None},
'taste': {'structure': None,
'flavor': [{'group': 'black_fruit', 'stats': {'count': 16, 'score': 2987}},
{'group': 'oak', 'stats': {'count': 11, 'score': 1329}},
{'group': 'red_fruit', 'stats': {'count': 10, 'score': 1413}},
{'group': 'spices', 'stats': {'count': 6, 'score': 430}},
{'group': 'non_oak', 'stats': {'count': 5, 'score': 126}},
{'group': 'floral', 'stats': {'count': 3, 'score': 300}},
{'group': 'earth', 'stats': {'count': 3, 'score': 249}},
{'group': 'microbio', 'stats': {'count': 2, 'score': 66}},
{'group': 'vegetal', 'stats': {'count': 1, 'score': 100}},
{'group': 'dried_fruit', 'stats': {'count': 1, 'score': 100}}]},
'statistics': {'status': 'Normal',
'ratings_count': 1002,
'ratings_average': 3.5,
'labels_count': 11180,
'vintages_count': 25},
'style': None,
'has_valid_ratings': True}

Is one of the numbers in this list in between the two given integers?

I have a list with barline ticks and midi notes that can overlap the barlines. So I made a list of 'barlineticks':
barlinepos = [0, 768.0, 1536.0, 2304.0, 3072.0, 3840.0, 4608.0, 5376.0, 6144.0, 6912.0, 0, 576.0, 1152.0, 1728.0, 2304.0, 2880.0, 3456.0, 4032.0, 4608.0, 5184.0, 5760.0, 6336.0, 6912.0, 7488.0]
And a MidiFile:
{'type': 'time_signature', 'numerator': 4, 'denominator': 4, 'time': 0, 'duration': 768, 'ID': 0}
{'type': 'set_tempo', 'tempo': 500000, 'time': 0, 'ID': 1}
{'type': 'track_name', 'name': 'Tempo Track', 'time': 0, 'ID': 2}
{'type': 'track_name', 'name': 'New Instrument', 'time': 0, 'ID': 3}
{'type': 'note_on', 'time': 0, 'channel': 0, 'note': 48, 'velocity': 100, 'ID': 4, 'duration': 956}
{'type': 'time_signature', 'numerator': 3, 'denominator': 4, 'time': 768, 'duration': 6911, 'ID': 5}
{'type': 'note_on', 'time': 768, 'channel': 0, 'note': 46, 'velocity': 100, 'ID': 6, 'duration': 575}
{'type': 'note_off', 'time': 956, 'channel': 0, 'note': 48, 'velocity': 0, 'ID': 7}
{'type': 'note_off', 'time': 1343, 'channel': 0, 'note': 46, 'velocity': 0, 'ID': 8}
{'type': 'end_of_track', 'time': 7679, 'ID': 9}
And I want to check if the midi note is overlapping a barline. Every note_on message has a 'time' and a 'duration' value. I have to check if one of the barlineticks(in the list) is inside the range of the note('time' and 'duration'). I tried:
if barlinepos in range(0, 956):
print(True)
Of course this doesn't work because barlinepos is a list. How can I check if one of the values in the list results in True?
Simple iteration to solve the requirement:
for i in midifile:
start, end = i["time"], i["time"]+i["duration"]
for j in barlinepos:
if j >= start and j<= end:
print(True)
break
print(False)

How do i fetch first key value pair (i.e., 'type': 45) from a dictionary which contains key as list with dictionaries

{'alarms': [{'date': '20170925T235525-0700',
'id': 8,
'ip': '172.26.70.4',
'severity': 4,
'type': 45},
{'date': '20170925T235525-0700',
'id': 7,
'ip': '172.26.70.4',
'severity': 4,
'type': 45},
{'date': '20170925T235525-0700',
'id': 6,
'ip': '172.26.70.4',
'severity': 4,
'type': 45},
{'date': '20170925T220858-0700',
'id': 5,
'ip': '172.26.70.4',
'severity': 6,
'type': 44},
{'date': '20170925T220857-0700',
'id': 4,
'ip': '172.26.70.4',
'severity': 6,
'type': 44},
{'date': '20170925T220857-0700',
'id': 3,
'ip': '172.26.70.4',
'severity': 6,
'type': 44},
{'date': '20170925T220856-0700',
'id': 2,
'severity': 6,
'type': 32},
{'date': '20170925T220850-0700', 'id': 1, 'severity': 6, 'type': 1},
{'date': '20170925T220850-0700',
'id': 0,
'severity': 6,
'type': 33}]}
Need to fetch first key value pair (i.e., 'type': 45)
Kindly guide, I am trying it on Python 2.7.
Your data is a dictionary where the `"alarms" key is associated with a list of dictionaries.
That dictionary is in the list associated with the "alarms" key. So you can fetch it with:
data['alarms'][0]
with data the variable that stores this structure. So:
>>> data['alarms'][0]
{'date': '20170925T235525-0700', 'severity': 4, 'id': 8, 'ip': '172.26.70.4', 'type': 45}
you will need to do :
def return_correct_dict(data):
for d in data['alarms']:
if d.get('type',"") == 45:
return d
You have a dictionary of a list of dictionaries.
Suppose your dictionary is stored in a variable named dict.
dict['alarm'][0]['type'] will give you the value 45.

Specific Sort of elements to add in new list Python/Django

I have this :
[
[{ 'position': 1, 'user_id': 2, 'value': 4, 'points': 100}],
[{ 'position': 2, 'user_id': 6, 'value': 3, 'points': 88}],
[{ 'position': 3, 'user_id': 5, 'value': 2, 'points': 77}],
[{ 'position': 4, 'user_id': 7, 'value': 1, 'points': 66}],
[{ 'position': 5, 'user_id': 3, 'value': 1, 'points': 9}],
[{ 'position': 6, 'user_id': 11, 'value': 0, 'points': 9}],
[{ 'position': 7, 'user_id': 1, 'value': 0, 'points': 3}],
[{ 'position': 8, 'user_id': 10, 'value': 0, 'points': 3}],
[{ 'position': 9, 'user_id': 4, 'value': 0, 'points': 2}],
[{ 'position': 10, 'user_id': 8, 'value': 0, 'points': 2}]
]
is organized by points.
The idea is to choose the user_id and generate a new list with the selected 5 users.
Example:
user_id=3:
[{ 'position': 3, 'user_id': 5, 'value': 2, 'points': 77}],
[{ 'position': 4, 'user_id': 7, 'value': 1, 'points': 66}],
[{ 'position': 5, 'user_id': 3, 'value': 1, 'points': 9}],
[{ 'position': 6, 'user_id': 11, 'value': 0, 'points': 9}],
[{ 'position': 7, 'user_id': 1, 'value': 0, 'points': 3}]
It returns user_id 3 in the middle with 2 users hight and 2 users lower
user_id=2
[{ 'position': 1, 'user_id': 2, 'value': 4, 'points': 100}],
[{ 'position': 2, 'user_id': 6, 'value': 3, 'points': 88}],
[{ 'position': 3, 'user_id': 5, 'value': 2, 'points': 77}],
[{ 'position': 4, 'user_id': 7, 'value': 1, 'points': 66}],
[{ 'position': 5, 'user_id': 3, 'value': 1, 'points': 9}],
As user_id hasn't higher users it returns 4 lower users. So is always same logic.
user_id=9:
[{ 'position': 6, 'user_id': 11, 'value': 0, 'points': 9}],
[{ 'position': 7, 'user_id': 1, 'value': 0, 'points': 3}],
[{ 'position': 8, 'user_id': 10, 'value': 0, 'points': 3}],
[{ 'position': 9, 'user_id': 4, 'value': 0, 'points': 2}],
[{ 'position': 10, 'user_id': 8, 'value': 0, 'points': 2}]
on user_id=9 We only have 1 user lower so we add 3 higher users
If for example we just have 2 users in list, it should return that 2 users.
Main rules:
If we have 5 users or more, as to return 5 users.
if we have 4 users, as to return 4 users
How is a good way to do it?
thanks
This is basically only an update of my answer to your original question.
a = [
[{ 'position': 1, 'user_id': 2, 'value': 4, 'points': 100}],
[{ 'position': 2, 'user_id': 6, 'value': 3, 'points': 88}],
[{ 'position': 3, 'user_id': 5, 'value': 2, 'points': 77}],
[{ 'position': 4, 'user_id': 7, 'value': 1, 'points': 66}],
[{ 'position': 5, 'user_id': 3, 'value': 1, 'points': 9}],
[{ 'position': 6, 'user_id': 11, 'value': 0, 'points': 9}],
[{ 'position': 7, 'user_id': 1, 'value': 0, 'points': 3}],
[{ 'position': 8, 'user_id': 10, 'value': 0, 'points': 3}],
[{ 'position': 9, 'user_id': 4, 'value': 0, 'points': 2}],
[{ 'position': 10, 'user_id': 8, 'value': 0, 'points': 2}]
]
# Sort it if not already sorted
# a.sort(key=lambda x: x[0]['position'])
def find_index(l, user_id):
i = 0
while l[i][0]['user_id'] != user_id:
i += 1
return i
def get_subset(l, i):
return l[:(i + 1 + max(2, 4 - i))][-5:]
get_subset(a, find_index(a, 3))

How to do a function to generate a rank/score number?

I need to do a function that generate a rank number(integer) based in N parameter received in that function. Each number need to be the same order of magnitude of that parameters, compare numbers of the same field and when the numbers of the same field tie than use second field to deadlock, e.g:
func({'field1': 100, 'field2': 3500225})
func({'field1': 50, 'field2': 5465481362135)
The number generated by the first function must be higher than the second because 100 is greater than 50.
func({'field1': 100, 'field2': 3500225})
func({'field1': 100, 'field2': 5465481362135)
The number generated by the second function must be higher than the first because field1 is tie than to deadloack it need to use the second field, so 5465481362135 is greater than 3500225.
func({'field1': 100, 'field2': 3500225})
func({'field1': 100, 'field2': 3500225, 'field3': 5465481362135, ...N})
In the sample above the second function need to be higher than the first because the first call function doesn't have the field3 so we can set their value to zero. Note that we can have N fields.
I have tried this code bellow but the value after the dot isn't right because a simple sum doesn't consider the priority of fields.:
Config = [
{'id': 1, 'parent_id': None, 'Description': 'root', 'level': 1},
{'id': 2, 'parent_id': 1, 'Description': 'PF', 'level': 1},
{'id': 3, 'parent_id': 1, 'Description': 'FP', 'level': 2},
{'id': 4, 'parent_id': 2, 'Description': 'Bank', 'level': 1},
{'id': 5, 'parent_id': 2, 'Description': 'Input', 'level': 2},
{'id': 6, 'parent_id': 4, 'Description': 'ST', 'level': 1},
{'id': 7, 'parent_id': 4, 'Description': 'CF', 'level': 2},
{'id': 8, 'parent_id': 4, 'Description': 'BB', 'level': 3},
{'id': 9, 'parent_id': 5, 'Description': 'DDS', 'level': 1},
{'id': 10, 'parent_id': 5, 'Description': 'Col', 'level': 2},
{'id': 11, 'parent_id': 3, 'Description': 'Qtd.Event', 'level': 1},
{'id': 12, 'parent_id': 3, 'Description': 'Unix_Date', 'level': 2},
]
def hierarchy_field(field):
if field[0]['parent_id'] is None:
return field
else:
[field.append(x) for x in hierarchy_field([dic for dic in Config if dic['id'] == field[0]['parent_id']])]
return field
def calc_priority(fields):
for field in [dic for dic in Config if dic['Description'] in [x for x in fields]]:
score_level = sum(x['level'] for x in hierarchy_field([field]))*-1
score_level += score_level
score_value = sum(fields.values())
score = str(score_level)+'.'+str(score_value)
return score
# dic[field['Description']] = score_level
# return sorted(dic, reverse=True)
value = calc_priority({'Qtd.Event': 871, 'Unix_Date': 564645})
print(value)

Categories

Resources