Develop Reference

What is the best way to return a variable or call a function to maximize code reuse?

I was wondering if i could get some input from some season python exports, i have a couple questions
I am extracting data from an api request and calculating the total vulnerabilities,
what is the best way i can return this data so that i can call it in another function
what is the way i can add up all the vulnerabilities (right now its just adding it per 500 at a time, id like to do the sum of every vulnerability
def _request():
third_party_patching_filer = {
"asset": "asset.agentKey IS NOT NULL",
"vulnerability" : "vulnerability.categories NOT IN ['microsoft patch']"}
headers = _headers()
print(headers)
url1 = f"https://us.api.insight.rapid7.com/vm/v4/integration/assets"
resp = requests.post(url=url1, headers=headers, json=third_party_patching_filer, verify=False).json()
jsonData = resp
#print(jsonData)
has_next_cursor = False
nextKey = ""
if "cursor" in jsonData["metadata"]:
has_next_cursor = True
nextKey = jsonData["metadata"]["cursor"]
while has_next_cursor:
url2 = f"https://us.api.insight.rapid7.com/vm/v4/integration/assets?&size=500&cursor={nextKey}"
resp2 = requests.post(url=url2, headers=headers, json=third_party_patching_filer, verify=False).json()
cursor = resp2["metadata"]
print(cursor)
if "cursor" in cursor:
nextKey = cursor["cursor"]
print(f"next key {nextKey}")
#print(desktop_support)
for data in resp2["data"]:
for tags in data['tags']:
total_critical_vul_osswin = []
total_severe_vul_osswin = []
total_modoer_vuln_osswin = []
if tags["name"] == 'OSSWIN':
print("OSSWIN")
critical_vuln_osswin = data['critical_vulnerabilities']
severe_vuln_osswin = data['severe_vulnerabilities']
modoer_vuln_osswin = data['moderate_vulnerabilities']
total_critical_vul_osswin.append(critical_vuln_osswin)
total_severe_vul_osswin.append(severe_vuln_osswin)
total_modoer_vuln_osswin.append(modoer_vuln_osswin)
print(sum(total_critical_vul_osswin))
print(sum(total_severe_vul_osswin))
print(sum(total_modoer_vuln_osswin))
if tags["name"] == 'DESKTOP_SUPPORT':
print("Desktop")
total_critical_vul_desktop = []
total_severe_vul_desktop = []
total_modorate_vuln_desktop = []
critical_vuln_desktop = data['critical_vulnerabilities']
severe_vuln_desktop = data['severe_vulnerabilities']
moderate_vuln_desktop = data['moderate_vulnerabilities']
total_critical_vul_desktop.append(critical_vuln_desktop)
total_severe_vul_desktop.append(severe_vuln_desktop)
total_modorate_vuln_desktop.append(moderate_vuln_desktop)
print(sum(total_critical_vul_desktop))
print(sum(total_severe_vul_desktop))
print(sum(total_modorate_vuln_desktop))
else:
pass
else:
has_next_cursor = False

If you have a lot of parameters to pass, consider using a dict to combine them. Then you can just return the dict and pass it along to the next function that needs that data. Another approach would be to create a class and either access the variables directly or have helper functions that do so. The latter is a cleaner solution vs a dict, since with a dict you have to quote every variable name, and with a class you can easily add additional functionally beyond just being a container for a bunch of instance variables.
If you want the total across all the data, you should put these initializations:
total_critical_vul_osswin = []
total_severe_vul_osswin = []
total_modoer_vuln_osswin = []
before the while has_next_cursor loop (and similarly for the desktop totals). The way your code is currently, they are initialized each cursor (ie, each 500 samples based on the URL).

django.core.exceptions.ValidationError: ['“TRUE” value must be either True or False.']

I am trying to upload data in the django ORM by a script for which I have written this
for index, row in df.iterrows():
allocated = row['is_allocated']
delivery_required_on = row['delivery_required_on']
linked = row['linked']
raised_by = row['raised_by']
raised_for = Company.objects.get(pk=row['raised_for']) ### double check
rejected = row['is_rejected']
reason = row['reason']
remarks = row['remarks']
created = row['created_at']
owner = User.objects.get(pk=row['New owner'])
j = literal_eval(row['flows'])
flows = []
mobj = MaterialRequest.objects.create(owner=owner, is_allocated=allocated,
delivery_required_on=delivery_required_on, linked=linked,
raised_by=raised_by, raised_for=raised_for, is_rejected=rejected,
reason=reason, remarks=remarks, created=created)
It is running fine when the data is something like the following:
But as soon as is_allocated reaches False it shows the following error:
django.core.exceptions.ValidationError: ['“TRUE” value must be either
True or False.']
I am unable to find something related to this

it seems like the is_allocated property of your model is a boolean. So you should assign a boolean value to it. But your column values in the data frame are strings TRUE and FALSE.
replacing this line
allocated = row['is_allocated']
with
allocated = (row['is_allocated'] == 'TRUE')
might help.
if you have None other than True and False values, consider that too.

It is because you are trying to store a string in a boolean field. One solution is to change your string type to boolean. Maybe having a function like following in your code solves your problem:
def to_boolean(raw_value: str) -> bool:
if not isinstance(raw_value, str):
raw_value = str(raw_value)
raw_value = raw_value.strip()
return {'true': True, 'false': False}.get(raw_value.lower(), False)
and then use it in your loop like (where ever you think your field type is boolean):
for index, row in df.iterrows():
allocated = to_boolean(row['is_allocated'])
delivery_required_on = row['delivery_required_on']
linked = row['linked']
raised_by = row['raised_by']
raised_for = Company.objects.get(pk=row['raised_for']) ### double check
rejected = to_boolean(row['is_rejected'])
reason = row['reason']
remarks = row['remarks']
created = row['created_at']
owner = User.objects.get(pk=row['New owner'])
j = literal_eval(row['flows'])
flows = []
mobj = MaterialRequest.objects.create(owner=owner, is_allocated=allocated,
delivery_required_on=delivery_required_on, linked=linked,
raised_by=raised_by, raised_for=raised_for, is_rejected=rejected,
reason=reason, remarks=remarks, created=created)

How can I Update Boolean Value Using SQLAlchemy and Python?

I am trying to update a boolean value within a database using Python and SQLAlchemy. Here is my code:
def update_record_to_hide_or_show(e2e_id, hide_error, event_time, study_id):
connection_string = _get_connection_string()
db = create_engine(connection_string)
roi_e2e_events = define_roi_e2e_events_table()
with db.connect() as conn:
if hide_error == "True":
update = roi_e2e_events.update().values(hide_error=True).where(roi_e2e_events.c.e2e_id == e2e_id)\
.where(roi_e2e_events.c.event_time == event_time)\
.where(roi_e2e_events.c.study_id == study_id)
print(update)
result = conn.execute(update)
else:
update = roi_e2e_events.update().values(hide_error=False).where(roi_e2e_events.c.e2e_id == e2e_id) \
.where(roi_e2e_events.c.event_time == event_time). \
where(roi_e2e_events.c.study_id == study_id)
result = conn.execute(update)
return result
I am able to enter the first portion of the conditional without problem, there is not an execution error displayed as I attempt to submit the query to the database, I've created the Metadata in a separate function and the update query looks like this:
UPDATE roi_e2e_events SET hide_error=:hide_error WHERE roi_e2e_events.e2e_id = :e2e_id_1 AND roi_e2e_events.event_time = :event_time_1 AND roi_e2e_events.study_id = :study_id_1
I don't see that the boolean value is changed to "True" after running this, what am I doing wrong here?

Without an example of the table + schema or what the arguments for this function look like (particularly hide_error), it's a little hard to be sure, but it looks like therecould be an issue on the hide_error == "True" line, since it's checking if that hide_error is the string "True", rather than the boolean True.
If it is in fact a boolean, we can actually get around the whole issue of checking what value it is by just using the not operator. Something like this:
def update_record_to_hide_or_show(e2e_id, hide_error, event_time, study_id):
connection_string = _get_connection_string()
db = create_engine(connection_string)
roi_e2e_events = define_roi_e2e_events_table()
with db.connect() as conn:
# no if statement here
update_query = roi_e2e_events.update().values(
hide_error=not hide_error # but notice the `not` here
).where(
roi_e2e_events.c.e2e_id == e2e_id
).where(
roi_e2e_events.c.event_time == event_time
).where(
roi_e2e_events.c.study_id == study_id
)
result = conn.execute(update)
return result
Also, if hide_error is being retrieved from the database, you can bundle it all up into a single UPDATE query like this
from sqlalchemy import not_ # this is used to invert the value of a boolean column in a query - on the database, rather than a value we have stored locally in a variable
def update_record_to_hide_or_show(e2e_id, event_time, study_id): # notice that `hide_error` is gone
connection_string = _get_connection_string()
db = create_engine(connection_string)
roi_e2e_events = define_roi_e2e_events_table()
with db.connect() as conn:
# still no if statement here
update_query = roi_e2e_events.update().values(
hide_error=not_(roi_e2e_events.c.hide_error) # here we use column, rather than its value, much like the `where` clauses
).where(
roi_e2e_events.c.e2e_id == e2e_id
).where(
roi_e2e_events.c.event_time == event_time
).where(
roi_e2e_events.c.study_id == study_id
)
result = conn.execute(update)
return result
where the update_query should look something like this:
UPDATE roi_e2e_events
SET hide_error=NOT roi_e2e_events.hide_error
WHERE roi_e2e_events.e2e_id = :e2e_id_1
AND roi_e2e_events.event_time = :event_time_1
AND roi_e2e_events.study_id = :study_id_1

Troubleshooting uncooperative For Loop of SQLalchemy results

Looking for a second set of eyes here. I cannot figure out why the following loop will not continue past the first iteration.
The 'servicestocheck' sqlalchemy query returns 45 rows in my test, but I cannot iterate through the results like I'm expecting... and no errors are returned. All of the functionality works on the first iteration.
Anyone have any ideas?
def serviceAssociation(current_contact_id,perm_contact_id):
servicestocheck = oracleDB.query(PORTAL_CONTACT).filter(
PORTAL_CONTACT.contact_id == current_contact_id
).order_by(PORTAL_CONTACT.serviceID).count()
print(servicestocheck) # returns 45 items
servicestocheck = oracleDB.query(PORTAL_CONTACT).filter(
PORTAL_CONTACT.contact_id = current_contact_id
).order_by(PORTAL_CONTACT.serviceID).all()
for svc in servicestocheck:
#
# Check to see if already exists
#
check_existing_association = mysqlDB.query(
CONTACTTOSERVICE).filter(CONTACTTOSERVICE.contact_id ==
perm_contact_id,CONTACTTOSERVICE.serviceID ==
svc.serviceID).first()
#
# If no existing association
#
if check_existing_association is None:
print ("Prepare Association")
assoc_contact_id = perm_contact_id
assoc_serviceID = svc.serviceID
assoc_role_billing = False
assoc_role_technical = False
assoc_role_commercial = False
if svc.contact_type == 'Billing':
assoc_role_billing = True
if svc.contact_type == 'Technical':
assoc_role_technical = True
if svc.contact_type == 'Commercial':
assoc_role_commercial = True
try:
newAssociation = CONTACTTOSERVICE(
assoc_contact_id, assoc_serviceID,
assoc_role_billing,assoc_role_technical,
assoc_role_commercial)
mysqlDB.add(newAssociation)
mysqlDB.commit()
mysqlDB.flush()
except Exception as e:
print(e)
This function is called from a script, and it is called from within another loop. I can't find any issues with nested loops.

Ended up being an issue with SQLAlchemy ORM (see SqlAlchemy not returning all rows when querying table object, but returns all rows when I query table object column)
I think the issue is due to one of my tables above does not have a primary key in real life, and adding a fake one did not help. (I don't have access to the DB to add a key)
Rather than fight it further... I went ahead and wrote raw SQL to move my project along.
This did the trick:
query = 'SELECT * FROM PORTAL_CONTACT WHERE contact_id = ' + str(current_contact_id) + 'ORDER BY contact_id ASC'
servicestocheck = oracleDB.execute(query)

Python Flask and SQLAlchemy, selecting all data from a column

I am attempting to query all rows for a column called show_id. I would then like to compare each potential item to be added to the DB with the results. Now the simplest way I can think of doing that is by checking if each show is in the results. If so pass etc. However the results from the below snippet are returned as objects. So this check fails.
Is there a better way to create the query to achieve this?
shows_inDB = Show.query.filter(Show.show_id).all()
print(shows_inDB)
Results:
<app.models.user.Show object at 0x10c2c5fd0>,
<app.models.user.Show object at 0x10c2da080>,
<app.models.user.Show object at 0x10c2da0f0>
Code for the entire function:
def save_changes_show(show_details):
"""
Save the changes to the database
"""
try:
shows_inDB = Show.query.filter(Show.show_id).all()
print(shows_inDB)
for show in show_details:
#Check the show isnt already in the DB
if show['id'] in shows_inDB:
print(str(show['id']) + ' Already Present')
else:
#Add show to DB
tv_show = Show(
show_id = show['id'],
seriesName = str(show['seriesName']).encode(),
aliases = str(show['aliases']).encode(),
banner = str(show['banner']).encode(),
seriesId = str(show['seriesId']).encode(),
status = str(show['status']).encode(),
firstAired = str(show['firstAired']).encode(),
network = str(show['network']).encode(),
networkId = str(show['networkId']).encode(),
runtime = str(show['runtime']).encode(),
genre = str(show['genre']).encode(),
overview = str(show['overview']).encode(),
lastUpdated = str(show['lastUpdated']).encode(),
airsDayOfWeek = str(show['airsDayOfWeek']).encode(),
airsTime = str(show['airsTime']).encode(),
rating = str(show['rating']).encode(),
imdbId = str(show['imdbId']).encode(),
zap2itId = str(show['zap2itId']).encode(),
added = str(show['added']).encode(),
addedBy = str(show['addedBy']).encode(),
siteRating = str(show['siteRating']).encode(),
siteRatingCount = str(show['siteRatingCount']).encode(),
slug = str(show['slug']).encode()
)
db.session.add(tv_show)
db.session.commit()
except Exception:
print(traceback.print_exc())

I have decided to use the method above and extract the data I wanted into a list, comparing each show to the list.
show_compare = []
shows_inDB = Show.query.filter().all()
for item in shows_inDB:
show_compare.append(item.show_id)
for show in show_details:
#Check the show isnt already in the DB
if show['id'] in show_compare:
print(str(show['id']) + ' Already Present')
else:
#Add show to DB

For querying a specific column value, have a look at this question: Flask SQLAlchemy query, specify column names. This is the example code given in the top answer there:
result = SomeModel.query.with_entities(SomeModel.col1, SomeModel.col2)
The crux of your problem is that you want to create a new Show instance if that show doesn't already exist in the database.
Querying the database for all shows and looping through the result for each potential new show might become very inefficient if you end up with a lot of shows in the database, and finding an object by identity is what an RDBMS does best!
This function will check to see if an object exists, and create it if not. Inspired by this answer:
def add_if_not_exists(model, **kwargs):
if not model.query.filter_by(**kwargs).first():
instance = model(**kwargs)
db.session.add(instance)
So your example would look like:
def add_if_not_exists(model, **kwargs):
if not model.query.filter_by(**kwargs).first():
instance = model(**kwargs)
db.session.add(instance)
for show in show_details:
add_if_not_exists(Show, id=show['id'])
If you really want to query all shows upfront, instead of putting all of the id's into a list, you could use a set instead of a list which will speed up your inclusion test.
E.g:
show_compare = {item.show_id for item in Show.query.all()}
for show in show_details:
# ... same as your code