Python : optimization/multiple if then on less lanes? - python

I'm trying some optimisation on my code and I struggle with multiple if then statements. The point is to update SQL database with psycopg2 and I must set some variables with "None" to be "null" in SQL.
If I do if/elif/elif, the script will leave on the first valid condition but I want every condition to be checked, here is the script :
if variable_a == "":
variable_a = None
if variable_b == "":
variable_b = None
if variable_c == "":
variable_c = None
if variable_d == "":
variable_d = None
Is there a way to do it in one row such as assigning multiple variable values?
variable_a, variable_b, variable_c, variable_d = "", "", "", ""
Bonus question : the whole point of the script is to srap data from website, getting the desired data with ReGex then updating SQL databases. The full script is about 24.000.000 iterations with all the code inside a "for i in" loop, i'm looking for a way to optimize it because it takes actually 12 days running... Do you have any tips ? Libraries ?
Thanks in advance! :)

You can define a simplified function for that case:
var_a, var_b, var_c, var_d = "", "", "", ""
to_None = lambda v: None if v == "" else v
var_a, var_b, var_c, var_d = to_None(var_a), to_None(var_b), to_None(var_c), to_None(var_d)
print(var_a, var_b, var_c, var_d)
None None None None

You can do this like
variable_a = variable_a or None
for every veriable. Also if your variables are in an array, you can do this with list comprehension:
arr = [i or None for i in arr]

Related

Python check sqlite data exits?

I wrote few lines of code to check if data exist in database as below:
def check_if_data_exists(self):
with self.connection as mycur:
result = mycur.execute(f'select 1 from Summary1 where "Date Time" = "20220722.2008";')
for row in result:
if row == (1,):
print("yes its here")
When the data exists it successfully printed "yes its here". However if I wrote lines as this:
def check_if_data_exists(self):
with self.connection as mycur:
result = mycur.execute(f'select 1 from Summary1 where "Date Time" = "20220723.2008";')
for row in result:
if row == (1,):
print("yes its here")
else:
print("No its not here")
It fails to print "No its not here". Instead the program just run with nothing happens. I'm using Pycharm by the way.
Anyone knows what's going on? how can I get the false conditioned result pls?
Alternatively, any better code better way to achieve the same result would be appreciated.
Many thanks.
What is happening here is that in the event that no matching data be found, the result set is empty, and therefore the list which gets returned will be completely empty and also falsy. Try using this version:
def check_if_data_exists(self):
with self.connection as mycur:
result = mycur.execute("SELECT 1 FROM Summary1 WHERE \"Date Time\" = '20220723.2008'")
if result:
print("yes its here")
else:
print("No its not here")
Perhaps a more safe way to do this would be to use an exists query, and then just check the (single) boolean value returned:
def check_if_data_exists(self):
with self.connection as mycur:
mycur.execute("SELECT EXISTS (SELECT 1 FROM Summary1 WHERE \"Date Time\" = '20220723.2008')")
row = mycur.fetchone()
if row[0]:
print("yes its here")
else:
print("No its not here")
In the second version above, you would always get back a result set with a single record and boolean value.

Calling functions and parameters through the code

I would like to write a program where I can call functions to run the code.
I wanted to let user to choice input parameters as follows:
q=input("Choose a query: ")
question=input("Would you like to run the code? ")
if question == 'y':
e_date=s_date
fun1(q, s_date, e_date) # to collect data
fun2(q, s_date) # to save data into file
Please see below an extract of the function that I would need to run. Fun1 collects data, in particular links and titles. In order to do this, I need a query, start date and end date (I add this information manually, as shown in the code above).
fun1 seems work fine.
The second function, fun2, should save data collected from fun1 into a file csv. However I am having some issues with the append_write, specifically:
UnboundLocalError: local variable 'append_write' referenced before assignment
I think the problem could be also due to links and titles (maybe also in q and start date) as I do not know how to call them into fun2 and use them through the code.
Could you please have a look and tell me how I could fix this issue? Thanks.
def fun1(q,s_date,e_date):
...
links = []
titles = []
while True:
soup = BeautifulSoup(driver.page_source,'lxml')
result_div = soup.find_all('div', attrs={'class': 'g'})
time.sleep(3)
for r in result_div:
try:
link = r.find('a', href=True)
time.sleep(2)
title = None
title = r.find('h3')
if isinstance(title,Tag):
title = title.get_text()
if link != '' and title != '':
links.append(link['href'])
titles.append(title)
except Exception as e:
continue
...
return links, titles
def fun2(q,s_date):
f = 'path'+q+'.csv'
if os.path.exists(f):
res_1 = input("Append data: ")
if res_1 == 'y':
append_write = 'a'
header_TF = False
if res_1 == 'n':
res_2 = input("Write a new file: ")
if res_2 == 'y':
append_write = 'w'
header_TF = True
else:
print("Wrong choice!")
else:
print("There was an error.\n")
df = pd.DataFrame({"Date": s_date, "Links" : links, "Titles" : titles})
df.to_csv(f, mode=append_write, header=header_TF, index=False)
The error message is related to the call fun2(q, s_date) # to save data into file and df.to_csv(f, mode=append_write, header=header_TF, index=False)
UnboundLocalError: local variable 'append_write' referenced before assignment
The error message is telling you that you're trying to use the variable append_write before you've assigned anything to it. You don't provide the stack trace that says where exactly this happens, but an obvious place it might occur is in fun2:
def fun2(q,s_date):
f = 'path'+q+'.csv'
if os.path.exists(f):
res_1 = input("Append data: ")
if res_1 == 'y':
append_write = 'a'
header_TF = False
if res_1 == 'n':
res_2 = input("Write a new file: ")
if res_2 == 'y':
append_write = 'w'
header_TF = True
else:
print("Wrong choice!")
else:
print("There was an error.\n")
df = pd.DataFrame({"Date": s_date, "Links" : links, "Titles" : titles})
df.to_csv(f, mode=append_write, header=header_TF, index=False)
So append_write will never be initialized if ANY of the following are true:
f does not exist
res_1 is not exactly either 'y' or 'n'
res_1 is 'n' but res_2 is not exactly 'y'
Maybe in those circumstances your code should return instead of trying to continue? Or maybe you should supply a default value for append_write? I'm not really sure what the function is supposed to be doing.

Attempting to modify CSV value based on the input received

Below is an example of the code, essentially I am taking input received and attempting to modify an existing CSV value. The problem is the code runs and produces the "print" output as expected, but the values are not being modified to "True" or "False" based on the if and else statements:
def test():
sn_cmn = pd.read_csv('cmn_location.csv')
wan = re.findall(r'Tunnel12', str(router_output)) or re.findall(r'Tunnel13', str(router_output))
hostname = re.search(r'...-....-R.', str(router_output))
hostname_string = str(hostname)
site_code = hostname_string[-13:-10]
if wan == ['Tunnel12'] or wan == ['Tunnel13']:
answer = 'Yes'
if answer == 'Yes':
for site in sn_cmn.u_site_code:
sn_cmn.loc[sn_cmn['u_site_code'] == site_code, 'u_iwan_3'] = 'TRUE'
else:
for site in sn_cmn.u_site_code:
sn_cmn.loc[sn_cmn['u_site_code'] == site_code, 'u_iwan_3'] = 'FALSE'
sn_cmn.to_csv('new_cmn_location.csv')

Editing a list of dictionaries from a list created from a txt file

I am trying to exclude and remove some dictionaries from a list. I have searched for awhile through the site and haven't found anything specific to this. The list of dictionaries was created from a txt file located: http://s000.tinyupload.com/?file_id=48953557772434487729
I'm trying to sort out and exclude the things I don't need. I thought my syntax was right, but apparently not.
I only included necessary code to cut down on the clutter. I am having problems at the action_genre point and excluding and deleting the dictionaries there. When prompted enter "s" and then "a" to access those two menus.
def load_movies():
global movies_list
movies_list= []
file_ref = open("movies.txt", 'r')
line = file_ref.readline()
for line in file_ref:
line = line.strip()
current = {}
if line == '':
break
movie_data = line.split("\t")
current["title"] = movie_data[0]
current["year"] = movie_data[1]
current["length"] = movie_data[2]
current["rating"] = movie_data[3]
current["action"] = int(movie_data[4][0]) == 1
current["animation"] = int(movie_data[4][1]) == 1
current["comedy"] = int(movie_data[4][2]) == 1
current["drama"] = int(movie_data[4][3]) == 1
current["documentary"] = int(movie_data[4][4]) == 1
current["romance"] = int(movie_data[4][5]) == 1
movies_list.append(current)
del current
file_ref.close()
def menu():
movie_selector =("Movie Selector - Please enter an option below:\nL - List all movies\nY - List all movies by year\n"
"T - Search by title\nS - Search by genre, rating, and maximum length\nQ - Quit the program\nOption:")
movie_selector_input = input(movie_selector).upper()
if movie_selector_input == "L":
list_movies()
if movie_selector_input == "Y":
list_by_year()
if movie_selector_input == "T":
search_by_title()
if movie_selector_input == "S":
search()
if movie_selector_input == "Q":
print("Thanks for using my program! Goodbye.")
exit()
else:
print("Invalid input")
print("Please try again")
print()
return menu()
def search():
genre_input = input("Please make a selection from the following genres.\n(Action(A), Animation(N), Comedy(C), "
"Drama(D), Documentary(O), or Romance(R)):").lower()
if genre_input == 'a':
action_genre()
elif genre_input == 'n':
animation_genre()
elif genre_input == 'c':
comedy_genre()
elif genre_input == 'd:':
drama_genre()
elif genre_input == 'o':
documentary_genre()
elif genre_input == 'r':
romance_genre()
else:
print("Invalid genre")
print()
menu()
#this is where I can't get the syntax to work
def action_genre():
for current in movies_list:
if current["action"] == "False":
del current
break
for i in movies_list:#using this to test output
print(i)
load_movies()
menu()
I'm narrowing down the list by excluding things that don't fit the parameters. In the action_genre function, I'm trying to delete all the dictionaries that don't equal current["action"] == True. I've tried using "True" and "False" as strings, as well as the bools True and False for comparisons, and still an error. Unfortunately, I have to use the Boolean logic per my professors directions.
His e.g.:
Professor's example. Apparently since I'm new I can't embed images. :/
I'm in programming 101, so thank you for your patience as I learn this, and thank you in advance for the help.
Okay, so the issue runs a bit deeper than the if condition being incorrect. In get_action() you effictively do not modify the actual movies_list object, but rather the local variable current, as proven by this simple test:
def action_genre():
for current in movies_list:
print(current)
if not current["action"]:
del current
print(current)
The second print(current) will result in an UnboundLocalError saying that current does not exist anymore, while in movies_list the entry it just deleted continues to exist. But in general, using del in loops indeed causes problems because that's how iterables and del itself behave. I encourage you to read up more on this on other sources or SO if you wish, like here.
Using the answer from the provided link above, we can use list comprehension to filter the movies:
def action_genre():
filtered_movies_list = [movie for movie in movies_list if movie['action']]
print(filtered_movies_list)
This creates a new list (so it does not modify movies_list), which includes all dictionary entries where item['action'] == True.
I hope this helps.
You are trying to compare a string with a boolean. Look at the following:
a= 1==1
print a
True
print type(a)
<class 'bool'> # a is a boolean
b='True' #assign string 'True' to b
print type(b)
<class 'str'>
print a==b #compare boolean True to string 'True'
False
b = True # assign boolean True to b
print a==b #compare boolean True to boolean True
True
so you need if current["action"] == False instead of if current["action"] == "False"

python if condition not working properly as anticipated

In a onClick_button event I've a if condition to show a messagedialog if condition fails or else execute the rest of the statements.
Basically if condition is checking whether a textctrl has value or not. If there is value execute the else statement.
this works for the first time with out any value in tc(textctrls) with a msgdlg, but when click ok on the dlg and add some value in the tc, still the msgdlg pops out when it should execute the else.
Your help is very appreciated.
I've checked all the indentation.
def onClick_button_new(self, event):
self.list_box.Clear()
varstr = "csv"
if [(self.tc1.GetValue() == "") or (self.tc2.GetValue() == "")]:
dial = wx.MessageDialog(None, 'No file specified. Please specify relevant file', 'Error', wx.OK)
dial.ShowModal()
else:
file1 = subprocess.check_output("cut -d '.' -f2 <<< %s" %self.var1, shell = True, executable="bash")
file1type = file1.strip()
print file1type
file2 = subprocess.check_output("cut -d '.' -f2 <<< %s" %self.var2, shell = True, executable="bash")
file2type = file2.strip()
if varstr in (file1type, file2type):
print "yes"
else:
dial = wx.MessageDialog(None, ' Invalid file format. Please specify relevant file', 'Error', wx.OK)
dial.ShowModal()
depending on your input
[(self.tc1.GetValue() == "") or (self.tc2.GetValue() == "")]
is either [True] or [False]. in any case a non-empty list. this will be interpreted as True;
if [False]:
do_something()
in this example do_something() will always be executed.
to fix this you need to remove the brackets []:
if (self.tc1.GetValue() == "") or (self.tc2.GetValue() == ""):
...
You have your boolean logic mixed up and created a list object (which is always non-empty and always true). You want to use and and not use a list:
if self.tc1.GetValue() == "" and self.tc2.GetValue() == "":
You should never use a [...] list for a boolean test, because that just creates a list object that is not empty, and thus always considered true in a boolean context, regardless of the contained comparison results:
>>> [0 == 1 or 0 == 1]
[False]
>>> bool([0 == 1 or 0 == 1])
True
Don't need to compare your method result to empty string :
if not self.tc1.GetValue() and not self.tc2.GetValue():
See that post : Most elegant way to check if the string is empty in Python?

Categories

Resources