I would like to write a program where I can call functions to run the code.
I wanted to let user to choice input parameters as follows:
q=input("Choose a query: ")
question=input("Would you like to run the code? ")
if question == 'y':
e_date=s_date
fun1(q, s_date, e_date) # to collect data
fun2(q, s_date) # to save data into file
Please see below an extract of the function that I would need to run. Fun1 collects data, in particular links and titles. In order to do this, I need a query, start date and end date (I add this information manually, as shown in the code above).
fun1 seems work fine.
The second function, fun2, should save data collected from fun1 into a file csv. However I am having some issues with the append_write, specifically:
UnboundLocalError: local variable 'append_write' referenced before assignment
I think the problem could be also due to links and titles (maybe also in q and start date) as I do not know how to call them into fun2 and use them through the code.
Could you please have a look and tell me how I could fix this issue? Thanks.
def fun1(q,s_date,e_date):
...
links = []
titles = []
while True:
soup = BeautifulSoup(driver.page_source,'lxml')
result_div = soup.find_all('div', attrs={'class': 'g'})
time.sleep(3)
for r in result_div:
try:
link = r.find('a', href=True)
time.sleep(2)
title = None
title = r.find('h3')
if isinstance(title,Tag):
title = title.get_text()
if link != '' and title != '':
links.append(link['href'])
titles.append(title)
except Exception as e:
continue
...
return links, titles
def fun2(q,s_date):
f = 'path'+q+'.csv'
if os.path.exists(f):
res_1 = input("Append data: ")
if res_1 == 'y':
append_write = 'a'
header_TF = False
if res_1 == 'n':
res_2 = input("Write a new file: ")
if res_2 == 'y':
append_write = 'w'
header_TF = True
else:
print("Wrong choice!")
else:
print("There was an error.\n")
df = pd.DataFrame({"Date": s_date, "Links" : links, "Titles" : titles})
df.to_csv(f, mode=append_write, header=header_TF, index=False)
The error message is related to the call fun2(q, s_date) # to save data into file and df.to_csv(f, mode=append_write, header=header_TF, index=False)
UnboundLocalError: local variable 'append_write' referenced before assignment
The error message is telling you that you're trying to use the variable append_write before you've assigned anything to it. You don't provide the stack trace that says where exactly this happens, but an obvious place it might occur is in fun2:
def fun2(q,s_date):
f = 'path'+q+'.csv'
if os.path.exists(f):
res_1 = input("Append data: ")
if res_1 == 'y':
append_write = 'a'
header_TF = False
if res_1 == 'n':
res_2 = input("Write a new file: ")
if res_2 == 'y':
append_write = 'w'
header_TF = True
else:
print("Wrong choice!")
else:
print("There was an error.\n")
df = pd.DataFrame({"Date": s_date, "Links" : links, "Titles" : titles})
df.to_csv(f, mode=append_write, header=header_TF, index=False)
So append_write will never be initialized if ANY of the following are true:
f does not exist
res_1 is not exactly either 'y' or 'n'
res_1 is 'n' but res_2 is not exactly 'y'
Maybe in those circumstances your code should return instead of trying to continue? Or maybe you should supply a default value for append_write? I'm not really sure what the function is supposed to be doing.
Related
I don't expect any coding answers, more just guidance. For my project I have to date-mine apple stock prices from a csv-file and implement it in my code. I provided a sample output below.
https://imgur.com/rPOPN1I
Right now, I am not getting any error messages but my code is not posting the columns requested from the csv.file. Are my definitions at fault or am I missing something else?
# Project No.: 5
# Author: burntchickennuget
# Description: making definitions, reading from a .csv file, validating input, data-mining,
f_list = list()
file_object = ()
my_tot = dict()
new_list = list()
def get_input_descriptor():
while True:
filename = input("Enter a file name: ")
if filename == 'table.csv':
with open(filename, "r") as infile:
infile.readlines()[1:]
# for line in lines:
# print(line.rstrip())
break
else:
print("Bad file name, try again")
return filename
def get_data_list(file_object, column_number):
dict = {}
for val in file_object:
date = val.split(",")[0]
data = float(val.split(",")[column_number])
dict[date] = data
return dict.items()
def average_data(new_list):
for date, price in new_list:
my_tot[date] = my_tot.get(date, 0) + float(price)
my_times[date] = my_times.get(date, 0) + 1
for key in my_tot:
f_list.append((float(my_tot[key] / my_times[key]), key))
def main():
get_input_descriptor()
column_number = int(input("Which column: "))
date_list = get_data_list(file_object, column_number)
final_list = average_data(date_list)
x = sorted(f_list)
print('Lowest 6:')
for tup in x[:6]:
print
tup[0], tup[1]
print('Highest 6:')
x = sorted(f_list, reverse=True)
for tup in x[:6]:
print
tup[0], tup[1]
while 1:
flag = input("do you want to continue? ")
if flag == '' or not flag[0].lower() in ['y', 'n']:
print("Please answer with a yes or no")
else:
break
if flag[0].lower() == 'y':
column = input("Which column: ")
print(column)
if flag[0].lower() == 'n':
print("Bye!")
if __name__ == "__main__":
main()
What can I try next?
Take a look around your get_input_descriptor method.
What are you returning from the method?
Where is the returned information being stored in the main method (is it being stored at all?)
What are you doing with the lines that you read from the file?
What is the file_object you are passing into get_data_list
My main advice would be to add print everywhere for debugging. See what it stored in your variables at different points in the program and see where a variable doesn't contain what you think it should
I have a text file set out in this layout:
Greg,Computer Science,Hard,5
Alex,Computer Science,Medium,2
Fiona,Maths,Easy,0
Cassie,Maths,Medium,5
Alex,Maths,Medium,1
In my program I want the user to be able to choose a certain name and see their results. My code for this looks like this:
name = input("Enter name: ")
for each in file:
each = each.split(",")
realName = each[0]
subject = each[1]
difficulty = each[2]
score = each[3]
if name == realName:
print(subject, difficulty, score)
break
else:
print()
print("Invalid name.")
name = input("Re-enter your name: ")
A few things are wrong with it though and I can't figure out what to do:
If the user entered "Alex", only one of his results will be displayed.
If a wrong name is inputted once, every other name inputted will return as "Invalid".
If the correct name is inputted and the results are displayed, the program will continue to ask for a name.
Does anybody have any solutions to these problems?
If you're going to query your file repeatedly, I'd recommend pre-loading your data once into a dictionary, and printing data as and when needed. Something like this:
data = {}
with open('file.txt', 'r') as file:
for line in file:
realName, subject, difficulty, score = each.split(',')
data.setdefault(realName, []).append((subject, difficulty, score))
while True:
name = input('>>> ')
data.get(name, 'Invalid Name')
This solves problems one and two. If you just want to break after the first valid name is input, you can query the return value of dict.get:
while True:
name = input('>>> ')
result = data.get(name)
if result:
print(result)
break
print('Invalid name')
This solves problem three.
You're better off using the csv module since your file syntax is simple CSV.
Then you can loop through the rows (each row will be an array of values).
import csv
def parse_csv_file(csv_file, operation, value, index):
with open(csv_file, newline='') as file:
reader = csv.reader(file, delimiter=',',
quotechar='|')
return operation(reader,
value, index)
def find_first_row(csv_reader, value, index):
for row in csv_reader:
if row[index] == value:
return row
return None
def main():
query = input('Enter a name: ')
result = parse_csv_file('file.csv',
find_first_row,
query, 0)
if result:
print(result)
else:
print('Nothing found!')
main()
I'm creating a work log in python where a user can enter a task or can lookup a task by date. My initial prompt asks user to either enter a task or lookup by date. If the user starts by looking up by date--the program works correctly and displays all dates. If a user starts by adding a task, and then looking up tasks by date, the program displays an 'object does not support indexing error'. I think for some reason, the list is getting emptied, but i can't for the life of me understand where or when this might be happening. Here is main worklog file:
import csv
import re
import datetime
from task import Task
from task_list import TaskList
class Worklog():
def __init__(self):
self.filename = "work_log.csv"
self.tasklist = TaskList()
self.tasklist.read_task_from_file(self.filename)
def search_by_date(self):
for d, i in enumerate(self.tasklist.dates()):
print(d+1, ':', i)
# while True:
# datereq = input("Select Number To See Tasks For A Date").strip().lower()
# try:
# datereq = int(datereq)
# return datereq
# except ValueError:
# print("Invalid Entry. Please try again")
# continue
def search_by_time(self):
pass
def exact_search(self):
pass
def pattern_search(self):
pass
def add_task(self):
task = Task()
task.input_task()
task.input_minutes()
task.input_notes()
task.date = datetime.date.today()
self.tasklist.app_task(task)
self.tasklist.save_task_to_file(self.filename,task)
def lookup_task(self):
if len(self.tasklist.task_list) == 0:
print("Your task log is empty.\n")
input("Hit Enter to add a new task.")
else:
while True:
lookup = input("Lookup by Date(D), Time(T), Exact Search(E) or Pattern(P): ")
lookup.lower()
if lookup == 'd':
self.search_by_date()
break
elif lookup == 't':
self.search_by_time()
break
elif lookup == 'e':
self.exact_search()
break
elif lookup == 'p':
self.pattern_search()
break
else:
print("Sorry invalid option. Please try again")
def start_message(self):
while True:
q = input("Add New Task(1) or Lookup Task(2) or Quit(3): ".strip().lower())
if q == "1":
self.add_task()
elif q == "2":
self.lookup_task()
elif q == "3":
exit()
else:
print("Sorry that's an invalid entry. Please try again.")
continue
if __name__ == '__main__':
log = Worklog()
log.start_message()
The error is pointing to the dates function which is shown below along with a few of the other methods for my 'task-list' class. Is there an issue with the way that I am indexing this list? Or am i right in that the task_list list is getting reset somehow when the user enters the second step of the loop. Thanks:
def app_task(self, task):
self.task_list.append(task)
def save_task_to_file(self,filename,task):
with open(filename, 'a', newline="\n", encoding="utf-8") as csvfile:
# creating a csv writer object
csvwriter = csv.writer(csvfile, delimiter=",")
# writing the data rows
csvwriter.writerow([task.date, task.task, task.minutes, task.notes])
def read_task_from_file(self,filename):
with open(filename, 'r') as csvfile:
task_reader = csv.reader(csvfile, delimiter=',')
for row in task_reader:
task = Task()
self.task_list.append(row)
return self.task_list
def dates(self):
self.read_task_from_file("work_log.csv")
dates = []
for row in self.task_list:
date = row[0]
if row[0] not in dates:
dates.append(date)
return sorted(dates)
Note--here is an example of what the work_log.csv file looks like:
2017-03-23,gardening,30,not really
2017-03-23,bowling,30,none
2017-03-23,bowling,90,none
2017-03-23,bowling,93,none
2017-03-23,baseball,2,none
2017-03-23,bowling,20,none
2017-03-23,base,0,jj
2017-03-23,bowling,33,none
Added per Jake's recommendation:
def get_date(self):
for row in self.task_list:
d = row[0]
return d
def dates(self):
dates = []
for row in dates:
date = row.get_date()
if date not in dates:
dates.append(date)
return sorted(dates)
The issue appears to be where you call date = row[0]; this is because in the loop row will be a Task object as you are iterating over self.task_list. In this case you are trying to index into a Task object, which is not set up for indexing.
The simple solution for this would be to replace row[0] with row.date; which will directly access the date field of the row object, without needing to bother about indexing at all.
I have been testing my code for the past few hours and I am stumped. This program takes a text file of names, turns the names into a list, prints the names, sorts the names, prints the sorted names, then allows you to search through the list. Everything seems to be working fine, but the one issue I have is exiting the while loop. If y or Y is selected you can search again, but that also happens if anything else is selected. I added a print statement outside the loop so if anything other than y is selected then the program should end with that last printed string, but it doesn't seem to be working. Does anyone have any ideas about why it isn't working and what I could change to get it to work?
Thank you for your time.
#define the main function
def main():
#create a variable to control the loop
keep_going = 'y'
#setup loop to search for name
while keep_going == 'y' or keep_going == 'Y':
#call input name function
names = input_name()
#call print name function
print_name(names)
#sort the printed list
names.sort()
#call the print name function
print_name(names)
#call the output name function
output_name(names)
#call the search name function
search_name(names)
#add user input for another search
search_again = input('Would you like to make another search?(y for yes): ')
#print if anything other than y or Y is selected
print()
print('Goodbye!')
#define the input function
def input_name():
#open the names.txt file
infile = open('names.txt', 'r')
#read contents into a list
names = infile.readlines()
#close the file
infile.close()
#strip the \n from each element
index = 0
while index < len(names):
names[index] = names[index].rstrip('\n')
index += 1
#return the list back to main function
return names
#define the print name function
def print_name(names):
#print the contents of the list
for name in names:
print(name)
#define the output name function
def output_name(names):
#open file for writing
outfile = open('sorted_names.txt', 'w')
#write the list to the file
for item in names:
outfile.write(item + '\n')
#close the file
outfile.close()
#return to main function
return
#define the search name function
def search_name(names):
#add a user input to search the file
search = input('Enter a name: ')
#determine whether the name is in the list
if search in names:
#get the names index
name_index = names.index(search)
#print the name was found and give the items index
print(search, "was found in list. This item's index is", name_index)
else:
#print the item was not found
print(search, 'was not found in the list.')
main()
#create a variable to control the loop
keep_going = 'y'
#setup loop to search for name
while keep_going == 'y' or keep_going == 'Y':
#add user input for another search
search_again = input('Would you like to make another search?(y for yes): ')
You never set keep_going to something else. Instead, you ask the user to enter y to continue but store it in search_again (which is then thrown away). You will want to change that to store the value in keep_going.
You are testing keep_going, but setting search_again
Replace line 29 which states:
search_again = input('Would you like to make another search?(y for yes): ')
with the following:
keep_going = input('Would you like to make another search?(y for yes): ')
Because when you enter y or Y then you are defining the variable search_again to that letter not the variable keep_going which is needed to change for the loop to stop.
I am attempting to query the twitter search engine (search.twitter.com), convert the results into json, and then prepare the results as a csv for a research project. I am a python novice, but I have managed to code 2/3 of the program myself. However, I have a difficult time converting my json file into the csv format. I have tried various suggested techniques without success. What am I doing wrong here?
Here is what I have so far:
import twitter, os, json, csv
qname = raw_input("Please enter the term(s) you wish to search for: ")
date = int(raw_input("Please enter today's date (no dashes or spaces): "))
nname = raw_input("Please enter a nickname for this query (no spaces): ")
q1 = raw_input("Would you like to set a custom directory? Enter Yes or No: ")
if q1 == 'No' or 'no' or 'n' or 'N':
dirname = 'C:\Users\isaac\Desktop\TPOP'
elif q1 == 'Yes' or 'yes' or 'y' or 'Y':
dirname = raw_input("Please enter the directory path:")
ready = raw_input("Are you ready to begin? Enter Yes or No: ")
while ready == 'Yes' or 'yes' or 'y' or 'Y':
twitter_search = twitter.Twitter(domain = "search.Twitter.com")
search_results = []
for page in range (1,10):
search_results.append(twitter_search.search(q=qname, rpp=1, page=page))
ready1 = raw_input("Done! Are you ready to continue? Enter Yes or No: ")
if ready1 == 'Yes' or 'yes' or 'y' or 'Y':
break
ready3 = raw_input("Do you want to save output as a file? Enter Yes or No: ")
while ready3 == 'Yes' or 'yes' or 'y' or 'Y':
os.chdir(dirname)
filename = 'results.%s.%06d.json' %(nname,date)
t = open (filename, 'wb+')
s = json.dumps(search_results, sort_keys=True, indent=2)
print >> t,s
t.close()
ready4 = raw_input("Done! Are you ready to continue? Enter Yes or No: ")
if ready4 == 'Yes' or 'yes' or 'y' or 'Y':
break
ready5 = raw_input("Do you want to save output as a csv/excel file? Enter Yes or No: ")
while ready5 == 'Yes' or 'yes' or 'y' or 'Y':
filename2 = 'results.%s.%06d.csv' %(nname,date)
z = json.dumps(search_results, sort_keys=True, indent=2)
x=json.loads(z)
json_string = z
json_array = x
columns = set()
for entity in json_array:
if entity == "created_at" or "from_user" or "from_user_id" or "from_user_name" or "geo" or "id" or "id_str" or "iso_language_code" or "text":
columns.update(set(entity))
writer = csv.writer(open(filename2, 'wb+'))
writer.writerow(list(columns))
for entity in json_array:
row = []
for c in columns:
if c in entity: row.append(str(entity[c]))
else: row.append('')
You have several different problems going on.
First off, the syntax of
x == 'a' or 'b' or 'c'
probably doesn't do what you think it does. You should use
x in ('a', 'b', 'c')
instead.
Second, your ready5 variable never changes and won't work right in the loop. Try
while True:
ready5 = raw_input("Do you want to save output as a csv/excel file? Enter Yes or No: ")
if ready5 not in (...):
break
And finally, there's something wrong with your dumping/loading code. What you're getting from twitter should be a JSON string. There's some code you've left out from your question, so I can't tell for sure, but I don't think you want to be using json.dumps at all. You're reading from JSON (using json.loads) and writing to CSV (using csv.writer.writerow).
A different approach would be to have tablib do the actual conversion for you:
import tablib
data = tablib.Dataset()
data.json = search_results
filename = 'results.%s.%06d.csv' %(nname,date)
csv_file = open(filename, 'wb')
csv_file.write(data.csv)
After some searching around, I found the answer here: http://michelleminkoff.com/2011/02/01/making-the-structured-usable-transform-json-into-a-csv/
The code should look something like this:(if you are search the twitter python api)
filename2 = '/path/to/my/file.csv'
writer = csv.writer(open(filename2, 'w'))
z = json.dumps(search_results, sort_keys=True, indent=2)
parsed_json=json.loads(z)
#X needs to be the number of page you pulled less one. So 5 pages would be 4.
while n<X:
for tweet in parsed_json[n]['results']:
row = []
row.append(str(tweet['from_user'].encode('utf-8')))
row.append(str(tweet['created_at'].encode('utf-8')))
row.append(str(tweet['text'].encode('utf-8')))
writer.writerow(row)
n = n +1
Thanks Everyone for the help!