I am attempting to query the twitter search engine (search.twitter.com), convert the results into json, and then prepare the results as a csv for a research project. I am a python novice, but I have managed to code 2/3 of the program myself. However, I have a difficult time converting my json file into the csv format. I have tried various suggested techniques without success. What am I doing wrong here?
Here is what I have so far:
import twitter, os, json, csv
qname = raw_input("Please enter the term(s) you wish to search for: ")
date = int(raw_input("Please enter today's date (no dashes or spaces): "))
nname = raw_input("Please enter a nickname for this query (no spaces): ")
q1 = raw_input("Would you like to set a custom directory? Enter Yes or No: ")
if q1 == 'No' or 'no' or 'n' or 'N':
dirname = 'C:\Users\isaac\Desktop\TPOP'
elif q1 == 'Yes' or 'yes' or 'y' or 'Y':
dirname = raw_input("Please enter the directory path:")
ready = raw_input("Are you ready to begin? Enter Yes or No: ")
while ready == 'Yes' or 'yes' or 'y' or 'Y':
twitter_search = twitter.Twitter(domain = "search.Twitter.com")
search_results = []
for page in range (1,10):
search_results.append(twitter_search.search(q=qname, rpp=1, page=page))
ready1 = raw_input("Done! Are you ready to continue? Enter Yes or No: ")
if ready1 == 'Yes' or 'yes' or 'y' or 'Y':
break
ready3 = raw_input("Do you want to save output as a file? Enter Yes or No: ")
while ready3 == 'Yes' or 'yes' or 'y' or 'Y':
os.chdir(dirname)
filename = 'results.%s.%06d.json' %(nname,date)
t = open (filename, 'wb+')
s = json.dumps(search_results, sort_keys=True, indent=2)
print >> t,s
t.close()
ready4 = raw_input("Done! Are you ready to continue? Enter Yes or No: ")
if ready4 == 'Yes' or 'yes' or 'y' or 'Y':
break
ready5 = raw_input("Do you want to save output as a csv/excel file? Enter Yes or No: ")
while ready5 == 'Yes' or 'yes' or 'y' or 'Y':
filename2 = 'results.%s.%06d.csv' %(nname,date)
z = json.dumps(search_results, sort_keys=True, indent=2)
x=json.loads(z)
json_string = z
json_array = x
columns = set()
for entity in json_array:
if entity == "created_at" or "from_user" or "from_user_id" or "from_user_name" or "geo" or "id" or "id_str" or "iso_language_code" or "text":
columns.update(set(entity))
writer = csv.writer(open(filename2, 'wb+'))
writer.writerow(list(columns))
for entity in json_array:
row = []
for c in columns:
if c in entity: row.append(str(entity[c]))
else: row.append('')
You have several different problems going on.
First off, the syntax of
x == 'a' or 'b' or 'c'
probably doesn't do what you think it does. You should use
x in ('a', 'b', 'c')
instead.
Second, your ready5 variable never changes and won't work right in the loop. Try
while True:
ready5 = raw_input("Do you want to save output as a csv/excel file? Enter Yes or No: ")
if ready5 not in (...):
break
And finally, there's something wrong with your dumping/loading code. What you're getting from twitter should be a JSON string. There's some code you've left out from your question, so I can't tell for sure, but I don't think you want to be using json.dumps at all. You're reading from JSON (using json.loads) and writing to CSV (using csv.writer.writerow).
A different approach would be to have tablib do the actual conversion for you:
import tablib
data = tablib.Dataset()
data.json = search_results
filename = 'results.%s.%06d.csv' %(nname,date)
csv_file = open(filename, 'wb')
csv_file.write(data.csv)
After some searching around, I found the answer here: http://michelleminkoff.com/2011/02/01/making-the-structured-usable-transform-json-into-a-csv/
The code should look something like this:(if you are search the twitter python api)
filename2 = '/path/to/my/file.csv'
writer = csv.writer(open(filename2, 'w'))
z = json.dumps(search_results, sort_keys=True, indent=2)
parsed_json=json.loads(z)
#X needs to be the number of page you pulled less one. So 5 pages would be 4.
while n<X:
for tweet in parsed_json[n]['results']:
row = []
row.append(str(tweet['from_user'].encode('utf-8')))
row.append(str(tweet['created_at'].encode('utf-8')))
row.append(str(tweet['text'].encode('utf-8')))
writer.writerow(row)
n = n +1
Thanks Everyone for the help!
Related
I don't expect any coding answers, more just guidance. For my project I have to date-mine apple stock prices from a csv-file and implement it in my code. I provided a sample output below.
https://imgur.com/rPOPN1I
Right now, I am not getting any error messages but my code is not posting the columns requested from the csv.file. Are my definitions at fault or am I missing something else?
# Project No.: 5
# Author: burntchickennuget
# Description: making definitions, reading from a .csv file, validating input, data-mining,
f_list = list()
file_object = ()
my_tot = dict()
new_list = list()
def get_input_descriptor():
while True:
filename = input("Enter a file name: ")
if filename == 'table.csv':
with open(filename, "r") as infile:
infile.readlines()[1:]
# for line in lines:
# print(line.rstrip())
break
else:
print("Bad file name, try again")
return filename
def get_data_list(file_object, column_number):
dict = {}
for val in file_object:
date = val.split(",")[0]
data = float(val.split(",")[column_number])
dict[date] = data
return dict.items()
def average_data(new_list):
for date, price in new_list:
my_tot[date] = my_tot.get(date, 0) + float(price)
my_times[date] = my_times.get(date, 0) + 1
for key in my_tot:
f_list.append((float(my_tot[key] / my_times[key]), key))
def main():
get_input_descriptor()
column_number = int(input("Which column: "))
date_list = get_data_list(file_object, column_number)
final_list = average_data(date_list)
x = sorted(f_list)
print('Lowest 6:')
for tup in x[:6]:
print
tup[0], tup[1]
print('Highest 6:')
x = sorted(f_list, reverse=True)
for tup in x[:6]:
print
tup[0], tup[1]
while 1:
flag = input("do you want to continue? ")
if flag == '' or not flag[0].lower() in ['y', 'n']:
print("Please answer with a yes or no")
else:
break
if flag[0].lower() == 'y':
column = input("Which column: ")
print(column)
if flag[0].lower() == 'n':
print("Bye!")
if __name__ == "__main__":
main()
What can I try next?
Take a look around your get_input_descriptor method.
What are you returning from the method?
Where is the returned information being stored in the main method (is it being stored at all?)
What are you doing with the lines that you read from the file?
What is the file_object you are passing into get_data_list
My main advice would be to add print everywhere for debugging. See what it stored in your variables at different points in the program and see where a variable doesn't contain what you think it should
I am trying to make a contacts python project but I don't know how to fin da single user. I have tried to find from the file but I am not able to, kindly give the code for that part. If you find any other mistakes kindly resolve that too
This is the code.
# -*- coding: utf-8 -*-
"""
Created on Wed Feb 19 17:56:45 2020
#author: Teerth Jain
"""
import time
from transpositionDecrypt import decryptMessage as dm
from transpositionEncrypt import encryptMessage as em
key = 8
users = {}
users1 = {}
print("Welcome to Teerth's Contacts Saver and Reader...")
time.sleep(0.5)
r_or_w = input("Do you want or (r)ead or (a)dd or (d)el contacts: ")
if r_or_w == 'r':
what = input("Do you want to read a single contact(y, n): ")
if what == 'y':
#here is the part where you need to give me advice#
if what == 'n':
print("Displaying the whole list...")
q = open('contacts.txt', 'r')
for liness in q:
print(dm(key, liness.strip()))
if r_or_w == 'a':
while True:
addwho = input("Please enter the name: ")
number = input("Please enter the number: ")
file2 = open('contacts.txt', 'r')
y = dm(key, file2.read())
if addwho in y:
print("User in contact list, please try again")
addwho = input("Please enter the name: ")
number = input("Please enter the number: ")
users1.update(addwho = number)
file1 = open('contacts.txt', 'a')
q = f'{addwho} : {number}'
file1.write(em(key, q))
file1.write("\n")
file1.close()
print("Number is ciphered and added.")
again = input("Do you want to add another contact(y, n): ")
if again == 'n':
break
There are no such errors now but if you can modify the code to make it better, kindly do it
Thanks
This is my code for entering student details. Once the user has entered the details and inputs yes, the details are exported to StudentDetails.csv (Microsoft Excel) where it should go below the headers but ends up going somewhere else.
def EnterStudent():
uchoice_loop = False
ask_loop = False
while uchoice_loop == False:
surname = raw_input("What is the surname?")
forename = raw_input("What is the forname?")
date = raw_input("What is the date of birth? {Put it in the format D/M/Y}")
home_address = raw_input("What is the home address?")
home_phone = raw_input("What is the home phone?")
gender = raw_input("What is their gender?")
tutor_group = raw_input("What is their tutor group?")
email = (forename.lower() + surname.lower() + ("#school.com"))
print(surname+" "+forename+" "+date+" "+home_address+" "+home_phone+" "+gender+" "+tutor_group+" "+email)
ask = raw_input("Are these details correct?"+"\n"+"Press b to go back, or yes to add entered data on your student.").lower()
if ask == "yes":
f = open("StudentDetails.csv","rt")
lines = f.readlines()
f.close()
lines.append(surname+","+forename+","+date+","+home_address+","+home_phone+","+gender+","+tutor_group+","+email+"\n")
f = open("StudentDetails.csv", "w")
f.writelines(lines)
f.close()
uchoice_loop = True
printMenu()
elif ask == "b":
uchoice_loop = False
else:
print("Plesase enter 'b' to go back or 'yes' to continue")
This is my csv file.
enter image description here
There's a few things you can do to make this work. You dont need to open the StudentDetails.csv and read all of the lines. Instead you can make a lines string variable and append it the the StudentDetails.csv like in the example below
#f = open("StudentDetails.csv","rt")
#lines = f.readlines()
#f.close()
lines = surname+","+forename+","+date+","+home_address+","+home_phone+","+gender+","+tutor_group+","+email
# the "a" appends the lines variable to the csv file instead of writing over it like the "w" does
f = open("StudentDetails.csv", "a")
f.writelines(lines)
f.close()
uchoice_loop = True
Eric is right in that you best open the file in append-mode (see https://docs.python.org/3.6/library/functions.html#open) instead of cumbersomely reading and rewriting your file over and over again.
I want to add to this that you probably will enjoy using the standard library's csv module as well (see https://docs.python.org/3.6/library/csv.html), especially if you want to use your output file in Excel afterwards.
Then, I'd also advise you to not use variables for while loop conditionals, but learning about the continue and break statements. If you want to break out of the outer loop in the example, research try, except and raise.
Finally, unless you really have to use Python 2.x, I recommend you to start using Python 3. The code below is written in Python 3 and will not work in Python 2.
#!/usr/bin/env python
# -*- codig: utf-8 -*-
import csv
def enterStudent():
b_or_yes = 'Press b to go back, or yes to save the entered data: '
while True:
surname = input('What is the surname? ')
forename = input('What is the first name? ')
date = input(
'What is the date of birth? {Put it in the format D/M/Y} ')
home_address = input('What is the home address? ')
home_phone = input('What is the home phone? ')
gender = input('What is the gender? ')
tutor_group = input('What is the tutor group? ')
email = forename.lower() + surname.lower() + '#school.com'
studentdata = (
surname,
forename,
date,
home_address,
home_phone,
gender,
tutor_group,
email)
print(studentdata)
while True:
reply = input('Are these details correct?\n' + b_or_yes).lower()
if reply == 'yes':
with open('studentdetails.csv', 'a', newline='') as csvfile:
studentwriter = csv.writer(csvfile, dialect='excel')
studentwriter.writerow(studentdata)
break
elif reply == 'b':
break
if __name__ == '__main__':
enterStudent()
Best of luck!
Here is my code -
#Using pickle
#using pickle with dictionaries
import pickle
checkDeets = True
passWrong = "You have entered incorrect details, please try again"
x = input("want to enter data? - ")
if x == "yes":
file = open("data.pickle" , "wb")
signUpU = input("enter user - ") #Later used as sign in details
signUpP = input("enter pass - ") # as above
ID_in = {signUpU : signUpP} #Is meant to store the two user details
pickle.dump(ID_in, file)
file.close()
y = input("want to log in? ")
if y == "yes":
file = open("data.pickle" , "rb")
ID_out = pickle.load(file)
while checkDeets == True:
signInU = input("enter username - ")
signInP = input("enter pass - ")
if signInU in ID_out:
if signInP == ID_out[signInU][0]:
print("Login accepted")
checkDeets = False
else:
print("1")
print(passWrong)
else:
print("2")
print(passWrong)
Here is my inputs -
want to enter data? - yes
enter user - user123
enter pass - pass123
want to log in? no
>>> x = open("data.pickle" , "rb")
>>> x
<_io.BufferedReader name='data.pickle'>
this last part is where i get confused, as it seems that my dictionary data is not being saved. And this is causing my to have other errors in my log in part of my code, where the user details are not recognized.
New to pickling, sorry if there are any obvious mistakes. Using python 3
open() returns a file object, your repl output is expected. If you want to see what the data inside of it contains pass it to pickle.load() like so:
want to enter data? - yes
enter user - foo
enter pass - bar
want to log in? no
>>> import pickle
>>> pickle.load(open("data.pickle" , "rb"))
{'foo': 'bar'}
>>>
And you can see your data is being saved and loaded without issue. The second part of your code doesn't work because of this:
if signInP == ID_out[signInU][0]:
ID_out[signInU] is a string, the password, so ID_out[signInU][0] is the first character of that password. If the password is "bar" this line compares "bar" (the string we're checking against the stored password) to "b" (the first letter of the stored password) and obviously these are not the same string. Just drop the [0] and this code should do want you're going for.
I wrote this sample program that is meant to open a text file (database1.txt) from the computer, and display the results that are currently in the file. Then prompt the use if their name is in the document, if it is it should print the contents of the text file then close, otherwise it should prompt the user to enter their first name, then the program writes the name into the same text document, then prints the contents of the text file again so that the user can see the new added data. I have typed the code, and somehow it keeps saying I have a syntax error. I checked a few times and I cannot fix the error. I was wondering if someone could take a look and if they might be able to explain the error to me. Thank you
#This program reads/writes information from/to the database1.txt file
def database_1_reader ():
print('Opening database1.txt')
f = open('database1.txt', 'r+')
data = f.read()
print data
print('Is your name in this document? ')
userInput = input('For Yes type yes or y. For No type no or n ').lower()
if userInput == "no" or userInput == "n"
newData = input('Please type only your First Name. ')
f.write(newData)
f = open ('database1.txt', 'r+')
newReadData = f.read()
print newReadData
f.close()
elif userInput == "yes" or userInput == "ye" or userInput == "y"
print data
f.close()
else:
print("You b00n!, You did not make a valid selection, try again ")
f.close()
input("Presss any key to exit the program")
database_1_reader()
print is a function in py3.x:
print newReadData
should be :
print (newReadData)
Demo:
>>> print "foo"
File "<ipython-input-1-45585431d0ef>", line 1
print "foo"
^
SyntaxError: invalid syntax
>>> print ("foo")
foo
statements like this:
elif userInput == "yes" or userInput == "ye" or userInput == "y"
can be reduced to :
elif userInput in "yes"