Develop Reference

Call a range of dates from an API using Python

Currently writing a program using an API from MarketStack.com. This is for school, so I am still learning.
I am writing a stock prediction program using Python on PyCharm and I have written the connection between the program and the API without issues. So, I can certainly get the High, Name, Symbols, etc. What I am trying to do now is call a range of dates. The API says I can call up to 30 years of historical data, so I want to call all 30 years for a date that is entered by the user. Then the program will average the high on that date in order to give a trend prediction.
So, the problem I am having is calling more than one date. As I said, I want to call all 30 dates, and then I will do the math, etc.
Can someone help me call a range of dates? I tried installing Pandas and that wasn't being accepted by PyCharm for some reason.. Any help is greatly appreciated.
import tkinter as tk
import requests
# callouts for window size
HEIGHT = 650
WIDTH = 600
# function for response
def format_response(selected_stock):
try:
name1 = selected_stock['data']['name']
symbol1 = selected_stock['data']['symbol']
high1 = selected_stock['data']['eod'][1]['high']
final_str = 'Name: %s \nSymbol: %s \nEnd of Day ($ USD): %s' % (name1, symbol1, high1)
except:
final_str = 'There was a problem retrieving that information'
return final_str
# function linking to API
def stock_data(entry):
params = {'access_key': 'xxx'}
response = requests.get('http://api.marketstack.com/v1/tickers/' + entry + '/' + 'eod', params=params)
selected_stock = response.json()
label2['text'] = format_response(selected_stock)
# function for response
def format_response2(stock_hist_data):
try:
name = stock_hist_data['data']['name']
symbol = stock_hist_data['data']['symbol']
high = stock_hist_data['data']['eod'][1]['high']
name3 = stock_hist_data['data']['name']
symbol3 = stock_hist_data['data']['symbol']
high3 = stock_hist_data['data']['eod'][1]['high']
final_str2 = 'Name: %s \nSymbol: %s \nEnd of Day ($ USD): %s' % (name, symbol, high)
final_str3 = '\nName: %s \nSymbol: %s \nEnd of Day ($ USD): %s' % (name3, symbol3, high3)
except:
final_str2 = 'There was a problem retrieving that information'
final_str3 = 'There was a problem retrieving that information'
return final_str2 + final_str3
# function for response in lower window
def stock_hist_data(entry2):
params2 = {'access_key': 'xxx'}
response2 = requests.get('http://api.marketstack.com/v1/tickers/' + entry2 + '/' + 'eod', params=params2)
hist_data = response2.json()
label4['text'] = format_response2(hist_data)

Updating access token (global variable) inside a function/loop problem

I'm extracting data using Spotify API wrapper. The access token (which is global variable) is valid for only 1 hour so I need to update it during the for loop in some defined function. I tried to update it using try/except, but I got the following error:
UnboundLocalError: local variable 'spotify' referenced before assignment.
Here is the relevant code:
token = credentials.get_access_token()
spotify = spotipy.Spotify(auth = token)
...
def main():
...
df_af = generate_audio_features_df(track_ids)
...
def generate_audio_features_df(track_ids):
col_list = ['id', 'danceability']
result = []
count = 0
for j in track_ids:
try:
r = spotify.audio_features(j)[0]
features_list = [r['id'], r['danceability']]
result.append(features_list)
#display progress
count += 1
print("Added ", count, " track")
except spotipy.client.SpotifyException:
token = credentials.get_access_token()
spotify = spotipy.Spotify(auth = token)
df = pd.DataFrame(data = result, columns = col_list)
return df
if __name__ == "__init__":
main()
I would like the code to update a token and get back to the loop.

Issue Running Python 3.7 in PyCharm to Execute Twitter Bot (via Tweepy)

I am attempting to run Python 3.7 code within PyCharm to execute a Twitter bot from Tweepy (Twitter's API).
-I have a developer account with Twitter, which has been approved.
-I have my correct consumer keys and access tokens within the code (these are left blank in this ticket).
-I have my correct Twitter handle within the code as well (left blank in this ticket's code as well).
-Tweepy has been added as a project interpreter to my bot project, and is using the same Python 3.7 interpreter.
#!/usr/bin/python
import tweepy
import random
import time
#Do not Touch Lists
followed = []
friends = []
liked = []
time_start = time.time()
first_run = 0
# 1. Log onto twitter
consumer_key = ''
consumer_secret = ''
access_token = ''
access_token_secret = ''
handle = '#MyHandle'
delay_between_search = 30
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)
api = tweepy.API(auth)
def findtweets():
# Find 'new' tweets (under hashtags/search terms)
terms = ["test"]
query = random.choice(terms)
search = tweepy.Cursor(api.search, q=query, result_type="recent", lang="en").items(35) #Change the amount of tweets being searched for 50-75
print("Searching under term..." + query)
# Like 'new' tweets (only if the user has more than 10 followers & less than 15000 tweets)
for tweet in search:
if (tweet.user.followers_count < 10 and tweet.user.statuses_count < 15000):
print("Ignoring user " + tweet.user.screen_name)
continue
try:
api.create_favorite(id=tweet.id)
print("Following user " + tweet.user.screen_name)
# Follow the user, and then mute them.
time.sleep(6)
api.create_friendship(id=tweet.user.id)
time.sleep(3)
api.create_mute(id=tweet.user.id)
# Store their name in a file to be reviewed later.
followed.append(tweet.user.id)
liked.append(tweet.id)
except tweepy.TweepError as e:
#Catching errors
if (e.args[0][0]['code'] == 139):
print ("Error with tweet " + str(tweet.id) + ", already liked!")
liked.append(tweet.id)
continue
if (e.args[0][0]['code'] == 88):
print ("Rate limited..")
time.sleep(60*15)
continue
print ("Error with tweet " + str(tweet.id))
# Unfollow() provides a method for unfollowing users that have not
followed you back. Unfollow() also
# adds users that have followed you back to your "friends" list to
be unfollowed at a later time.
def unfollow():
print(" ~ Starting unfollow process ~ ")
for user in followed:
try:
status = api.show_friendship(source_screen_name=handle,
target_id=user)
except:
print ("Rate Limit Exceeded")
time.sleep(900)
# If the user is not following me back:
if (str(status[1].following) == "False"):
try:
# Unfollow them
api.destroy_friendship(id=user)
time.sleep(10)
print("Unfollowed: " + str(user))
followed.remove(user)
continue
except:
print("Could not Unfollow: " + str(user))
# Save the names of those who followed us back.
friends.append(user)
print("Friended: " + str(user))
followed.remove(user)
# Unfollow_friends(): Unfollows users that DO follow you back.
def unfollow_friends():
print(" ~ Starting unfollow_friends process ~ ")
for user in friends:
try:
api.destroy_friendship(id=user)
print("Unfollowed: " + str(user))
friends.remove(user)
except:
print("Could not Unfollow: " + str(user))
# Unlike statuses liked with the bot
def unlike():
for tweet in liked:
try:
api.destroy_favorite(id=tweet)
print("Unliked: " + str(tweet))
liked.remove(tweet)
except tweepy.TweepError as e:
if(e.args[0][0]['code'] == 144):
print("This status no longer exists... removing..")
liked.remove(tweet)
continue
print("An unknown error occured")
# Write our actions to separate files in case of a error
def write_to_file(filename, list):
for item in list:
filename.write(str(item) + "\n")
# Read from our files on first run.
if (first_run == 0):
try:
with open('followed_users.txt') as f:
followed = f.read().splitlines()
if (len(followed) > 100):
unfollow()
with open('liked_tweets.txt') as f:
liked = f.read().splitlines()
if (len(liked) > 100):
unlike()
with open('friend_users.txt') as f:
friends = f.read().splitlines()
if (len(friends) > 100):
unfollow_friends()
except:
print("Files not found...waiting for first run.")
first_run = 1
while (1 > 0):
since_launch = int(time.time() - time_start)
print ("Running Twitter Bot... " + str(since_launch/60) + " minutes since last mass-action..")
# Tweeting
if (time.time() > time_start+(3600*3)):
time_start = time.time()
unfollow_friends()
if (time.time() > time_start+3600 and len(followed) > 100):
unfollow()
unlike()
findtweets()
time.sleep(delay_between_search)
# Logging
print("Logging our files...")
liked_tweets = open("liked_tweets.txt", "w")
write_to_file(liked_tweets, liked)
liked_tweets.close()
followed_users = open("followed_users.txt", "w")
write_to_file(followed_users, followed)
followed_users.close()
friend_users = open("friend_users.txt", "w")
write_to_file(friend_users, friends)
friend_users.close()
Running the code should expect for the bot to search and follow/like Tweets based on the search term, but I'm not getting that far. I am experiencing the following error(s):
/usr/local/bin/python3.7 /Users/myname/Downloads/yt-python-twitter-master/twitter_bot.py
Files not found...waiting for first run.
Running Twitter Bot... 0.0 minutes since last mass-action..
Searching under term...test
Traceback (most recent call last):
File "/Users/myname/Downloads/yt-python-twitter-master/twitter_bot.py", line 48, in findtweets
api.create_favorite(id=tweet.id)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tweepy/binder.py", line 250, in _call
return method.execute()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tweepy/binder.py", line 234, in execute
raise TweepError(error_msg, resp, api_code=api_error_code)
tweepy.error.TweepError: Twitter error response: status code = 429
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/myname/Downloads/yt-python-twitter-master/twitter_bot.py", line 158, in <module>
findtweets()
File "/Users/myname/Downloads/yt-python-twitter-master/twitter_bot.py", line 60, in findtweets
if (e.args[0][0]['code'] == 139):
TypeError: string indices must be integers
Process finished with exit code 1

The code you provided doesn't seem to be properly formatted.
However, a 429 response status code means that you're being rate limited by Twitter's API.
The rate limit for the favorites/create endpoint that API.create_favorite uses is 1000 requests / 24-hour window.
As for your handling of the resulting error, e.args[0] is the reason for the error in the form of a string.
See https://github.com/tweepy/tweepy/blob/master/tweepy/error.py.
e.args[0][0] is therefore the first character of that reason.
Hence, the error you're encountering is because you're attempting to access a string (a single character) with the index 'code'.

Bug in python code preventing successful recursion?

I have been working on a script to ingest a file (accounts.txt) which contains email addresses, for which each will then be verified against an API to see if they appear in a data dump. The script appears to work, however there is a bug present whereby once it finds a positive hit, it will disregard any other match...
For example;
If my "accounts.txt" file contains the following entries:
a#a.com
b#b.com
Even though both of those should return results, as soon as the script is run, the match on a#a.com will be found however b#b.com will not return anything.
I cannot seem to figure out why this is happening, ideally I want all of the hits outputted.
FYI, the script is querying 'haveibeenpwned' which is a site that locates email addresses found in credential dumps.
Any help finding my bug would be greatly appreciated. Below is my current script.
#!/usr/bin/env python
import argparse
import json
import requests
import time
breaches_by_date = {}
breaches_by_account = {}
breaches_by_name = {}
class Breach(object):
def __init__(self, e, n, d):
self.email = e
self.name = n
self.date = d
def __repr__(self):
return "%s: %s breached on %s" % (self.email, self.name, self.date)
def accCheck(acc):
global breaches_by_date, breaches_by_account, breaches_by_name
r = requests.get('https://haveibeenpwned.com/api/v2/breachedaccount/%s?truncateResponse=false' % acc)
try:
data = json.loads(r.text)
except ValueError:
print("No breach information for %s" % acc)
return
for i in data:
name, date = (i['Name'], i['BreachDate'])
breach = Breach(acc, name, date)
try: breaches_by_account[acc].append(breach)
except: breaches_by_account[acc] = [breach]
try: breaches_by_name[name].append(breach)
except: breaches_by_name[name] = [breach]
try: breaches_by_date[date].append(breach)
except: breaches_by_date[date] = [breach]
def readFromFile(fname="accounts.txt"):
accounts=[]
with open(fname, "r+") as f:
accounts = [l.strip() for l in f.readlines()]
return accounts
if __name__ == '__main__':
accounts = readFromFile()
for email_addr in accounts:
accCheck(email_addr)
print
print("Breaches by date")
for date, breaches in breaches_by_date.items():
for breach in breaches:
print(breach)
print
print("Breaches by account")
for acc, breaches in breaches_by_account.items():
print(acc)
for breach in breaches:
print("%s breached on %s" % (breach.name, breach.date))
print
print("Breaches by name")
for name, breaches in breaches_by_name.items():
print("%s breached for the following accounts:" % name)
for breach in breaches:
print("%s on %s" % (breach.email, breach.date))
print

I am not 100% sure to know where your problem comes from, but I would opt for a code like:
emails_to_check = open("/path/to/yourfile").read().split("\n")
for email in emails_to_check:
if is_email_blacklisted(email):
do_something()

return actual tweets in tweepy?

I was writing a twitter program using tweepy. When I run this code, it prints the Python ... values for them, like
<tweepy.models.Status object at 0x95ff8cc>
Which is not good. How do I get the actual tweet?
import tweepy, tweepy.api
key = XXXXX
sec = XXXXX
tok  = XXXXX
tsec = XXXXX
auth = tweepy.OAuthHandler(key, sec)
auth.set_access_token(tok, tsec)
api = tweepy.API(auth)
pub = api.home_timeline()
for i in pub:
        print str(i)

In general, you can use the dir() builtin in Python to inspect an object.
It would seem the Tweepy documentation is very lacking here, but I would imagine the Status objects mirror the structure of Twitter's REST status format, see (for example) https://dev.twitter.com/docs/api/1/get/statuses/home_timeline
So -- try
print dir(status)
to see what lives in the status object
or just, say,
print status.text
print status.user.screen_name

Have a look at the getstate() get method which can be used to inspect the returned object
for i in pub:
print i.__getstate__()

The api.home_timeline() method returns a list of 20 tweepy.models.Status objects which correspond to the top 20 tweets. That is, each Tweet is considered as an object of Status class. Each Status object has a number of attributes like id, text, user, place, created_at, etc.
The following code would print the tweet id and the text :
tweets = api.home_timeline()
for tweet in tweets:
print tweet.id, " : ", tweet.text

from actual tweets,if u want specific tweet,u must have a tweet id,
and use
tweets = self.api.statuses_lookup(tweetIDs)
for tweet in tweets:
#tweet obtained
print(str(tweet['id'])+str(tweet['text']))
or if u want tweets in general
use twitter stream api
class StdOutListener(StreamListener):
def __init__(self, outputDatabaseName, collectionName):
try:
print("Connecting to database")
conn=pymongo.MongoClient()
outputDB = conn[outputDatabaseName]
self.collection = outputDB[collectionName]
self.counter = 0
except pymongo.errors.ConnectionFailure as e:
print ("Could not connect to MongoDB:")
def on_data(self,data):
datajson=json.loads(data)
if "lang" in datajson and datajson["lang"] == "en" and "text" in datajson:
self.collection.insert(datajson)
text=datajson["text"].encode("utf-8") #The text of the tweet
self.counter += 1
print(str(self.counter) + " " +str(text))
def on_error(self, status):
print("ERROR")
print(status)
def on_connect(self):
print("You're connected to the streaming server.
l=StdOutListener(dbname,cname)
auth=OAuthHandler(Auth.consumer_key,Auth.consumer_secret)
auth.set_access_token(Auth.access_token,Auth.access_token_secret)
stream=Stream(auth,l)
stream.filter(track=stopWords)
create a class Stdoutlistener which is inherited from StreamListener
override function on_data,and tweet is returned in json format,this function runs every time tweet is obtained
tweets are filtered accrding to stopwords
which is list of u words u wants in ur tweets

On a tweepy Status instance you can can access the _json attribute, which returns a dict representing the original Tweet contents.
For example:
type(status)
# tweepy.models.Status
type(status._json)
# dict
status._json.keys()
# dict_keys(['favorite_count', 'contributors', 'id', 'user', ...])