Develop Reference

Python Bot with tweepy

I'm trying to code a bot on twitter using the tweepy lib but I'm not getting results. I need help for code to reply to tweets that mentioned me.
search = '#MoviesRandom'
numberOfTweets = 10
phrase = movies() # Here im using a function declared by me before. Doesn't having errors here
for tweet in tweepy.Cursor(api.search, search).items(numberOfTweets):
try:
tweetId = tweet.user.idusername
username = tweet.user.screen_name
api.update_status("#" + username + " " + phrase, in_reply_to_status_id=tweetId)
print("Replied with " + phrase)
except tweepy.TweepError as e:
print(e.reason)

It's likely caused by this line here
tweetId = tweet.user.idusername
There is no such object called idusername and as #Andy mentioned it, it should just be the id object.
tweetId = tweet.user.id

Reddit to Twitter python bot error (python: praw & tweepy used)

I have a problem with this part of my code:
timeline = tweepy.Cursor(api.user_timeline).items(1)
for submission in reddit.subreddit('StonerPhilosophy').top('hour' , limit=1):
if len(submission.title) <= 280:
try:
api.update_status(status = submission.title)
sleep(120)
for tweet in timeline:
api.update_status(status = 'Credits: Posted by /u/' + str(submission.author) + 'url: redd.it/' + str(submission.id) , in_reply_to_status_id = tweet.id)
except:
print('Fail')
elif len(submission.title) <= 560:
try:
s = submission.title
first_half = s[0:len(s)//2]
second_half = s[len(s)//2 if len(s)%2 == 0 else ((len(s)//2)+1):]
api.update_status(status = first_half)
for tweet in timeline:
api.update_status(status = second_half) , in_reply_to_status_id = tweet.id)
sleep(120)
for tweet in timeline:
api.update_status(status = 'Credits: Posted by /u/' + str(submission.author) + 'url: redd.it/' + str(submission.id)) , in_reply_to_status_id = tweet.id)
except:
print('Fail')
when I try to run it I get 'Invalid Syntax' as an error. The problem is with 'tweet.id' (but for some reason the second and the third, no problem with the first one... :/)
what i am doing with 'tweet.id' is basically getting the id of my last tweet so that i can reply to it with either the credits or the second part of the tweet (I have to break some tweet in two parts if they are longer than 280 char) and then another reply with the credits. I have been trying to fix this for hours :/

Not sure what your API calls should look like, but assuming
api.update_status() takes a status, and a tweetID; you are only passing it a status. If you look at the brackets on it,
api.update_status(status = second_half) is all that you would call, if it didn't fail during run because of the rest of the line.
you might have better luck with:
api.update_status(status = second_half, in_reply_to_status_id = tweet.id)
More information on your API/code would be helpful.

Sending form request to SciHub no longer working using urllib urllib2 python

I had a little script I was very pleased with that would read one or more bibliographic references from the clipboard, and get info on the academic paper from Google Scholar, and then feed it into SciHub to get the pdf. For some reason it has stopped working and I have spent ages trying to work out why.
Testing reveals that the Google (scholarly.py) part of the program is working correctly, it's the SciHub part that is the issue.
Any ideas?
Here is an example reference:
Appleyard, S.J., Angeloni, J. and Watkins, R. (2006) Arsenic-rich groundwater in an urban area experiencing drought and increasing population density, Perth, Australia. Applied Geochemistry 21(1), 83-97.
'''Program to automatically find and download items from a bibliography or references list.
This program uses the 'scihub' website to obtain the full-text paper where
available, if no entry is found the paper is ignored and the failed downloads
are listed at the end'''
import scholarly
import win32clipboard
import urllib
import urllib2
import webbrowser
import re
'''Select and then copy the bibliography entries you want to download the
papers for, python reads the clipboard'''
win32clipboard.OpenClipboard()
c = win32clipboard.GetClipboardData()
win32clipboard.EmptyClipboard()
'''Cleans up the text. removes end lines and double spaces etc.'''
c = c.replace('\n', ' ')
c = c.replace('\r', ' ')
while c.find(' ') != -1:
c = c.replace(' ', ' ')
win32clipboard.SetClipboardText(c)
win32clipboard.CloseClipboard()
print "Working..."
'''bit of regex to extract the title of the paper,
IMPORTANT: bibliography has to be in
author date format or you will need to revise this,
at the moment it looks for year date in brackets, then copies all the text until it
reaches a full-stop, assuming that this is the paper title. If it is not, it
will either fail or will be using inappropriate search terms.'''
paper_info= re.findall(r"(\d{4}[a-z]*)([). ]+)([ \"])+([\w\s_():,-]*)(.)",c)
print "Analysing titles"
print "The following titles found:"
print "*************************"
list_of_titles= list()
for i in paper_info:
print '%s...' % (i[3][:50])
Paper_title=str(i[3])
list_of_titles.append(Paper_title)
failed=list()
for title in list_of_titles:
try:
search_query = scholarly.search_pubs_query(title)
info= (next(search_query))
print "Querying Google Scholar"
print "**********************"
print "Looking up paper title:"
print "**********************"
print title
print "**********************"
url=info.bib['url']
print "Journal URL found "
print url
#url=next(search_query)
print "Sending URL: ", url
site='http://sci-hub.cc/'
data = urllib.urlencode({'request': url})
print data
results = urllib2.urlopen(site, data) #this is where it fails
with open("results.html", "w") as f:
f.write(results.read())
webbrowser.open_new("results.html")
except:
print "**********************"
print "No valid journal found for:"
print title
print "**********************"
print "Continuing..."
failed.append(title)
continue
if len(failed)==0:
print 'Complete'
else:
print '*************************************'
print 'The following titles did not download: '
print '*************************************'
print failed
print "Please check that these are valid entries"

This works now, I added "User-Agent" header and re-jigged the URLlib stuff. It seems more obvious what it is doing now. A process of trial and error trying lots of different snippets of code picked up from around the web. Hope my boss doesn't ask me what I've achieved today. Someone should create a forum where people can get answers to coding problems...
'''Program to automatically find and download items from a bibliography or references list here are some journal papers in bibliographic format. Just copy the text to clipboard and run the script.
Ghaffour, N., T. M. Missimer and G. L. Amy (2013). "Technical review and evaluation of the economics of water desalination: Current and future challenges for better water supply sustainability." Desalination 309(0): 197-207.
Gutiérrez Ortiz, F. J., P. G. Aguilera and P. Ollero (2014). "Biogas desulfurization by adsorption on thermally treated sewage-sludge." Separation and Purification Technology 123(0): 200-213.
This program uses the 'scihub' website to obtain the full-text paper where
available, if no entry is found the paper is ignored and the failed downloads are listed at the end'''
import scholarly
import win32clipboard
import urllib
import urllib2
import webbrowser
import re
'''Select and then copy the bibliography entries you want to download the
papers for, python reads the clipboard'''
win32clipboard.OpenClipboard()
c = win32clipboard.GetClipboardData()
win32clipboard.EmptyClipboard()
'''Cleans up the text. removes end lines and double spaces etc.'''
c = c.replace('\n', ' ')
c = c.replace('\r', ' ')
while c.find(' ') != -1:
c = c.replace(' ', ' ')
win32clipboard.SetClipboardText(c)
win32clipboard.CloseClipboard()
print "Working..."
'''bit of regex to extract the title of the paper,
IMPORTANT: bibliography has to be in
author date format or you will need to revise this,
at the moment it looks for date in brackets, then copies all the text until it
reaches a full-stop, assuming that this is the paper title. If it is not, it
will either fail or will be using inappropriate search terms.'''
paper_info= re.findall(r"(\d{4}[a-z]*)([). ]+)([ \"])+([\w\s_():,-]*)(.)",c)
print "Analysing titles"
print "The following titles found:"
print "*************************"
list_of_titles= list()
for i in paper_info:
print '%s...' % (i[3][:50])
Paper_title=str(i[3])
list_of_titles.append(Paper_title)
paper_number=0
failed=list()
for title in list_of_titles:
try:
search_query = scholarly.search_pubs_query(title)
info= (next(search_query))
paper_number+=1
print "Querying Google Scholar"
print "**********************"
print "Looking up paper title:"
print title
print "**********************"
url=info.bib['url']
print "Journal URL found "
print url
#url=next(search_query)
print "Sending URL: ", url
site='http://sci-hub.cc/'
r = urllib2.Request(url=site)
r.add_header('User-Agent','Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11')
r.add_data(urllib.urlencode({'request': url}))
res= urllib2.urlopen(r)
with open("results.html", "w") as f:
f.write(res.read())
webbrowser.open_new("results.html")
if not paper_number<= len(list_of_titles):
print "Next title"
else:
continue
except Exception as e:
print repr(e)
paper_number+=1
print "**********************"
print "No valid journal found for:"
print title
print "**********************"
print "Continuing..."
failed.append(title)
continue
if len(failed)==0:
print 'Complete'
else:
print '*************************************'
print 'The following titles did not download: '
print '*************************************'
print failed
print "Please check that these are valid entries"

Why is one of my functions running twice?

The function search_for_song(pbody) is running twice, i can't figure out why.
would like some help, just started learning python a few days ago.
Here's the full code:
#a bot that replies with youtube songs that were mentioned in the comments
import traceback
import praw
import time
import sqlite3
import requests
from lxml import html
import socket
import errno
import re
import urllib
from bs4 import BeautifulSoup
import sys
import urllib2
'''USER CONFIGURATION'''
APP_ID = ""
APP_SECRET = ""
APP_URI = ""
APP_REFRESH = ""
# https://www.reddit.com/comments/3cm1p8/how_to_make_your_bot_use_oauth2/
USERAGENT = "Python automatic youtube linkerbot"
# This is a short description of what the bot does.
# For example "Python automatic replybot v2.0 (by /u/GoldenSights)"
SUBREDDIT = "kqly"
# This is the sub or list of subs to scan for new posts. For a single sub, use "sub1". For multiple subreddits, use "sub1+sub2+sub3+..."
DO_SUBMISSIONS = False
DO_COMMENTS = True
# Look for submissions, comments, or both.
KEYWORDS = ["linksong"]
# These are the words you are looking for
KEYAUTHORS = []
# These are the names of the authors you are looking for
# The bot will only reply to authors on this list
# Keep it empty to allow anybody.
#REPLYSTRING = "**Hi, I'm a bot.**"
# This is the word you want to put in reply
MAXPOSTS = 100
# This is how many posts you want to retrieve all at once. PRAW can download 100 at a time.
WAIT = 30
# This is how many seconds you will wait between cycles. The bot is completely inactive during this time.
CLEANCYCLES = 10
# After this many cycles, the bot will clean its database
# Keeping only the latest (2*MAXPOSTS) items
'''All done!'''
try:
import bot
USERAGENT = bot.aG
except ImportError:
pass
print('Opening SQL Database')
sql = sqlite3.connect('sql.db')
cur = sql.cursor()
cur.execute('CREATE TABLE IF NOT EXISTS oldposts(id TEXT)')
print('Logging in...')
r = praw.Reddit(USERAGENT)
r.set_oauth_app_info(APP_ID, APP_SECRET, APP_URI)
r.refresh_access_information(APP_REFRESH)
def replybot():
print('Searching %s.' % SUBREDDIT)
subreddit = r.get_subreddit(SUBREDDIT)
posts = []
if DO_SUBMISSIONS:
posts += list(subreddit.get_new(limit=MAXPOSTS))
if DO_COMMENTS:
posts += list(subreddit.get_comments(limit=MAXPOSTS))
posts.reverse()
for post in posts:
#print ("Searching for another the next comment")
# Anything that needs to happen every loop goes here.
pid = post.id
try:
pauthor = post.author.name
except AttributeError:
# Author is deleted. We don't care about this post.
continue
if pauthor.lower() == r.user.name.lower():
# Don't reply to yourself, robot!
print('Will not reply to myself.')
continue
if KEYAUTHORS != [] and all(auth.lower() != pauthor for auth in KEYAUTHORS):
# This post was not made by a keyauthor
continue
cur.execute('SELECT * FROM oldposts WHERE ID=?', [pid])
if cur.fetchone():
# Post is already in the database
continue
if isinstance(post, praw.objects.Comment):
pbody = post.body
else:
pbody = '%s %s' % (post.title, post.selftext)
pbody = pbody.lower()
if not any(key.lower() in pbody for key in KEYWORDS):
# Does not contain our keyword
continue
cur.execute('INSERT INTO oldposts VALUES(?)', [pid])
sql.commit()
print('Replying to %s by %s' % (pid, pauthor))
try:
if search_for_song(pbody):
# pbody=pbody[8:]
# pbody=pbody.replace("\n", "")
temp=pbody[8:].lstrip()
post.reply("[**"+temp+"**]("+search_for_song(pbody)+") \n ---- \n ^^This ^^is ^^an ^^automated ^^message ^^by ^^a ^^bot, ^^if ^^you ^^found ^^any ^^bug ^^and/or ^^willing ^^to ^^contact ^^me. [**^^Press ^^here**](https://www.reddit.com/message/compose?to=itailitai)")
except praw.errors.Forbidden:
print('403 FORBIDDEN - is the bot banned from %s?' % post.subreddit.display_name)
def search_for_song(pbody):
#print("in search_for_song")
song=pbody
if len(song)>8:
song=song[8:]
if song.isspace()==True or song=='':
return False
else:
print("Search if %s exists in the database" % song )
#HEADERS = {'User-Agent': 'Song checker - check if songs exists by searching this website, part of a bot for reddit'}
author, song_name = song_string_generator(song)
url = 'http://www.songlyrics.com/'+author+'/'+song_name+'-lyrics/'
print url
#page = requests.get(url, HEADERS)
check=1
while check==1:
try:
headers = { 'User-Agent' : 'Mozilla/5.0 (Windows NT 6.1; rv:40.0) Gecko/20100101 Firefox/40.0' }
req = urllib2.Request(url, None, headers)
page= urllib2.urlopen(req)
check=2
except socket.error as error:
pass
except Exception:
print('An error occured while tryinc to verify song existence')
return False
soup = BeautifulSoup(page.read(), "lxml")
if "Please check the spelling and try again" not in soup.get_text():
print ("Song was found in the database!")
result=first_youtube(song)
return result
else:
print ("Song was not found in the database!")
return False
def song_string_generator(song):
#print("in song_string_generator")
song=song
author,song_name= '',''
try:
if "-" in song:
l=song.split('-', 1 )
print ("2 ",l)
author=l[0]
song_name=l[1]
elif "by" in song:
l=song.split('by', 1 )
print ("2 ",l)
author=l[1]
song_name=l[0]
song_name=" ".join(song_name.split())
author=" ".join(author.split())
print (author,song_name)
if author == 'guns and roses':
author="guns n' roses"
song_name=song_name.replace("\n", "")
author=author.replace("\n", "")
author=author.replace(" ", "-")
song_name=song_name.replace(" ", "-")
author=author.replace("'", "-")
song_name=song_name.replace("'", "-")
song_name=song_name.rstrip()
song_name=" ".join(song_name.split())
return author, song_name
except:
print ("No song was mentioned in the comment!")
return False
def first_youtube(textToSearch):
reload(sys)
sys.setdefaultencoding('UTF-8')
query_string = textToSearch
try:
html_content = urllib.urlopen("http://www.youtube.com/results?search_query=" + query_string)
search_results = re.findall(r'href=\"\/watch\?v=(.{11})', html_content.read().decode())
result="http://www.youtube.com/watch?v=" + search_results[0]
return result
except IOError:
print ("IOError Occured while contacting Youtube!")
except Exception:
print ("A non IOError Occured while contacting Youtube!")
return False
cycles = 0
while True:
try:
replybot()
cycles += 1
except Exception as e:
traceback.print_exc()
if cycles >= CLEANCYCLES:
print('Cleaning database')
cur.execute('DELETE FROM oldposts WHERE id NOT IN (SELECT id FROM oldposts ORDER BY id DESC LIMIT ?)', [MAXPOSTS * 2])
sql.commit()
cycles = 0
print('Running again in %d seconds \n' % WAIT)
time.sleep(WAIT)
This is the output I'm getting:
Opening SQL Database
Logging in...
Searching kqly.
Will not reply to myself.
Will not reply to myself.
Will not reply to myself.
Will not reply to myself.
Will not reply to myself.
Will not reply to myself.
Will not reply to myself.
Will not reply to myself.
Will not reply to myself.
Will not reply to myself.
Will not reply to myself.
Will not reply to myself.
Will not reply to myself.
Will not reply to myself.
Will not reply to myself.
Will not reply to myself.
Will not reply to myself.
Will not reply to myself.
Will not reply to myself.
Will not reply to myself.
Will not reply to myself.
Replying to d0kwcrs by itailitai
Search if guns and roses - paradise city exists in the database
('2 ', [u' guns and roses ', u' paradise city'])
(u'guns and roses', u'paradise city')
http://www.songlyrics.com/guns-n--roses/paradise-city-lyrics/
Song was found in the database!
Search if guns and roses - paradise city exists in the database
('2 ', [u' guns and roses ', u' paradise city'])
(u'guns and roses', u'paradise city')
http://www.songlyrics.com/guns-n--roses/paradise-city-lyrics/
Song was found in the database!
Running again in 30 seconds
it's a bot for reddit that replies with the youtube video of a song that was mentioned in the comments, if anyone wants to know.

With a cursory reading of your code you have
if search_for_song(pbody):
# do stuff..
post.reply("[**"+temp+"**]("+search_for_song(pbody)+") \n ---- \n ^^This ^^is ^^an ^^automated ^^message ^^by ^^a ^^bot, ^^if ^^you ^^found ^^any ^^bug ^^and/or ^^willing ^^to ^^contact ^^me. [**^^Press ^^here**](https://www.reddit.com/message/compose?to=itailitai)")
You call the function in the start of the if and in your post.reply line
RESPONDING TO COMMENTS
If you need to check the results but don't want to call twice simply save the output
res = search_for_song(pbody):
if res:
#...
post.reply(... + res + ...)

I've just quickly searched for the function call search_for_song, I suppose the following piece of code is resulting in 2 function calls.
if search_for_song(pbody):
# pbody=pbody[8:]
# pbody=pbody.replace("\n", "")
temp=pbody[8:].lstrip()
post.reply("[**"+temp+"**]("+search_for_song(pbody)+")
Once at the if statement, and once inside the post.reply statement.

Twython: Get a list of all your followers except those who are on your muted list?

I have this block of code that gets my twitter followers using Twython. How would I modify this to instead do:
"Get all your twitter followers who do not appear on your muted followers list" ?
I tried doing a request twice, one for followers, and one for twitter.list_mutes, then comparing the two lists as sets. But the twitter api wasn't too happy about that.
Suggestions?
followers = []
next_cursor = -1
while(next_cursor):
get_followers = twitter.get_followers_list(screen_name=username, count=200, cursor=next_cursor)
for follower in get_followers["users"]:
followers.append(follower["screen_name"].encode("utf-8"))
next_cursor = get_followers["next_cursor"]
print "%s followers found" % str(len(followers))

Alright. Found a way to do it, but its not elegant. It works tho. I just made different cursor to pass to a different loop. However there are probably way better ways to do this, so please feel free to comment here.
https://github.com/sharkwheels/twitter_muter/blob/master/randoMute.py
### GET FOLLOWERS ##################
while(next_cursor):
get_followers = twitter.get_followers_list(screen_name=username, count=200, cursor=next_cursor)
for follower in get_followers["users"]:
followers.append(follower["screen_name"].encode("utf-8"))
next_cursor = get_followers["next_cursor"]
print "%s followers found" % str(len(followers))
print " "
print "get muted users"
## GET ALREADY MUTED FOLLOWERS ##################
while(butts_cursor):
get_muted = twitter.list_mutes(screen_name=username, count=200, cursor=butts_cursor)
for x in get_muted["users"]:
silent_followers.append(x["screen_name"].encode("utf-8"))
butts_cursor = get_muted["next_cursor"]
print " "
print "%s silent_followers found" % str(len(silent_followers))
print silent_followers