Develop Reference

GAE Python URL Fetch throws InvalidURLError [duplicate]

This question already has an answer here:
Python GAE urlfetch credentials
(1 answer)
Closed 7 years ago.
GAE Python URL Fetch throws InvalidURLError while the same URL works perfectly with Postman ( Google Chrome App ).
CODE
url = "https://abcdefgh:28dfd95928dfd95928dfd95928dfd95928dfd95928dfd959#twilix.exotel.in/v1/Accounts/abcdefgh/Sms/send"
form_fields = {
"From": "08039511111",
"To": "+919844100000",
"Body": "message for you"
}
form_data = urllib.urlencode (form_fields)
try:
result = urlfetch.fetch(url=url,
payload=form_data,
method=urlfetch.POST,
headers={'Content-Type': 'application/x-www-form-urlencoded' }
)
logging.info ("result = ["+repr (result)+"] ")
except Exception:
logging.error ("Exception. ["+traceback.format_exc ()+"] ")
OUTPUT LOGS
2016-01-21 15:48:23.368 +0530 E Exception. [
Traceback (most recent call last): File "main.py", line 27, in get method=urlfetch.POST,
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/urlfetch.py", line 271, in fetch return rpc.get_result()
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 613, in get_result return self.__get_result_hook(self)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/urlfetch.py", line 389, in _get_fetch_result 'Invalid request URL: ' + url + error_detail) InvalidURLError: Invalid request URL: https://abcdefgh:28dfd95928dfd95928dfd95928dfd95928dfd95928dfd959#twilix.exotel.in/v1/Accounts/abcdefgh/Sms/send ]
For security purpose, I have replaced sensitive text in the URL with similar different characters.

The code indicates an INVALID_URL RPC error code was received from the urlfetch service.
The most common occurence seems to be due to the URL length limit (check if your unedited URL hits that): Undocumented max length for urlfetch URL?
A long time ago it was also seen for very slow URLs (in Go land, but I suspect the urlfetch service itself is the same serving all language sandboxes) - unsure if this still stands, I also see a DEADLINE_EXCEEDED error code as well which might have been introduced specifically for such case in the meantime): Google App Engine Go HTTP request to a slow page
The failure might also be related to incorrect parsing of the rather unusual "host" portion of your URL foo:blah#hostname. Check if it you're getting the same error if dropping the foo:blah# portion. If it's indeed the case you might want to file an issue with Google - the URL seems valid, works with curl as well.

I found the problem and the solution.
We need to specify the HTTP auth info using headers.
urlfetch.make_fetch_call ( rpc,
url,
method = urlfetch.POST,
headers = { "Authorization" : "Basic %s" % base64.b64encode ( URL_USERNAME+":"+URL_PASSOWRD ) },
)
Courtesy
https://stackoverflow.com/a/8454580/1443563 by raugfer

Getting twitter friends number with Python

I need to get twitter friends from an user. With twitter "friends" I mean user A follows user B, and also user B follows A.
Ok, so my idea was getting a list of people who are followed by user A, and if they are followed by user B, it increases numFriends.
First part works fine, but when I do that second request it falls apart and throws me an ugly 'error 400' :(
I read about the restrictions of twitter and all that, but it seems weird that second request so I don't know if it's doing ok.
Thank you in advance, I'm a noob at python and twitter api, and my mother tongue is not english, so I really hope it is everything clear. I will thank any help about this.
Here is the code :)
def getNumFriends(user):
dataUser=0
numFriends=0
url ="https://api.twitter.com/1.1/friends/ids.json?cursor=-1&screen_name=%s&count=5000"%(user.screen_name)
auth = OAuth1(getConsumerKey(), getConsumerSecret(), getAccessToken(), getAccessTokenSecret())
response = requests.get(url, auth=auth)
if response.status_code == 200:
dataUser = response.json()
userIDs = dataUser['ids']
else:
print "Error code %s" %response.status_code
#Here comes the problem :S
for friend in userIDs:
url = "https://api.twitter.com/1.1/friendships/show.json?source_id=%s&target_screen_name=%s"%(friend, user)
response = requests.get(url, auth)
if response.status_code == 200:
dataCompare = response.json()
mutualfriends = dataCompare['relationship']['target']['followed_by']
if mutualfriends =='true':
numFriends=numFriends+1
else:
print "First request OK. Second request error code %s" %response.status_code
break
return numFriends

Your code is fine, there are only a couple of minor mistakes. The error code 400, "bad request", does not give you very concrete info, but it tells you that something in the way you've written the url is wrong.
It should be:
url = "https://api.twitter.com/1.1/friendships/show.json?source_id=%s&target_screen_name=%s"%(friend, user.screen_name)
i.e it should be user.screen_name instead of user.
Besides, the second argument of requests.get() is not auth, thus you should specify always the name of the argument,
response = requests.get(url, auth=auth)
which wasn't the case in your second call.
BTW, just curious, any reason why you are not using a library like twython?
Hope it helps.
** EDIT after comments: **
There was another minor mistake in the code. Note that your variable mutualfriends is already boolean, in order to check its value you can do it like this,
if mutualfriends:
...
BTW, to check the type of a variable,
print(type(mutualfriends))

Python - Check if url exists fails

I'm trying to create a python function that will connect to a URL and check if a list of directory's exists on that website. So the input consists of a target and the directory's. My ultimate goal is to write some sort of DirBuster like program.
This is my function untill now:
def checkDir(checkDir_target):
breakurl = urlparse(target)
conn = httplib.HTTPConnection(breakurl.netloc)
conn.request('HEAD', checkDir_target)
response = conn.getresponse()
print response.status
complete = target + x
if (response.status < 400):
print(" [X] " + complete)
global total_resp
total_resp += 1
found.append(complete)
else:
print(" [ ] " + complete)
The only problem I'm having right now is that dynamic created pages like wordpress pages also return HTTP Status 200 codes. So even when I'm testing on a non-existing url the website will still return a HTTP 200 OK.
Example: testing on www.wordpressexamplesite.com/DIRECTORYTHATDOESNTEXISTS/ gives a HTTP 200 code as well as website URL's that DOES exist.
This means that the whole check in the checkDir function is not doing it's work like I want it to.
Can one of you guys give me some ideas on how to resolve this?

Unfortunately for you, when the server returns a "200 OK", then that means the URL does, in fact, exist and has the contents returned. Those contents might be a page that says "This doesn't exist". To identify that you would need to work on some artificial intelligence that can render and read the content that was returned and comprehend it like a human would.
I consider it bad web site design (and even worse for AJAX APIs) to always return "200 OK" and embed the "real" status in the payload, but that is how some people code it.

Try use requests lib:
import requests
def checkDir(checkDir_target):
breakurl = urlparse(target)
response = requests.get(breakurl.netloc, headers=checkDir_target)
complete = target + x
if response.status < 400:
print(" [X] " + complete)
global total_resp
total_resp += 1
found.append(complete)
else:
print(" [ ] " + complete)
I think this can works for you.

How to keep a program running if there is an Traceback error

I made a simple script for amusment that takes the latest comment from http://www.reddit.com/r/random/comments.json?limit=1 and speaks through espeak. I ran into a problem however. If Reddit fails to give me the json data, which it commonly does, the script stops and gives a traceback. This is a problem, as it stops the script. Is there any sort of way to retry to get the json if it fails to load. I am using requests if that means anything
If you need it, here is the part of the code that gets the json data
url = 'http://www.reddit.com/r/random/comments.json?limit=1'
r = requests.get(url)
quote = r.text
body = json.loads(quote)['data']['children'][0]['data']['body']
subreddit = json.loads(quote)['data']['children'][0]['data']['subreddit']

For the vocabulary, the actual error you're having is an exception that has been thrown at some point in a program because of a detected runtime error, and the traceback is the program thread that tells you where the exception has been thrown.
Basically, what you want is an exception handler:
try:
url = 'http://www.reddit.com/r/random/comments.json?limit=1'
r = requests.get(url)
quote = r.text
body = json.loads(quote)['data']['children'][0]['data']['body']
subreddit = json.loads(quote)['data']['children'][0]['data']['subreddit']
except Exception as err:
print err
so that you jump over the part that needs the thing that couldn't work. Have a look at that doc as well: HandlingExceptions - Python Wiki
As pss suggests, if you want to retry after the url failed to load:
done = False
while not done:
try:
url = 'http://www.reddit.com/r/random/comments.json?limit=1'
r = requests.get(url)
except Exception as err:
print err
done = True
quote = r.text
body = json.loads(quote)['data']['children'][0]['data']['body']
subreddit = json.loads(quote)['data']['children'][0]['data']['subreddit']
N.B.: That solution may not be optimal, since if you're offline or the URL is always failing, it'll do an infinite loop. If you retry too fast and too much, Reddit may also ban you.
N.B. 2: I'm using the newest Python 3 syntax for exception handling, which may not work with Python older than 2.7.
N.B. 3: You may also want to choose a class other than Exception for the exception handling, to be able to select what kind of error you want to handle. It mostly depends on your app design, and given what you say, you might want to handle requests.exceptions.ConnectionError, but have a look at request's doc to choose the right one.
Here's what you may want, but please think this through and adapt it to your use case:
import requests
import time
import json
def get_reddit_comments():
retries = 5
while retries != 0:
try:
url = 'http://www.reddit.com/r/random/comments.json?limit=1'
r = requests.get(url)
break # if the request succeeded we get out of the loop
except requests.exceptions.ConnectionError as err:
print("Warning: couldn't get the URL: {}".format(err))
time.delay(1) # wait 1 second between two requests
retries -= 1
if retries == 0: # if we've done 5 attempts, we fail loudly
return None
return r.text
def use_data(quote):
if not quote:
print("could not get URL, despites multiple attempts!")
return False
data = json.loads(quote)
if 'error' in data.keys():
print("could not get data from reddit: error code #{}".format(quote['error']))
return False
body = data['data']['children'][0]['data']['body']
subreddit = data['data']['children'][0]['data']['subreddit']
# … do stuff with your data here
if __name__ == "__main__":
quote = get_reddit_comments()
if not use_data(quote):
print("Fatal error: Couldn't handle data receipt from reddit.")
sys.exit(1)
I hope this snippet will help you correctly design your program. And now that you've discovered exceptions, please always remember that exceptions are for handling things that shall stay exceptional. If you throw an exception at some point in one of your programs, always ask yourself if this is something that should happen when something unexpected happens (like a webpage not loading), or if it's an expected error (like a page loading but giving you an output that is not expected).

XML parser syntax error

So I'm working with a block of code which communicates with the Flickr API.
I'm getting a 'syntax error' in xml.parsers.expat.ExpatError (below). Now I can't figure out how it'd be a syntax error in a Python module.
I saw another similar question on SO regarding the Wikipedia API which seemed to return HTML intead of XML. Flickr API returns XML; and I'm also getting the same error when there shouldn't be a response from Flickr (such as flickr.galleries.addPhoto)
CODE:
def _dopost(method, auth=False, **params):
#uncomment to check you aren't killing the flickr server
#print "***** do post %s" % method
params = _prepare_params(params)
url = '%s%s/%s' % (HOST, API, _get_auth_url_suffix(method, auth, params))
payload = 'api_key=%s&method=%s&%s'% \
(API_KEY, method, urlencode(params))
#another useful debug print statement
#print url
#print payload
return _get_data(minidom.parse(urlopen(url, payload)))
TRACEBACK:
Traceback (most recent call last):
File "TESTING.py", line 30, in <module>
flickr.galleries_create('test_title', 'test_descriptionn goes here.')
File "/home/vlad/Documents/Computers/Programming/LEARNING/curatr/flickr.py", line 1006, in galleries_create
primary_photo_id=primary_photo_id)
File "/home/vlad/Documents/Computers/Programming/LEARNING/curatr/flickr.py", line 1066, in _dopost
return _get_data(minidom.parse(urlopen(url, payload)))
File "/usr/lib/python2.6/xml/dom/minidom.py", line 1918, in parse
return expatbuilder.parse(file)
File "/usr/lib/python2.6/xml/dom/expatbuilder.py", line 928, in parse
result = builder.parseFile(file)
File "/usr/lib/python2.6/xml/dom/expatbuilder.py", line 207, in parseFile
parser.Parse(buffer, 0)
xml.parsers.expat.ExpatError: syntax error: line 1, column 62
(Code from http://code.google.com/p/flickrpy/ under New BSD licence)
UPDATE:
print urlopen(url, payload) == <addinfourl at 43340936 whose fp = <socket._fileobject object at 0x29400d0>>
Doing a urlopen(url, payload).read() returns HTML which is hard to read in a terminal :P but I managed to make out a 'You are not signed in.'
The strange part is that Flickr shouldn't return anything here, or if permissions are a problem, it should return a 99: User not logged in / Insufficient permissions error as it does with the GET function (which I'd expect would be in valid XML).
I'm signed in to Flickr (in the browser) and the program is properly authenticated with delete permissions (dangerous, but I wanted to avoid permission problems.)

SyntaxError normally means an error in Python syntax, but I think here that expatbuilder is overloading it to mean an XML syntax error. Put a try:except block around it, and print out the contents of payload and to work out what's wrong with the first line of it.
My guess would be that flickr is rejecting your request for some reason and giving back a plain-text error message, which has an invalid xml character at column 62, but it could be any number of things. You probably want to check the http status code before parsing it.
Also, it's a bit strange this method is called _dopost but you seem to actually be sending an http GET. Perhaps that's why it's failing.

This seems to fix my problem:
url = '%s%s/?api_key=%s&method=%s&%s'% \
(HOST, API, API_KEY, method, _get_auth_url_suffix(method, auth, params))
payload = '%s' % (urlencode(params))
It seems that the API key and method had to be in the URL not in the payload. (Or maybe only one needed to be there, but anyways, it works :-)