So I was trying to dive into Python because I want to write an Agent for the Plex Media Server. This Agent will access the MyAnimeList.net API with HTTP Authentication (more about that here) my Username and passwords work but I don't have a clue why I still get a 401 error from the server as response.
Here is some code (I'm using python 2.5 because plex said so) :)
import urllib2
username = "someUser"
password = "somePass"
url = "http://myanimelist.net/api/anime/search.xml?q=bleach"
password_mgr = urllib2.HTTPPasswordMgrWithDefaultRealm()
top_level_url = "http://myanimelist.net/"
password_mgr.add_password(None, top_level_url, username, password)
handler = urllib2.HTTPBasicAuthHandler(password_mgr)
opener = urllib2.build_opener(urllib2.HTTPHandler, handler)
request = urllib2.Request(url)
print request.get_full_url()
f = urllib2.urlopen(request).read()
print(f)
and this is what I get as response
> http://myanimelist.net/api/anime/search.xml?q=bleach Traceback (most
> recent call last): File
> "C:\Users\Daraku\Desktop\MAL.bundle\Contents\Code\__init__.py", line
> 16, in <module>
> f = urllib2.urlopen(request).read() File "C:\Python25\lib\urllib2.py", line 121, in urlopen
> return _opener.open(url, data) File "C:\Python25\lib\urllib2.py", line 380, in open
> response = meth(req, response) File "C:\Python25\lib\urllib2.py", line 491, in http_response
> 'http', request, response, code, msg, hdrs) File "C:\Python25\lib\urllib2.py", line 418, in error
> return self._call_chain(*args) File "C:\Python25\lib\urllib2.py", line 353, in _call_chain
> result = func(*args) File "C:\Python25\lib\urllib2.py", line 499, in http_error_default
> raise HTTPError(req.get_full_url(), code, msg, hdrs, fp) HTTPError: HTTP Error 401: Unauthorized
Bear in mind that this is my first time programming in Python and im kind of confused with many examples in the web because they did something with the urllib2 so that it doesn't exists in python 3.0, i think, anymore
Any ideas? Or any better ways to do this?
Related
I'm trying to make a terminal app to crawl a website and return the time of the entered city name. this is my code so far:
import re
import urllib.request
city = input('Enter city name: ')
url = 'https://time.is/'
rawData = urllib.request.urlopen(url).read()
decodedData = rawData.decode('utf-8')
print(decodedData)
after the last line i get this error:
Traceback (most recent call last):
File "<pyshell#13>", line 1, in <module>
rawData = urllib.request.urlopen(url).read()
File "~/Python\Python35-32\lib\urllib\request.py", line 163, in urlopen
return opener.open(url, data, timeout)
File "~/Python\Python35-32\lib\urllib\request.py", line 472, in open
response = meth(req, response)
File "~/Python\Python35-32\lib\urllib\request.py", line 582, in http_response
'http', request, response, code, msg, hdrs)
File "~/Python\Python35-32\lib\urllib\request.py", line 510, in error
return self._call_chain(*args)
File "~/Python\Python35-32\lib\urllib\request.py", line 444, in _call_chain
result = func(*args)
File "~/Python\Python35-32\lib\urllib\request.py", line 590, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden
why do i get this error? what's wrong?
[EDIT]
the reason is time.is banns requests. Always remember to read terms and conditions when doing web scraping. free APIs can be found to do the same job too.
When this happens, I usually open the debugger and try to find out whats being called when I access the website. It seems like time.is doesn't like having scripts call their website.
A quick search yielded this:
1532027279136 0 161_(UTC,_UTC+00:00) 1532027279104
Time.is is for humans. To use from scripts and apps, please ask about our API. Thank you!
Here are some APIs you could use to build your project. https://www.programmableweb.com/category/time/api
I am using an embedded python (2.4.3) in digi Connectport X4 here is the code to post to an Azure IoT Hub using HTTPS:
import urllib, sys, datetime, time
import urllib2
iot_device_id = 'HTTP_Device'
iot_endpoint = 'https://winiothub.azure-devices.net/devices/' + iot_device_id + '/messages/events?api-version=2016-02-03'
sas_header = {'Authorization': 'SharedAccessSignature sr=winiothub.azure-devices.net%2Fdevices%2FHTTP_Device&sig=o7dndsA%2FJOnkzYRUhqAwMrQXVhOTpIJqJqILyGDdQAc%3D&se=1522414643'}
while True:
#try:
body_data = { 'gateway_serial': '123', 'readingvalue':'66.00', 'date': str(datetime.datetime.now())}
iot_request = urllib2.Request(iot_endpoint, str(body_data), sas_header)
resp = urllib2.urlopen(iot_request)
contents = resp.read()
resp.close()
time.sleep(1)
#except:
# print 'error'
# time.sleep(1)
The code actually post the message to the hub, but is throwing the following error:
Traceback (most recent call last):
File "C:\Users\JeffreyBiesecker\documents\visual studio 2017\Projects\NewGateAzure\NewGateAzure\NewGateAzure2.py", line 14, in ?
urllib2.urlopen(iot_request)
File "C:\Python24\lib\urllib2.py", line 130, in urlopen
return _opener.open(url, data)
File "C:\Python24\lib\urllib2.py", line 364, in open
response = meth(req, response)
File "C:\Python24\lib\urllib2.py", line 471, in http_response
response = self.parent.error(
File "C:\Python24\lib\urllib2.py", line 402, in error
return self._call_chain(*args)
File "C:\Python24\lib\urllib2.py", line 337, in _call_chain
result = func(*args)
File "C:\Python24\lib\urllib2.py", line 480, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 204: No Content
Press any key to continue . . .
I get an error if running the code in the embedded Digi gateway or if I run in Python in Visual Studio using version 2.4.3.
urllib2.HttpError includes the response code that's received. Because of this, you can catch the exception, test for 204, and continue safely if it is the case. Otherwise, you can handle (or rethrow) the exception.
try:
resp = urllib2.urlopen(iot_request)
except urllib2.HTTPError as e:
if e.code == 204: pass
else: raise
I have been searching for a way to implement 2 legged oauth in python 3 to work with the brightcloud api. They offer several code examples using java, php, ruby, .NET C# here: https://bcws.brightcloud.com/code-samples.php. I tried following the same logic to convert the java example into python, however, I'm relatively new to python and I quickly came unstuck.
I have tried implementing using rauth, however, the basic setup utilises a request_token_url which is not provided by brightcloud. I also tried implementing with the following code which was based on this answer -
How do I send a POST using 2-legged oauth2 in python?
import oauth2
import urllib #for url-encode
import urllib.request
import time #Unix timestamp import oauth2
# construct request url
base_url = "http://thor.brightcloud.com/rest"
uri_info_path = "/uris"
url = urllib.parse.quote_plus("http://www.booking.com")
# api key and secret
consumer_key = 'MY_CONSUMER_KEY'
consumer_secret = 'MY_CONSUMER_SECRET'
# contruct endpoint
endpoint = rest_endpoint + uri_info_path + '/' + url
# build request
def build_request(url, method):
params = {
'oauth_version': "1.0",
'oauth_nonce': oauth2.generate_nonce(),
'oauth_timestamp': int(time.time())
}
consumer = oauth2.Consumer(key=consumer_key, secret=consumer_secret)
params['oauth_consumer_key'] = consumer.key
req = oauth2.Request(method=method, url=url, parameters=params)
signature_method = oauth2.SignatureMethod_HMAC_SHA1()
req.sign_request(signature_method, consumer, None)
return req
# call
request = build_request(endpoint,'GET')
u = urllib.request.urlopen(request.to_url())
data = u.read()
print (data)
There is a problem with this line:
u = urllib.request.urlopen(request.to_url())
Which generates the following traceback:
Traceback (most recent call last):
File "bright.py", line 37, in
u = urllib.request.urlopen(request.to_url())
File "/usr/lib/python3.5/urllib/request.py", line 163, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib/python3.5/urllib/request.py", line 472, in open
response = meth(req, response)
File "/usr/lib/python3.5/urllib/request.py", line 582, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python3.5/urllib/request.py", line 510, in error
return self._call_chain(*args)
File "/usr/lib/python3.5/urllib/request.py", line 444, in _call_chain
result = func(*args)
File "/usr/lib/python3.5/urllib/request.py", line 590, in >http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 401: Unauthorized
Any help would be much appreciated.
I try to write a simple python3 script that gets some playlist informations via the youtube API. However I always get a 401 Error whereas it works perfectly when I enter the request string in a browser or making a request with w-get. I'm relatively new to python and I guess I'm missing some important point here.
This is my script. Of course I actually use a real API-Key.
from urllib.request import Request, urlopen
from urllib.parse import urlencode
api_key = "myApiKey"
playlist_id = input('Enter playlist id: ')
output_file = input('Enter name of output file (default is playlist id')
if output_file == '':
output_file = playlist_id
url = 'https://www.googleapis.com/youtube/v3/playlistItems'
params = {'part': 'snippet',
'playlistId': playlist_id,
'key': api_key,
'fields': 'items/snippet(title,description,position,resourceId/videoId),nextPageToken,pageInfo/totalResults',
'maxResults': 50,
'pageToken': '', }
data = urlencode(params)
request = Request(url, data.encode('utf-8'))
response = urlopen(request)
content = response.read()
print(content)
Unfortunately it rises a error at response = urlopen(request)
Traceback (most recent call last):
File "gpd-helper.py", line 35, in <module>
response = urlopen(request)
File "/usr/lib/python3.4/urllib/request.py", line 153, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib/python3.4/urllib/request.py", line 461, in open
response = meth(req, response)
File "/usr/lib/python3.4/urllib/request.py", line 571, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python3.4/urllib/request.py", line 499, in error
return self._call_chain(*args)
File "/usr/lib/python3.4/urllib/request.py", line 433, in _call_chain
result = func(*args)
File "/usr/lib/python3.4/urllib/request.py", line 579, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 401: Unauthorized
I looked up the documentation but couldn't find any hint. According to the docs other authentication than the api key is not required for listing a public playlist.
After diving deeper into the docs of python and google I found the solution to my problem.
Pythons Request object automatically creates a POST request when the data parameter is given but the youtube api expects GET (with post params)
The Solution is to ether supply the GET argument for the method parameter in python 3.4
request = Request(url, data.encode('utf-8'), method='GET')
or concatenate the url with the urlencoded post data
request = Request(url + '?' + data)
I'm making a page that displays details of an event, and details of any user that has registered an interest in coming to that event. I am using Facebook's Python-SDK (https://github.com/facebook/python-sdk) and I have had no problem getting the access token of the currently logged-in user. However, when the user visits this page of event details, the intention is that they can see names and profile pics of the users involved in this event. I use this code to populate this list of users:
for liftoffer in self.liftoffers:
if (liftoffer.user.key() == dbuser.key()):
self.hasliftoffer = True
self.template_values['myliftoffer'] = liftoffer
logging.warn("Request: fbuser = graph.get_object(%s, fields=\"name, picture, username\")" % str(liftoffer.user.fbid))
logging.warn("access_token: %s" % self.current_user.access_token)
fbuser = graph.get_object(str(liftoffer.user.fbid), fields="name, picture, username")
newuser = ListUser(fbuser['name'], fbuser['picture'], fbuser['username'], liftoffer.user.key())
self.drivers.append(newuser)
The graph has already been defined at this point using:
graph = facebook.GraphAPI(self.current_user.access_token)
Here is the error I get:
ERROR 2011-04-27 17:45:42,007 __init__.py:427] HTTP Error 400: Bad Request
Traceback (most recent call last):
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/webapp/__init__.py", line 634, in __call__
handler.get(*groups)
File "/Users/wadben/Documents/Dev/Python/facebook-python-sdk-322930c/examples/oauth/sp-oauth-local/sparewheels.py", line 83, in get
self.get_secure()
File "/Users/wadben/Documents/Dev/Python/facebook-python-sdk-322930c/examples/oauth/sp-oauth-local/sparewheels.py", line 485, in get_secure
fbuser = graph.get_object(str(liftoffer.user.fbid), fields="name, picture, username")
File "/Users/wadben/Documents/Dev/Python/facebook-python-sdk-322930c/examples/oauth/sp-oauth-local/facebook.py", line 88, in get_object
return self.request(id, args)
File "/Users/wadben/Documents/Dev/Python/facebook-python-sdk-322930c/examples/oauth/sp-oauth-local/facebook.py", line 172, in request
urllib.urlencode(args), post_data)
File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/urllib2.py", line 121, in urlopen
return _opener.open(url, data)
File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/urllib2.py", line 380, in open
response = meth(req, response)
File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/urllib2.py", line 491, in http_response
'http', request, response, code, msg, hdrs)
File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/urllib2.py", line 418, in error
return self._call_chain(*args)
File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/urllib2.py", line 353, in _call_chain
result = func(*args)
File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/urllib2.py", line 499, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 400: Bad Request
The logging lines show me that the request took in a valid Facebook ID and access_token, so I don't understand what is going wrong here. Is it not possible to get the picture and username of other Facebook users apart from the one currently logged in?
http://code.google.com/p/googleappengine/issues/detail?id=4824