I am trying to fetch data from quandl using urllib2.Please check code below.
import json
from pymongo import MongoClient
import urllib2
import requests
import ssl
#import quandl
codes = [100526];
for id in codes:
url = 'https://www.quandl.com.com//api/v3/datasets/AMFI/"+str(id)+".json?api_key=XXXXXXXX&start_date=2013-08-30'
req = urllib2.Request(url)
response = urllib2.urlopen(req)
data = response.read()
print data
OR
for id in codes:
url = "https://www.quandl.com.com//api/v3/datasets/AMFI/"+str(id)+".json?api_key=XXXXXXXX&start_date=2013-08-30"
request = requests.get(url,verify=False)
print request
I am getting HTTPERROR exception 404 in 1st case. and when I use request module I get SSL error even after using verify=false. I am looking through previous posts but most of them are related to HTTP request.
Thanks for help.
J
This is working for me, but you get a warning about the SSL certificate but you don't need to care about it.
import requests
codes = [100526];
for id in codes:
url = "https://www.quandl.com.com//api/v3/datasets/AMFI/"+str(id)+".json?api_key=XXXXXXXX&start_date=2013-08-30"
request = requests.get(url, verify=False)
print request.text
request.text has your response data.
You seem to be using a wrong URL (.com.com instead of .com) as well as a combination of different quotes in the first version of your code. Use the following instead and it should work:
import urllib2
import requests
codes = [100526]
for id in codes:
url = "https://www.quandl.com//api/v3/datasets/AMFI/"+str(id)+".json?start_date=2013-08-30"
req = urllib2.Request(url)
response = urllib2.urlopen(req)
print response.read()
for id in codes:
url = "https://www.quandl.com//api/v3/datasets/AMFI/"+str(id)+".json?start_date=2013-08-30"
response = requests.get(url,verify=False)
print response.text
To disable the warning about the SSL certificate, use the following code before making the request using requests:
from requests.packages.urllib3.exceptions import InsecureRequestWarning
requests.packages.urllib3.disable_warnings(InsecureRequestWarning)
Related
I'm trying to download automatically the first image which appears in the google image search but I'm not able to read the website source and an error occurs ("HTTP Error 403: Forbidden").
Any ideas? Thank you for your help!
That's my code:
from urllib.request import urlopen
from bs4 import BeautifulSoup
word = 'house'
r = urlopen('https://www.google.pl/search?&dcr=0&tbm=isch&q='+word)
data = r.read()
Apparently you have to pass the headers argument because the website is blocking you thinking you are a bot requesting data. I found an example of doing this here HTTP error 403 in Python 3 Web Scraping.
Also, the urlopen object didn't support the headers argument, so I had to use the Request object instead.
from urllib.request import urlopen, Request
from bs4 import BeautifulSoup
word = 'house'
r = Request('https://www.google.pl/search?&dcr=0&tbm=isch&q='+word, headers={'User-Agent': 'Mozilla/5.0'})
response = urlopen(r).read()
disclaimer- noob here!
i keep running into the issue that says I imported requests but am not using it. not sure how to proceed. I looked in my package library and it is there under site-packages, however I do not see the 'get' function in it. I am using PyCharm, and have also checked this Spyder. Same problem
import json
import requests
from requests import get as geturl
url = 'http://api.open-notify.org/iss-now.json'
response = geturl(url)
print(response)
result = json.loads(response.text)
print(result('iss_position'))
You are not using the name requests. You can safely remove the import requests line. Your from requests import get as geturl line suffices.
As a side note, you can load JSON responses directly without having to import json:
result = response.json()
so all you need is
from requests import get as geturl
url = 'http://api.open-notify.org/iss-now.json'
response = geturl(url)
result = response.json()
print(result['iss_position'])
>>> from requests import get as geturl
>>> url = 'http://api.open-notify.org/iss-now.json'
>>> response = geturl(url)
>>> result = response.json()
>>> print(result)
{'message': 'success', 'iss_position': {'latitude': '-40.8208', 'longitude': '103.2043'}, 'timestamp': 1509893252}
You can either use
import requests
requests.get(url, params=None, **kwargs)
Or
from requests import get as geturl
geturl(url, params=None, **kwargs)
You have used both, which is redundent and the requests is not being used as requests.get() that's was the reason why you were getting that message.
When I call the socialcast api from python 2.7 using requests, I get a response "reason" but I don't see that text in my actual API. Here's my code:
import requests
parameters = {"username" = "myUsername", "password" = "myPassword"}
response = requests.get("https://hub.sas.com/api/groups/808/messages.json", parameters)
response.json()
The beginning of the JSON that I'm passing through is this:
{"messages":[{"id":126433,"user":{"id":4468,"name":
So I would expect something else to come back, but what it returns is:
{u'reason': u''}
Is this an error or is there something I'm not understanding?
I solved my problem by using this code:
import requests
from requests.auth import HTTPBasicAuth
r = requests.get('https://hub.sas.com/api/groups/808/messages.json', auth=HTTPBasicAuth('username', 'password'))
data = r.json()
for message in data['messages']:
print(message['user']['name'])
I'm not sure that the from requests.auth import HTTPBasicAuth or the data = r.json() were necessary but it ended up working for me so I left them in there.
I'm trying to use python urllib2 to access the files in a SharePoint 2013 folder using the SharePoint REST API. https://msdn.microsoft.com/en-us/library/office/dn450841.aspx
I can't seem to get the response as a json object even after adding the header 'accept':'application/json;odata=verbose'.
I am using Python 2.7.5, urllib2, and can not use the python requests module due to restrictions and permission constraints.
If I use the .../_api/web/GetFolderByServerRelativeUrl('/File Name')/Files url I get an AttributeError("'str' object has no attribute'read'"). If I just use the normal url to the site I get an error that there is not a 'JSON object'in the response.
I don't know what I am doing wrong and why I can't get at any json from this request?
import sys
import urllib
import urllib2
import json
from ntlm import HTTPNtlmAuthHandler
try:
url = 'https://sharepointsite/subfolder/subfolder/subfolder/Forms/AllItems.aspx/_api/web/GetFolderByServerRelativeUrl(/Folder Name)/Files'
user = r'DOMAIN\USER'
password = '*********'
req = urllib2.Request(url)
req.add_header('accept', 'application/json;odata=verbose')
passman = urllib2.HTTPPasswordMgrWithDefaultRealm()
passman.add_password(None, url, user, password)
handler = HTTPNtlmAuthHandler.HTTPNtlmAuthHandler(passman)
opener = urllib2.build_opener(handler)
urllib2.install_opener(opener)
response = urllib2.urlopen(req)
data_json = json.load(response.read())
except Exception as e:
print '{0}'.format(e)
What the request actually returns is a string representation of the json object.
You can use use the json.loads() function instead of the json.load() function in order to convert the string into a json object.
I'm trying to use github oauth. I'n using urllib and urllib2 and have this code:
def github_login(request):
post_data = [('client_id','****'),('redirect_uri','http://localhost:8000/callback')]
result = urllib2.urlopen('https://github.com/login/oauth/authorize', urllib.urlencode(post_data))
content = result.read()
And after sending query I have httperror 403. I had already configured allowed_hosts in settings.py
From my expirence I know that working with urllib is rly hard, I would suggest to use requests
http://requests.readthedocs.org/en/latest/
You can easly send the get:
r = requests.get('https://api.github.com/user', auth=('user', 'pass'))