Develop Reference

urllib.request on Python 3 failed to get certificate

I am writing a program which can read json of ipsw.me in python3:
import urrlib.request as ur
url = "https://api.ipsw.me/v4/device/" + device
json_file = ur.urlopen(url)
with open("signedipsw.json",'wb') as output:
output.write(json_file.read())
data = json.load(open("signedipsw.json"))
i = 0
with open("signedipsw.json"):
print("signed firmwares for %s:" % device)
for i in range(0, len(data["firmwares"])):
if data["firmwares"][i]["signed"] == True :
print("%s - %s" % (data["firmwares"][i]["version"], data["firmwares"][i]["buildid"]))
i+=1
I used to write this in Python2 but I used urrlib2 module and it worked perfectly. Now, on Python3 I use urrlib.request but I received an error:
json_file = ur.urlopen(url)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/urllib/request.py", line 214, in urlopen
return opener.open(url, data, timeout)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/urllib/request.py", line 517, in open
response = self._open(req, data)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/urllib/request.py", line 534, in _open
result = self._call_chain(self.handle_open, protocol, protocol +
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/urllib/request.py", line 494, in _call_chain
result = func(*args)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/urllib/request.py", line 1385, in https_open
return self.do_open(http.client.HTTPSConnection, req,
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/urllib/request.py", line 1345, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)>
How can I fix this? Thanks

It looks like urllib (not urrlib) is not able to verify you're talking to the real ipsw.me. They're using Cloudflare's certificates.
You'll have a much better time using the Requests module. It comes with a more recent certificate bundle, which will likely fix your problem.
import requests
device = "..."
resp = requests.get(f"https://api.ipsw.me/v4/device/{device}")
resp.raise_for_status()
data = resp.json()
print(f"signed firmwares for {device}:")
for fw in data["firmwares"]:
if fw["signed"]:
print(f"{fw['version']} - {fw['buildid']}")

Having issues getting firebase data through Oauth pin-auth session for Nest

Base script:
from sanction import Client
# client_id & client_secret are omitted but are valid
client_pin = input('Enter PIN:')
access_token_url = 'https://api.home.nest.com/oauth2/access_token'
c = Client(
token_endpoint=access_token_url,
client_id=client_id,
client_secret=client_secret)
c.request_token(code = client_pin)
[See edits for history]
Running c.request('/devices') returned:
Traceback (most recent call last):
File "C:\py\nest_testing_sanction.py", line 36, in <module>
c.request("/devices")
File "C:\Python34\lib\site-packages\sanction-0.4.1-py3.4.egg\sanction\__init__.py", line 169, in request
File "C:\Python34\lib\site-packages\sanction-0.4.1-py3.4.egg\sanction\__init__.py", line 211, in transport_query
File "C:\Python34\lib\urllib\request.py", line 258, in __init__
self.full_url = url
File "C:\Python34\lib\urllib\request.py", line 284, in full_url
self._parse()
File "C:\Python34\lib\urllib\request.py", line 313, in _parse
raise ValueError("unknown url type: %r" % self.full_url)
ValueError: unknown url type: 'None/devices?access_token=c.[some long session token]'
Given the output it seems like I need to be putting in a generic URL so I tried c.request('wss://developer-api.nest.com'):
Traceback (most recent call last):
File "C:\py\nest_testing_sanction.py", line 36, in <module>
data = c.request(query_url)
File "C:\Python34\lib\site-packages\sanction-0.4.1-py3.4.egg\sanction\__init__.py", line 171, in request
File "C:\Python34\lib\urllib\request.py", line 153, in urlopen
return opener.open(url, data, timeout)
File "C:\Python34\lib\urllib\request.py", line 455, in open
response = self._open(req, data)
File "C:\Python34\lib\urllib\request.py", line 478, in _open
'unknown_open', req)
File "C:\Python34\lib\urllib\request.py", line 433, in _call_chain
result = func(*args)
File "C:\Python34\lib\urllib\request.py", line 1257, in unknown_open
raise URLError('unknown url type: %s' % type)
urllib.error.URLError: <urlopen error unknown url type: nonewss>
I also tried https as per:
- same result
By contrast, this works (for a firebase.io virtual device):
firebase = firebase.FirebaseApplication('https://nesttest.firebaseio.com', None)
thermostat_result = firebase.get('/devices', 'thermostats')

In Python I would use something like sanction to keep things simple. You should be able to get it to work with the Nest API using code like: (untested, using token flow rather than pin flow)
from sanction.client import Client
# instantiating a client to get the auth URI
c = Client(auth_endpoint="https://home.nest.com/login/oauth2",
client_id=config["nest.client_id"])
# instantiating a client to process OAuth2 response
c = Client(token_endpoint="https://api.home.nest.com/oauth2/access_token",
client_id=config["nest.client_id"],
client_secret=config["nest.client_secret"])
The library is well documented, so you should be able to figure it out from here if something is missing.

This is more of a comment, but the system does not let me comment just yet.
To your question about where to put the web pin simply add code = pin to the request_token call.
c.request_token(code = nest_client_pin)
This still does not fully solve the issue as I can only use a PIN once. After I have used it once, every subsequent call will fail again as you describe. Still researching that.

Freebase python

I tried this python example from freebase and run it in my windows and ubuntu machine.
http://mql.freebaseapps.com/ch04.html
import sys # Command-line arguments, etc.
import simplejson # JSON encoding.
import urllib # URI encoding.
import urllib2 # High-level URL content fetching.
# These are some constants we'll use.
SERVER = 'api.freebase.com' # Metaweb server
SERVICE = '/api/service/mqlread' # Metaweb service
# Compose our MQL query as a Python data structure.
# The query is an array in case multiple bands share the same name.
band = sys.argv[1] # The desired band, from command line.
query = [{'type': '/music/artist', # Our MQL query in Python.
'name': band, # Place the band in the query.
'album': [{ 'name': None, # None is Python's null.
'release_date': None,
'sort': 'release_date' }]}]
# Put the query in an envelope
envelope = {
'query': query, # The query property specifies the query.
'escape': False # Turns off HTML escaping.
}
# These five lines are the key code for using mqlread
encoded = simplejson.dumps(envelope) # JSON encode the envelope.
params = urllib.urlencode({'query':encoded}) # Escape request parameters.
url ='http://%s%s?%s' % (SERVER,SERVICE,params) # The URL to request.
f = urllib2.urlopen(url) # Open the URL as a file.
response = simplejson.load(f) # Read and JSON parse response.
# Check for errors and exit with a message if the query failed.
if response['code'] != '/api/status/ok': # If not okay...
error = response['messages'][0] # First msg object.
sys.exit('%s: %s' % (error['code'], error['message'])) # Display code,msg.
# No errors, so handle the result
result = response['result'] # Open the response envelope, get result.
# Check the number of matching bands
if len(result) == 0:
sys.exit('Unknown band')
elif len(result) > 1:
print "Warning: multiple bands named " + band + ". Listing first only."
result = result[0] # Get first band from array of matches.
if not result['album']: # Exit if band has no albums
sys.exit(band + ' has no known albums.')
for album in result['album']: # Loop through the result albums.
name = album['name'] # Album name.
date = album['release_date'] # Release date timestamp or null.
if not date: date = ''
else: date = ' [%s]' % date[0:4] # Just the 4-digit year in brackets.
print "%s%s" % (name, date) # Print name and date.
However, it did not work. Could someone explain it why i got this error?
Traceback (most recent call last):
File "mql.py", line 29, in <module>
f = urllib2.urlopen(url) # Open the URL as a file.
File "C:\Python27\lib\urllib2.py", line 126, in urlopen
return _opener.open(url, data, timeout)
File "C:\Python27\lib\urllib2.py", line 394, in open
response = self._open(req, data)
File "C:\Python27\lib\urllib2.py", line 412, in _open
'_open', req)
File "C:\Python27\lib\urllib2.py", line 372, in _call_chain
result = func(*args)
File "C:\Python27\lib\urllib2.py", line 1199, in http_open
return self.do_open(httplib.HTTPConnection, req)
File "C:\Python27\lib\urllib2.py", line 1168, in do_open
h.request(req.get_method(), req.get_selector(), req.data, headers)
File "C:\Python27\lib\httplib.py", line 955, in request
self._send_request(method, url, body, headers)
File "C:\Python27\lib\httplib.py", line 989, in _send_request
self.endheaders(body)
File "C:\Python27\lib\httplib.py", line 951, in endheaders
self._send_output(message_body)
File "C:\Python27\lib\httplib.py", line 811, in _send_output
self.send(msg)
File "C:\Python27\lib\httplib.py", line 773, in send
self.connect()
File "C:\Python27\lib\httplib.py", line 754, in connect
self.timeout, self.source_address)
File "C:\Python27\lib\socket.py", line 562, in create_connection
sock.connect(sa)
One more thing, I am using python 2.7 and i don't have any special proxy configuration for the internet setting.

api.freebase.com has been retired and the Python client library was never updated to work with the new endpoint.

It would be better if you can enclose your code within a try-except block as below:
try:
#all the logic to create a URL and store in a variable say url for MQLReader service
invoke urllib2.urlopen(url)
#perform other processing.
except urllib2.URLError, e:
print str(e)
This will give you an idea about what is going wrong.
I tried your code sample and I get an error saying that:
A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
Hence catching the exception and examining it will give you the root cause of the problem.

Python authentication through PasswordMgr still get 401

So I was trying to dive into Python because I want to write an Agent for the Plex Media Server. This Agent will access the MyAnimeList.net API with HTTP Authentication (more about that here) my Username and passwords work but I don't have a clue why I still get a 401 error from the server as response.
Here is some code (I'm using python 2.5 because plex said so) :)
import urllib2
username = "someUser"
password = "somePass"
url = "http://myanimelist.net/api/anime/search.xml?q=bleach"
password_mgr = urllib2.HTTPPasswordMgrWithDefaultRealm()
top_level_url = "http://myanimelist.net/"
password_mgr.add_password(None, top_level_url, username, password)
handler = urllib2.HTTPBasicAuthHandler(password_mgr)
opener = urllib2.build_opener(urllib2.HTTPHandler, handler)
request = urllib2.Request(url)
print request.get_full_url()
f = urllib2.urlopen(request).read()
print(f)
and this is what I get as response
> http://myanimelist.net/api/anime/search.xml?q=bleach Traceback (most
> recent call last): File
> "C:\Users\Daraku\Desktop\MAL.bundle\Contents\Code\__init__.py", line
> 16, in <module>
> f = urllib2.urlopen(request).read() File "C:\Python25\lib\urllib2.py", line 121, in urlopen
> return _opener.open(url, data) File "C:\Python25\lib\urllib2.py", line 380, in open
> response = meth(req, response) File "C:\Python25\lib\urllib2.py", line 491, in http_response
> 'http', request, response, code, msg, hdrs) File "C:\Python25\lib\urllib2.py", line 418, in error
> return self._call_chain(*args) File "C:\Python25\lib\urllib2.py", line 353, in _call_chain
> result = func(*args) File "C:\Python25\lib\urllib2.py", line 499, in http_error_default
> raise HTTPError(req.get_full_url(), code, msg, hdrs, fp) HTTPError: HTTP Error 401: Unauthorized
Bear in mind that this is my first time programming in Python and im kind of confused with many examples in the web because they did something with the urllib2 so that it doesn't exists in python 3.0, i think, anymore
Any ideas? Or any better ways to do this?

I'm getting an error when trying to open a website url with Python 3.1, urllib & json: operation was attempted on something that is not a socket

I'm getting an error when trying to open a website url with Python 3.1, urllib & json
urllib.error.URLError:
Here's the code. The first website loads fine. The second one
import json
import urllib.request
import urllib.parse
import util
# This one works fine
response = urllib.request.urlopen('http://python.org/')
html = response.read()
print(html)
# parms - CSV filename, company, ....
p_filename = "c:\\temp\\test.csv"
jg_token = "zzzzzzzzzzzzzzzzzzzzzzzzz"
jg_proto = "https://"
jg_webst = "www.jigsaw.com/rest/"
jg_cmd_searchContact = "searchContact.json"
jg_key_companyName = "companyName"
jg_key_levels = "levels"
jg_key_departments = "departments"
jg_args = {
"token":jg_token,
jg_key_companyName: "Technical Innovations",
jg_key_departments: "HR"
}
jg_url = jg_proto + jg_webst + jg_cmd_searchContact + "?" + urllib.parse.urlencode(jg_args)
# This one generates teh error
result = json.load(urllib.request.urlopen(jg_url))
urllib.error.URLError:
File "c:\dev\xdev\PyJigsaw\searchContact.py", line 46, in
result = json.load(urllib.request.urlopen(jg_url))
File "c:\dev\tdev\Python31\Lib\urllib\request.py", line 121, in urlopen
return _opener.open(url, data, timeout)
File "c:\dev\tdev\Python31\Lib\urllib\request.py", line 349, in open
response = self._open(req, data)
File "c:\dev\tdev\Python31\Lib\urllib\request.py", line 367, in _open
'_open', req)
File "c:\dev\tdev\Python31\Lib\urllib\request.py", line 327, in _call_chain
result = func(*args)
File "c:\dev\tdev\Python31\Lib\urllib\request.py", line 1098, in https_open
return self.do_open(http.client.HTTPSConnection, req)
File "c:\dev\tdev\Python31\Lib\urllib\request.py", line 1075, in do_open
raise URLError(err)

Please edit the title and tags and maybe even the question body:
This has nothing to do with JSON and everything to do with Windows. It's also at a lower level than urllib. (Probably in the SSL code.) Distilled:
Both of the following approaches fail on Python 3.1.2 for Vista, but work fine on Linux (Python 3.1.3)
print( HTTPSConnection(hostname).request('GET',url).getresponse().read() )
print( urllib.request.urlopen('https://'+hostname+url).read() )
Change them to not use SSL, and then they work fine on Windows:
print( HTTPConnection(hostname).request('GET',url).getresponse().read() )
print( urllib.request.urlopen('http://'+hostname+url).read() )

On Vista, I just upgraded from Python 3.1.2 to Python 3.2 and this is no longer a problem.
The following now works just fine:
print( urllib.request.urlopen('https://'+hostname+url).read() )