Develop Reference

Trying to load a page and cycle through proxies each time

I'm currently trying to learn Python by doing small little silly projects to try and get my head around certain bits but I have hit a bit of a brick wall. I'm wanting to make something that will visit a page using a proxy list I have in a .txt file. I want it to load up the web page with the first proxy in the file, then load up the page with the second proxy and so on. However, I keep on getting this error:
Traceback (most recent call last):
File "c:\Users\Admin.vscode\extensions\ms-python.python-2019.6.24221\pythonFiles\ptvsd_launcher.py", line 43, in
main(ptvsdArgs)
File "c:\Users\Admin.vscode\extensions\ms-python.python-2019.6.24221\pythonFiles\lib\python\ptvsd__main__.py", line 434, in main
run()
File "c:\Users\Admin.vscode\extensions\ms-python.python-2019.6.24221\pythonFiles\lib\python\ptvsd__main__.py", line 312, in run_file
runpy.run_path(target, run_name='main')
File "C:\Users\Admin\AppData\Local\Programs\Python\Python37-32\lib\runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "C:\Users\Admin\AppData\Local\Programs\Python\Python37-32\lib\runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "C:\Users\Admin\AppData\Local\Programs\Python\Python37-32\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "c:\Users\Admin\Documents\PythonScripts\ebay-traffic.py", line 10, in
r = requests.get(url, proxies = line)
File "C:\Users\Admin\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\api.py", line 75, in get
return request('get', url, params=params, **kwargs)
File "C:\Users\Admin\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\api.py", line 60, in request
return session.request(method=method, url=url, **kwargs)
File "C:\Users\Admin\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\sessions.py", line 524, in request
prep.url, proxies, stream, verify, cert
File "C:\Users\Admin\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\sessions.py", line 699, in merge_environment_settings
no_proxy = proxies.get('no_proxy') if proxies is not None else None
AttributeError: 'str' object has no attribute 'get'
The proxy file looks like this:
I've tried various stupid things like putting the proxy file in the int(), but that obviously doesn't work (but I was trying a lot of silly things).
import requests
proxyList = 'proxies.txt'
file = open(proxyList, "r")
url = input('Website: ')
for line in file:
print(line, end="")
r = requests.get(url, proxies = line)
print('Finished.')
input()
I expect it to print each line of the proxy file when it loads up the page when connected to the proxy.

You need to pass proxies as a dict
import requests
proxyList = 'proxies.txt'
file = open(proxyList, "r")
url = input('Website: ')
for line in file:
print(line, end="")
proxies = {'http': line.strip(), 'https': line.strip()}
r = requests.get(url, proxies=proxies)
print('Finished.')
input()

You need to provide the proxies as a dict to python requests, i.e.:
import requests
url = input('Website:\n')
with open('proxies.txt') as f:
proxies = [x.strip() for x in list(f)]
for p in proxies:
r = requests.get(url, proxies={'http': p, 'https': p})
print(r.text)
Demo

Having issues getting firebase data through Oauth pin-auth session for Nest

Base script:
from sanction import Client
# client_id & client_secret are omitted but are valid
client_pin = input('Enter PIN:')
access_token_url = 'https://api.home.nest.com/oauth2/access_token'
c = Client(
token_endpoint=access_token_url,
client_id=client_id,
client_secret=client_secret)
c.request_token(code = client_pin)
[See edits for history]
Running c.request('/devices') returned:
Traceback (most recent call last):
File "C:\py\nest_testing_sanction.py", line 36, in <module>
c.request("/devices")
File "C:\Python34\lib\site-packages\sanction-0.4.1-py3.4.egg\sanction\__init__.py", line 169, in request
File "C:\Python34\lib\site-packages\sanction-0.4.1-py3.4.egg\sanction\__init__.py", line 211, in transport_query
File "C:\Python34\lib\urllib\request.py", line 258, in __init__
self.full_url = url
File "C:\Python34\lib\urllib\request.py", line 284, in full_url
self._parse()
File "C:\Python34\lib\urllib\request.py", line 313, in _parse
raise ValueError("unknown url type: %r" % self.full_url)
ValueError: unknown url type: 'None/devices?access_token=c.[some long session token]'
Given the output it seems like I need to be putting in a generic URL so I tried c.request('wss://developer-api.nest.com'):
Traceback (most recent call last):
File "C:\py\nest_testing_sanction.py", line 36, in <module>
data = c.request(query_url)
File "C:\Python34\lib\site-packages\sanction-0.4.1-py3.4.egg\sanction\__init__.py", line 171, in request
File "C:\Python34\lib\urllib\request.py", line 153, in urlopen
return opener.open(url, data, timeout)
File "C:\Python34\lib\urllib\request.py", line 455, in open
response = self._open(req, data)
File "C:\Python34\lib\urllib\request.py", line 478, in _open
'unknown_open', req)
File "C:\Python34\lib\urllib\request.py", line 433, in _call_chain
result = func(*args)
File "C:\Python34\lib\urllib\request.py", line 1257, in unknown_open
raise URLError('unknown url type: %s' % type)
urllib.error.URLError: <urlopen error unknown url type: nonewss>
I also tried https as per:
- same result
By contrast, this works (for a firebase.io virtual device):
firebase = firebase.FirebaseApplication('https://nesttest.firebaseio.com', None)
thermostat_result = firebase.get('/devices', 'thermostats')

In Python I would use something like sanction to keep things simple. You should be able to get it to work with the Nest API using code like: (untested, using token flow rather than pin flow)
from sanction.client import Client
# instantiating a client to get the auth URI
c = Client(auth_endpoint="https://home.nest.com/login/oauth2",
client_id=config["nest.client_id"])
# instantiating a client to process OAuth2 response
c = Client(token_endpoint="https://api.home.nest.com/oauth2/access_token",
client_id=config["nest.client_id"],
client_secret=config["nest.client_secret"])
The library is well documented, so you should be able to figure it out from here if something is missing.

This is more of a comment, but the system does not let me comment just yet.
To your question about where to put the web pin simply add code = pin to the request_token call.
c.request_token(code = nest_client_pin)
This still does not fully solve the issue as I can only use a PIN once. After I have used it once, every subsequent call will fail again as you describe. Still researching that.

SSL Error occurs on one computer but not the other?

I can't figure out why all of a sudden the below code that uses Asana's API generates the below SSL error. Something must have changed on my laptop, since it runs perfectly on my other computer.
from asana import asana
class Login(object):
def __init__(self):
api = 'API'
self.asana_api = asana.AsanaAPI(api, debug=False)
self.user_id = 7359085011308L
class Test(Login):
def Test(self):
Id = 2467584555313L
print self.asana_api.list_tasks(Id,self.user_id)
Traceback (most recent call last):
File "/Users/Chris/Dropbox/AsanaPullPush.py", line 75, in <module>
if __name__ == "__main__": main()
File "/Users/Chris/Dropbox/AsanaPullPush.py", line 72, in main
print Test().Test()
File "/Users/Chris/Dropbox/AsanaPullPush.py", line 15, in Test
print self.asana_api.list_tasks(Id,self.user_id)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/asana/asana.py", line 174, in list_tasks
return self._asana(target)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/asana/asana.py", line 74, in _asana
r = requests.get(target, auth=(self.apikey, ""))
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/requests/api.py", line 55, in get
return request('get', url, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/requests/api.py", line 44, in request
return session.request(method=method, url=url, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/requests/sessions.py", line 383, in request
resp = self.send(prep, **send_kwargs)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/requests/sessions.py", line 486, in send
r = adapter.send(request, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/requests/adapters.py", line 389, in send
raise SSLError(e)
requests.exceptions.SSLError: [Errno 1] _ssl.c:507: error:0D0890A1:asn1 encoding routines:ASN1_verify:unknown message digest algorithm

We recently changed our SSL key in response to the Heartbleed bug you may have heard about. http://blog.asana.com/2014/04/heartbleed/
It looks like your laptop may not have the right SSL. See https://github.com/pypa/pip/issues/829 for discussion of a similar issue.
You should be able to check SSL version on the two machines with python -c "import ssl; print ssl.OPENSSL_VERSION". If indeed the laptop is behind, you'll need to update your python's SSL.

Freebase python

I tried this python example from freebase and run it in my windows and ubuntu machine.
http://mql.freebaseapps.com/ch04.html
import sys # Command-line arguments, etc.
import simplejson # JSON encoding.
import urllib # URI encoding.
import urllib2 # High-level URL content fetching.
# These are some constants we'll use.
SERVER = 'api.freebase.com' # Metaweb server
SERVICE = '/api/service/mqlread' # Metaweb service
# Compose our MQL query as a Python data structure.
# The query is an array in case multiple bands share the same name.
band = sys.argv[1] # The desired band, from command line.
query = [{'type': '/music/artist', # Our MQL query in Python.
'name': band, # Place the band in the query.
'album': [{ 'name': None, # None is Python's null.
'release_date': None,
'sort': 'release_date' }]}]
# Put the query in an envelope
envelope = {
'query': query, # The query property specifies the query.
'escape': False # Turns off HTML escaping.
}
# These five lines are the key code for using mqlread
encoded = simplejson.dumps(envelope) # JSON encode the envelope.
params = urllib.urlencode({'query':encoded}) # Escape request parameters.
url ='http://%s%s?%s' % (SERVER,SERVICE,params) # The URL to request.
f = urllib2.urlopen(url) # Open the URL as a file.
response = simplejson.load(f) # Read and JSON parse response.
# Check for errors and exit with a message if the query failed.
if response['code'] != '/api/status/ok': # If not okay...
error = response['messages'][0] # First msg object.
sys.exit('%s: %s' % (error['code'], error['message'])) # Display code,msg.
# No errors, so handle the result
result = response['result'] # Open the response envelope, get result.
# Check the number of matching bands
if len(result) == 0:
sys.exit('Unknown band')
elif len(result) > 1:
print "Warning: multiple bands named " + band + ". Listing first only."
result = result[0] # Get first band from array of matches.
if not result['album']: # Exit if band has no albums
sys.exit(band + ' has no known albums.')
for album in result['album']: # Loop through the result albums.
name = album['name'] # Album name.
date = album['release_date'] # Release date timestamp or null.
if not date: date = ''
else: date = ' [%s]' % date[0:4] # Just the 4-digit year in brackets.
print "%s%s" % (name, date) # Print name and date.
However, it did not work. Could someone explain it why i got this error?
Traceback (most recent call last):
File "mql.py", line 29, in <module>
f = urllib2.urlopen(url) # Open the URL as a file.
File "C:\Python27\lib\urllib2.py", line 126, in urlopen
return _opener.open(url, data, timeout)
File "C:\Python27\lib\urllib2.py", line 394, in open
response = self._open(req, data)
File "C:\Python27\lib\urllib2.py", line 412, in _open
'_open', req)
File "C:\Python27\lib\urllib2.py", line 372, in _call_chain
result = func(*args)
File "C:\Python27\lib\urllib2.py", line 1199, in http_open
return self.do_open(httplib.HTTPConnection, req)
File "C:\Python27\lib\urllib2.py", line 1168, in do_open
h.request(req.get_method(), req.get_selector(), req.data, headers)
File "C:\Python27\lib\httplib.py", line 955, in request
self._send_request(method, url, body, headers)
File "C:\Python27\lib\httplib.py", line 989, in _send_request
self.endheaders(body)
File "C:\Python27\lib\httplib.py", line 951, in endheaders
self._send_output(message_body)
File "C:\Python27\lib\httplib.py", line 811, in _send_output
self.send(msg)
File "C:\Python27\lib\httplib.py", line 773, in send
self.connect()
File "C:\Python27\lib\httplib.py", line 754, in connect
self.timeout, self.source_address)
File "C:\Python27\lib\socket.py", line 562, in create_connection
sock.connect(sa)
One more thing, I am using python 2.7 and i don't have any special proxy configuration for the internet setting.

api.freebase.com has been retired and the Python client library was never updated to work with the new endpoint.

It would be better if you can enclose your code within a try-except block as below:
try:
#all the logic to create a URL and store in a variable say url for MQLReader service
invoke urllib2.urlopen(url)
#perform other processing.
except urllib2.URLError, e:
print str(e)
This will give you an idea about what is going wrong.
I tried your code sample and I get an error saying that:
A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
Hence catching the exception and examining it will give you the root cause of the problem.

Can't get Custom DNS server working in Python

I'm having real trouble getting python to use a custom dns server.
I have followed this Tell urllib2 to use custom DNS
If I don't specify a self.host and self.port, it will go through without blocking.
Here is the code:
import urllib2
import httplib
import socket
class MyHTTPConnection (httplib.HTTPConnection):
def connect (self):
if self.host == 'www.porn.com':
self.host = '208.67.222.123' #OpenDNS FamilyShield
self.port = 53
self.sock = socket.create_connection ((self.host, self.port))
class MyHTTPHandler (urllib2.HTTPHandler):
def http_open (self, req):
return self.do_open (MyHTTPConnection, req)
opener = urllib2.build_opener(MyHTTPHandler)
urllib2.install_opener (opener)
f = urllib2.urlopen ('http://www.porn.com/videos/anime-toon.html')
data = f.read ()
print data
I keep getting a "raise BadStatusLine(line)" error
Error log:
Traceback (most recent call last):
File "K:\Desktop\rte\dns2.py", line 16, in <module>
f = urllib2.urlopen ('http://www.porn.com/videos/anime-toon.html')
File "C:\Python27\lib\urllib2.py", line 126, in urlopen
return _opener.open(url, data, timeout)
File "C:\Python27\lib\urllib2.py", line 394, in open
response = self._open(req, data)
File "C:\Python27\lib\urllib2.py", line 412, in _open
'_open', req)
File "C:\Python27\lib\urllib2.py", line 372, in _call_chain
result = func(*args)
File "K:\Desktop\rte\dns2.py", line 12, in http_open
return self.do_open (MyHTTPConnection, req)
File "C:\Python27\lib\urllib2.py", line 1170, in do_open
r = h.getresponse(buffering=True)
File "C:\Python27\lib\httplib.py", line 1027, in getresponse
response.begin()
File "C:\Python27\lib\httplib.py", line 407, in begin
version, status, reason = self._read_status()
File "C:\Python27\lib\httplib.py", line 371, in _read_status
raise BadStatusLine(line)
BadStatusLine: ''
EDIT: Going on isedev response, that I was going about it the wrong way.
It doesn't seem to register with urllib2 the changes to the namesservers
import dns.resolver
import urllib2
resolver = dns.resolver.Resolver()
resolver.nameservers = ['208.67.222.123']
answer = resolver.query('www.porn.com','A')
web_url = 'http://www.porn.com/videos/anime-toon.html'
req1 = urllib2.Request(web_url)
req1.add_header('User-Agent', 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.0.3) Gecko/2008092417 Firefox/3.0.3')
response1 = urllib2.urlopen(req1)
html=response1.read()
print html

I think you've misunderstood what's being done in the "Custom DNS" answer you refer to. The example given in that solution is not in fact setting up a custom DNS server - the MyResolver class is given as example only and performs a hard-coded name-to-IP for 'news.bbc.co.uk'.
So what your code is actually doing is redirecting an HTTP request to 'www.porn.com' (port 80) to the OpenDNS Family Shield DNS server (on port 53)... which will obviously lead to the error you're getting.
So what you need to do is replace:
if self.host == 'www.porn.com':
self.host = '208.67.222.123' #OpenDNS FamilyShield
self.port = 53
with code that actually resolves 'www.porn.com' against the chosen DNS server directly (using dnspython for instance).
Assuming you've got the dnspython package installed, you could do something like:
import urllib2
import httplib
import socket
import dns.resolver
class MyHTTPConnection (httplib.HTTPConnection):
def connect (self):
if self.host == 'www.porn.com':
resolver = dns.resolver.Resolver()
resolver.nameservers = ['208.67.222.123']
answer = resolver.query(self.host,'A')
self.host = answer.rrset.items[0].address
self.sock = socket.create_connection ((self.host, self.port))
class MyHTTPHandler (urllib2.HTTPHandler):
def http_open (self, req):
return self.do_open (MyHTTPConnection, req)
opener = urllib2.build_opener(MyHTTPHandler)
urllib2.install_opener (opener)
f = urllib2.urlopen ('http://www.porn.com/videos/anime-toon.html')
data = f.read ()
print data
This code returns '404 - not found' and network trace shows HTTP request to 'hit-adult.opendns.com', which is what 'www.porn.com' resolves to when using the '208.67.222.123' nameserver:
dig #208.67.222.123 www.porn.com A
;; ANSWER SECTION:
www.porn.com. 0 IN A 67.215.65.130
nslookup 67.215.65.130
130.65.215.67.in-addr.arpa name = hit-adult.opendns.com.
The above is an example only. Real code would require error checking, etc...