Develop Reference

Python requests-html with Tor

The requirement is to scrap anonymously or change ip after certain number of calls. I use the https://github.com/kennethreitz/requests-html module to parse the HTML, but i get the below error,
socks.SOCKS5Error: 0x01: General SOCKS server failure
Code
import socks
import socket
import requests_html
socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5, addr='127.0.0.1', port=int('9150'))
socket.socket = socks.socksocket
session = requests_html.HTMLSession()
r = session.get('http://icanhazip.com')
r.html.render(sleep=5)
print(r.html.text)
But it works perfectly fine with requests module,
import socks
import socket
import requests
socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5, addr='127.0.0.1', port=int('9150'))
socket.socket = socks.socksocket
print(requests.get("http://icanhazip.com").text)
Any help to solve the issue with requests-html module would be highly appreciated.

Try:
session = requests_html.HTMLSession(browser_args=["--no-sandbox","--proxy-server=127.0.0.1:9150"])
Depends on how your proxy is set up to use tor but this worked for me!

AttributeError: type object 'socket' has no attribute 'socket'

import socket
from socket import *
conn = socket.socket(socket.AF_PACKET, socket.SOCK_RAW, socket.ntohs(3))
I'm facing this error:
AttributeError: type object 'socket' has no attribute 'socket'

If you import all library don't need to name library before method:
from socket import *
socket.socket # AttributeError: type object 'socket' has no attribute 'socket'
from socket import *
socket # <class 'socket.socket'>
or maybe :
import socket
socket.socket # <class 'socket.socket'>

import socket
server_socket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
print("Socket is created")
I fixed these changes my codes and work.

You've imported socket and then imported * from socket.
As socket has a socket function in it python is likely getting confused and using the socket inside socket when you're calling socket.socket.
I would drop the from socket import * and work from there.

socket(AF_INET, SOCK_DGRAM)
python is a little simpler

"If you import all library don't need to name library before method:"
Option 1 - import library:
import socket
conn = socket.socket( socket.AF_INET, socket.SOCK_STREAM )
Option 2 - from library import all:
from socket import *
conn = socket( AF_INET, SOCK_STREAM )

Socket with Proxy in Python 3.4

I have a code that I want to pass all packages through the proxy, how can I do it using:
import socket
import #Here your socket library
# The rest of the code here
Which library do you recommend me to use with Socket? Is it really possible what I'm saying?

Using Socksipy in Python 3.4 I can let the code with 3 more lines that are:
import socksipy
s=socket.socket( )
s = socks.socksocket()
s.setproxy(socks.PROXY_TYPE_SOCKS5, 'localhost', 9050)
s.connect((HOST, PORT))
Socksipy can be found at: http://socksipy.sourceforge.net/ and its very simple to use, you just need to read their wiki so than you can work with. The URL for the wiki is: https://code.google.com/p/socksipy-branch/

Stem as python tor client - stuck on loading descriptors

I'm trying to connect to tor with python stem, while trying to connect (using th emodified example) it just won't work...here's my code:
(I'm using python 3.4.1)
import socket,urllib, sys, socks, stem.process
from stem.util import term
SOCKS_PORT = 7000
# Set socks proxy and wrap the urllib module
socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5, '127.0.0.1', SOCKS_PORT) socket.socket = socks.socksocket
# Perform DNS resolution through the socket
def getaddrinfo(*args): return [(socket.AF_INET, socket.SOCK_STREAM, 6, '', (args[0], args[1]))]
socket.getaddrinfo = getaddrinfo
def query(url): """ Uses urllib to fetch a site using SocksiPy for Tor over the SOCKS_PORT. """
try:
return urllib.urlopen(url).read() except:
return "Unable to reach %s" % url
def print_bootstrap_lines(line): if "Bootstrapped " in line:
print(term.format(line, term.Color.BLUE))
print(term.format("Starting Tor:\n", term.Attr.BOLD))
tor_process = stem.process.launch_tor_with_config( tor_cmd = "C:\Users\Nadav\Desktop\Tor Browser\Tor\\tor.exe" , config = {
'SocksPort': str(SOCKS_PORT),
'ExitNodes': '{ru}', }, init_msg_handler = print_bootstrap_lines, )
print(term.format("\nChecking our endpoint:\n", term.Attr.BOLD)) print(term.format(query("https://www.atagar.com/echo.php"), term.Color.BLUE))
tor_process.kill

The socks port can be different from the port to manipulate tor using stem / stem.control
Hopefully this helps to get things working for you:
import requesocks as requests
from stem import Signal
from stem.control import Controller
# proxies for requests
proxies = {'http': 'socks5://127.0.0.1:9150',
'https': 'socks5://127.0.0.1:9150'}
# when using the Controller
with Controller.from_port(port=9151) as controller:
controller.authenticate()
controller.signal(Signal.NEWNYM)
Notice that the port for sock is different from the port for the Controller. You can find the port for the controller in your torrc file (for me it was called torrc-defaults).
Looks something like this:
# Bind to this address to listen to connections from SOCKS-speaking
# applications.
SocksPort 9150
ControlPort 9151
Hope this helps!

Python urllib over TOR? [duplicate]

This question already has answers here:
How to route urllib requests through the TOR network? [duplicate]
(3 answers)
Closed 7 years ago.
Sample code:
#!/usr/bin/python
import socks
import socket
import urllib2
socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS4, "127.0.0.1", 9050, True)
socket.socket = socks.socksocket
print urllib2.urlopen("http://almien.co.uk/m/tools/net/ip/").read()
TOR is running a SOCKS proxy on port 9050 (its default). The request goes through TOR, surfacing at an IP address other than my own. However, TOR console gives the warning:
"Feb 28 22:44:26.233 [warn] Your
application (using socks4 to port 80)
is giving Tor only an IP address.
Applications that do DNS resolves
themselves may leak information.
Consider using Socks4A (e.g. via
privoxy or socat) instead. For more
information, please see
https://wiki.torproject.org/TheOnionRouter/TorFAQ#SOCKSAndDNS."
i.e. DNS lookups aren't going through the proxy. But that's what the 4th parameter to setdefaultproxy is supposed to do, right?
From http://socksipy.sourceforge.net/readme.txt:
setproxy(proxytype, addr[, port[, rdns[, username[, password]]]])
rdns - This is a boolean flag than
modifies the behavior regarding DNS
resolving. If it is set to True, DNS
resolving will be preformed remotely,
on the server.
Same effect with both PROXY_TYPE_SOCKS4 and PROXY_TYPE_SOCKS5 selected.
It can't be a local DNS cache (if urllib2 even supports that) because it happens when I change the URL to a domain that this computer has never visited before.

The problem is that httplib.HTTPConnection uses the socket module's create_connection helper function which does the DNS request via the usual getaddrinfo method before connecting the socket.
The solution is to make your own create_connection function and monkey-patch it into the socket module before importing urllib2, just like we do with the socket class.
import socks
import socket
def create_connection(address, timeout=None, source_address=None):
sock = socks.socksocket()
sock.connect(address)
return sock
socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5, "127.0.0.1", 9050)
# patch the socket module
socket.socket = socks.socksocket
socket.create_connection = create_connection
import urllib2
# Now you can go ahead and scrape those shady darknet .onion sites

The problem is that you are importing urllib2 before you set up the socks connection.
Try this instead:
import socks
import socket
socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS4, '127.0.0.1', 9050, True)
socket.socket = socks.socksocket
import urllib2
print urllib2.urlopen("http://almien.co.uk/m/tools/net/ip/").read()
Manual request example:
import socks
import urlparse
SOCKS_HOST = 'localhost'
SOCKS_PORT = 9050
SOCKS_TYPE = socks.PROXY_TYPE_SOCKS5
url = 'http://www.whatismyip.com/automation/n09230945.asp'
parsed = urlparse.urlparse(url)
socket = socks.socksocket()
socket.setproxy(SOCKS_TYPE, SOCKS_HOST, SOCKS_PORT)
socket.connect((parsed.netloc, 80))
socket.send('''GET %(uri)s HTTP/1.1
host: %(host)s
connection: close
''' % dict(
uri=parsed.path,
host=parsed.netloc,
))
print socket.recv(1024)
socket.close()

I've published an article with complete source code showing how to use urllib2 + SOCKS + Tor on http://blog.databigbang.com/distributed-scraping-with-multiple-tor-circuits/
Hope it solves your issues.