I'm trying to request things using TOR, first through HTTP and then HTTPS.
I tried really all python libraries for TOR, and ALL of them has the same proble, the HTTPS connection.
My problem is not the real communication with HTTPS, is that if i try to get new TOR ip, with HTTP request it refresh the IP address, but non the same with HTTPS ! So, i'm still using the TOR network, but always with the same ip ..
from stem import Signal
from stem.control import Controller
import requests
import time
import os
session = requests.session()
# signal TOR for a new connection
session.proxies = {'http': 'socks5://127.0.0.1:9150',
'https': 'socks5://127.0.0.1:9150'}
def new_connection():
with Controller.from_port(port = 9151) as controller:
controller.authenticate(password="####")
# NUOVA CONNESSIONE
print 'Trying to connect with new IP...'
controller.signal(Signal.NEWNYM)
time.sleep(5) #5 sec
print "New connection as "+session.get("http://ipecho.net/plain").text
response = session.get("https://www.google.it/").text
print response
One more thing ..
I was using the tor version for OSX, but i can't get the control port for it, so i'm using the Tor Browser (9150) with the control port 9151
When you get a new tor identity, you may still have the same ip address, cuz tor only have certain amount of service You can try something like this:
def get_new_tor_ip():
try:
ip_current = session.get("http://ipecho.net/plain").text
except:
ip_current = OWN_IP
ip_old = ip_current
socket.socket = OWN_SOCKET
with Controller.from_port(address='127.0.0.1', port=9051) as controller:
controller.authenticate() # optional
socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5, "127.0.0.1", 9050)
socket.socket = socks.socksocket
while True:
controller.signal(Signal.NEWNYM)
try:
ip_current = str(session.get("http://ipecho.net/plain").text
if ip_current != (ip_old and OWN_IP):
break
sleep_time = random.randint(1, 3)
time.sleep(sleep_time)
except:
pass
If you are using stem library it will handle it for you:
from stem import Signal
from stem.control import Controller
with Controller.from_port(port = 9051) as controller:
controller.authenticate()
controller.signal(Signal.NEWNYM)
if you didn't set any hashed password for tor you can remove controller.authenticate().
if you want to set password for tor controller port do as below:
tor --hash-password my_password
it will generate something like this string 16:3F347B435567CBE1608384B9236897C7322A8EB241CFE5855863F32FB9. now modify torrc file:
sudo vim /etc/tor/torrc
uncomment these lines:
ControlPort 9051
HashedControlPassword 16:E600ADC1B52C80BB6022A0E999A7734571A451EB6AE50FED489B72E3DF
CookieAuthentication 1
Related
I know similar questions have been asked several times:
General SOCKS server failure with python tor but working from tor browser
General SOCKS server failure when switching identity using stem
General SOCKS server failure while using tor proxy
I checked all related posts and googled a lot, but still got stuck.
I'm on Win10. I download Tor browser, run it and make sure it's on port 127.0.0.1:9150 with cmd netstat -aon in administrator.
Then I run the following example code in Python:
import socks
import socket
socks.set_default_proxy(socks.SOCKS5, "127.0.0.1", 9150)
socket.socket = socks.socksocket
The last line socket.socket = socks.socksocket gives the Error message.
socks.GeneralProxyError: Socket error: 0x01: General SOCKS server failure
It's supposed to return a socket object which is assigned to socket.socket that opens a socket. Like this example:
https://deshmukhsuraj.wordpress.com/2015/03/08/anonymous-web-scraping-using-python-and-tor/
Can anyone tell me what's wrong?
Thanks.
Update
Thanks to drew010's answer, this code will work (with Tor browser running and it's port = 9150):
import requests
proxies = {
'http': 'socks5h://127.0.0.1:9150',
'https': 'socks5h://127.0.0.1:9150'
}
url = 'http://icanhazip.com'
# request without Tor (original IP)
r = requests.get(url)
print(r.text)
# request with Tor (Tor IP)
r = requests.get(url, proxies=proxies)
print(r.text)
# Force change IP
from stem.control import Controller
from stem import Signal
with Controller.from_port(port = 9151) as controller:
controller.authenticate('mypassword')
controller.signal(Signal.NEWNYM)
# Changed Tor IP
r = requests.get(url, proxies=proxies)
print(r.text)
Note that we need to set password in torrc before.
by doing "socket.socket = socks.socksocket" you're actually replacing each future socket objects to actually be a socksocket object, which means after that you can just use regular sockets and they will go through your socks proxy.
I have Tor running on a remote server (Ubuntu) on port 9150 with the control port on 9151. I've confirmed both are running via netstat -ant.
Here is my code which is eliciting the SOCKS5Error: 0x01: General SOCKS server failure error.
import socks
import socket
socks.set_default_proxy(socks.SOCKS5, server_ip, 9150)
socket.socket = socks.socksocket
I can make requests from any library and successfully get responses back with a tor ip address.
However the following is what causes the error:
from stem import Signal
from stem.control import Controller
with Controller.from_port(port = 9151) as controller:
controller.authenticate(password)
controller.signal(Signal.NEWNYM)
If I run the above without setting up the proxy using socks (first snippet), I can issue signals with no trouble.
You can't open a new controller once you've connected to Tor. Try opening a controller right at the top of your script. Then both the Tor connection and signaller use the same controller object.
This seems to work with Python3:
import time
import socket
import socks
import requests
from bs4 import BeautifulSoup
from stem import Signal
from stem.control import Controller
controller = Controller.from_port(port=9051)
def connectTor():
socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5 , "127.0.0.1", 9050, True)
socket.socket = socks.socksocket
def renew_tor():
controller.authenticate(<INSERT YOUR PASSPHRASE HERE>)
controller.signal(Signal.NEWNYM)
def show_my_ip():
url = "http://www.showmyip.gr/"
r = requests.Session()
page = r.get(url)
soup = BeautifulSoup(page.content, "lxml")
ip_address = soup.find("span",{"class":"ip_address"}).text.strip()
print(ip_address)
for i in range(10):
renew_tor()
connectTor()
showmyip()
time.sleep(10)
Your first snippet is proxying traffic over tor, but Stem's Controller.from_port() method uses the socket module too. As such Stem attempts to connect to your local control port, gets proxied through a tor exit node, then can't connect.
I saw this error happens when you try to open a new connection to port 9051, while an old connection is still open. I solved the problem in this way.
#----------------Cut Here----------------------
import stem
from stem import Signal
from stem.control import Controller
from stem.connection import connect
import time
#
# Create a new controller
#
controller = Controller.from_port()
Password = "My_Personal_Password"
#
def renew_tor():
global controller
global Password
print ('Renewing Tor Circuit')
if "stem.control.Controller" not in str(controller):
#if global controller exist no more
controller = Controller.from_port()
# debug output
print (controller)
# authenticare the connection with the server control port
controller.authenticate(Password)
print ('Tor running version is : %s' % controller.get_version() )
# force a new circuit
controller.signal(Signal.NEWNYM)
# wait for new circuit
time.sleep(10)
print ('New Tor circuit estabilished')
if __name__ == "__main__":
for i in range (10000):
print ( " Attempt n. : %i " % i)
renew_tor()
#----------------Cut Here(end)--------------------------------------------
From your personal password you can create a hash with the command
tor --hash password My_Personal_Password
and the resulting string has the format
16:CA850F5648.........
This must be inserted in the file /etc/tor/torrc
under:
HashedControlPassword 16: CA850F5648 .........
I'm trying to connect to tor with python stem, while trying to connect (using th emodified example) it just won't work...here's my code:
(I'm using python 3.4.1)
import socket,urllib, sys, socks, stem.process
from stem.util import term
SOCKS_PORT = 7000
# Set socks proxy and wrap the urllib module
socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5, '127.0.0.1', SOCKS_PORT) socket.socket = socks.socksocket
# Perform DNS resolution through the socket
def getaddrinfo(*args): return [(socket.AF_INET, socket.SOCK_STREAM, 6, '', (args[0], args[1]))]
socket.getaddrinfo = getaddrinfo
def query(url): """ Uses urllib to fetch a site using SocksiPy for Tor over the SOCKS_PORT. """
try:
return urllib.urlopen(url).read() except:
return "Unable to reach %s" % url
def print_bootstrap_lines(line): if "Bootstrapped " in line:
print(term.format(line, term.Color.BLUE))
print(term.format("Starting Tor:\n", term.Attr.BOLD))
tor_process = stem.process.launch_tor_with_config( tor_cmd = "C:\Users\Nadav\Desktop\Tor Browser\Tor\\tor.exe" , config = {
'SocksPort': str(SOCKS_PORT),
'ExitNodes': '{ru}', }, init_msg_handler = print_bootstrap_lines, )
print(term.format("\nChecking our endpoint:\n", term.Attr.BOLD)) print(term.format(query("https://www.atagar.com/echo.php"), term.Color.BLUE))
tor_process.kill
The socks port can be different from the port to manipulate tor using stem / stem.control
Hopefully this helps to get things working for you:
import requesocks as requests
from stem import Signal
from stem.control import Controller
# proxies for requests
proxies = {'http': 'socks5://127.0.0.1:9150',
'https': 'socks5://127.0.0.1:9150'}
# when using the Controller
with Controller.from_port(port=9151) as controller:
controller.authenticate()
controller.signal(Signal.NEWNYM)
Notice that the port for sock is different from the port for the Controller. You can find the port for the controller in your torrc file (for me it was called torrc-defaults).
Looks something like this:
# Bind to this address to listen to connections from SOCKS-speaking
# applications.
SocksPort 9150
ControlPort 9151
Hope this helps!
I'm playing around with mechanize on a website that appears differently based on your ip.
Is there a way to change you ip in mechanize?
I've tried:
br.set_proxies({"http": '127.0.0.1:80'})
but that timesout. Is there something else I'm supposed to do to make this work?
no, I do not believe this is possible. IP address is set on outgoing packets by your network stack, outside of mechanize's control.
You can use tor with menchanize it will allowed you tu use different IP and anonymous.
import socks
import socket
def create_connection(address, timeout=None, source_address=None):
sock = socks.socksocket()
sock.connect(address)
return sock
And This code before create the browser of mechanize
socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5, "127.0.0.1", 9050)
socket.socket = socks.socksocket
socket.create_connection = create_connection
This question already has answers here:
How to route urllib requests through the TOR network? [duplicate]
(3 answers)
Closed 7 years ago.
Sample code:
#!/usr/bin/python
import socks
import socket
import urllib2
socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS4, "127.0.0.1", 9050, True)
socket.socket = socks.socksocket
print urllib2.urlopen("http://almien.co.uk/m/tools/net/ip/").read()
TOR is running a SOCKS proxy on port 9050 (its default). The request goes through TOR, surfacing at an IP address other than my own. However, TOR console gives the warning:
"Feb 28 22:44:26.233 [warn] Your
application (using socks4 to port 80)
is giving Tor only an IP address.
Applications that do DNS resolves
themselves may leak information.
Consider using Socks4A (e.g. via
privoxy or socat) instead. For more
information, please see
https://wiki.torproject.org/TheOnionRouter/TorFAQ#SOCKSAndDNS."
i.e. DNS lookups aren't going through the proxy. But that's what the 4th parameter to setdefaultproxy is supposed to do, right?
From http://socksipy.sourceforge.net/readme.txt:
setproxy(proxytype, addr[, port[, rdns[, username[, password]]]])
rdns - This is a boolean flag than
modifies the behavior regarding DNS
resolving. If it is set to True, DNS
resolving will be preformed remotely,
on the server.
Same effect with both PROXY_TYPE_SOCKS4 and PROXY_TYPE_SOCKS5 selected.
It can't be a local DNS cache (if urllib2 even supports that) because it happens when I change the URL to a domain that this computer has never visited before.
The problem is that httplib.HTTPConnection uses the socket module's create_connection helper function which does the DNS request via the usual getaddrinfo method before connecting the socket.
The solution is to make your own create_connection function and monkey-patch it into the socket module before importing urllib2, just like we do with the socket class.
import socks
import socket
def create_connection(address, timeout=None, source_address=None):
sock = socks.socksocket()
sock.connect(address)
return sock
socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5, "127.0.0.1", 9050)
# patch the socket module
socket.socket = socks.socksocket
socket.create_connection = create_connection
import urllib2
# Now you can go ahead and scrape those shady darknet .onion sites
The problem is that you are importing urllib2 before you set up the socks connection.
Try this instead:
import socks
import socket
socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS4, '127.0.0.1', 9050, True)
socket.socket = socks.socksocket
import urllib2
print urllib2.urlopen("http://almien.co.uk/m/tools/net/ip/").read()
Manual request example:
import socks
import urlparse
SOCKS_HOST = 'localhost'
SOCKS_PORT = 9050
SOCKS_TYPE = socks.PROXY_TYPE_SOCKS5
url = 'http://www.whatismyip.com/automation/n09230945.asp'
parsed = urlparse.urlparse(url)
socket = socks.socksocket()
socket.setproxy(SOCKS_TYPE, SOCKS_HOST, SOCKS_PORT)
socket.connect((parsed.netloc, 80))
socket.send('''GET %(uri)s HTTP/1.1
host: %(host)s
connection: close
''' % dict(
uri=parsed.path,
host=parsed.netloc,
))
print socket.recv(1024)
socket.close()
I've published an article with complete source code showing how to use urllib2 + SOCKS + Tor on http://blog.databigbang.com/distributed-scraping-with-multiple-tor-circuits/
Hope it solves your issues.