Develop Reference

How do I import a python library while behind a proxy?

I am trying to import a python library using:
import cenpy as cp
but I get an error message:
ConnectionError: HTTPSConnectionPool(host='api.census.gov', port=443): Max retries exceeded with url: /data.json (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x0000013167B552B0>: Failed to establish a new connection: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond'))
I have had this issue before while calling a website. It has to do with the proxy settings. I resolved those other issues using code like this:
import requests
s = requests.Session()
s.proxies = {
"https":"https://user:pass#server:port",
"http":"http://user:pass#server:port"
}
and then:
s.get('http://web.address')
Is there anyway to implement the request session so that I am able to import the library?
Using Python 3.9.12

So I did some more digging and found out the library does place a call to the API during import. There seems to be a workaround for this but it is not implemented their code yet. I tried a few more things and I wanted to share what worked for me. You have to make sure that the code below runs prior to importing the library making the call. This code should allow for all other call/get requests to run through the proxy without having to use a requests session.
The snippets below will set the proxy environment variables
import os
os.environ['http_proxy'] = 'http://<user>:<pass>#<proxy>:<port>'
os.environ['https_proxy'] = 'http://<user>:<pass>#<proxy>:<port>'
Or to be more thorough:
import os
proxy = 'http://<user>:<pass>#<proxy>:<port>'
os.environ['http_proxy'] = proxy
os.environ['HTTP_PROXY'] = proxy
os.environ['https_proxy'] = proxy
os.environ['HTTPS_PROXY'] = proxy
Remember that this should be at the very top of your script, or at least prior to any connection requests. Also, make sure you are using the correct IP address for the proxy, as that tripped me up as well.
Credit goes here and here.

Connect to .onion websites on tor using python?

Here is the code that i have till now
import socks
import socket
import requests
import json
socks.setdefaultproxy(proxy_type=socks.PROXY_TYPE_SOCKS5, addr="127.0.0.1", port=9050)
socket.socket = socks.socksocket
data = json.loads(requests.get("http://freegeoip.net/json/").text)
and it works fine. The problem is when i use a .onion url it shows error
Failed to establish a new connection: [Errno -2] Name or service not known
After researching a little i found that although the http request is made over tor the resolution still occours over clearnet. What is the proper way so i can also have the domain resolved over tor network to connect to .onion urls ?

Try to avoid the monkey patching if possible. If you're using modern version of requests, then you should have this functionality already.
import requests
import json
proxies = {
'http': 'socks5h://127.0.0.1:9050',
'https': 'socks5h://127.0.0.1:9050'
}
data = requests.get("http://altaddresswcxlld.onion",proxies=proxies).text
print(data)
It's important to specify the proxies using the socks5h:// scheme so that DNS resolution is handled over SOCKS so Tor can resolve the .onion address properly.

There is a more simple solution for this, but therefore you will need Kali Linux. If you have this OS, you can install tor service and kalitorify, start tor service with: sudo service tor start and start kalitorify with sudo kalitorify -t. Now your trafic will be send through tor and you can access .onion sites just as they would be normal sites.

Python3.4 HTTP connection using ip, port, path

im trying to establish a HTTP connection but getting the error:
"http.client.InvalidURL: numeric port: '80/test/path'
Code:
Import http.client
self.conn = http.client.HTTPConnection("192.168.1.1:80/test/path") #"[ip]/[path],[port] does'nt work either
# some post and put requests
working Code:
Import requests
requests.post('192.168.1.1:80/test/path',data='testdata')
How can I establish a HTTP Connection by using ip, port and path?

tor via python - connection ok, but no showing up as on tor

i am using the stem example of connection to the tor network, this should connect a client to the tor network, it seems to be doing this but when i check the ip address it is incorrect and not of a tor ip, any ideas as to why this and more importantly how can i fix this issue :)
import StringIO
import socket
import urllib
import socks # SocksiPy module
import stem.process
from stem.util import term
SOCKS_PORT = 7000
# Set socks proxy and wrap the urllib module
socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5, '127.0.0.1', SOCKS_PORT)
socket.socket = socks.socksocket
# Perform DNS resolution through the socket
def getaddrinfo(*args):
return [(socket.AF_INET, socket.SOCK_STREAM, 6, '', (args[0], args[1]))]
socket.getaddrinfo = getaddrinfo
def query(url):
"""
Uses urllib to fetch a site using SocksiPy for Tor over the SOCKS_PORT.
"""
try:
return urllib.urlopen(url).read()
except:
return "Unable to reach %s" % url
# Start an instance of Tor configured to only exit through Russia. This prints
# Tor's bootstrap information as it starts. Note that this likely will not
# work if you have another Tor instance running.
def print_bootstrap_lines(line):
if "Bootstrapped " in line:
print term.format(line, term.Color.BLUE)
print term.format("Starting Tor:\n", term.Attr.BOLD)
tor_process = stem.process.launch_tor_with_config(
config = {
'SocksPort': str(SOCKS_PORT),
'ExitNodes': '{ru}',
},
init_msg_handler = print_bootstrap_lines,
)
I get the output :
richard#Tornado:~/Documents/Masters Project$ python russiaExample.py
Starting Tor:
May 26 21:56:49.000 [notice] Bootstrapped 80%: Connecting to the Tor network.
May 26 21:56:50.000 [notice] Bootstrapped 85%: Finishing handshake with first hop.
May 26 21:56:50.000 [notice] Bootstrapped 90%: Establishing a Tor circuit.
May 26 21:56:50.000 [notice] Bootstrapped 100%: Done.
however when i visit https://check.torproject.org/ to check i am using tor, it says i am now, and my normal ip is shown,
what is causing this issue, as the output shown above seems to suggest it has established a circuit all ok to Tor, but seems as although it is not using it ?
i am on the right lines here ?
Thanks guys

You have to set up your browser to use Tor as a proxy.
If you are using Firefox:
Go to Edit, preference, advanced and choose "configure how firefox connects to the internet" settings.
In socks host enter 127.0.0.1 and under port enter 7000.
Go to whatismyip.com and you will see a new ip.
Or check tor,project to see you are using Tor successfully.

Using SocksiPy with SSL

I'm trying to use SocksIPy with ssl module (from stdlib) to grab a site's remote certificate but SocksIPy won't play with ssl.
The below code will connect to check.torproject.org and state we are not using Tor (meaning SocksIPy is not working) (bad).
Not sure if SocksIPy is the best solution for this but I haven't been able to find any other way to proxify a raw socket (or get pycurl/urllib2 to use SOCKS proxies and give SSL certs!).
To clarify, my issue is that the socket is not being proxied. I'd like to get the ssl certificate with a proxy of my choosing, that's not happening.
Seems right now, I can either have proxy or SSL but not both. Help!
import socks
import ssl
s = socks.socksocket()
s.setproxy(socks.PROXY_TYPE_SOCKS5, "127.0.0.1", 9050)
ss = ssl.wrap_socket(s)
ss.connect(('check.torproject.org', 443))
ss.write("""GET / HTTP/1.0\r
Host: check.torproject.org\r\n\r\n""")
# print ss.getpeercert()
print ss.read(), ss.read(), ss.read()
ss.close()

I have tested this code while running tcpdump so it should work.
import socks
import ssl
s = socks.socksocket()
s.setproxy(socks.PROXY_TYPE_SOCKS5,"127.0.0.1",port=9050)
s.connect(('83.94.121.246', 443))
ss = ssl.wrap_socket(s)
print ss.send("hello")
ss.close()
I didn't review the ssl.py but I guess you have to call connect on the socks object and not the ssl object.

Put ssl.wrap_socket below connect. It doesn't work properly otherwise.
Use validation and CA certfile Getting the certificate from the server requires creating the SSL object with validation turned on and giving it a CA certificates file. If you can't find one on your system you could download the one provided by the CURL project based on Mozilla's as a local file: http://curl.haxx.se/docs/caextract.html
Note: the SocksIPy project hasn't been updated in quite a while and doesn't support Python 3.
Fixed version of original code:
import socks
import ssl
s = socks.socksocket()
s.setproxy(socks.PROXY_TYPE_SOCKS5, "127.0.0.1", port=9050)
s.connect(('check.torproject.org', 443))
ss = ssl.wrap_socket(s, cert_reqs=ssl.CERT_REQUIRED, ca_certs="cacert.pem")
print "Peer cert: ", ss.getpeercert()
ss.write("""GET / HTTP/1.0\r\nHost: check.torproject.org\r\n\r\n""")
content = []
while True:
data = ss.read()
if not data: break
content.append(data)
ss.close()
content = "".join(content)
assert "This browser is configured to use Tor" in content