Develop Reference

Using mitmproxy inside a Python script to connect to upstream proxy (with user & password)

I am trying to use mitmproxy behind a company proxy that requires a user/password login.
The setup is:
Local PC's browser -> mitmproxy (on local PC) -> company proxy -> internet.
Based on this SO thread, this is how you use mitmproxy within a Python program. This example works fine when there's no proxy.
from mitmproxy.options import Options
from mitmproxy.proxy.config import ProxyConfig
from mitmproxy.proxy.server import ProxyServer
from mitmproxy.tools.dump import DumpMaster
class Addon(object):
def __init__(self):
pass
def request(self, flow):
# examine request here
pass
def response(self, flow):
# examine response here
pass
if __name__ == "__main__":
options = Options(listen_host='0.0.0.0', listen_port=8080, http2=True)
m = DumpMaster(options, with_termlog=False, with_dumper=False)
config = ProxyConfig(options)
m.server = ProxyServer(config)
m.addons.add(Addon())
try:
print('starting mitmproxy')
m.run()
except KeyboardInterrupt:
m.shutdown()
Assuming the company proxy is at IP "1.2.3.4" port 3128 and requires a
login USER and PASSWORD, how can I change this script to have mitproxy
use that proxy instead of going to the internet directly?
Addition info: I am not using mitmdump with the script-parameter to run this script.
The goal is to run this from Python 3.8 with a pip-installed mitmproxy

How to set proxy settings on MacOS using python

How to change the internet proxy settings using python in MacOS to set Proxy server and Proxy port
I do that with windows using this code:
import _winreg as winreg
INTERNET_SETTINGS = winreg.OpenKey(winreg.HKEY_CURRENT_USER, r'Software\Microsoft\Windows\CurrentVersion\Internet Settings', 0, winreg.KEY_ALL_ACCESS)
def set_key(name, value):
_, reg_type = winreg.QueryValueEx(INTERNET_SETTINGS, name)
winreg.SetValueEx(INTERNET_SETTINGS, name, 0, reg_type, value)
set_key('ProxyEnable', 0)
set_key('ProxyOverride', u'*.local;<local>') # Bypass the proxy for localhost
set_key('ProxyServer', u'proxy.example.com:8080')
is this possible to do it on MacOS ?

After a long time of search, I found this way of how to change proxy on MacOs using python.
We need to use networksetup via terminal.
To set HTTP proxy server on MacOS using python:
import os
proxy = "proxy.example.com"
port = 8080
def Proxy_on():
os.system('networksetup -setwebproxy Ethernet '+proxy+' '+port)
Proxy_on()
and to turn it off:
import os
proxy = "proxy.example.com"
port = 8080
def Proxy_off():
os.system('networksetup -setwebproxystate Ethernet off')
Proxy_off()
If the network service isn't named just "Ethernet", you may need to parse networksetup -listallnetworkservices or -listnetworkserviceorder to get the correct name.

Connecting to the Tor network with Python without getting "Proxy server is refusing connection"

I've been trying to use Tor via Python only to come across the "Proxy server is refusing connection" error.
I've trying this method using the Stem library:
http://www.thedurkweb.com/automated-anonymous-interactions-with-websites-using-python-and-tor/
Any help to fixing this error?
Here is the code:
import stem.process
from stem import Signal
from stem.control import Controller
from splinter import Browser
proxyIP = "127.0.0.1"
proxyPort = 9150
proxy_settings = {"network.proxy.type":1,
"network.proxy.ssl": proxyIP,
"network.proxy.ssl_port": proxyPort,
"network.proxy.socks": proxyIP,
"network.proxy.socks_port": proxyPort,
"network.proxy.socks_remote_dns": True,
"network.proxy.ftp": proxyIP,
"network.proxy.ftp_port": proxyPort
}
browser = Browser('firefox', profile_preferences=proxy_settings)
def interactWithSite(browser):
browser.visit("http://dogdogfish.com/python-2/generating-b2b-sales-data-in-python/")
browser.fill("comment", "But the thing is... Why would anyone ever want to do this? I must have thought that times...")
browser.fill("author", "Pebblor El Munchy")
browser.fill("email", "barack#tehwhitehouz.gov")
browser.fill("url", "https://upload.wikimedia.org/wikipedia/en/1/16/Drevil_million_dollars.jpg")
button = browser.find_by_name("submit")
button.click()
interactWithSite(browser)

I deleted the SSL and FTP proxy and port settings and it worked. I also used port 9150.
Here is the working code:
import stem.process
from stem import Signal
from stem.control import Controller
from splinter import Browser
proxyIP = "127.0.0.1"
proxyPort = 9150
proxy_settings = {"network.proxy.type":1,
"network.proxy.socks": proxyIP,
"network.proxy.socks_port": proxyPort,
"network.proxy.socks_remote_dns": True,
}
browser = Browser('firefox', profile_preferences=proxy_settings)
def interactWithSite(browser):
browser.visit("http://dogdogfish.com/python-2/generating-b2b-sales-data-in-python/")
browser.fill("comment", "But the thing is... Why would anyone ever want to do this? I must have thought that times...")
browser.fill("author", "Pebblor El Munchy")
browser.fill("email", "barack#tehwhitehouz.gov")
browser.fill("url", "https://upload.wikimedia.org/wikipedia/en/1/16/Drevil_million_dollars.jpg")
button = browser.find_by_name("submit")
button.click()
interactWithSite(browser)

Remote executing of program via xterm run using paramiko python ssh library

Flow of the program is:
Connect to OpenSSH server on Linux machine using Paramiko library
Open X11 session
Run xterm executable
Run some other program (e.g. Firefox) by typing executable name in the terminal and running it.
I would be grateful if someone can explain how to cause some executable to run in a terminal which was open by using the following code and provide sample source code (source):
import select
import sys
import paramiko
import Xlib.support.connect as xlib_connect
import os
import socket
import subprocess
# run xming
XmingProc = subprocess.Popen("C:/Program Files (x86)/Xming/Xming.exe :0 -clipboard -multiwindow")
ssh_client = paramiko.SSHClient()
ssh_client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh_client.connect(SSHServerIP, SSHServerPort, username=user, password=pwd)
transport = ssh_client.get_transport()
channelOppositeEdges = {}
local_x11_display = xlib_connect.get_display(os.environ['DISPLAY'])
inputSockets = []
def x11_handler(channel, (src_addr, src_port)):
local_x11_socket = xlib_connect.get_socket(*local_x11_display[:3])
inputSockets.append(local_x11_socket)
inputSockets.append(channel)
channelOppositeEdges[local_x11_socket.fileno()] = channel
channelOppositeEdges[channel.fileno()] = local_x11_socket
transport._queue_incoming_channel(channel)
session = transport.open_session()
inputSockets.append(session)
session.request_x11(handler = x11_handler)
session.exec_command('xterm')
transport.accept()
while not session.exit_status_ready():
readable, writable, exceptional = select.select(inputSockets,[],[])
if len(transport.server_accepts) > 0:
transport.accept()
for sock in readable:
if sock is session:
while session.recv_ready():
sys.stdout.write(session.recv(4096))
while session.recv_stderr_ready():
sys.stderr.write(session.recv_stderr(4096))
else:
try:
data = sock.recv(4096)
counterPartSocket = channelOppositeEdges[sock.fileno()]
counterPartSocket.sendall(data)
except socket.error:
inputSockets.remove(sock)
inputSockets.remove(counterPartSocket)
del channelOppositeEdges[sock.fileno()]
del channelOppositeEdges[counterPartSocket.fileno()]
sock.close()
counterPartSocket.close()
print 'Exit status:', session.recv_exit_status()
while session.recv_ready():
sys.stdout.write(session.recv(4096))
while session.recv_stderr_ready():
sys.stdout.write(session.recv_stderr(4096))
session.close()
XmingProc.terminate()
XmingProc.wait()
I was thinking about running the program in child thread, while the thread running the xterm is waiting for the child to terminate.

Well, this is a bit of a hack, but hey.
What you can do on the remote end is the following: Inside the xterm, you run netcat, listen to any data coming in on some port, and pipe whatever you get into bash. It's not quite the same as typing it into xterm direclty, but it's almost as good as typing it into bash directly, so I hope it'll get you a bit closer to your goal. If you really want to interact with xterm directly, you might want to read this.
For example:
terminal 1:
% nc -l 3333 | bash
terminal 2 (type echo hi here):
% nc localhost 3333
echo hi
Now you should see hi pop out of the first terminal. Now try it with xterm&. It worked for me.
Here's how you can automate this in Python. You may want to add some code that enables the server to tell the client when it's ready, rather than using the silly time.sleeps.
import select
import sys
import paramiko
import Xlib.support.connect as xlib_connect
import os
import socket
import subprocess
# for connecting to netcat running remotely
from multiprocessing import Process
import time
# data
import getpass
SSHServerPort=22
SSHServerIP = "localhost"
# get username/password interactively, or use some other method..
user = getpass.getuser()
pwd = getpass.getpass("enter pw for '" + user + "': ")
NETCAT_PORT = 3333
FIREFOX_CMD="/path/to/firefox &"
#FIREFOX_CMD="xclock&"#or this :)
def run_stuff_in_xterm():
time.sleep(5)
s = socket.socket(socket.AF_INET6 if ":" in SSHServerIP else socket.AF_INET, socket.SOCK_STREAM)
s.connect((SSHServerIP, NETCAT_PORT))
s.send("echo \"Hello there! Are you watching?\"\n")
s.send(FIREFOX_CMD + "\n")
time.sleep(30)
s.send("echo bye bye\n")
time.sleep(2)
s.close()
# run xming
XmingProc = subprocess.Popen("C:/Program Files (x86)/Xming/Xming.exe :0 -clipboard -multiwindow")
ssh_client = paramiko.SSHClient()
ssh_client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh_client.connect(SSHServerIP, SSHServerPort, username=user, password=pwd)
transport = ssh_client.get_transport()
channelOppositeEdges = {}
local_x11_display = xlib_connect.get_display(os.environ['DISPLAY'])
inputSockets = []
def x11_handler(channel, (src_addr, src_port)):
local_x11_socket = xlib_connect.get_socket(*local_x11_display[:3])
inputSockets.append(local_x11_socket)
inputSockets.append(channel)
channelOppositeEdges[local_x11_socket.fileno()] = channel
channelOppositeEdges[channel.fileno()] = local_x11_socket
transport._queue_incoming_channel(channel)
session = transport.open_session()
inputSockets.append(session)
session.request_x11(handler = x11_handler)
session.exec_command("xterm -e \"nc -l 0.0.0.0 %d | /bin/bash\"" % NETCAT_PORT)
p = Process(target=run_stuff_in_xterm)
transport.accept()
p.start()
while not session.exit_status_ready():
readable, writable, exceptional = select.select(inputSockets,[],[])
if len(transport.server_accepts) > 0:
transport.accept()
for sock in readable:
if sock is session:
while session.recv_ready():
sys.stdout.write(session.recv(4096))
while session.recv_stderr_ready():
sys.stderr.write(session.recv_stderr(4096))
else:
try:
data = sock.recv(4096)
counterPartSocket = channelOppositeEdges[sock.fileno()]
counterPartSocket.sendall(data)
except socket.error:
inputSockets.remove(sock)
inputSockets.remove(counterPartSocket)
del channelOppositeEdges[sock.fileno()]
del channelOppositeEdges[counterPartSocket.fileno()]
sock.close()
counterPartSocket.close()
p.join()
print 'Exit status:', session.recv_exit_status()
while session.recv_ready():
sys.stdout.write(session.recv(4096))
while session.recv_stderr_ready():
sys.stdout.write(session.recv_stderr(4096))
session.close()
XmingProc.terminate()
XmingProc.wait()
I tested this on a Mac, so I commented out the XmingProc bits and used /Applications/Firefox.app/Contents/MacOS/firefox as FIREFOX_CMD (and xclock).
The above isn't exactly a secure setup, as anyone connecting to the port at the right time could run arbitrary code on your remote server, but it sounds like you're planning to use this for testing purposes anyway. If you want to improve the security, you could make netcat bind to 127.0.0.1 rather than 0.0.0.0, setup an ssh tunnel (run ssh -L3333:localhost:3333 username#remote-host.com to tunnel all traffic received locally on port 3333 to remote-host.com:3333), and let Python connect to ("localhost", 3333).
Now you can combine this with selenium for browser automation:
Follow the instructions from this page, i.e. download the selenium standalone server jar file, put it into /path/to/some/place (on the server), and pip install -U selenium (again, on the server).
Next, put the following code into selenium-example.py in /path/to/some/place:
#!/usr/bin/env python
from selenium import webdriver
from selenium.common.exceptions import NoSuchElementException
from selenium.webdriver.common.keys import Keys
import time
browser = webdriver.Firefox() # Get local session of firefox
browser.get("http://www.yahoo.com") # Load page
assert "Yahoo" in browser.title
elem = browser.find_element_by_name("p") # Find the query box
elem.send_keys("seleniumhq" + Keys.RETURN)
time.sleep(0.2) # Let the page load, will be added to the API
try:
browser.find_element_by_xpath("//a[contains(#href,'http://docs.seleniumhq.org')]")
except NoSuchElementException:
assert 0, "can't find seleniumhq"
browser.close()
and change the firefox command:
FIREFOX_CMD="cd /path/to/some/place && python selenium-example.py"
And watch firefox do a Yahoo search. You might also want to increase the time.sleep.
If you want to run more programs, you can do things like this before or after running firefox:
# start up xclock, wait for some time to pass, kill it.
s.send("xclock&\n")
time.sleep(1)
s.send("XCLOCK_PID=$!\n") # stash away the process id (into a bash variable)
time.sleep(30)
s.send("echo \"killing $XCLOCK_PID\"\n")
s.send("kill $XCLOCK_PID\n\n")
time.sleep(5)
If you want to do perform general X11 application control, I think you might need to write similar "driver applications", albeit using different libraries. You might want search for "x11 send {mouse|keyboard} events" to find more general approaches. That brings up these questions, but I'm sure there's lots more.
If the remote end isn't responding instantaneously, you might want to sniff your network traffic in Wireshark, and check whether or not TCP is batching up the data, rather than sending it line by line (the \n seems to help here, but I guess there's no guarantee). If this is the case, you might be out of luck, but nothing is impossible. I hope you don't need to go that far though ;-)
One more note: if you need to communicate with CLI programs' STDIN/STDOUT, you might want to look at expect scripting (e.g. using pexpect, or for simple cases you might be able to use subprocess.Popen.communicate](http://docs.python.org/2/library/subprocess.html#subprocess.Popen.communicate)).

How to detect the current open webbrowser and open new page in that same browser using Python?

I am making a website where I am using some html forms, which passes the values to a python script and in return the python script opens a new page/tab in the web browser. I am using the webbrowser module for it. Although I can choose the default browser or any other browser using "webbrowser.get([name])"; but my concern is, as this will be a public webpage, so anyone can open the page in any browser of their choice.The problem I am facing is : Lets say my default browser is "firefox", and I open the page in "chrome", so when the python script opens the new page it opens that in "firefox" instead of "chrome".
Here are my questions :
How do I detect the current web browser the user is using?
How to open the new page in that browser?
The code looks like this :
#!C:\Python27\python.exe -u
# -- coding: UTF-8 --
import MySQLdb
import sys
import cgi
import re
import cgitb
import webbrowser
cgitb.enable()
print "Content-Type: text/plain;charset=utf-8"
print
try:
db = MySQLdb.connect(host = "localhost", user = "root", passwd = "", db = "pymysql")
except MySQLdb.Error, e:
print "Error %d: %s" % (e.args[0], e.args[1])
sys.exit()
----- Do some analysis with the database ----
----- Create some kml files ----
#Use the kml files to display points in the map.
#Open the page where openlayers is present
webbrowser.open_new_tab('http://localhost/simulator.html')

The only reason that you are probably convinced that this is a working approach is most likely because you are running the server on your local machine. The python code you are executing is server-side so it has no control over the client. The client would normally be on a remote machine. In your case since your client is also on the server, you get the effect of seeing your python script open a browser tab with the webbrowser module.
This is impossible in a standard client server web situation. The client will be remote and your server side code cannot control their machine. You may only serve back http requests which will simply be something their browser receives and renders. If you want to open tabs it will need to be a javascript solution on the client side.
A more realistic solution would be to have your server serve back proper client side code. If the form is submitted via ajax then your response could contain javaacript that would open a new page.

Your python code will run on a server. The person who visits your website will only recieve html pages. See this about opening new tabs. But please read more about how websites work. The webbrowser module is used for client side applications and not for websites.