I'm using mitmproxy with python as a http proxy
I run the proxy with following command:
mitmdump -s proxy.py -U http://upstreamproxy
The proxy.py is like following:
#!/usr/bin/mitmdump
from __future__ import print_function
import pprint
import datetime
import os
import re
pp = pprint.PrettyPrinter(indent=4)
def request(context, flow):
print("DEBUG")
oldhost = flow.request.host
flow.request.url = re.sub(r"www.verycd.com",r"115.182.66.26",flow.request.url)
# flow.request.host = oldhost #<---This will modify the url also
print("DEBUG")
What I expect is to change the www.verycd.com to IP in url but keep the host field still using www.verycd.com, like following:
GET http://115.182.66.26/ HTTP/1.1
TE: deflate,gzip;q=0.3
Connection: TE, close
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding: gzip, deflate
Accept-Language: zh,en-US;q=0.5
Host: www.verycd.com
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:43.0) Gecko/20100101 Firefox/43.0
But the mitmproxy always change the url and host at the same time
I got following which the server does not accept
GET http://115.182.66.26/ HTTP/1.1
TE: deflate,gzip;q=0.3
Connection: TE, close
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding: gzip, deflate
Accept-Language: zh,en-US;q=0.5
Host: 115.182.66.26
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:43.0) Gecko/20100101 Firefox/43.0
Already answered by project owner
https://github.com/mitmproxy/mitmproxy/issues/890
Related
I'm developing a proxy and I came across this problem where I can't resend the request and get the HTML output(ofc I thought about splitting and these things but it won't help later)
GET http://www.example.com/ HTTP/1.1
Host: www.example.com
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:74.0) Gecko/20100101 Firefox/74.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Language: fr,fr-FR;q=0.8,en-US;q=0.5,en;q=0.3
Accept-Encoding: gzip, deflate
Connection: keep-alive
Upgrade-Insecure-Requests: 1
now I wanna know what do we call this and I wanna know if I can use it without extracting the URL and headers and requesting it
i found a solution by using socket
request="""
GET http://www.example.com/ HTTP/1.1
Host: www.example.com
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:74.0) Gecko/20100101 Firefox/74.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Language: fr,fr-FR;q=0.8,en-US;q=0.5,en;q=0.3
Accept-Encoding: gzip, deflate
Connection: keep-alive
Upgrade-Insecure-Requests: 1
"""
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
webserver=re.findall("(?P<url>https?://[^\s]+)", request)[0]
s.connect((webserver, 80))
s.send(request)
html = s.recv(1024)
conn.send(html)
POST /somelink
Host: hostname
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:47.0) Gecko/20100101
Firefox/47.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Content-Type: application/x-www-form-urlencoded
Content-Length: 683
Cookie: xxx
Connection: keep-alive
access_token=xxx
How can I form an api call with these details?
I already tried:
payload={ "POST":"somelink",
"Host":"hostname",
"User-Agent":"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:47.0) Gecko/20100101 Firefox/47.0",
"Accept":"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"Accept-Language":"en-US,en;q=0.5",
"Accept-Encoding":"gzip, deflate, br",
"Content-Type": "application/x-www-form-urlencoded",
"Content-Length": "683",
"Cookie":" xxx",
"Connection":"keep-alive",
"access_token":"xxx"}
headers = {}
r = requests.post(url, data=json.dumps(payload), headers=headers)
But if I try this way I get 'Connection aborted.', gaierror(-2, 'Name or service not known'.
Can anyone help me with this ?
gaierror means that the DNS lookup is failing: What does this socket.gaierror mean?
Which means Python is unable to resolve the host name to an IP address to initiate the request. Example:
>>> from socket import gethostbyname
>>> gethostbyname('www.google.com')
'172.217.26.196'
>>> gethostbyname('www.wrongurlthatdoesnotexist.com')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
socket.gaierror: [Errno -2] Name or service not known
This is the code that is important:
elif msgheader.startswith( 'POST' ):
print "The POST msg:\n", msgheader
And this is the response:
The POST msg:
POST / HTTP/1.1
Host: 192.168.1.102:63166
Connection: keep-alive
Content-Length: 12
Cache-Control: max-age=0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Origin: http://192.168.1.102:63166
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36
Content-Type: application/x-www-form-urlencoded
Referer: http://192.168.1.102:63166/
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.8
comment=test
Now my question is how i can print out individual rows of the response like the comment field. I tried using this code but i get an error:
print['comment']
Exception (with type 'exceptions.TypeError'): string indices must be integers, not str
You have to use something like this supposing you are using urllib2:
print msgheader.getheader('content-type')
With requests lib you should use:
print msgheader['content-type']
I have written a web server in python, the client uses web-browser to send the requests like this:
http://localhost:13555/ChessBoard_x16_y16.bmp
On server side when I am printing the client-request it is like this:
GET /ChessBoard_x16_y16.bmp HTTP/1.1
Host: localhost:13555
Connection: keep-alive
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0
.8
User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like
Gecko) Chrome/43.0.2357.81 Safari/537.36
Accept-Encoding: gzip, deflate, sdch
Accept-Language: en-US,en;q=0.8
GET /favicon.ico HTTP/1.1
Host: localhost:13555
Connection: keep-alive
User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like
Gecko) Chrome/43.0.2357.81 Safari/537.36
Accept: */*
Referer: http://localhost:13555/ChessBoard_x16_y16.bmp
Accept-Encoding: gzip, deflate, sdch
Accept-Language: en-US,en;q=0.8
But I only want to get & print the actual URL like:
Referer: http://localhost:13555/ChessBoard_x16_y16.bmp
please tell how can I do this?
Do a regular reg-ex search in result set within python:
import re
for line in <your results set>:
line = line.rstrip()
if re.search('^Referer:', line) :
print line
If you don't have reg-ex:
for line in <your results set>:
line = line.rstrip()
if '^Referer:' in line) :
print line
Making a simple python web server using sockets to start understanding how they work, but I think I'm lost on this on. My python server is supposed to access a basic html file in the same directory and display it, once for every time it's requested. But this code for some reason sends the request 3 to 5 times...
from socket import *
server = socket(AF_INET, SOCK_STREAM)
port = 12030
server.bind((gethostname(), port))
server.listen(1)
while True:
print 'Ready to serve'
conection, addr = server.accept()
try:
print 'Working'
message = conection.recv(1024)
filename = message.split()[1] #cuts off the '/' in the request page
f = open(filename[1:])
print message
outputdata = f.read()
print outputdata
conection.send('HTTP/1.1 200 OK\r\n')
for i in range(0, len(outputdata)):
conection.send(outputdata[i])
conection.close()
except IOError:
print 'IO ERROR'
print message
print outputdata
conection.close()
except KeyboardInterrupt:
server.close()
conection.close()
break;
This is the output from me opening the page in the browser.
-en 14:59:54 # ・ー ・
python project.py
Ready to serve
Working
<html><body><h1>Wurld</body></html>
Ready to serve
Working
IO ERROR
<html><body><h1>Wurld</body></html>
Ready to serve
Working
IO ERROR
<html><body><h1>Wurld</body></html>
Ready to serve
I've tried adding a server.listen(1)
and a conection.send("Content-Type:text/html\r\n") , but neither of these do anything.
I'm not sure what the problem could be other than blocking how many times can be requested per minute?
Updated to print message every time
-en 15:33:26 # ・ー ・
python project.py
Ready to serve
Working
GET /HelloWorld.html HTTP/1.1
Host: seppala:12030
Connection: keep-alive
Cache-Control: max-age=0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.120 Safari/537.36
DNT: 1
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
<html><body><h1>Wurld</body></html>
Ready to serve
Working
IO ERROR
GET /favicon.ico HTTP/1.1
Host: seppala:12030
Connection: keep-alive
Accept: */*
DNT: 1
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.120 Safari/537.36
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
<html><body><h1>Wurld</body></html>
Ready to serve
Working
IO ERROR
GET /favicon.ico HTTP/1.1
Host: seppala:12030
Connection: keep-alive
Accept: */*
DNT: 1
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.120 Safari/537.36
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
<html><body><h1>Wurld</body></html>
Ready to serve
It seems your browser is requesting favicon.ico . Try adding a favicon.ico to your document root, or perhaps try a different browser. This problem isn't because of your script.