I am trying to implement an mitmproxy addon script, in order to tamper with a particular https packet data - which is by the way decrypted on the fly through mitmproxy's certificate injection.
I am following this Stack Overflow answer to a rather similar question, as well as this tutorial from the mitmproxy docs, but without any success so far.
The packet I'm targeting comes from https://api.example.com/api/v1/user/info.
Now here is the whole python script I wrote so as to tamper with this packet data, based upon the aforementioned sources :
from mitmproxy import ctx
class RespModif:
def _init_(self):
self.num = 0
def response(self, flow):
ctx.log.info("RespModif triggered")
if flow.request.url == "https://api.example.com/api/v1/user/info":
ctx.log.info("RespModif triggered -- In the if statement")
self.num = self.num + 1
ctx.log.info("RespModif -- Proceeded to %d response modifications "
"of the targetted URLs" % self.num)
addons = [
RespModif()
]
Checking out the events log, I'm able to see that the first log information ("RespModif triggered") is being reported onto the log, but the two other log infos (done from inside the if statement) are never reported, which means I think that the if statement does never succeed.
Is there something wrong with my code ?
How can I get the if statement to succeed ?
PS: The target URL is definitely correct, plus I'm using it with a registered account from the client application that is being sniffed with mitmproxy.
Have you tried to use pretty_url attribute ?
Something like :
if flow.request.pretty_url == "https://api.example.com/api/v1/user/info":
....
pretty_url attribute handles full domain name whereas url only deals with corresponding ip address.
Also logging the content of pretty_url should allow to see what exact URL is going through and give more visibility as to what the code is actually doing.
Related
I'd like to setup a "timeout" for the ldap library (python-ldap-2.4.15-2.el7.x86_64) and python 2.7
I'm forcing my /etc/hosts to resolve an non existing IP address in
order to raise the timeout.
I've followed several examples and took a look at the documentation or questions like this one without luck.
By now I've tried forcing global timeouts before the initialize:
ldap.set_option(ldap.OPT_NETWORK_TIMEOUT, 1)
ldap.set_option(ldap.OPT_TIMEOUT, 1)
ldap.protocol_version=ldap.VERSION3
Forced the same values at object level:
ldap_obj = ldap.initialize("ldap://%s:389" % LDAPSERVER ,trace_level=9)
ldap_obj.set_option(ldap.OPT_NETWORK_TIMEOUT, 1)
ldap_obj.set_option(ldap.OPT_TIMEOUT, 1)
I've also tried with:
ldap.network_timeout = 1
ldap.timelimit = 1
And used both methods, search and search_st (The synchronous form with timeout)
Finally this is the code:
def testLDAPConex( LDAPSERVER ) :
"""
Check ldap
"""
ldap.set_option(ldap.OPT_NETWORK_TIMEOUT, 1)
ldap.set_option(ldap.OPT_TIMEOUT, 1)
ldap.protocol_version=ldap.VERSION3
ldap.network_timeout = 1
ldap.timelimit = 1
try:
ldap_obj = ldap.initialize("ldap://%s:389" % LDAPSERVER ,trace_level=9)
ldap_result_id = ldap_obj.search("ou=people,c=this,o=place", ldap.SCOPE_SUBTREE, "cn=user")
I've printed the constants of the object OPT_NETWORK_TIMEOUT and OPT_TIMEOUT, the values were assigned correctly.
The execution time is 56s everytime and I'm not able to control the amount of seconds for the timeout.
Btw, the same code in python3 does work as intended:
real 0m10,094s
user 0m0,072s
sys 0m0,013s
After some testing I've decided to rollback the VM to a previous state.
The Networking Team were doing changes across the network configuration, that might have changed the behaviour over the interface and some kind of bug when the packages were being sent over the wire in:
ldap_result_id = ldap_obj.search("ou=people,c=this,o=place", ldap.SCOPE_SUBTREE, "cn=user")
After restoring the VM, which included a restart, the error of not being able to reach LDAP raises as expected.
I am currently trying to find a way to check whether or not the name servers can respond to either TCP or UDP packets.
My idea behind that was, to get all the name servers from a website (for example google.com), store them in a list, and then try to send TCP and UDP messages to all of them.
Although I am getting the name servers, my interpreter shows a problem when I am trying to make a query on udp(check udpPacket on the code) saying:
"TypeError: coercing to Unicode: need string or buffer, NS found"
I am new in Python(coming from C and C++) and I am guessing this is just incompatible types.
I checked dnspython's documentation and could not find what kind of type NS is (probably it's a type by itself) and why it cannot be passed as an argument.
What do you think the problem is? Is there maybe a better way to solve that kind of problem?
def getNSResults(url):
#create an empty list where we can store all the nameservers we found
nameServers = []
nameServers = dns.resolver.query(url,dns.rdatatype.NS, raise_on_no_answer=False)
#create a dictionary where based on all the nameservers.
#1st label refers to the ns name of our url that we inserted.
#2nd label shows wether or not we received a UDP response or not.
#3rd label shows wether or not we received a TCP response or not.
results = {}
for nameServer in nameServers:
#make a dns ns query, acts as a dumb message since whatever we send we just care of what we get back
query = dns.message.make_query(dns.name.from_text(url), dns.rdatatype.ANY)
query.flags |= dns.flags.AD
query.find_rrset(query.additional, dns.name.root, 65535, dns.rdatatype.OPT, create=True, force_unique=True)
#try sending a udp packet to see if it's listening on UDP
udpPacket = dns.query.udp(query,nameServer)
#try sending a tcp packet to see if it's listening on TCP
tcpPacket = dns.query.tcp(None,nameServer)
#add the results in a dictionary and return it, to be checked later by the user.
results.update({"nsName" == nameServer, "receivedUDPPacket" == isNotNone(udpPacket),"receivedTCPPacket" == isNotNone(tcpPacket)})
Thanks in advance!
Looking at your code, I see some DNS problems, some Python problems, and some dnspython problems. Let's see if we can't learn something together.
DNS
First, the parameter to your function getNSResults is called url. When you send DNS queries, you query for a domain name. A URL is something totally different (e.g. https://example.com/index.html). I would rename url to something like domain_name, domain, or name. For more on the difference between URLs and domain names, see https://www.copahost.com/blog/domain-vs-url/.
Second, let's talk about what you're trying to do.
i am currently trying to find a way to check wether or not the name servers can respond to either tcp or udp packets.
My idea behind that was, to get all the name servers from a website (for example google.com), store them in a list, and then, try to send tcp and udp messages to all of them.
That sounds like a great approach. I think you might be missing a few details here. so let me explain the steps you can take to do this:
Do an NS query for a domain name. You already have this step in your code. What you'll actually get from that query is just another domain name (or multiple domain names). For example, if you run dig +short NS google.com, you'll get this output:
ns3.google.com.
ns1.google.com.
ns4.google.com.
ns2.google.com.
At this step, we have a list of one or more names of authoritative servers. Now we need an IP address to use to send them queries. So we'll do a type A query for each of the names we got from step 1.
Now we have a list of IP addresses. We can send a DNS query over UDP and one over TCP to see if they're supported.
Python
For the most part, your Python syntax is okay.
The biggest red flag I see is the following code:
results.update({"nsName" == nameServer,
"receivedUDPPacket" == isNotNone(udpPacket),
"receivedTCPPacket" == isNotNone(tcpPacket)})
Let's break this down a bit.
First, you have results, which is a dict.
Then you have this:
{"nsName" == nameServer,
"receivedUDPPacket" == isNotNone(udpPacket),
"receivedTCPPacket" == isNotNone(tcpPacket)}
which is a set of bools.
What I think you meant to do was something like this:
results.update({
"nsName": nameServer,
"receivedUDPPacket": true,
"receivedTCPPacket": true
})
Function and variables names in Python are usually written in lowercase, with words separated by underscores (e.g. my_variable, def my_function()). Class names are usually upper camel case (e.g. class MyClass).
None of this is required, you can name your stuff however you want, plenty of super popular libraries and builtins break this convention, just figured I'd throw it out there because it can be helpful when reading Python code.
dnspython
When you're not sure about the types of things, or what attributes things have, remember these four friends, all builtin to Python:
1. pdb
2. dir
3. type
4. print
pdb is a Python debugger. Just import pdb, and the put pdb.set_trace() where you want to break. Your code will stop there, and then you can check out the values of all the variables.
dir will return the attributes and methods of whatever you pass to it. Example: print(dir(udpPacket)).
type will return the type of an object.
print as you probably already know, will print out stuff so you can see it.
I'm going to leave this part for you to test out.
Run dir() on everything if you don't know what it is.
I also should probably mention help(), which is super useful for built-in stuff.
The summary for this section is that sometimes documentation isn't all there, or hard to find, especially when you're new to a language/library/whatever.
So you have to figure stuff out on your own, and that means using all the tools I've just mentioned, looking at the source code, things like that.
Summary
I hope this was helpful. I know it's a lot, it's probably too much, but just be patient and know that DNS and Python are some very useful and fun things to learn about.
I went ahead and wrote something up that is a start at what I think you're hoping to achieve.
I recommend walking through the whole thing and making sure you understand what's going on.
If you don't understand something, remember pdb and dir (and there's always Google, SO, etc).
import dns.resolver
import dns.message
import dns.rdatatype
import json
import sys
def check_tcp_and_udp_support(name):
# this will give me the first default system resolver from /etc/resolv.conf
# (or Windows registry)
where = dns.resolver.Resolver().nameservers[0]
q = dns.message.make_query(name, dns.rdatatype.NS)
ns_response = dns.query.udp(q, where)
ns_names = [t.target.to_text() for ans in ns_response.answer for t in ans]
# this code is the same as the one-liner above
# ns_names = []
# for ans in ns_response.answer:
# for t in ans:
# ns_names.append(t.target.to_text())
results = {}
for ns_name in ns_names:
# do type A lookup for nameserver
q = dns.message.make_query(ns_name, dns.rdatatype.A)
response = dns.query.udp(q, where)
nameserver_ips = [item.address for ans in response.answer for item in ans.items if ans.rdtype == dns.rdatatype.A]
# now send queries to the nameserver IPs
for nameserver_ip in nameserver_ips:
q = dns.message.make_query('example.com.', dns.rdatatype.A)
try:
udp_response = dns.query.udp(q, nameserver_ip)
supports_udp = True
except dns.exception.Timeout:
supports_udp = False
try:
tcp_response = dns.query.tcp(q, nameserver_ip)
supports_tcp = True
except dns.exception.Timeout:
supports_tcp = True
results[nameserver_ip] = {
'supports_udp': supports_udp,
'supports_tcp': supports_tcp
}
return results
def main():
results = check_tcp_and_udp_support('google.com')
# this is just fancy JSON printing
# you could do print(results) instead
json.dump(results, sys.stdout, indent=4)
if __name__ == '__main__':
main()
Again, I hope this is helpful. It's hard when I don't know exactly what's going on in your head, but this is what I've got for you.
I have subclassed BaseHTTPRequestHandler and implemented do_GET(), do_POST() and do_PUT().
I thought that everything works fine until I put my Qt application to the test. The GET request works and the server sends an XML document with some data that the application processes. The POST request also works and is used to generate test cases through REST.
As for the PUT things are looking weird.
Here is the PUT handler on the server side:
def do_PUT(self):
"""
Processes PUT request. It can be tested using wget, curl or any
other tool with support for http PUT. Use this to send reports
about the current status of device and all of its slave devices
Example:
curl -X PUT -d "status=downloading" http://127.0.0.1:8090/so/updateProgress
"""
print('Processing PUT request...')
params_parsed = urlparse(self.path)
params = parse_qs(params_parsed.query)
if len(params) > 0 or '/so/updateProgress' not in params_parsed.path:
print('Use "/so/updateProgress" to report on update progress')
self.__set_headers(data_type='text/html')
self.send_error(501)
return
self.__set_headers(data_type='text/xml')
report_raw = self.rfile.read(int(self.headers.getheader('Content-length')))
print('Received report')
print('Parsing report...')
# Generate XML from the raw data and validate it
report = SoRequestHandler.Report.from_xml(report_raw)
print('Parsing of report complete')
report.write_to_file(log_path='report_log', append=True)
self.send_response(200)
In my Qt application I have a class called ReportPublisher, which takes some reports from all the devices it is connected to (through MQTT), aggregates the reports into a single one and sends it to the server, which logs it in a file:
void ReportPublisher::publishHttpReport(HttpReport report)
{
QString reportXml = report.xml().toString();
if (reportXml.isEmpty())
{
LOG(ERROR) << "Something went wrong with the generation of HTTP report";
return;
}
LOG(INFO) << "Generated report for SO server: " << reportXml;
QByteArray reportRaw = reportXml.toUtf8();
QUrl target(this->urlServer);
target.setPath(this->urlPath);
QNetworkRequest req(target);
req.setHeader(QNetworkRequest::ContentTypeHeader, QString("application/xml"));
this->reply = this->nam->put(req, reportRaw);
// TODO Read back response (expected code 200)?
}
I have to be honest. I have never done a PUT request in Qt and all I've done so far were GET requests so there might be some really fundamental yet easy to spot error in the code above.
The data that the server receives looks like this:
<updateProgress xmlns='service.so.de/so'>
<deviceTypeId>...</deviceTypeId>
<packageId>...</packageId>
<version>...</version>
<status>...</status>
</updateProgress>
If I use curl like this
root#obunit:~$ curl -X PUT -i 'http://192.168.120.61:8090/so/updateProgress' -d "<updateProgress xmlns='service.so.de/so'><deviceTypeId>...</deviceTypeId><packageId>...</packageId><version>...</version><status>...</status></updateProgress>"
where 192.168.120.61:8090 is the IP address and the port the server is located at/listening to for incoming requests I have no problem.
However with data coming from my Qt application I get
192.168.120.172 - - [11/Apr/2018 15:32:37] code 400, message Bad HTTP/0.9 request type ('PUT')
192.168.120.172 - - [11/Apr/2018 15:32:37] "PUT HTTP/1.1" 400 -
in my log (with 192.168.120.172 being the IP address of the system where my software is running.
From my scarce knowledge code 400 means invalid syntax, which can be due to following 2 reasons (at least what I can think of right now):
malformed data (invalid XML, JSON etc.)
incorrect encoding of the data, which is basically equivalent to the malformed data but coming from a different source.
I have tried converting the XML document that I'm generating to QByteArray using QDomDocument::toByteArray(int i). I have also tried (as you can see in the code) to convert the document to QString and then to a UTF-8 QByteArray but I can't get my server to process the data.
What's even stranger is that (as you can see in my do_PUT()) implementation my PUT handler starts with printing a message, which never appears in the logs hence the issue arises from somewhere deeper in the code of BaseHTTPRequestHandler.
The problem was rather simple yet very annoying.
Following the advice of #MrEricSir I did use Wireshark just to find out that get some weird TCP retramsission. Retransmission usually occurs due to network congestion, which in my case was not a possibility however it did indicate a network issue that has nothing to do with the XML content I was sending.
So I started my remote debugging session again and look closer to where I was emitting the PUT request. And there it was! There was a / missing between the base URL (IP:PORT) and the path. So instead of
PUT 192.168.120.61:8090/so/updateProgress
I was doing
PUT 192.168.120.61:8090so/updateProgress
which ultimately led to the PUT request not succeeding at all (hence the code 400).
Just a hint for anyone reading this in the future - whenever you manually set that path using QUrl::setPath(QString)
QUrl target(this->urlServer);
target.setPath(this->urlPath);
ALWAYS make sure that the string you pass as a path starts with / because Qt isn't going to add one!
I have the following case:
Two servers running each their own name server (NS)
Each of these servers has the same object registering with a different URI. The URI includes the local server's hostname to make it unique
A third server, the client, tries to target the right server to query information available only that server
Please note that all of these 3 servers can communicate with each other.
My questions and issues:
The requests from the third server always goes to the first server, no matter what; except when I shutdown the first NS. Is there something definitely wrong with that I'm doing? I guess I do, but I can't figure it out...
Is running separate nameservers the root cause? What would be the alternative if this not allowed? I run multiple name servers for redundancy as some other upcoming operations can run on any of the two first servers. When I list the content of each name server (locally on each server), I get the right registration (which includes the hostname).
Is the use of Pyro4.config.NS_HOST parameter wrong, see below the usage in the code? What would be the alternative?
My configuration:
Pyro 4-4.63-1.1
Python 2.7.13
Linux OpenSuse (kernel version 4.4.92)
The test code is listed below. I got rid of the details like try blocks and imports, etc...
My server skeleton code (which runs on the first two servers):
daemon = Pyro4.Daemon(local_ip_address)
ns = Pyro4.locateNS()
uri = daemon.register(TestObject())
ns.register("test.object#%s" % socket.gethostname(), uri)
daemon.requestLoop()
The local_ip_address is the one supplied below by the user to contact the correct name server (on the correct server).
The name server is started on each of the first tow servers as follows:
python -m Pyro4.naming -n local_ip_address
The local_ip_address is the same as above.
My client skeleton code (which runs on the third server):
target_server_hostname = user_provided_hostname
target_server_ip = user_provided_ip
Pyro4.config.NS_HOST = target_server_ip
uri = "test.object#%s" % target_server_hostname
proxy = Pyro4.Proxy(uri)
proxy._pyroTimeout = my_timeout
proxy._pyroMaxRetries = my_retries
rc, reason = proxy.testAction(target_server_hostname)
if rc != 0:
print reason
else:
print "Hostname matches"
If more information is required, please let me know.
Djurdjura.
I think figured it out. Hope this will be useful to anyone else looking for a similar use case.
You just need to specify where to look for the name server itself. The client code becomes something like the following:
target_server_hostname = user_provided_hostname
target_server_ip = user_provided_ip
# The following statement finds the correct name server
ns = Pyro4.locateNS(host=target_server_ip)
name = "test.object#%s" % target_server_hostname
uri = ns.lookup(name)
proxy = Pyro4.Proxy(uri)
proxy._pyroTimeout = my_timeout
proxy._pyroMaxRetries = my_retries
rc, reason = proxy.testAction(target_server_hostname)
if rc != 0:
print reason
else:
print "Hostname matches"
In my case, I guess the alternative would be using a single common name server running on ... the third server (the client server). This server is always on and ready before the other ones. I didn't try this approach one yet.
Regards.
D.
PS. Thanks Irmen for your answer.
As you will be able to see from my previous questions, I have been working on a project, and really want to know how I can get this last part finished.
Quick summary of project: I have a Raspberry Pi that is running a Web Server (Lighttpd) and Flask. It has an RF USB Transmitter connected, which controls the power of a plug via a Python script. (Power.pyon GitHub). This works.
I now need to create an Endpoint in Flask so that Salesforce can send it some JSON, and it will understand it.
I want to keep this as simple as I can, so I understand what it's actually doing. In my last question, someone did provide me with something, but I thought it'd be better have a specific question relating to it, rather than trying to cover too much in one.
All I need to be able to send is 'power=on/off', 'device=0,1,2', 'time=(secondsasinteger)' and 'pass=thepassword' I can send this as URL variables, or a POST to my existing power.py linked above, and it does it.
I would like a simple, clear way of sending this from Salesforce in JSON, to Flask and make it understand the request.
Literally all I need to do now is go to: ip/cgi-bin/power.py?device=0&power=on&time=10&pass=password
That would load a Python script, and turn device 0 on for 10 seconds. (0 is unlimited).
How can I convert that to JSON? What code do I need to put into Flask for it to be able to comprehend that? Can I forward the variables onto the power.py so the Flask file only has to find the variables and values?
I have downloaded Postman in Chrome, and this allows me to send POST's to the Pi to test things.
Where can I find out more info about this, as a beginner?
Can I send something like this?
'requestNumber = JSONRequest.post(
"ip/api.py",
{
deviceid: 0,
pass: "password",
time: 60,
power: "on"
},'
I don't know how you can get saleforce to send a POST request with an associated JSON, but capturing it with Flask is fairly easy. Consider the following example:
from flask import request
from yourmodule import whatever_function_you_want_to_launch
from your_app import app
#app.route('/power/', methods=['POST'])
def power():
if request.headers['Content-Type'] == 'application/json':
return whatever_function_you_want_to_launch(request.json)
else:
return response("json record not found in request", 415)
when saleforce visits the url http://example.com/power/ your applications executes the power() function passing it, as a parameter, a dictionary containing the JSON contents. The whatever_function_you_want_to_launch function can use the dictionary to trigger whatever action you want to take, and return a response back to the power() function. The power() function would return this respose back to salesforce.
For example:
def whatever_function_you_want_to_launch(data):
device = data['deviceid']
power = data['power']
message = ""
if power == "on":
turn_power_on(device)
message = "power turned on for device " + device
else:
turn_power_off(device)
message = "power turned off for device " + device
return make_response(message, 200)
this is just a short example, of course. You'd need to add some additional stuff (e.g. handle the case that the JSON is malformed, or does not contain one of the requested keys).
in order to test the whole stuff you can also use curl command (available on Linux, don't know on other OSs) with this type of syntax:
curl -H "Content-type: application/json" -X POST http://localhost:5000/power/ -d '{"deviceid": "0", "pass": "password", "time": "60", "power": "on"}'