So I need to build an HTTP server that will contact a client and send him data like pictures or calculations and create a page with those things. I guess you understood that I do not really know what I'm doing... :(
I know python and the basic(+) of the client-server project but I don't understand that HTTP protocol and didn't understand anything from what I read on the internet...
Can anyone explain to me how to work with this protocol? What is the form of HTTP packets?
Here an example of 1 problem that I don't understand: I have been asked to get a packet (which I did) and understand what is the request there, then send back the name of the file the client wants and after it the file itself. I printed the packet and didn't understand where is the request or what the client wants...
Thank you very very much!
Can anyone explain to me how to work with this protocol? What is the form of HTTP packets?
The specification might be helpful.
Concerning the webz, you find a lot of specification on the RFCs.
More to HTTP below.
(Since you seem to be new to programming, I figured I might want to tell you the following:)
Usually one doesn't directly interact with HTTP(S) packets. Instead you use a framework, such as flask, django, aiohttp and many more. The choice of framework depends on the use-case. E.g.:
You need a database, authentication and any imaginable feature? Go with Django.
You just want to create a WebApplication without a bloated framework? Go with Flask.
You need the bare minimum or want to act as a client? Go with aiohttp.
More frameworks are listed here.
The advantage of using such frameworks is that they usually include useful things, that are battletested (i.e. usually no bugs), while you don't have to figure out pecularities of certain protocols.
You just import the framework and write awesomeness! :)
(Anyways, here is a little very oversimplified overview for completeness)
So, HTTP is an text protocol over TCP, which basically means that you send text over a simple tcp socket. When you receive your request you have to "parse" (i.e. comprehend its contents). Luckily for us the requests are standarized and follow the same scheme.
The smallest request would look like this:
GET / HTTP/1.0
Host: www.server.com
The first line starts with a verb (also called request method), in our example the verb is GET. The / denotes the path. Think of file paths on your HDD. The last part of the first line, namely HTTP/1.0, tells the receiver with which version of HTTP we are operating on. Currently the there is HTTP 1.0 and HTTP 1.1; however, I wouldn't bother with HTTP 1.1 yet and stick with HTTP 1.0, if you're implementing the requests your self.
Lastly the Host: www.server.com line tells us which server we want to talk to, since multiple instances of an HTTP server could be running under the same ip. This is used to revole the subdomain.
If you send this request to an HTTP Server, you're likely to receive an response like this:
HTTP/1.0 200 OK
Server: Apache/1.3.29 (Unix) PHP/4.3.4
Content-Length: 1337
Connection: close
Content-Type: text/html
<DATA>
This response contains the status in the first line HTTP/1.0 200 OK. The number and the 'OK' represent a status code, telling us that everything is fine. There are many status codes with their own meaning and usages.
The lines following the first are so-called Response-Headers. They provide additional useful information about the response. For instance, when we open a site like 'stackoverflow.com', the server transmits an HTML file to us for the browser to interpret. Before we can do that, we need to know the size of the HTML file.
Luckily the server tells us beforehand with Content-Length: 1337 line, that the file is 1337 bytes big. The file itself would be present where the <DATA> placeholder stands.
There are, yet again, many of these headers.
As you can see, there are many things to account for when working with HTTP, showing that it is not feasible, without a very good reason, to implement a HTTP client/server from scratch.
Instead it's preferred to use one of the frameworks (for python) listed above.
As a last note:
In the process of trying to explain the concepts as simple as possible I probably left-out or oversimplified some things. If you find any mistake, please let me know.
Related
I have a super special proxy i need to use to access certain hosts ( it turns all other traffic away ), and a bunch of complex libraries and applications that can only take a single http proxy configuration parameter for all their http requests. Which are of course a mix of restricted/proxied traffic and traffic that this proxy is refusing to handle.
I've found an example script showing how to manipulate the upstream proxy host/address in upstream mode, but couldn't find any indication in public API, that "breaking out" of upstream mode in a script is possible, to have mitmproxy directly handle traffic instead of sending it upstream, given certain conditions are met ( request target host mostly )
What am i missing? Should i be trying to do this in "regular" mode?
I invoke PAC in the title because it has the DIRECT keyword that allows the library/application to continue processing the request without going to a proxy.
thanks!
i've found evidence that this is in fact not possible and unlikely to be implemented https://github.com/mitmproxy/mitmproxy/issues/2042#issuecomment-280857954 although this issue and comment is very old, there are some recent related and unanswered questions such as How can I switch mitmproxy mode based on attributes of the proxied request
So instead, i'm pivoting to tinyproxy which does seem to provide this exact functionality https://github.com/tinyproxy/tinyproxy/blob/1.10.0/etc/tinyproxy.conf.in#L143
A shame because the replay/monitoring/interactive editing features of mitmproxy would've been amazing to have
I am trying to make little automated-testing script. It shoud be able to make HTTP request based on string provided, something like follows:
import coollib # non-existent library
r = coollib.make_raw_request(
# Lets assume, python's tripple quoted string spacing is not a problem.
"""
GET / HTTP/1.1\n\r
Host: example.com\n\r
My-Faulty-Header: status\n\r
"""
)
print(r.response_body)
Intention behind this is to insert a little mistakes into request, to test, how web server copes with faulty requests.
Any idea how to do this? Any insights are welcome.
As Ronald Aaronson mentioned in the comment, your weapon of choice here should probably be the socket library. There's an example in this related question: Creating a raw HTTP request with sockets
I am writing a simple SOAP client application in Python.
WSDL file can be found here: https://clients.nationalmailing.com.au/ServiceTest/OrderService.svc?wsdl
Unfortunately the server declared usage of wsHttpBinding in its WSDL file and I had to learn how many troubles it brings to not-.NET developers.
I have working C# code (and it is pretty simple there) and used Fiddler to capture the traffic and analyze messages. Now I know the structure to follow. Client sends 2 subsequental messages.
I managed to create and send first request and receive a response from the server. BUT second request is a way more complex. I have found a library signxml which helped me to create <Signature> structure with all the fields that should present (as per captured traffic).
But the server continues to answer with "Error 500: An error occurred when verifying security for the message."
I realized that in the first message I put just random values for the following structure:
<s:Body>
<trust:RequestSecurityToken xmlns:trust="http://docs.oasis-open.org/ws-sx/ws-trust/200512">
<trust:TokenType>http://docs.oasis-open.org/ws-sx/ws-secureconversation/200512/sct</trust:TokenType>
<trust:RequestType>http://docs.oasis-open.org/ws-sx/ws-trust/200512/Issue</trust:RequestType>
<trust:Entropy>
<trust:BinarySecret
u:Id="uuid-0649fd7a-9ae2-4f9f-964c-e3aa5d68e8cd-1"
Type="http://docs.oasis-open.org/ws-sx/ws-trust/200512/Nonce">h/MaeQVSL5Br30Hnt/SAl274flYfZVZyx2Fri9zNuEY=</trust:BinarySecret>
</trust:Entropy>
<trust:KeySize>256</trust:KeySize>
</trust:RequestSecurityToken>
</s:Body>
The value of BinarySecret is just a random string encoded with Base64. I think this should be an issue on this stage. I also do not use the same parameters from server's response.
Could anyone explain how should I use Entropy.BinarySecret - should it take part in the calculations of Signature and how it is used?
Answering my own question. Yes, the issue was in improper usage of Entropy parameter.
To sign the message you need to generate a key, it consists of two parts (client entropy and server's entropy). They get combined with P_SHA1 algorithm into a key.
To anyone who find this post in the future: for Python have a look on signxml library and section 4 of ws-trust spec.
I apologise if this is a daft question. I'm currently writing against a Django API (which I also maintain) and wish under certain circumstances to be able to generate multiple partial responses in the case where a single request yields a large number of objects, rather than sending the entire JSON structure as a single response.
Is there a technique to do this? It needs to follow a standard such that client systems using different request libraries would be able to make use of the functionality.
The issue is that the client system, at the point of asking, does not know the number of objects that will be present in the response.
If this is not possible, then I will have to chain requests on the client end - for example, getting the first 20 objects & if the response suggests there will be more, requesting the next 20 etc. This approach is an OK work-around, but any subsequent requests rely on the previous response. I'd rather ask once and have some kind of multi-part response.
As far as I know, No you can't send Multipart http response not yet atleast. Multipart response is only valid in http requests. Why? Because no browser as I know of completely supports this.
Firefox 3.5: Renders only the last part, others are ignored.
IE 8: Shows all the content as if it were text/plain, including the boundaries.
Chrome 3: Saves all the content in a single file, nothing is rendered.
Safari 4: Saves all the content in a single file, nothing is rendered.
Opera 10.10: Something weird. Starts rendering the first part as plain/text, and then clears everything. The loading progress bar hangs on 31%.
(Data credits Diego Jancic)
After the AppEngine API update that came out a few weeks ago, the wonderful "Disallowed HTTP Response Headers" section appeared in the Python Response class documentation here, which explains that the listed headers cannot be set for security purposes.
That is all well and good except that now all of my blob downloads have unknown lengths, causing all major browsers show unknown length progress indicators! Suffice it to say that users (and myself) find this quite annoying for large downloads, as there is no way to guess how long the download will take, or how far along they may be. I fixed this before by setting the Content-Length header based on the blob's info records in the datastore, but now that that is disallowed, is there another way to accomplish this? Any ideas much appreciated!
Are your files transferred with
Transfer-Encoding: Chunked
Then it is possible over HTTP to send these files without the Content-Length: header. See the HTTP/1.1 RFC on Chunked Transfer Coding. I guess you should be able to define your own handler for the methods such as get etc, and create yourself the HTTP responses using webapp.WSGIApplication. OTOH, Adrian Holovaty never received an answer to this same question.