Develop Reference

Formatting of a GET request

I'm sending data in a TCP client in python and the tutorial I'm following is telling me to send this:
"GET / HTTP/1.1\r\nHost: google.com\r\n\r\n"
I've tried looking up information about the formatting here and I'm confused about what the GET is actually requesting or what data would be sent back by this request, and also what is the purpose of the carriage returns and newlines?

If want to write low-level HTTP GET in Python then you can create a TCP Socket and write the GET command optionally with header parameters then read the response.
The HTTP request starts with a Request-line (e.g. GET / HTTP/1.1 with a terminating CRLF or "\r\n"). The request line is followed by zero or more headers each ending with a CRLF. A final CRLF sequence marks the end of the request line and header part of the HTTP request followed by an optional message body. The request structure is defined in section 5 of the HTTP 1.1 spec
import socket
# host and port map to URL http://localhost:8000/
host = "localhost"
port = 8000
try:
sock = socket.socket()
sock.connect((host, port))
sock.sendall("GET / HTTP/1.1\r\nHost: google.com\r\n\r\n".encode())
# keep reading from socket until no more data in response
while True:
response = sock.recv(8096)
if len(response) == 0:
break
print(response)
except Exception as ex:
print("I/O Error:", ex)
The first line of the HTTP response is the status line including status code terminated with \r\n and followed by response headers.
HTTP/1.1 200 OK\r\n
Content-type: text/plain\r\n
Content-length: 14\r\n
\r\n
This is a test
You need to parse the status line and headers to determine how to decode the message body of the HTTP response.
Details of the HTTP response are in section 6 of the HTTP 1.1 Spec.
Alternatively, the requests module implements the HTTP spec in a simple API.
Example to make a HTTP GET using requests API.
import requests
url = 'http://localhost:8000/'
response = requests.get(url)
print("Status code:", response.status_code)
print("Content:", response.text)

how to show html page before it was fully downloaded

I am sending some data after html content (it has a little delay) in the same response during keep-alive session and want browser to show html before the whole response is downloaded.
For example, I have text 'hello, ' and a function that computes 'world' with delay (let it be 1 sec). So I want browser to show 'hello, ' immediately and 'world' with its delay. Is it possible within one request (so, without ajax)
Here is example python code of what I do (highlighted: https://pastebin.com/muUJyR36):
import socket
from time import sleep
sock = socket.socket()
sock.bind(('', 9090))
sock.listen(1)
conn, addr = sock.accept()
def give_me_a_world():
sleep(1)
return b'world'
while True:
data = conn.recv(1024)
response = b'HTTP/1.1 200 OK\r\n'\
b'Content-Length: 12\r\n'\
b'Connection: keep-alive\r\n'\
b'\r\n'\
b'hello, '
conn.send(response) # send first part
conn.send(give_me_a_world()) # make a delay and send other part
conn.close()

First and foremost, read How the web works: HTTP and CGI explained to understand why and where your current code violates HTTP and thus doesn't and shouldn't work.
Now, as per Is Content-Length or Transfer-Encoding is mandatory in a response when it has body , after fixing the violation, you should
omit the Content-Length header and close the socket after sending all the data, OR
calculate the length of the entire data to send beforehand and specify it in the Content-Length header

You could use Transfer-Encoding: chunked and omit Content-Length.
It works fine on text browsers like curl and Links WWW Browser. But, modern graphical browsers don't really start rendering until it reaches some sort of buffer boundaries.
import socket
from time import sleep
sock = socket.socket()
sock.bind(('', 9090))
sock.listen(1)
conn, addr = sock.accept()
def give_me_a_world():
sleep(1)
return b'5\r\n'\
b'world\r\n'\
b'0\r\n'\
b'\r\n'
while True:
data = conn.recv(1024)
response = b'HTTP/1.1 200 OK\r\n'\
b'Transfer-Encoding: chunked\r\n'\
b'Connection: keep-alive\r\n'\
b'\r\n'\
b'7\r\n'\
b'hello, \r\n'
conn.send(response) # send first part
conn.send(give_me_a_world()) # make a delay and send other part
conn.close()

Web server not sending response

I am trying to write a simple HTTP client program using raw sockets in Python 3. However, the server does not return a response despite having been sent a simple HTTP request. My question is why the server doesn't return a response.
Here is my code:
from socket import *
BUF_LEN = 8192 * 100000
info = getaddrinfo('google.com', 80, AF_INET)
addr = info[-1][-1]
print(addr)
client = socket(AF_INET, SOCK_STREAM)
client.connect(addr)
client.send(b"GET /index.html HTTP1.1\r\nHost: www.google.com\r\n")
print(client.recv(BUF_LEN).decode("utf-8")) # print nothing

You've missed a blank line at the end and mis-specified the HTTP version without a slash:
>>> client.send(b"GET /index.html HTTP1.1\r\nHost: www.google.com\r\n")
Should be:
>>> client.send(b"GET /index.html HTTP/1.1\r\nHost: www.google.com\r\n\r\n")
50
>>> client.recv(BUF_LEN).decode("utf-8")
u'HTTP/1.1 302 Found\r\nCache-Control: private\r\nContent-Type: text/html; charset=UTF-8\r\nLocation: http://www.google.co.uk/index.html?gfe_rd=cr&ei=fIR7WJ7QGejv8AeZzbWgCw\r\nContent-Length: 271\r\nDate: Sun, 15 Jan 2017 14:17:32 GMT\r\n\r\n<HTML><HEAD><meta http-equiv....
The blank line tells the server its the end of the headers, and since this is a GET request there's no payload and so it can then return the content.
Without the / in the HTTP/1.1 spec Google's servers will return an Error: 400 Bad Request response.

Python proxy - receiving http header from the server

I'm building a proxy server in Python and I got a question.
First I'll be showing you a part of my code that presents the receiving data from the client: If there is data from the client, it downloads the content of the requested website (By using the urllib library) and then sending to the client 200 OK with the content length and the content itself:
data = currentSocket.recv(4096)
if data == "":
open_client_sockets.remove(currentSocket)
print 'Conn is closed'
else:
dataSplit = data.split("\r\n")
Host = HostFliter(dataSplit)
print Host, " Host"
if Host == "":
break
contentURL = urllib.urlopen(Host)
content_to_send = contentURL.read()
currentSocket.send("HTTP/1.1 200 OK\r\nContent-Length:"+str(len(content_to_send))+"\r\n\r\n"+str(content_to_send))
contentURL.close()
**The variable "Host" contains the url of the website.
Now for the question:
Where do I get the headers from the server and then send them to the client?
**The libraries I use: socket, select, urllib.
**This is for the select library:
rlist, wlist, xlist = select.select([serverSocket] + open_client_sockets, open_client_sockets, [])

The HTTP response syntax is as follows
HTTP/1.1 200 OK
Content-Type: text/html
Content-Length: 38
<html><body>Hello world!</body></html>
So you need to send headers just before \r\n separated by \n in above format.

Python HTTPS Proxy Tunnelling

I'm trying to make an http proxy in python. So far I've got everything except https working, hence the next step is to implement the CONNECT method.
I'm slightly confused with the chain of events that need to occur when doing https tunnelling.
From my understanding I should have this when connecting to google:
Broswer -> Proxy
CONNECT www.google.co.uk:443 HTTP/1.1\r\n\r\n
Then the proxy should establish a secure connection to google.co.uk, and confirm it by sending:
Proxy -> Browser
HTTP/1.1 200 Connection established\r\n\r\n
At this point I'd expect the browser to now go ahead with whatever it was going to do in the first place, however, I either get nothing, or get a string of bytes that I can't decode(). I've been reading anything and everything to do with ssl tunnelling, and I think I'm supposed to be forwarding any and all bytes from browser to server, as well as the other way around. However, when doing this, I get a:
HTTP/1.0 400 Bad Request\r\n...\r\n
Once I've sent the 200 code, what should I be doing next?
My code snippet for the connect method:
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
if headers["Method"] == "CONNECT":
client = ssl.wrap_socket(client)
try:
client.connect(( headers["Host"], headers["Port"] ))
reply = "HTTP/1.0 200 Connection established\r\n"
reply += "Proxy-agent: Pyx\r\n"
reply += "\r\n"
browser.sendall( reply.encode() )
except socket.error as err:
print(err)
break
while True:
now not sure
Help is much appreciated!

After finding this answer to a related question: HTTPS Proxy Implementation (SSLStream)
I realised that the initial connection on port 443 of the target server (in this case google.co.uk) should NOT be encrypted. I therefore removed the
client = ssl.wrap_socket(client)
line to continue with a plain text tunnel rather than ssl. Once the
HTTP/1.1 200 Connection established\r\n\r\n
message is sent, the browser and end server will then form their own ssl connection through the proxy, and so the proxy doesn't need to do anything related to the actual https connection.
The modified code (includes byte forwarding):
# If we receive a CONNECT request
if headers["Method"] == "CONNECT":
# Connect to port 443
try:
# If successful, send 200 code response
client.connect(( headers["Host"], headers["Port"] ))
reply = "HTTP/1.0 200 Connection established\r\n"
reply += "Proxy-agent: Pyx\r\n"
reply += "\r\n"
browser.sendall( reply.encode() )
except socket.error as err:
# If the connection could not be established, exit
# Should properly handle the exit with http error code here
print(err)
break
# Indiscriminately forward bytes
browser.setblocking(0)
client.setblocking(0)
while True:
try:
request = browser.recv(1024)
client.sendall( request )
except socket.error as err:
pass
try:
reply = client.recv(1024)
browser.sendall( reply )
except socket.error as err:
pass
References:
HTTPS Proxy Implementation (SSLStream)
https://datatracker.ietf.org/doc/html/draft-luotonen-ssl-tunneling-03
http://www.ietf.org/rfc/rfc2817.txt