Develop Reference

Encoding an HTTP request in Python

Short version: Is there any easy API for encoding an HTTP request (and decoding the response) without actually transmitting and receiving the encoded bytes as part of the process?
Long version: I'm writing some embedded software which uses paramiko to open an SSH session with a server. I then need to make an HTTP request across an SSH channel opened with transport.open_channel('direct-tcpip', <remote address>, <source address>).
requests has is transport adapters, which lets you substitute your own transport. But the send interface provided by BaseAdapter just accepts a PreparedRequest object which (a) doesn't provide the remote address in any useful way; you need to parse the URL to find out the host and port and (b) doesn't provide an encoded version of the request, only a dictionary of headers and the encoded body (if any). It also gives you no help in decoding the response. HTTPAdapter defers the whole lot, including encoding the request, making the network connection, sending the bytes, receiving the response bytes and decoding the response, to urllib3.
urllib3 likewise defers to http.client and http.client's HTTPConnection class has encoding and network operations all jumbled up together.
Is there a simple way to say, "Give me a bunch of bytes to send to an HTTP server," and "Here's a bunch of bytes from an HTTP server; turn them into a useful Python object"?

This is the simplest implementation of this that I can come up with:
from http.client import HTTPConnection
import requests
from requests.structures import CaseInsensitiveDict
from urllib.parse import urlparse
from argparse import ArgumentParser
class TunneledHTTPConnection(HTTPConnection):
def __init__(self, transport, *args, **kwargs):
self.ssh_transport = transport
HTTPConnection.__init__(self, *args, **kwargs)
def connect(self):
self.sock = self.ssh_transport.open_channel(
'direct-tcpip', (self.host, self.port), ('localhost', 0)
)
class TunneledHTTPAdapter(requests.adapters.BaseAdapter):
def __init__(self, transport):
self.transport = transport
def close(self):
pass
def send(self, request, **kwargs):
scheme, location, path, params, query, anchor = urlparse(request.url)
if ':' in location:
host, port = location.split(':')
port = int(port)
else:
host = location
port = 80
connection = TunneledHTTPConnection(self.transport, host, port)
connection.request(method=request.method,
url=request.url,
body=request.body,
headers=request.headers)
r = connection.getresponse()
resp = requests.Response()
resp.status_code = r.status
resp.headers = CaseInsensitiveDict(r.headers)
resp.raw = r
resp.reason = r.reason
resp.url = request.url
resp.request = request
resp.connection = connection
resp.encoding = requests.utils.get_encoding_from_headers(response.headers)
requests.cookies.extract_cookies_to_jar(resp.cookies, request, r)
return resp
if __name__ == '__main__':
import paramiko
parser = ArgumentParser()
parser.add_argument('-p', help='Port the SSH server listens on', default=22)
parser.add_argument('host', help='SSH server to tunnel through')
parser.add_argument('username', help='Username on SSH server')
parser.add_argument('url', help='URL to perform HTTP GET on')
args = parser.parse_args()
client = paramiko.SSHClient()
client.load_system_host_keys()
client.connect(args.host, args.p, username=args.username)
transport = client.get_transport()
s = requests.Session()
s.mount(url, TunneledHTTPAdapter(transport))
response = s.get(url)
print(response.text)
There are various options to BaseAdapter.send that it doesn't handle, and it completely ignores issues like connection pooling and so on, but it gets the job done.

You could write your own SOCKS4 proxy, run it on localhost, then point your HTTP requests at it. For example, https://urllib3.readthedocs.io/en/latest/advanced-usage.html describes how to use a SOCKS proxy with urllib3.
SOCKS4 is basically a simple handshake followed by raw HTTP/TCP traffic. The handshake conveys the target IP address and port. So your proxy can do the handshake to satisfy the client that it is a SOCKS server, then the proxy can send the "real" traffic straight to the SSH session (and proxy the responses in the reverse direction).
The cool thing about this approach is that it will work with tons of clients--SOCKS has been widespread for a long time.

HTTP Retrieval error Twilio

I'm new to Twilio and I am trying to send and receive sms via python. Here is my code
import os
from twilio.rest import TwilioRestClient
from twilio import twiml
from flask import Flask, request, redirect
app = Flask(__name__)
port = int(os.environ.get('PORT', 5000))
# put your own credentials here
ACCOUNT_SID = "..."
AUTH_TOKEN = "..."
client = TwilioRestClient(ACCOUNT_SID, AUTH_TOKEN)
def respond (recipient, twilio_account, body):
message = client.messages.create(
to=recipient,
from_=twilio_account,
body=body,
)
#app.route('/sms', methods=['POST'])
def receive_sms():
number = request.form['From']
body = request.form['Body']
# print "Message received from {0} saying {1}".format(number, body)
response_message = "Message received from {0} saying {1}".format(number, body)
resp = twiml.Response()
resp.message(response_message)
return str(resp)
if __name__ == '__main__':
app.run(debug=True, host="0.0.0.0", port=port)
I keep getting a 11200 error everytime I text my Twilio number. What is happening?

This error indicates that Twilio didn't receive a response from your code within 15 seconds.
I don't see any reason your code would run slowly, but Twilio may not be able to access it at all. Have you taken any steps to let requests into your local network from the Internet?
One useful strategy is to use something like ngrok while you're still in development to tunnel traffic in from the outside world. After installing ngrok you can run
ngrok http 5000
to tunnel traffic in to your localhost port 5000. Then configure Twilio to connect to the ngrok tunnel.

Flask generating requests with random port

Am trying to generate a get request from a flask server to another, on destination side when I print the port, I keep getting random ports with each request
CLIENT:
from flask import Flask, redirect, request
import requests
app = Flask(__name__)
#app.route('/acqlock/<resource_name>')
def acquire_resource(resource_name):
print request.url
response = requests.get('http://127.0.0.1:8080/acqlock/' + resource_name)
return response.text
if __name__ == "__main__":
app.run(host="localhost", port=8081)
SERVER :
from flask import Flask, redirect, request
app = Flask(__name__)
#app.route('/')
#app.route('/acqlock/<resource_name>')
def acquire_lock(resource_name):
print request.url
print request.environ.get('REMOTE_PORT')
if __name__ == "__main__":
app.run(port=int("8080"))
it keeps printing http://127.0.0.1:8080/acqlock/file 58077 eventhough I was expecting it to print http://127.0.0.1:8081 8081 as the server is generating the request

This is normal. Remember that requests is the client here, it has to create a new HTTP connection to another server. TCP connections on the client side need a port as well, and when you create a socket to a remote server you are assigned a port number by the OS for the client-side of the connection.
That you are making this outgoing HTTP connection in the context of an incoming HTTP connection is neither here nor there. It is a different connection, in a different direction.
If you need to identify where the request came from, add information to the request. You can add custom headers, a cookie, query parameters, or post data.

Scraping web-page data with urllib with headers and proxy

I have got web-page data, but now I want to get it with proxy. How could I do it?
import urllib
def get_main_html():
request = urllib.request.Request(URL, headers=headers)
doc = lh.parse(urllib.request.urlopen(request))
return doc

From the documentation
urllib will auto-detect your proxy settings and use those. This is through the ProxyHandler, which is part of the normal handler chain when a proxy setting is detected. Normally that’s a good thing, but there are occasions when it may not be helpful. One way to do this is to setup our own ProxyHandler, with no proxies defined. This is done using similar steps to setting up a Basic Authentication handle.
Check this, https://docs.python.org/3/howto/urllib2.html#proxies

use :
proxies = {'http': 'http://myproxy.example.com:1234'}
print "Using HTTP proxy %s" % proxies['http']
urllib.urlopen("http://yoursite", proxies=proxies)

You can use socksipy
import ftplib
import telnetlib
import urllib2
import socks
#Set the proxy information
socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5, 'localhost', 9050)
#Route an FTP session through the SOCKS proxy
socks.wrapmodule(ftplib)
ftp = ftplib.FTP('cdimage.ubuntu.com')
ftp.login('anonymous', 'support#aol.com')
print ftp.dir('cdimage') ftp.close()
#Route a telnet connection through the SOCKS proxy
socks.wrapmodule(telnetlib)
tn = telnetlib.Telnet('achaea.com')
print tn.read_very_eager() tn.close()
#Route an HTTP request through the SOCKS proxy
socks.wrapmodule(urllib2)
print urllib2.urlopen('http://www.whatismyip.com/automation/n09230945.asp').read()
in your case:
import urllib
import socks
#Set the proxy information
socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5, 'localhost', 9050)
socks.wrapmodule(urllib)
def get_main_html():
request = urllib.request.Request(URL, headers=headers)
doc = lh.parse(urllib.request.urlopen(request))
return doc

How to send URL back as reply by web server in python

I have written this HTTP web server in python which simply sends reply "Website Coming Soon!" to the browser/client, but I want that this web server should sends back the URL given by the client, like if I write
http://localhost:13555/ChessBoard_x16_y16.bmp
then server should reply back the same url instead of "Website Coming Soon!" message.
please tell how can I do this?
Server Code:
import sys
import http.server
from http.server import HTTPServer
from http.server import SimpleHTTPRequestHandler
#import usb.core
class MyHandler(SimpleHTTPRequestHandler): #handles client requests (by me)
#def init(self,req,client_addr,server):
# SimpleHTTPRequestHandler.__init__(self,req,client_addr,server)
def do_GET(self):
response="Website Coming Soon!"
self.send_response(200)
self.send_header("Content-type", "application/json;charset=utf-8")
self.send_header("Content-length", len(response))
self.end_headers()
self.wfile.write(response.encode("utf-8"))
self.wfile.flush()
print(response)
HandlerClass = MyHandler
Protocol = "HTTP/1.1"
port = 13555
server_address = ('localhost', port)
HandlerClass.protocol_version = Protocol
try:
httpd = HTTPServer(server_address, MyHandler)
print ("Server Started")
httpd.serve_forever()
except:
print('Shutting down server due to some problems!')
httpd.socket.close()

You can do what you're asking, sort of, but it's a little complicated.
When a client (e.g., a web browser) connects to your web server, it sends a request that look like this:
GET /ChessBoard_x16_y16.bmp HTTP/1.1
Host: localhost:13555
This assumes your client is using HTTP/1.1, which is likely true of anything you'll find these days. If you expect HTTP/1.0 or earlier clients, life is much more difficult because there is no Host: header.
Using the value of the Host header and the path passed as an argument to the GET request, you can construct a URL that in many cases will match the URL the client was using.
But it won't necessarily match in all cases:
There may be a proxy in between the client and your server, in which case both the path and hostname/port seen by your code may be different from that used by the client.
There may be packet manipulation rules in place that modify the destination ip address and/or port, so that the connection seen by your code does not match the parameters used by the client.
In your do_GET method, you can access request headers via the
self.headers attribute and the request path via self.path. For example:
def do_GET(self):
response='http://%s/%s' % (self.headers['host'],
self.path)