Develop Reference

Python requests remove Connection header

I'm currently needing to make a single HTTP request to a bunch of servers that I have in a list however, these HTTP requests contain a 'Connection' header, which I need to remove. How would I do this?
I had the same issue with Accept-Encoding, but I was able to comment out the section that automatically applies that header in httpclient.py (I'm using the requests lib for this). Is there any way around this aside from using sockets and sending raw HTTP requests? Is there maybe another snippet that can be commented out to prevent the header from being automatically added?
I realize that removing the header is a bad idea in the real world, but there's justification for this and it 100% needs to be removed.
I've tried assign it as empty and NoneType, both appear to be failing. I'm wondering if this is something I can't change.

Seems you can actually patch this out by commenting out line 1180 in urllib2.py
headers["Connection"] = "close"

Python tornado AsyncHTTPClient fluke

I have a problem here:
import tornado.httpclient
from tornado.httpclient import AsyncHTTPClient
AsyncHTTPClient.configure("tornado.curl_httpclient.CurlAsyncHTTPClient")
# inside a function
client = AsyncHTTPClient()
result = yield client.fetch('http://some-site.com/#hash?&key=value', raise_error=False)
print(result.effective_url) # prints: http://some.site/some/path/
Note that key-values go after hash. Some site that I scrape gives redirects like this. If I comment out "AsyncHTTPClient.configure('tornado.curl_httpclient.CurlAsyncHTTPClient')" that all works fine, but I cannot use proxy to intersect and view the HTTP exchanges. And with this line staff after hash disappears... Can Anyone tell me why?

Everything after the # is called the "fragment", and it is not normally sent to the server. Instead, it is made available for the browser and javascript to use. At the level of HTTP, http://some-site.com/#hash?&key=value is equivalent to http://some-site.com/. AsyncHTTPClient should be stripping off the fragment whether you use curl or not; the difference you're seeing here is probably a bug.

You want to pass #fragment part. It is used by browsers to navigate through anchors on page or client side routing (more info rfc3985 3.5).
Fragment is not send to the server by browser. Also libcurl do not send the fragment part, as doc says:
While space is not typically a "legal" letter, libcurl accepts them.
When a user wants to pass in a '#' (hash) character it will be treated
as a fragment and get cut off by libcurl if provided literally. You
will instead have to escape it by providing it as backslash and its
ASCII value in hexadecimal: "\23".
You could replace # with %23 as well, but the server should know how to handle it, more likely it does not since it is the part handled by a browser.

POST parameters formatted as JSON

I am writing a web service using Django that will be consumed from a MS SharePoint workflow. In the SP workflow, I created a dictionary with 2 items (id:1, text:'foo'), and used this dictionary as the request content. However, instead of using the dictionary to format a traditional POST parameter list, it sends it as a JSON object in the body of the POST request, so instead of the expected:
id=1&text=foo
in the body of the request, there is this:
{"id":1,"text":"foo"}
which of course, in turn, does not get parsed correctly by Python/Django (I am not sure who exactly does the parsing). How can I either get it to parse JSON, or get SharePoint to send traditionally encoded POST parameters?
EDIT
I saw other posts that explain how to get the raw body and parse the JSON. I was looking for a solution that would either:
Make SharePoint send normal data, or
Get Django to respect the Content-type header that states the data is JSON

There is no need for any parsing at the framework level. The body of the post request is always available in request.body, so you can access it directly:
result = json.loads(request.body)

May be it will help you bit more to handle.
import json
import urlparse
json.dumps(urlparse.parse_qs("id=1&text=foo"))

How do you decode a binary encoded mail message in Python?

I'm writing a Google App engine app that processes incoming mail, and here's the code I'm currently using to process mail messages:
for content_type, body in email_bodies:
#8bit bug in mail messages - see bug report here
#http://code.google.com/p/googleappengine/issues/detail?id=2383
if body.encoding == '8bit':
body.encoding = '7bit'
#test for html content
if content_type == "text/html":
#parse html result
if content_type == "text/plain":
decoded_msg_body = body.decode()
However I just got a message that was using the binary encoding scheme, and when my program tried to process the message using body.decode(), I received a UnknownEncodingError. How should this program parse the binary content type? Also, how can I imitate this message type on my local version of GAE so I can debug and test it out?
I appreciate your help,
Kevin

Rather than reinventing the wheel, you should try Python's built in email parser.
http://docs.python.org/library/email.parser.html
It's designed to handle the lifting involved in getting all sorts of different email formats into a good Python object. Use it to do the parsing, and you'll get nicely predictable objects to work with.
The email module doesn't do message sending and receiving, it just helps put them together and parse them out.

Cookies and HTTP with Python

I wish to "retrieve" the cookies sent by the client in my subclass of BaseHTTPRequestHandler.
Firstly I'm unsure of the exact sequence of sending of headers, in a typical HTTP request and response this is my understanding of the sequence of events:
Client sends request (method, path, HTTP version, host, and ALL headers).
The server responds with a response code, followed by a bunch of headers of its own.
The server then sends the body of the response.
When exactly is the client's POST data sent? Does any overlapping occur in this sequence as described above?
Second, when is it safe to assume that the "Cookie" header has been received by the server. Should all of the client headers have been received by the time self.send_response is called by the server? When in the HTTP communication is the appropriate time to peek at cookie headers in self.headers?
Thirdly, what is the canonical way to parse cookies in Python. I currently believe a Cookie.SimpleCookie should be instantiated, and then data from the cookie headers some how fed into it. Further clouding this problem, is the Cookie classes clunkiness when dealing with the HTTPRequestHandler interfaces. Why does the output from Cookie.output() not end with a line terminator to fit into self.wfile.write(cookie.output()), or instead drop the implicitly provided header name to fit nicely into self.send_header("Set-Cookie", cookie.output())
Finally, the cookie classes in the Cookie module, give the illusion that they're dictionaries of dictionaries. Assigning to different keys in the cookie, does not pack more data into the cookie, but rather generates more cookies... all apparently in the one class, and each generating its own Set-Cookie header. What is the best practise for packing containers of values into cookie(s)?

HTTP is a request/response protocol, without overlap; the body of a POST comes as part of the request (when the verb is POST, of course).
All headers also come as part of the request, including Cookie: if any (there might be no such header of course, e.g. when the browser is running with cookies disabled or whatever). So peek at the headers whenever you've received the request and are serving it.
I'm not sure what your "thirdly" problem is. No newline gets inserted if none is part of the cookie -- why ever should it be? Edit: see later.
On the fourth point, I think you may be confusing cookies with "morsels". There is no limit to the number of Set-Cookie headers in the HTTP response, so why's that a problem?
Edit: you can optionally pass to output up to three arguments: the set of morsel attributes you want in the output for each morsel (default None meaning all attributes), the header string you want to use in front of each morsel (default Set-Cookie:), the separator string you want between morsels (default \r\n). So it seems that your intended use of a cookie is single-morsel (otherwise you couldn't stick the string representation into a single header, which you appear most keen to do): in that case
thecookie.output(None, '')
will give you exactly the string you want. Just make multiple SimpleCookie instances with one morsel each (since one morsel is what fits into a single header!-).

Here's a quick way to get the cookies with no 3rd-party-libraries. While it only answers a section of the question, it may be answering the one which most "visitors" will be after.
import Cookie
def do_GET(self):
cookies = {}
cookies_string = self.headers.get('Cookie')
if cookies_string:
cookies = Cookie.SimpleCookie()
cookies.load(cookies_string)
if 'my-cookie' in cookies:
print(cookies['my-cookie'].value)