Develop Reference

Understanding the Python requests module

So I'm currently learning the python requests module but I'm a bit confused and was wondering if someone could steer me in the right direction. I've seen some people post headers when they want to log into the website, but where do they get these headers from and when do you need them? I've also seen some people say you need an authentication token, but I've seen some other solutions not even use headers or an authentication token at all. This is supposedly the authentication token but I'm not sure where to go from here after I post my username and password.
<input type="hidden" name="lt" value="LT-970332-9KawhPFuLomjRV3UQOBWs7NMUQAQX7" />

Although your question is a bit vague, I'll try to help you.
Authentication
A web browser (client) can authenticate on the target server by providing data, usually the pair login/password, which is usually encoded for security reasons.
This data can be passed from client to server using the following parts of HTTP request:
URL parameters (http://httpbin.org/get?foo=bar)
headers
body (this is where POST parameters from HTML forms usually go)
Tokens
After successful authentication server generates a unique token and sends it to client. If server wants client to store token as a cookie, it includes Set-Cookie header in its response.
A token usually represents a unique identifier of a user session. In most cases token has an expiration date for security reasons.
Web browsers usually store token as a cookie in internal cookie storage and use them in all subsequent requests to corresponding website. A single website can use multiple tokens and other cookies for a single user.
Research
Every web site has its own authentication format, rules and restrictions, so first thing you need to do is a little research on target website. You need to get information about the client sends auth information to server, what server replies and where session data is being stored (usually you can find it in client request headers).
In order to do that, you may use a proxy (Burp for example) to intercept browser traffic. It can help you to get the data passed from client to server and back.
Try to authenticate and then browse some pages on target site using your web browser with a proxy. After that, using your proxy, examine what parts of HTTP request/response do client and browser use to store information about sessions and authentication.
After that you can finally use python and requests to do what you want.

AppEnginePlatformWarning - reason to use sockets?

In the Google App Engine standard environment, if you use urllib to make HTTPS requests, you'll get an AppEnginePlatformWarning which says you're using urlfetch instead of sockets.
I found the warning annoying, so I disabled it.
# Use the App Engine Requests adapter. This makes sure that Requests uses
# URLFetch.
requests_toolbelt.adapters.appengine.monkeypatch()
# squelch warning
requests.packages.urllib3.disable_warnings(
requests.packages.urllib3.contrib.appengine.AppEnginePlatformWarning
)
My question is - is there a good reason to switch to sockets? Specifically what is wrong with using urlfetch?

There's nothing wrong with using urlfetch, in fact it is the recommended method for issuing outbound HTTP(S) requests on GAE. From Issuing HTTP(S) Requests (emphasis on requests-related note mine):
App Engine uses the URL Fetch service to issue outbound HTTP(S)
requests.
For details about how the URL Fetch service is implemented and which
headers are sent in a URL Fetch request, see Outbound Requests.
Issuing an HTTP request
To issue an outbound HTTP request, use the urlfetch.fetch
method. For improved code portability, you can also use the Python
standard libraries urllib, urllib2, or httplib to issue HTTP
requests. When you use these libraries in App Engine, they perform
HTTP requests using App Engine's URL Fetch service. You can also use
the third-party requests library as long as you configure it to use
URLFetch.
The sockets support is rather the problematic one in GAE, it comes with a fairly long list of limitations and restrictions, see Sockets Python API Overview, in particular the Limitations and restrictions section.
The warning you see is not from GAE, it's from the 3rd-party requests library you use, which is why I highlighted the note in the above quote. IMHO it's safe to simply ignore/mask the warning in a GAE context.

Is it possible to use Jira cookie based auth with separate connections?

I'm trying to create a command line client for Jira, but I don't really want to store the username/password, and I don't want to have to put in my password with every single request.
Jira says they have a cookie based API, but it doesn't look like it works the way that I think it works.
Specifically, when using Python's requests library I can only re-use the cookie if I have a Session object that I think keeps a connection to Jira.
But if I try to say, make a requests.post request and requests.get requests to the REST URL, it fails with a 401 and tells me that I'm not authenticated. OTOH, if I create a Session, I can do
session.post(.../rest/auth/1/session)
print(session.get(.../rest/auth/1/session).status_code)
And I'll get the 200 that I expect.
I do notice that there's another cookie in the requests response headers:
atlassian.xsrf.token=SOMETHING|RANDOM|lout
but I didn't see anything about that in the documentation.
Is it possible to do this, or do I have to store the username/password if I want to break the connection in between requests?

You are correct, the session is required. From the documentation:
The client creates a new session for the user, via the JIRA REST API.
JIRA returns a session object, which has information about the session including the session cookie. The client stores this session object.
The client can now set the cookie in the header for all subsequent requests to the JIRA REST API.
In other words, the session is integral to the request, receipt and use of the cookie-based authentication token.
Also, the atlassian.xsrf.token would have been injected by atlassian to prevent cross-site forgery and hijacking of the session/cookie.
The way I see it, here are your simple-but-secure options:
For every invocation of your script, use the session to request-receive-retain the cookie (and then, once all API calls are complete, let everything get discarded)
Base64 encode your username and password, store it in a separate file (encrypted if you so choose), and have your script collect (and decrypt) it then place it in an authorization header. See Hiding a password in a python script (insecure obfuscation only).

If you follow the goal not to authorize every time you send a request to the API, you should send (POST) your authentication requests using the cookie-based authentication /rest/auth/1/session, not Basic Auth, to get the token. You will then use that obtained token subsequently in your further requests (in the Cookie header) to the API without a need to authorize every single request.
Watch out for the important missing piece in the API documentation: you should sent the username, NOT email, to authorize in a cookie-based manner. Even though both variants work for the Basic Auth, only user works for the cookie-based authentication.

Use python to download a file from a cookie authenticated page

I have a python script that needs to download a csv file from:
https://myasx.asx.com.au/home/watchlist/download.do
The issue I have is you have to log in to the website first, It is Cookie based authentication (HTML form login).
So far I have looked at urllib2 and Requests and haven't had much luck.

The requests library should do what you want. You can use Session objects to persist the authentication.
To quote the request docs -
The Session object allows you to persist certain parameters across requests. It also persists cookies across all requests made from the Session instance.
Post your code if you are still experiencing problems.

Preferred way to write a WSGI middleware for Flask

I would like to write a WSGI middleware that will work with my flask application. Any examples on how I can do that ? I found one on their documentation, but it says that it is not the preferred way to do it, so wondering what is ?
I would like to write a function which encrypts the session information and sends it as a cookie on response. On request it decrypts the cookie, and passes it forward as a hash. Basically client side session management as an encrypted cookie.

Em.. have you read the Flask or Werkzeug docs? Flask allows passing cryptographycally signed session data via cookies. If you want some secure cookie-based session management of your own, check the SecureCookieSessionInterface.