Develop Reference

Privacy Issue with Heroku App

When I try to look up some pages, I get the following:
Your connection is not private
Attackers might be trying to steal your information from www.xxx.com (for example, passwords, messages or credit cards). NET::ERR_CERT_COMMON_NAME_INVALID
Back to safetyHide advanced
This server could not prove that it is www.xxx.com; its security certificate is from *.herokuapp.com. This may be caused by a misconfiguration or an attacker intercepting your connection.
Proceed to www.xxx.com (unsafe)
What should I do to prevent this? Somehow google has https, but I know I haven't paid for a certificate and everything runs off http, but pages are indexed in google as http(s).

You must use the SSL Endpoint addon.
Heroku use a default *.herokuapp.com certificate. For custom domains, you must use the adon.
https://devcenter.heroku.com/articles/ssl-endpoint

Tornado's XSRF protection

I am using Facebook's Tornado web engine for Python for a project I'm doing and was planning on implementing the XSRF protection, but it left me a little confused.
On a typical request it sets an "_xsrf" cookie to the user's browser if it's not found and then matches that with the value embedded in an HTML form value the browser has sent with the request.
Well let's say an attacker did something like this:
<img src="blah.com/transfer_money?account=0098&destination=0099&_xsrf=
(whatever the client's cookie contains)" title="cool image" />
What's to prevent the attacker from using the cookie outright? As far as I can tell the cookies used for XSRF are not "secure" both from the check_xsrf_cookie method and the xsrf_token method that actually generates the XSRF token. Am I missing something...?

If I understand you correctly, you are asking what prevents attacker from accessing user's cookie in given domain.
Well, the answer is: browser security policy. The script from one domain cannot access cookie from other domain (most of the time). More details here: http://en.wikipedia.org/wiki/HTTP_cookie#Domain_and_Path
This can be circumvented by using XSS (Cross-Site Scripting) attack: injecting the script directly into the source of attacked page. Another approach is to break the client application (browser).
However, most of the time it is not possible for the attacker to retrieve user's cookie from other domain. Additional level of security would be to associate specific CSRF (or "XSRF") token with specific user (and to check it during validation).

POST request to Tastypie returns a non-SSL Location Header

I am doing a POST request to my Tastypie api, which creates a resource.
It normally returns the resource uri, through the Location header in the response.
The problem I'm having is the Location header contains a non-ssl url, even though my initial request (and the whole of my application) is under https.
From my request headers:
URL: https://example.com/api/v1/resource/
From my response headers:
Location: http://example.com/api/v1/resource/80/
Because this is a reusable application that is not always running under ssl, I do not want to hardcode an ugly string replace. Also, there is already a 301 redirect in place, from http to https, but I do not want the redirect to happen.
All help appreciated!
Update:
This actually didn't have anything to do with Tastypie, it was because of the servers/proxy configuration. See answer below for resolution details.

The reason is simple: seemingly request.is_secure() returns False in your case, so the URL is constructed using http instead of https.
There are couple of solutions, but you should first find what caused request.is_secure() to return False. I bet you are running behind some proxy or load balancer. If you did not change the logic behind URL generation, then this is probably the cause of your issue.
To fix that, you can take a look at SECURE_PROXY_SSL_HEADER setting in Django, which defines headers that indicate the SSL connection established with the proxy or load balancer:
If your Django app is behind a proxy, though, the proxy may be "swallowing" the fact that a request is HTTPS, using a non-HTTPS connection between the proxy and Django. In this case, is_secure() would always return False -- even for requests that were made via HTTPS by the end user.
In this situation, you'll want to configure your proxy to set a custom HTTP header that tells Django whether the request came in via HTTPS, and you'll want to set SECURE_PROXY_SSL_HEADER so that Django knows what header to look for.
But if you are designing a reusable app and the above is correct in your case, just make sure it is not something different. If you are sure this is the case, then leave that to the user - the headers responsible for secure request indication should be set explicitly, only by the programmer who uses your app. Otherwise this could mean a security issue.

Google App Engine URL Fetch Doesn't Work on Production

I am using google app engine's urlfetch feature to remotely log into another web service. Everything works fine on development, but when I move to production the login procedure fails. Do you have any suggestions on how to debug production URL fetch?
I am using cookies and other headers in my URL fetch (I manually set up the cookies within the header). One of the cookies is a session cookie.
There is no error or exception. On production, posting a login to the URL command returns the session cookies but when you request a page using the session cookies, they are ignored and you are prompted for login information again. On development once you get the session cookies you can access the internal pages just fine. I thought the problem was related to saving the cookies, but they look correct as the requests are nearly identical.
This is how I call it:
fetchresp = urlfetch.fetch(url=req.get_full_url(),
payload=req.get_data(),
method=method,
headers=all_headers,
allow_truncated=False,
follow_redirects=False,
deadline=10
)
Here are some guesses as to the problem:
The distributed nature of google's url fetch implementation is messing things up.
On production, headers are sent in a different order than in development, perhaps confusing the server.
Some of google's servers are blacklisted by the target server.
Here are some hypothesis that I've ruled out:
Google caching is too aggressive. But I still get the problem after turning off cache by using the header Cache-Control: no-store.
Google's urlfetch is too fast for the target server. But I still get the problem after inserting delays between calls.
Google appends some data to the User-Agent header. But I have added that header to development and I don't get the problem.
What other differences are there between the production URL fetch and the development URL fetch? Do you have any ideas for debugging this?
UPDATE 2
(First update was incorporated above)
I don't know if it was something I did (maybe adding delays or disabling caches mentioned above) but now the production environment works about 50% of the time. This definitely looks like a race condition. Unfortunately, I have no idea if the problem is in my code, google's code, or the target server's code.

As others have mentioned, the key differences between dev and prod are the originating IP, and how some of the request headers are handled. See here for a list of restricted headers. I don't know if this is documented, but in prod, your app ID is appended to the end of your user agent. I had an issue once where requests in prod only were getting detected as a search engine spider because my app ID contained the string "bot".
You mentioned that you're setting up cookies manually, including the session cookie. Does this mean that you established a session in Dev, and then you're trying to re-use it in prod? Is it possible that the remote server is logging the source IP that establishes a session, and requiring that subsequent requests come from the same IP?
You said that it doesn't work, but you don't get an exception. What exactly does this mean? You get an HTTP 200 and an empty response body? Another HTTP status? Your best bet may be to contact the owners of the remote service and see if they can tell you more specifically what was wrong with your request. Anything else is just speculation.

Check your server's logs to see if GAE is chopping any headers off. I've noticed that GAE (thought I think I've seen it on the dev server) will chop off headers it doesn't like.
Depending on the web service you're calling, it might also be less ok with GAE calling it than your local machine.

I ran across this issue while making a webapp with an analogous issue- when looking at urlfetch's documentation, it turns out that the maximum timeout for a fetch call is 60 seconds, but it defaults to 5 seconds.
5 seconds on my local machine was long enough to request URLs on my local machine, but on GAE it was only consistently completing its task in 5 seconds about 20% of the time.
I included the parameter deadline=60 and it has been working fine since.
Hope this helps others!

On what side is 'HTTP Error 403: request disallowed by robots.txt' generated?

I am trying out Mechanize to make some routine simpler. I have managed to bypass that error by using br.set_handle_robots(False). There are talks about how ethical it's to use it. What I wonder about is where this error is generated, on my side, or on server side? I mean does Mechanize throw the exception when it sees some robots.txt rule or does server decline the request when it detects that I use an automation tool?

The server detects the user-agent. If the user agent match one in robots.txt, the rules are applied by the client.
By default, mechanize returns "Python-urllib/2.7".
See http://en.wikipedia.org/wiki/Robots_exclusion_standard

The server blocks your activity with such response.
Is it your site? If not, follow the rules:
Obey robots.txt file
Put a delay between request, even if robots.txt doesn't require it.
Provide some contact information (e-mail or page URL) in the User-Agent header.
Otherwise be ready site owner blocking you based on User-Agent, IP or other information he thinks distinguish you from legitimate users.