Develop Reference

Google Appengine: Requests Alternative

I have a non-GAE application/request-handler that uses the Python requests module in a to post an uploaded imaged via a POST request, as binary:
headers = {"MyAuth" : "xyz"}
r = requests.post(base_uri, data=open('0.jpg')), headers=headers)
The user uploads an image, the uploaded image is saved locally, opened for reading, then sent to a remote classifier pipeline via post request - this returns some JSON regarding the image features, which can then be returned to the user.
I need to implement this behaviour in a GAE app, but know that GAE has no traditional file system, so I will have to use StringIO:
data = ... #some jpg => str
headers = {"MyAuth" : "xyz"}
r = requests.post(base_uri, data=StringIO.StringIO(data), headers=headers)
How could I completely replace the requests module in this example in a GAE friendly way?
Many thanks.

Commonly used module for making HTTP requests on app engine is urlfetch, it is available in the default runtime via google.appengine.api.urlfetch. Supposedly urllib2 and/or urllib3 are also options, but I have not used those myself so I can't say for sure.
You can also install requests in your app engine directory and upload it with the project, but I find that a bit of a hassle, since requests has its own dependencies that you will need to include as well.
Also see Using the Requests python library in Google App Engine

Although probably not the best solution to this problem, I managed to get requests 2.3.0 to work in the GAE project with:
pip install --target myproject/externals/ requests==2.3.0
I can now use requests as I would normally.

Language switching issue with Django 1.8.4 and uWSGI config with several processes

I'm facing a strange issue when I switch the language on a Django 1.8.4 project that I have deployed on production with Nginx + uWSGI. The problem is that when I switch the language, somehow every language I change is "remembered", and every time I refresh the page the site language changes to a different language that I have used before. It seems like each process of uWSGI is storing a language itself. When I run my project with the Django server locally, it works like a charm, and when I set just 1 process to the uWSGI config file, it also works properly, but as soon as I set just 2 processes it is broken again. I have been using:
Django 1.8.4
uWSGI==2.0.11.2
Below the uWSGI config file I have:
uwsgi.ini
[uwsgi]
socket=127.0.0.1:8080
chdir=/path/to/code
wsgi-file=wsgi.py
need-app=true
processes=8
master=true
enable-threads=true
thunder-lock=true
single-interpreter=true
plugin=python
As I commented above if I leave the config file with processes=1 it works as should.
What I mean with "switch the language" is exactly what is writing on this section of the Django doc: https://docs.djangoproject.com/en/1.8/topics/i18n/translation/#explicitly-setting-the-active-language. I have a utility like this:
def activate_language(request, language_code):
"""
Explicitly setting the active language.
As to the doc:
https://docs.djangoproject.com/en/1.8/topics/i18n/translation/#the-set-language-redirect-view#s-explicitly-setting-the-active-language
:param language_code: the language code: en, es, da, and so on.
:return:
"""
activate(language_code)
request.session[LANGUAGE_SESSION_KEY] = language_code
After the excellent questions done, I would like to put more details here. I'm using this session engine:
SESSION_ENGINE = 'django.contrib.sessions.backends.cached_db'
and I have tested that the value of the session on the DB is updated properly:
In [13]: Session.objects.get(pk='lrx8sdzbabztc8wifiab5i5gmqiwuxnn').get_decoded()
Out[13]: {u'_language': u'es'}
In [14]: Session.objects.get(pk='lrx8sdzbabztc8wifiab5i5gmqiwuxnn').get_decoded()
Out[14]: {u'_language': u'en'}
In [15]: Session.objects.get(pk='lrx8sdzbabztc8wifiab5i5gmqiwuxnn').get_decoded()
Out[15]: {u'_language': u'da'}
but still the issue persists even when this value changes as should. I have some clues about this though. When I use any of the following session engines:
"django.contrib.sessions.backends.file"
"django.contrib.sessions.backends.signed_cookies"
it works like a charm. So could it be related with the way of getting this value when using the cached_db engine?
I will appreciate any help you can give me about that issue. Thanks in advance to all the community.

Well, I finally got this fixed. Which I couldn't have done without the help of #iamkhush, who with his great questions directed me to the fix. The problem was solved setting up the Memcache. The key of this was on the Django session doc itself:
You should only use cache-based sessions if you’re using the Memcached cache backend.
And that was exactly my problem, that I was using django.contrib.sessions.backends.cached_db as SESSION_ENGINE but I was not using Memcache. So it is solved now.

SOLR mysolr pysolr Python 401 reply

If there is someone out there who has already worked with SOLR and a python library to index/query solr, would you be able to try and answer the following question.
I am using the mySolr python library but there are others out (like pysolr) there and I don't think the problem is related to the library itself.
I have a default multicore SOLR setup, so no authentication required normally. Don't need it to access the admin page at http://localhost:8080/solr/testcore/admin/ either
from mysolr import Solr
solr = Solr('http://localhost:8080/solr/testcore/')
response = solr.search(q='*:*')
print("response")
print(response)
This code used to work but now I get a 401 reply from SOLR ... just like that, no changes have been made to the python virtual env containing mysolr or the SOLR setup. Still...something must have changed somewhere but I'm out of clues.
What could be the causes of a SOLR 401 reponse?
Additional info: This script and mor advanced script do work on another PC, just not on the one I am working on. Also, adding "/select?q=:" behind the url in the browser does return the correct results. So the SOLR is setup correctly, it has probably something to do with my computer itself. Could windows settings (of any kind) have an impact on how SOLR responds to requests from python? The python env itself has been reinstalled several times to no avail.
Thanks in advance!

The problem was: proxy.
If this exact situation was ever to occur to someone and you are behind a proxy, check if your HTTP and HTTPS environmental variables are not set. If they are... this might cause the python session to try using the proxy while it shouldn't (connecting to localhost via proxy).
It didn't cause any trouble for months but out of the blue it did so whether you encounter this or not might be dependent on how your IT setup your proxy or made some other changes...somewhere.
thank you everyone!

Is there a working OAuth library for Python 3?

What's the most current form of Oauth for Python 3?
I'm trying to create a stock screener using my broker's API, which uses Oauth. Most of the info I find is out of date or conflicting. I've seen the following modules referenced:
Oauth - Seems to be the original, now outdated. I get an error of "'module' object has no attribute 'Consumer'"
Oauth2 - Newer version, apparently also outdated? The one most referenced one online. Glitches out in pip/can't figure out how to install it.
Oauthlib - IIRC, claims to be the new replacement for Oauth and Oauth2
Rauth.OAuth2Service - Also potentially replacement for Oauth and Oauth2?
Requests - ?
Oauth_hook - ?
pyoauth2 - I get an error about not having a module named "client" in pyoauth2's init.
None of them seem to work as expected, and I have a feeling that this is due to low Oauth 3 support. Have you gotten OAuth to work in Python 3? If so, how did you do it?

It looks like Requets_oauthlib works. Here's code I used that works in Python 3. I'm posting it because most of the example code I found used formats that I couldn't get working.
from requests_oauthlib import OAuth1
client_key = ''
client_secret = ''
resource_owner_key = ''
resource_owner_secret = ''
def query(queryurl):
headeroauth = OAuth1(client_key, client_secret, resource_owner_key,
resource_owner_secret, signature_type = 'auth_header')
return requests.get(queryurl, auth = headeroauth)
query('http://website.com')

Author of rauth here: rauth is a client library which currently does not officially support Python 3.
However, we are working on it, and there's an active branch (aptly named "python-3") over at GitHub which works. You're free to use it, but bear in mind that things may change slightly when we officially release support for it later on. With that said, it would be great to have people out in the real world testing it so that we can make changes to accommodate the Python 3 ecosystem.
Also note: oauthlib is not a replacement for rauth and not a client library. It attempts to be a generic solution, much like python-oauth2 was, but it doesn't provide a client, unlike python-oauth2.

How to access YQL in Python (Django)?

Hey, I need a simple example for the following task:
Send a query to YQL and receive a response
I am accessing public data from python backend of my Django app.
If I just copy/paste an example from YQL, it says "Please provide valid credentials".
I guess, I need OAuth authorization to do it.
So I got an API key and a shared secret.
Now, what should I do with them?
Should I use python oauth library? This one?
http://oauth.googlecode.com/svn/code/python/oauth/
But what is the code? How I pass my secret/API key along with my yql query?
I guess, many Django programmers would love to know this.

I've just released python-yql also available on pypi. It can do public, two-legged oauth a.k.a signed requests and facilitate 3-legged outh too.
It's brand new so there may be some bugs whilst I work on improving the test coverage but should hopefully do what you need. See the source for some idea on how to use it.
Installing to try it is as follows:
sudo easy_install yql
Bug/Feature requests can be filed here: https://bugs.launchpad.net/python-yql

If you only are accessing public data you can just make a direct rest call from python.
>>> import urllib2
>>> result = urllib2.urlopen("http://query.yahooapis.com/v1/public/yql?q=select%20title%2Cabstract%20from%20search.web%20where%20query%3D%22paul%20tarjan%22&format=json").read()
>>> print result[:100]
{"query":{"count":"10","created":"2009-11-03T04:47:01Z","lang":"en-US","updated":"2009-11-03T04:47:0
And then you can parse the result with simplejson.
>>> import simplejson
>>> data = simplejson.loads(result)
>>> data['query']['results']['result'][0]['title']
u'<b>Paul</b> <b>Tarjan</b> - Silicon Valley, CA | Facebook'

Ok, I sort of resolved the problem.
In YQL console example for data/html the following url was presented as an example:
http://query.yahooapis.com/v1/yql?q=select+*+from+html+where+url%3D%22http%3A%2F%2Ffinance.yahoo.com%2Fq%3Fs%3Dyhoo%22+and%0A++++++xpath%3D%27%2F%2Fdiv%5B%40id%3D%22yfi_headlines%22%5D%2Fdiv%5B2%5D%2Ful%2Fli%2Fa%27
It does not work!
But if you insert "/public" after "v1/" than it magically starts working!
http://query.yahooapis.com/v1/public/yql?q=select+*+from+html+where+url%3D%22http%3A%2F%2Ffinance.yahoo.com%2Fq%3Fs%3Dyhoo%22+and%0A++++++xpath%3D%27%2F%2Fdiv%5B%40id%3D%22yfi_headlines%22%5D%2Fdiv%5B2%5D%2Ful%2Fli%2Fa%27
But the question of how to pass my API key (for v1/yql access) is still open. Any advice?