I have python/django code hosted at dotcloud and redhat openshift. For handling different user, I use token and save it in dictionary. But when I get the value from dict, it sometimes throws an error(key value error).
import threading
thread_queue = {}
def download(request):
dl_val = request.POST["input1"]
client_token = str(request.POST["pagecookie"])
# save client token as keys and thread object as value in dictionary
thread_queue[client_token] = DownloadThread(dl_val,client_token)
thread_queue[client_token].start()
return render_to_response("progress.html",
{ "dl_val" : dl_val, "token" : client_token })
The code below is executed in 1 second intervals via javascript xmlhttprequest to server.
It will check variable inside another thread and return the value to user page.
def downloadProgress(request, token):
# sometimes i use this for check the content of dict
#resp = HttpResponse("thread_queue = "+str(thread_queue))
#return resp
prog, total = thread_queue[str(token)].getValue() # problematic line !
if prog == 0:
# prevent division by zero
return HttpResponse("0")
percent = float(prog) / float(total)
percent = round(percent*100, 2)
if percent >= 100:
try:
f_name = thread_queue[token].getFileName()[1]
except:
downloadProgress(request,token)
resp = HttpResponse('<a href="http://'+request.META['HTTP_HOST']+
'/dl/'+token+'/">'+f_name+'</a><br />')
return resp
else:
return HttpResponse(str(percent))
After testing for several days, it sometimes return :
thread_queue = {}
It sometimes succeeds :
thread_queue = {'wFVdMDF9a2qSQCAXi7za': , 'EVukb7QdNdDgCf2ZtVSw': , 'C7pkqYRvRadTfEce5j2b': , '2xPFhR6wm9bs9BEQNfdd': }
I never get this result when I'm running django locally via manage.py runserver, and accessing it with google chrome, but when I upload it to dotcloud or openshift, it always gives the above problem.
My question :
How can I solve this problem ?
Does dotcloud and openshift limit their python cpu usage ?
Or is the problem inside the python dictionary ?
Thank You.
dotCloud has 4 worker processes by default for the python service. When you run the dev server locally, you are only running one process. Like #martijn said, your issue is related to the fact that your dict isn't going to be shared between these processes.
To fix this issue, you could use something like redis or memcached to store this information instead. If you need a more long term storage solution, then using a database is probably better suited.
dotCloud does not limit the CPU usage, The CPU is shared amongst others on the same host, and allows bursting, but in the end everyone has the same amount of CPU.
Looking at your code, you should check to make sure there is a value in the dict before you access it, or at a minimum surround the code with a try except block, to handle the case when the data isn't there.
str_token = str(token)
if str_token in thread_queue:
prog, total = thread_queue[str_token].getValue() # problematic line !
else:
# value isn't there, do something else
Presumably dotcloud and openshift run multiple processes of your code; the dict is not going to be shared between these processes.
Note that that also means the extra processes will not have access to your extra tread either.
Use an external database for this kind of information instead. For long-running asynchronous jobs like these you also need to run them in a separate worker process. Look at Celery for an all-in-one solution for asynchronous job handling, for example.
Related
I have a Python Firebase SDK on the server, which writes to Firebase real-time DB.
I have a Javascript Firebase client on the browser, which registers itself as a listener for "child_added" events.
Authentication is handled by the Python server.
With Firebase rules allowing reads, the client listener gets data on the first event (all data at that FB location), but only a key with empty data on subsequent child_added events.
Here's the listener registration:
firebaseRef.on
(
"child_added",
function(snapshot, prevChildKey)
{
console.log("FIREBASE REF: ", firebaseRef);
console.log("FIREBASE KEY: ", snapshot.key);
console.log("FIREBASE VALUE: ", snapshot.val());
}
);
"REF" is always good.
"KEY" is always good.
But "VALUE" is empty after the first full retrieval of that db location.
I tried instantiating the firebase reference each time anew inside the listen function. Same result.
I tried a "value" event instead of "child_added". No improvement.
The data on the Firebase side looks perfect in the FB console.
Here's how the data is being written by the Python admin to firebase:
def push_value(rootAddr, childAddr, data):
try:
ref = db.reference(rootAddr)
posts_ref = ref.child(childAddr)
new_post_ref = posts_ref.push()
new_post_ref.set(data)
except Exception:
raise
And as I said, this works perfectly to put the data at the correct place in FB.
Why the empty event objects after the first download of the database, on subsequent events?
I found the answer. Like most things, it turned out to be simple, but took a couple of days to find. Maybe this will save someone else.
On the docs page:
http://firebase.google.com/docs/database/admin/save-data#section-push
"In JavaScript and Python, the pattern of calling push() and then
immediately calling set() is so common that the Firebase SDK lets you
combine them by passing the data to be set directly to push() as
follows..."
I suggest the wording should emphasize that you must do it that way.
The earlier Python example on the same page doesn't work:
new_post_ref = posts_ref.push()
new_post_ref.set({
'author': 'gracehop',
'title': 'Announcing COBOL, a New Programming Language'
})
A separate empty push() followed by set(data) as in this example, won't work for Python and Javascript because in those cases the push() implicitly also does a set() and so an empty push triggers unwanted event listeners with empty data, and the set(data) didn't trigger an event with data, either.
In other words, the code in the question:
new_post_ref = posts_ref.push()
new_post_ref.set(data)
must be:
new_post_ref = posts_ref.push(data)
with set() not explicitly called.
Since this push() code happens only when new objects are written to FB, the initial download to the client wasn't affected.
Though the documentation may be trying to convey the evolution of the design, it fails to point out that only the last Python and Javascript example given will work and the others shouldn't be used.
I would like to have a computational simulation running on a background process (started with redis rq) where I can query its current state, as well as change parameters using Django.
For the sake of simplicity: let's say I want to run the following code for a long time (which I would set up through a python worker):
def simulation(a=1):
value = 0
while a != None:
value += a
time.sleep(5)
Then, by visiting a URL, it would tell me the current value of value. I could also POST to a URL to change the value of a i.e. a=None to stop the simulation or a=-10 to change the behavior.
What is the best way to do this?
This best way I've found to do this is using cache
from django.core.cache import cache
def simulation(a=1):
value = 0
while a != None:
value += a
cache.set('value', value, 3600)
time.sleep(5)
a = cache.get('a', None)
This does work, but it's quite slow for my needs. Perhaps there's a method using sockets, but I wasn't abe to get it to work. The socket is blocked in the background process.
I have a pull queue being serviced by a backend and when the queue is empty I need to trigger another script.
At the moment I am using a very crude detection in the method that leases the tasks from the queue, so that if the task list returned is empty we presume that there are no more to lease and trigger the next step. However, while this works most of the time, occasionally a lease request seems to return an empty list even though there are tasks available.
Anyway, the better way to do it I think is to use the fetch_statistics method of the Queue. That way the script can monitor whats going on in the pull queue and know that there are no more items left in the queue. Now this is obviously available via the REST api for queues, but it seems rather backward to use this when I am using these internally.
So I am making the Queue.fetch_statistics() call, but it throws an error. I've tried putting the stated error into Google, but it returns nothing. Same here on stackoverflow.
It always throws:
AttributeError: type object 'QueueStatistics' has no attribute '_QueueStatistics__TranslateError'
My code is:
q = taskqueue.Queue('reporting-pull')
try:
logging.debug(q.fetch_statistics())
except Exception, e:
logging.exception(e)
Can anyone shed any light on this? I am I doing something really stupid here?
Just incase it is useful to anyone else, here is an example function to get you started getting queue info from your app. Its only an example, and could do with better error handling, but it should get you up and running. Previously we have used the Taskqueue client but I thought that was a bit overkill as we can lease and delete in the code anyway, so I used app identity, and it worked a treat.
from google.appengine.api import taskqueue
from google.appengine.api import app_identity
from google.appengine.api import urlfetch
try:
import json
except ImportError:
import simplejson as json
import logging
def get_queue_info(queue_name, stats=False):
'''
Uses the Queue REST API to fetch queue info
Args:
queue_name: string - the name of the queue
stats: boolean - get the stats info too
RETURNS:
DICT: from the JSON response or False on fail
'''
scope = 'https://www.googleapis.com/auth/taskqueue'
authorization_token, _ = app_identity.get_access_token(scope)
app_id = app_identity.get_application_id()
#note the s~ denoting HRD its not mentioned in the docs as far as
#I can see, but it wont work without it
uri = 'https://www.googleapis.com/taskqueue/v1beta1/projects/s~%s/taskqueues/%s?getStats=%s' % (app_id, queue_name, stats)
#make the call to the API
response = urlfetch.fetch(uri, method="GET", headers = {"Authorization": "OAuth " + authorization_token})
if response.status_code == 200:
result = json.loads(response.content)
else:
logging.error('could not get queue')
logging.error(response.status_code)
logging.error(response.content)
return False
return result
Don't forget to update your queue.yaml with the acl for your app identity
-name: queue_name
mode: pull
acl:
- user_email: myappid#appspot.gserviceaccount.com
I hope someone finds this useful.
In the meantime I have posted a Feature request so we can do this with the Queue object, please go and star it if you want it too. http://goo.gl/W8Pk1
The Task Queue Statistics API is now documented and publicly available. The error no longer occurs.
The immediate reason for the specific error you're getting seems to be a bug in the code; Queue.fetch_statistics() calls QueueStatistics.fetch() calls QueueStatistics._FetchMultipleQueues() which apparently encounters an apiproxy_errors.ApplicationError and then tries to call cls.__TranslateError() but there is no such method on the QueueStatistics class.
I don't know the deeper reason for the ApplicationError, but it may mean that the feature is not yet supported by the production runtime.
I'm trying to create and store 20000 random codes in my local datastore, before trying this in appspot... This is the model
class PromotionCode (db.Model):
code = db.StringProperty(required=True)
And this is the class that handles the populate request (only a logged admin may use it). It creates random alphanumeric codes and tries to store 20000 of them in the datastore:
class Populate(webapp.RequestHandler):
def GenerateCode(self):
chars = string.letters + string.digits
code = ""
for i in range(8):
code = code + choice(chars)
return code.upper()
def get(self):
codes = "";
code_list = []
for i in range(20000):
new_code = self.GenerateCode()
promotion_code = PromotionCode(code=new_code)
code_list.append(promotion_code)
codes = codes + "<br>" + new_code
db.put(code_list)
self.response.out.write("populating datastore...<br>")
self.response.out.write(codes)
I thought I could try batching all those put(), so I created a list of codes (code_list). It takes 2-5 minutes to do it locally.
Is it possible to do it faster without using the bulkuploader option? Because I'm getting the 500 server error, obviously. Or maybe doing it in consecutive calls or steps...
Why not just change your code above to insert 100 at a time, and just run something like:
for i in {1..200}
do
curl --cookie "ACSID=your-acsid-cookie" http://your-app-id.appspot.com/populatepath
sleep(5)
done
from your command line? The entries are random anyway, you don't need to remember any state.
You can get the ACSID cookie by logging in manually and inspecting the cookies from your browser.
The sleep between requests will prevent you from spinning up a gigantic number of instances or hitting short-term quotas.
The task queue suggestion is good if this is something you need to automate, but if it's a one-time thing you might as well keep it simple.
Can you batching the process in task queues.
Setting batch size high into task queue...
U can archive it doing faster
I don't understand why you have to create 20,000 in advance as opposed to creating each as needed on the fly, but I bet you could speed up your code quite a bit. Something like this (untested):
class Populate(webapp.RequestHandler):
chars = "AB...Z01...9"
def GenerateCode(self):
return ''.join(choice(chars) for _ in xrange(8))
def get(self):
code_list = []
for i in range(20000):
new_code = self.GenerateCode()
promotion_code = PromotionCode(code=new_code)
code_list.append(promotion_code)
db.put(code_list)
self.response.out.write("populating datastore...<br>")
self.response.out.write("done")
Not printing out the codes may save time.
I'm sure others here can do better...
If your task won't complete in the 30 second request deadline, you can break it up into chunks - which should be easy since they're all doing the same thing - and run them in tasks on the Task Queue. You should probably do all your work there anyway, so you don't force the user to wait for it to complete before returning a response.
Like Jeff, though, I'm puzzled why you'd want to generate 20,000 of these upfront rather than just generating them when you need them.
With the new release of GAE 1.5.0, we now have an easy way to do async datastore calls. Are we required to call get_result() after calling
'put_async'?
For example, if I have an model called MyLogData, can I just call:
put_async(MyLogData(text="My Text"))
right before my handler returns without calling the matching get_result()?
Does GAE automatically block on any pending calls before sending the result to the client?
Note that I don't really care to handle error conditions. i.e. I don't mind if some of these puts fail.
I don't think there is any sure way to know if get_result() is required unless someone on the GAE team verifies this, but I think it's not needed. Here is how I tested it.
I wrote a simple handler:
class DB_TempTestModel(db.Model):
data = db.BlobProperty()
class MyHandler(webapp.RequestHandler):
def get(self):
starttime = datetime.datetime.now()
lots_of_data = ' '*500000
if self.request.get('a') == '1':
db.put(DB_TempTestModel(data=lots_of_data))
db.put(DB_TempTestModel(data=lots_of_data))
db.put(DB_TempTestModel(data=lots_of_data))
db.put(DB_TempTestModel(data=lots_of_data))
if self.request.get('a') == '2':
db.put_async(DB_TempTestModel(data=lots_of_data))
db.put_async(DB_TempTestModel(data=lots_of_data))
db.put_async(DB_TempTestModel(data=lots_of_data))
db.put_async(DB_TempTestModel(data=lots_of_data))
self.response.out.write(str(datetime.datetime.now()-starttime))
I ran it a bunch of times on a High Replication Application.
The data was always there, making me believe that unless there is a failure in the datastore side of things (unlikely), it's gonna be written.
Here's the interesting part. When the data is written with put_async() (?a=2), the amount of time (to process the request) was on average about 2 to 3 times as fast as put()(?a=1) (not a very scientific test - just eyeballing it).
But the cpu_ms and api_cpu_ms were the same for both ?a=1 and ?a=2.
From the logs:
ms=440 cpu_ms=627 api_cpu_ms=580 cpm_usd=0.036244
vs
ms=149 cpu_ms=627 api_cpu_ms=580 cpm_usd=0.036244
On the client side, looking at the network latency of the requests, it showed the same results, i.e. `?a=2' requests were at least 2 times faster. Definitely a win on the client side... but it seems to not have any gain on the server side.
Anyone on the GAE team care to comment?
db.put_async works fine without get_result when deployed (in fire-and-forget style) but in locally it won't take action until get_result gets called more context
I dunno, but this works:
import datetime
from google.appengine.api import urlfetch
def main():
rpc = urlfetch.create_rpc()
urlfetch.make_fetch_call(rpc, "some://artificially/slow.url")
print "Content-type: text/plain"
print
print str(datetime.datetime.now())
if __name__ == '__main__':
main()
The remote URL sleeps 3 seconds and then sends me an email. The App Engine handler returns immediately, and the remote URL completes as expected. Since both services abstract the same underlying RPC framework, I would guess the datastore behaves similarly.
Good question, though. Perhaps Nick or another Googler can answer definitively.