I am creating a app on Google app engine and am wondering if there are ways to do automated testing with python.
Thanks!
We are generally not testing too much. We once had a "80% test coverage" rule but found this doesn't make us better or faster. Most code and data structures we use are designed quite defensively so there is seldom harm which can't be undone. Our users prefer fast turnaround times to 100% uptime.
We have two apps setup: my app.appspot.com and my app-test.appspot.com. The whole codebase is designer to ensure app-test.appspot.com never changes state in external systems.
occasionally we copy the data from app.appspot.com to app-test.appspot.com. It can get messy, because id generation counters for the datastore don't get updated but it works good enough.
We develop on both systems. Frontend development is done mostly on app.appspot.com and experiments with the backend are done on app-test.appspot.com.
We have three branches: master, rc and production.rc gets updated from master and production from rc. rc is deployed daily to rc.app.appspot.com by or operations them. production is deployed weekly to production.app.appspot.com (which is also reachable via an other app name.
Developers usually deply to dev-whoami.app.appspot.com for experimenting. We use the development server very little because wee need a lot of data from the datastore.
Now to testing: we mostly use acceptance tests. We have a little framework called resttest_dsl which we use to describe tests like this:
client.GET('/').responds_access_denied()
client.GET('/', auth='user').responds_html()
client.GET('/admin').responds_access_denied()
client.GET('/admin', auth='user').responds_access_denied()
client.GET('/admin', auth='admin').responds_html()
client.GET('/artikel/').responds_with_html_to_valid_auth()
client.GET('/artikel/?q=Ratzfratz', auth='user').responds_html()
client.GET('/api/ic/v3/a/14600/03/zukunft.json').responds_with_json_to_valid_auth()
client.GET('/kunden/SC50313/o/SO1194829/', auth='user').responds_html()
client.GET('/api/masterdata/artikel/v2/artnr/14600/bild=s210').redirects_to('...')
hostname and credentials have defaults but can be overwritten by environment variables. Most errors we ever have fixed have a regression test in there. We use Makefiles to drive the whole stuff. Eg.g:
deploy:
appcfg.py update -V dev-`whoami` -A app .
TESTHOST=dev-`whoami`.app.appspot.com make resttest
open http://dev-`whoami`.app.appspot.com/
Deployment always happens from the central git repository like this:
deploy_production:
rm -Rf tmp
mkdir tmp
(cd tmp ; git clone git#github.com:user/app.git)
(cd tmp/app ; git checkout production ; make dependencies)
(cd tmp/app ; git show-ref --hash=7 refs/remotes/origin/production > version.txt)
appcfg.py update -V "v`cat tmp/app/version.txt`" -A app tmp/app
(cd tmp/huWaWi ; TESTHOST="v`cat version.txt`".app.appspot.com make resttest)
appcfg.py update -V production -A app tmp/app
appcfg.py backends -V production -A app tmp/huWaWi app
We first deploy to a version tagged with the current revision on AppEngine. We then run resttest.py against this freshly deployed version. On failure the mmake stops execution. If no failure occurred the "production version" is deployed.
We also run mandantory pep8, pyflakes and pylint checks on source code checkin.
All in all we have very simple minded tests but run them a lot and against production code and data. For us this catches most of error we make which relatively little effort.
I use gaeunit - http://code.google.com/p/gaeunit/ - which may or may not suit your needs but once its going it's pretty easy to add to. I also added an xml output so that I can stuff the results back into a junit analyser so my jenkins can report back after code checkins that nothing broke.
David Robinson refers to the development unit testing.
If you are looking for automated user(production) testing using python, go for selenium rc or selenium webdriver(improved version & standalone).
You can do wonders with selenium RC.
Refer to http://seleniumhq.org/projects/webdriver/
Related
How should I implement continuous integration on my new application? Currently, this is how we're pushing to production - please bear with me, I know this is far from sane:
From local, git push origin production (the production codebase is kept on the production branch, modifications are either written directly there and committed, or files are checked out individually from another branch. Origin is the remote production server)
On the remote box, sudo stop gunicorn (application is running as a process)
cp ~/flaskgit/application.py ~/flask/applicaion.py (the git push origin from local pushes to an init -bare repo with a post-update hook that populates the files in ~/flaskgit. ~/flask is where the gunicorn service runs the application under a virtualenv)
sudo start gunicorn
we do our testing with the ~/flaskgit code running on a different port. once it looks good we do the CP
I would love to have something more fluid. I have used jenkins in the past, and loved the experience - but didn't set it up.
What resources / utilities should I look up in order to do this well?
Thank you!
buildbot, jenkins/hudson but these give you continuous integration in the sense you can run a "make" equivalent with every code base change through a commit hook. You could also look at vagrant if there is something there for you for creating repeatable vm's wrt to config/setup. Could tie it with a commit hook.
I commit every time I make some changes that I think might work: I don't do extensive testing before a commit. Also, my commits will soon be automatically pushed to a remote repository. (I'm the only developer, and I have to add features or rewrite parts of the code many times a day.)
I'd like to set up a remote computer to run regression tests automatically whenever I commit anything; and then email me back the differences report.
What's the easiest way to set this up?
All my code is in Python 3. My own system is Windows 7, ActiveState Python, TortoiseHG, and Wing IDE. I can set up the remote computer as either Linux or Windows. The application is all command-line, with text input and output.
Use a continious integration server such as Buildbot or Jenkins and configure it to monitor the repository. Then run the tests using that. Buildbot is written in Python so you should feel right at home with it.
If you feel it's wasteful to make Buildbot or Jenkins poll the repository (even though hg pull uses very few resources when there are no new changesets), then you can configure a changegroup hook in the repository to trigger a build in the CI server.
I would recommend setting up Buildbot. You can have it watch a remote repository (Mercurial is supported) and automatically kick off a build when the repository changes. In your case, a build would just be running your test suite.
Its waterfall display allows you to see which builds failed and when, in relation to commits from the repository. It can even notify you, with the offending commit, when something breaks.
Jenkins is another option, supporting most of the same features. There are even cloud hosting options, like ShiningPanda that can host it for you, and they offer free licensing for open-source projects.
How can I test web2py units that are intended to be run on GAE?
I learned to run tests using this method:
python web2py.py -S api -M -R applications/api/tests/test.py
But how do I run tests with dev_appserver.py & web2py.py?
Unless you are testing requests, all you need to to do run unit tests is to have the Google App Engine SDK in your Python path.
I found it a little bit annoying so I built testable-appengine to automate the setup process (and add a couple nice things to the virtualenv it builds). It also has some interesting examples of how you can install third-party libraries for deployment alongside the application. I'd love to see how it works with Web2Py.
Having properly configured a Development server and a Production server, I would like to set up a Staging environment on Google App Engine useful to test new developed versions live before deploying them to production.
I know two different approaches:
A. The first option is by modifying the app.yaml version parameter.
version: app-staging
What I don't like of this approach is that Production data is polluted with my staging tests because (correct me if I'm wrong):
Staging version and Production version share the same Datastore
Staging version and Production version share the same logs
Regarding the first point, I don't know if it could be "fixed" using the new namespaces python API.
B. The second option is by modifying the app.yaml application parameter
application: foonamestaging
with this approach, I would create a second application totally independent from the Production version.
The only drawback I see is that I'm forced to configure a second application (administrators set up).
With a backup\restore tool like Gaebar this solution works well too.
What kind of approach are you using to set up a staging environment for your web application?
Also, do you have any automated script to change the yaml before deploying?
If separate datastore is required, option B looks cleaner solution for me because:
You can keep versions feature for real versioning of production applications.
You can keep versions feature for traffic splitting.
You can keep namespaces feature for multi-tenancy.
You can easily copy entities from one app to another. It's not so easy between namespaces.
Few APIs still don't support namespaces.
For teams with multiple developers, you can grant upload to production permission for a single person.
I chose the second option in my set-up, because it was the quickest solution, and I didn't make any script to change the application-parameter on deployment yet.
But the way I see it now, option A is a cleaner solution. You can with a couple of code lines switch the datastore namespace based on the version, which you can get dynamically from the environmental variable CURRENT_VERSION_ID as documented here: http://code.google.com/appengine/docs/python/runtime.html#The_Environment
We went with the option B. And I think it is better in general as it isolates the projects completely. So for example playing around with some of the configurations on the staging server will not affect and wont compromise security or cause any other butterfly effect in your production environment.
As for the deployment script, you can have any application name you want in your app.yaml. Some dummy/dev name and when you deploy, just use an -A parameter:
appcfg.py -A your-app-name update .
That will simplify your deploy script quite much, no need to string replace or anything similar in your app.yaml
We use option B.
In addition to Zygmantas suggestions about the benefits of separating dev from prod at application level, we also use our dev application to test performance.
Normally the dev instance runs without much available in the way of resources, this helps to see where the application "feels" slow. We can then also independently tweak the performance settings to see what makes a difference (e.g. front-end instance class).
Of course sometimes we need to bite the bullet and tweak & watch on live. But it's nice to have the other application to play with.
Still use namespaces and versions, just dev is dirty and experimental.
Here is what the Google documentation says :
A general recommendation is to have one project per application per
environment. For example, if you have two applications, "app1" and
"app2", each with a development and production environment, you would
have four projects: app1-dev, app1-prod, app2-dev, app2-prod. This
isolates the environments from each other, so changes to the
development project do not accidentally impact production, and gives
you better access control, since you can (for example) grant all
developers access to development projects but restrict production
access to your CI/CD pipeline
With this in mind, add a dispatch.yaml file at the root directory, and in each directory or repository that represents a single service and contain that service, add a app.yaml file along with the associated source code, as explained here : Structuring web services in App Engine
Edit, check out the equivalent link in the python section if you're using python.
No need to create a separate project. You can use dispatch.yaml to route your staging URL to another service (staging) in the same project.
Create a custom domain staging.yourdomain.com
Create a separate app-staging.yaml, that specifies staging service.
...
service: staging
...
Create distpatch.yaml that contains something like
...
url: "*staging.mydomain.com/"
service: staging
url: "*mydomain.com/"
service: default
...
gloud app deploy app-staging.yaml dispatch.yaml
use of application in app.yaml has been shut down.
Instead Google recommends
gcloud app deploy --project [YOUR_PROJECT_ID]
Please see https://cloud.google.com/appengine/docs/standard/python/config/appref
I'd like to write some Python unit tests for my Google App Engine. How can I set that up? Does someone happen to have some sample code which shows how to write a simple test?
GAEUnit is a unit test framework that helps to automate testing of your Google App Engine application.
Update: The Python SDK now provides a testbed module that makes service stubs available for unit testing. Documentation here.
Google's Python SDK now allows for this via the unittest module. More details here.
One note that you might find useful: To actually execute the tests, you should use NoseGAE. From the commandline, use:
$ sudo easy_install nose
$ sudo easy_install NoseGAE
(you can alternatively use pip for a virtual environment installation)
Then cd into your app's source directory and run:
$ nosetests --with-gae
That will run all the unit tests for your app.
One working solution is using following combination (as described in http://www.cuberick.com/2008/11/unit-test-your-google-app-engine-models.html)
Nose
Nose GAE
GAE Testbed
Since, gae is based on webhooks it can be easy to set your own testing framework for all relevant urls in your app.yaml. You can test it on sample dataset on development server ( start devel server with --datastore_path option ) and assert writes to database or webhook responses.