Is there any software library that provides an interface for storing and querying data like the Google App Engine Datastore, but uses a local file or service instead of running on App Engine?
The specific features I am looking for are:
Stores data as Entities with Named Properties
Query support
Atomic transactions
Python language bindings
Runs on my local machine
either stores to a single file
or connects to a local database
service
Free and open source
Thanks
You can also check MongoDB. It is an open source document-oriented database system.
You may also want to check out Appscale (http://www.appscale.com). It lets you run your App Engine apps without modification outside of Google (on your laptop, on your local cluster / behind your firewall, or in Amazon EC2). AppScale is and does each of the requirements you list here. It automatically installs/configures/manages the datastore service (and all other APIs/services) for your apps to use, so you don't have to.
Have a look at ZODB - not exactly alike but similiar http://www.zodb.org/
from the docs
Some of the features that ZODB brings to you:
Transparent persistence for Python objects
Full ACID-compatible
transaction support (including savepoints) History/undo ability
Efficient support for binary large objects (BLOBs)
Pluggable storages
Scalable architecture
Related
I am building this ecommerce app with django, and I was thinking if the default database (sqlite) was fine enough? My django app is going to have around 200 different products, and for payments I will use stripe API. I do not expect too much traffic when the app is up, since it's a website only for the country I live in with 6 million inhabitants. So can I settle with the sqlite database?
Although your answer may seem opinion based, the right answer is no anyways. SQLite3 is a lightweight database, which is commonlu used when your project is small and you don't mind to start up a server.
SQLite comes with following advantages and disadvantages (source)
Advantages of SQLite
Small footprint: As its name implies, the SQLite library is very lightweight. Although the space it uses varies depending on the system where it’s installed, it can take up less than 600KiB of space. Additionally, it’s fully self-contained, meaning there aren’t any external dependencies you have to install on your system for SQLite to work.
User-friendly: SQLite is sometimes described as a “zero-configuration” database that’s ready for use out of the box. SQLite doesn’t run as a server process, which means that it never needs to be stopped, started, or restarted and doesn’t come with any configuration files that need to be managed. These features help to streamline the path from installing SQLite to integrating it with an application.
Portable: Unlike other database management systems, which typically store data as a large batch of separate files, an entire SQLite database is stored in a single file. This file can be located anywhere in a directory hierarchy, and can be shared via removable media or file transfer protocol.
Disadvantages of SQLite
Limited concurrency: Although multiple processes can access and query an SQLite database at the same time, only one process can make changes to the database at any given time. This means SQLite supports greater concurrency than most other embedded database management systems, but not as much as client/server RDBMSs like MySQL or PostgreSQL.
No user management: Database systems often come with support for users, or managed connections with predefined access privileges to the database and tables. Because SQLite reads and writes directly to an ordinary disk file, the only applicable access permissions are the typical access permissions of the underlying operating system. This makes SQLite a poor choice for applications that require multiple users with special access permissions.
Security: A database engine that uses a server can, in some instances, provide better protection from bugs in the client application than a serverless database like SQLite. For example, stray pointers in a client cannot corrupt memory on the server. Also, because a server is a single persistent process, a client-server database cancontrol data access with more precision than a serverless database, allowing for more fine-grained locking and better concurrency.
It depends where are you going to host it. Some servers do not accept SQLite3 as database and require something more complicated as PostgreSQL (like Heroku). But if you are hosting by yourself (Rapsberry Pi for instance) then it's going to be fine and simple, if the site is going to have small traffic.
I am writing a micro-service that will have to share database owned by a different micro-service.
I understand that from a micro-services architecture perspective, this is not a good design. Hence, I decided to separate out the database access as another micro-service, whose only task it to manage access to db.
I need pointers on how I may write such an app using Python which exposes API for read/write to a database?
I realize this is not a design perspective answer.
Did you have a chance to take a look at sandman, a Python library that can generate a REST API over a database?
Is it possible to setup the App Engine SDK on my local machine to use the live datastore while developing? Sometimes it's just easier for my workflow to work live.
If not, is there an easy way to download or sync the live data to development machine?
Thanks!
TL;DR: We do not support having the dev_appserver use the real app-engine datastore. Even with the suggested use of "remote_api", AFAIK, the dev_appserver does not know how to use it.
If you really want to make this work, you could write your own low-level API and have your own datastore abstraction that uses your API instead of the actual datastore, however this is a non trivial amount of work.
Another option is to have a servlet that can pre-populate your dev datastore with the data you need from checked in files. The checked in raw data could be non-real data or obfuscated real data. At dev_appserver startup, you hit this URL and your database becomes pre-populated with data. If you take this route, you get the bonus of not operating on your live data with dev code.
HTH!
Instead of "touching" live data from the development server I'd recommend downloading a copy of all your production data locally with appcfg.py download_data/upload_data if you wish to move changes from development to production you can explicitly use the same commands to override existing entities.
I am looking to back up user-generated data (user profiles, that may change from time to time) from my AppEngine python application into Google Cloud Storage. I could easily periodically brute-force back up all of the user-generated data, but it probably makes more sense to only update data that has changed (only writing it to the cloud storage if the user has changed their data). Later, in the case that data needs to be restored, I would like to take advantage of the object-versioning functionality of the Cloud Storage service to determine which objects need to be restored.
I am trying to understand exactly how the google cloud storage interacts with AppEngine based on the information regarding cloudstorage.open() found at https://developers.google.com/appengine/docs/python/googlecloudstorageclient/functions. However, there is no indication of how this service interacts with versioned objects that are stored in the cloud (versioned objects are documented here: https://developers.google.com/storage/docs/object-versioning).
So, my question is: how can an application running on the AppEngine access specific versions of objects that are stored in Google Cloud Storage.
If there is a better way of doing this, I would be interested in hearing about it as well.
The AppEngine GCS Client Library doesn't support versioning at this time. If you enable versioning on a bucket through other channels, the GCS Client Library will keep working fine, but in order to access or delete older generations of objects, you'll need to use either the XML API or the JSON API (as opposed to the appengine-specific API). There is a Python client for the JSON API that works fine from within appengine, but you'll lose a few of appengine's niceties by using it. See https://developers.google.com/appengine/docs/python/googlecloudstorageclient/#gcs_rest_api for more details.
Here's a bit of info on how to use versioning from the XML and JSON APIs: https://developers.google.com/storage/docs/generations-preconditions
Coming from a java background I'm used to having a bunch of properties files I can swap round at runtime dependent on what server I'm running on e.g. dev/production.
Is there a method in python to do similar, specifically on Google's App Engine framework?
At the minute I have them defined in .py files, obviously I'd like a better separation.
You can:
edit records in the datastore through the dashboard ( if you really have to )
upload new scripts / files ( you can access files in READ-ONLY )
export a WEB Service API to configuration records in the datastore ( probably not what you had in mind )
access a page somewhere through an HTTP end-point
I don't see what is wrong with using python files to configure your application (apart from cultural issues :) ). In fact I have an issue with frameworks which don't allow me to script the configuration parameters.
That said, please have a look http://aaron.oirt.rutgers.edu/myapp/docs/W1100_2300.GAEDeploy for
a discussion of how to configure WHIFF application resources to configure applications to work in and out of the GAE framework in a portable manner.