We're trying to enable a SQL query front-end to our Web application, which is WSGI and uses Python, with SQLAlchemy (core, not ORM) to query a PostgreSQL database. We have several data layer functions set up to assist in query construction, and we are now trying to set something up that allows this type of query:
select id from <table_name> where ... limit ...
In the front end, we have a text box which lets the user type in the where clause and the limit clause, so that the data can be queried flexibly and dynamically from the front end, that is, we want to enable ad hoc querying. So, the only thing that we now firsthand is:
select id from <table_name>
And the user will type in, for example:
where date > <some_date>
where location is not null limit 10 order by location desc
using the same back end function. The select, column and table should be managed by the data layer (i.e. it knows what they are, and the user should not need to know that). However, I'm not aware of any way to get SQLAlchemy to automatically parse both the where clause and the limit clause automatically. What we have right now is a function which can return the table name and the name of the id column, and then use that to create a text query, which is passed to SQLAlchemy, as the input to a text() call.
Is there any way I can do this with SQLAlchemy, or some other library? Or is there a better pattern of which I should be aware, which does not involve parsing the SQL while still allowing this functionality from the front-end?
Thanks a lot! All suggestions will be greatly appreciated.
I'm not sure I follow, but the general SQL-Alchemy usage is like:
results = db.session.query(User).filter(User.name == "Bob").order_by(User.age.desc()).limit(10)
That will query the User table to return the top ten oldest members named "Bob"
Related
I have researched this topic and have decided just to ask here since I can't seem to find anything. I'll explain below:
Context: Flask Application with a form the client fills out and posts to the server. The form inputs are used to create a query and return data.
I am using SQLalchemy currently to construct the query from scratch. At this point, I have successfully connected to my existing Redshift database and can query properly but I cannot figure out how to dynamically construct a simple Select x, y, z statement based on the user's form inputs.
The main problem being that Query() can't take in a python list of columns. It seems you must specify each column like table.c.column1 which doesn't work well with a dynamic query since I don't know what columns I want until the user submits the form.
My 2 ideas so far:
Loop through all column names and use Query.add_columns(table.c['colname'])
Use select([col1, col2, ...]) instead of Query()
Use load_columns() to load only specific columns in a table to query. Unfortunately seems to only work with model objects and not reflected tables unless I am mistaken
Both of these seem backwards to me as they do not really accomplish my goal effectively.
SQLAlchemy is quite flexible, so both 1 and 2 get the job done. If you've no need for ORM functionality, then perhaps #2 is more natural. If the user were to pass a list of column names such as
columns = request.args.getlist('columns')
you could then create your select() quite easily with a bunch of column() constructs:
stmt = select([column(c) for c in columns]).\
select_from(some_table)
or if you have the table at hand, like you hint in the question:
stmt = select([table.c[c] for c in columns])
and then all that is left is to execute your statement:
results = db.session.execute(stmt).fetchall()
I have the following models, where one Thing can have multiple Action:
class Thing(polymodel.PolyModel):
create_date = db.DateTimeProperty(required=True)
class Action(db.Model):
thing = db.ReferenceProperty(Thing, collection_name='action')
action_by = db.StringProperty(required=True)
I need to use raw GQL to count the number of Things that have no actions. It should be something like:
SELECT * FROM Thing WHERE action = []
I have the following limitations:
I must use raw GQL (because the actual query contains DISTINCT which is not supported on a regular Query).
I can't fetch the data and check the contents because I use remote api and would like only to count the data to save bandwith.
Can it be done?
Firstly, you're completely mistaken to say that you have to use GQL because normal queries don't support DISTINCT: there is nothing you can do with GQL that you can't do with a normal query. The datastore is not a database, and does not have an underlying query language that ORM calls must be translated to; on the contrary, in fact GQL calls are translated into RPC calls in exactly the same way as model calls, and there is no benefit to using GQL at all. In this specific case, the Query class has a distinct parameter.
However, another implication of the datastore not being an SQL database is that you cannot do JOINs. There is no way to select instances of Thing based on any property in Action, whether it's a specific field value or the absence of any relation. The only way to do this would be to get all distinct values of Action.thing, then all Things, and work out the set difference.
I have a table called User and in Web2Py I would to get Users that match certain ids say (1,5,11,16).
Do I have to write SQL query itself or can it be elegantly done using Web2Py style queries.
The belongs operator
Two questions:
i want to generate a View in my PostGIS-DB. How do i add this View to my geometry_columns Table?
What i have to do, to use a View with SQLAlchemy? Is there a difference between a Table and View to SQLAlchemy or could i use the same way to use a View as i do to use a Table?
sorry for my poor english.
If there a questions about my question, please feel free to ask so i can try to explain it in another way maybe :)
Nico
Table objects in SQLAlchemy have two roles. They can be used to issue DDL commands to create the table in the database. But their main purpose is to describe the columns and types of tabular data that can be selected from and inserted to.
If you only want to select, then a view looks to SQLAlchemy exactly like a regular table. It's enough to describe the view as a Table with the columns that interest you (you don't even need to describe all of the columns). If you want to use the ORM you'll need to declare for SQLAlchemy that some combination of the columns can be used as the primary key (anything that's unique will do). Declaring some columns as foreign keys will also make it easier to set up any relations. If you don't issue create for that Table object, then it is just metadata for SQLAlchemy to know how to query the database.
If you also want to insert to the view, then you'll need to create PostgreSQL rules or triggers on the view that redirect the writes to the correct location. I'm not aware of a good usage recipe to redirect writes on the Python side.
I'm trying to create a system in Python in which one can select a number of rows from a set of tables, which are to be formatted in a user-defined way. Let's say the table a has a set of columns, some of which include a date or timestamp value. The user-defined format for each column should be stored in another table, and queried and applied on the main query at runtime.
Let me give you an example: There are different ways of formatting a date column, e.g. using
SELECT to_char(column, 'YYYY-MM-DD') FROM table;
in PostgreSQL.
For example, I'd like the second parameter of the to_char() builtin to be queried dynamically from another table at runtime, and then applied if it has a value.
Reading the definition from a table as such is not that much of a problem, rather than creating a database scheme which would receive data from a user interface from which a user can select which formatting instructions to apply to the different columns. The user should be able to pick the user's set of columns to be included in the user's query, as well as the user's user defined formatting for each column.
I've been thinking about doing this in an elegant and efficient way for some days now, but to no avail. Having the user put in the user's desired definition in a text field and including it in a query would pretty much generate an invitation for SQL injection attacks (although I could use escape() functions), and storing every possible combination doesn't seem feasible to me either.
It seems to me a stored procedure or a sub-select would work well here, though I haven't tested it. Let's say you store a date_format for each user in the users table.
SELECT to_char((SELECT date_format FROM users WHERE users.id=123), column) FROM table;
Your mileage may vary.
Pull the dates out as Unix timestamps and format them in Python:
SELECT DATE_PART('epoch', TIMESTAMP(my_col)) FROM my_table;
my_date = datetime.datetime.fromtimestamp(row[0]) # Or equivalent for your toolkit
I've found a couple of advantages to this approach: unix timestamps are the most space-efficient common format (this approach is effectively language neutral) and because the language you're querying the database in is richer than the underlying database, giving you plenty of options if you start wanting to do friendlier formatting like "today", "yesterday", "last week", "June 23rd".
I don't know what sort of application you're developing but if it's something like a web app which will be used by multiple people I'd also consider storing your database values in UTC so you can apply user-specific timezone settings when formatting without having to consider them for all of your database operations.