Develop Reference

DolphinDB Python API: Get the number of records queried by SQL statements

I run the following script in Python:
t2 = t1.select("count(*)")
t2.toDF()
Output (of DataFrame type):
count
0 13136
If I run the script as follows:
t2.toDF().count # count is the column name corresponding to the result of count(*)
Output (of <class 'method'> data type):
<bound method DataFrame.count of count
0 13136>
How to get the number of records of type scalar?

Solution 1: use function pandas.DataFrame.count.
trade.select("*").toDF().count()["col"]
# col is the name of any column of the DataFrame returned by the SQL query
# Any SQL statement can be used before toDF()
Solution 2:
In DolphinDB, the output of t2=t1.select("count(*)") is stored in the “count“ column. t2.toDF().count does not return the result of the count column as there is a method count with the same name in pandas.DataFrame. It is recommended to specify aliases for the result columns returned by count(*).
t2 = t1.select("count(*) as cnt")
t2.toDF().cnt
Return a Seires. You can obtain the scalar with index.
t2.toDF().cnt[0]
See also: DolphinDB Python API

The toDF() function returns a DataFrame, please try t2.toDF()['count'][0].

How to convert database query with an in clause into a list of lists in SQLAlchemy

I have the following SQLAlchemy db query
test_query = Unit.query.filter(Unit.id_1.in_(('3D0U|1|A|G|1', '3D0U|1|A|C|160')))
I would like the result of this query to be a list of lists with rows corresponding to each element in the in clause as a separate list. Currently I'm getting all the rows in a single list
This is what I've tried
result = []
for row in test_query:
result.append[(row.id_2)]
When I print out the results, this is what I get
["3D0X|1|A|C|160", "4ERJ|1|A|C|160", "4ERL|1|A|C|160", "3D0X|1|A|G|1", "4ERJ|1|A|G|1", "4ERL|1|A|G|1"]
The desired output is:
[["3D0X|1|A|C|160", "4ERJ|1|A|C|160", "4ERL|1|A|C|160"], ["3D0X|1|A|G|1", "4ERJ|1|A|G|1", "4ERL|1|A|G|1"]]
Sample data from the Unit table is shown below
"id_1","chain_1","pdb_id_1","id_2","chain_2","pdb_id_2"
"3D0U|1|A|G|1","A","3D0U","3D0X|1|A|G|1","A","3D0X"
"3D0U|1|A|G|1","A","3D0U","4ERJ|1|A|G|1","A","4ERJ"
"3D0U|1|A|G|1","A","3D0U","4ERL|1|A|G|1","A","4ERL"
"3D0U|1|A|C|160","A","3D0U","3D0X|1|A|C|160","A","3D0X"
"3D0U|1|A|C|160","A","3D0U","4ERJ|1|A|C|160","A","4ERJ"
"3D0U|1|A|C|160","A","3D0U","4ERL|1|A|C|160","A","4ERL"
Any help is appreciated. Thanks

This could be solved in the following way: first order by Unit.id_1 in the query, and then
result = [[unit.id_2 for unit in units] for id_1, units in itertools.groupby(test_query, lambda x: x.id_1)]
In case of postgresql there is also sqlalchemy.func.array_agg that could be used to construct an array of id_2 grouped by id_1.
For reference, python documentation for itertools describes what itertools.groupby does.

sqlite3: How to use SELECT + WHERE to search a random row? [duplicate]

This question already has answers here:
SQLite - ORDER BY RAND()
(5 answers)
Closed 5 years ago.
Eg. In the Table1 there is a column ColName, some of the items in ColName are "Mike".
The code to search one of the them:
searchString = " SELECT * FROM Table1 WHERE ColName = 'Mike' "
cur.execute(searchString).fetchone()
The Problem: The code above allways gives me the first row, where "Mike" in ColName appears.
I actually want, by everytime running the sqlite code, to get a random search result from the column ColName, whose value is "Mike". How could I change the code?
Thanks for the help!

If you want a random value, then you need to iterate over cur.execute(searchString) for some random amount, then extract the column(s).
fetchone() always returns the top result
The alternative includes trying to randomly sort the query results in SQL

Getting COUNT from sqlalchemy

I have:
res = db.engine.execute('select count(id) from sometable')
The returned object is sqlalchemy.engine.result.ResultProxy.
How do I get count value from res?
Res is not accessed by index but I have figured this out as:
count=None
for i in res:
count = res[0]
break
There must be an easier way right? What is it? I didn't discover it yet.
Note: The db is a postgres db.

While the other answers work, SQLAlchemy provides a shortcut for scalar queries as ResultProxy.scalar():
count = db.engine.execute('select count(id) from sometable').scalar()
scalar() fetches the first column of the first row and closes the result set, or returns None if no row is present. There's also Query.scalar(), if using the Query API.

what you are asking for called unpacking, ResultProxy is an iterable, so we can do
# there will be single record
record, = db.engine.execute('select count(id) from sometable')
# this record consist of single value
count, = record

The ResultProxy in SQLAlchemy (as documented here http://docs.sqlalchemy.org/en/latest/core/connections.html?highlight=execute#sqlalchemy.engine.ResultProxy) is an iterable of the columns returned from the database. For a count() query, simply access the first element to get the column, and then another index to get the first element (and only) element of that column.
result = db.engine.execute('select count(id) from sometable')
count = result[0][0]
If you happened to be using the ORM of SQLAlchemy, I would suggest using the Query.count() method on the appropriate model as shown here: http://docs.sqlalchemy.org/en/latest/orm/query.html?highlight=count#sqlalchemy.orm.query.Query.count

SELECT * in SQLAlchemy?

Is it possible to do SELECT * in SQLAlchemy?
Specifically, SELECT * WHERE foo=1?

Is no one feeling the ORM love of SQLAlchemy today? The presented answers correctly describe the lower-level interface that SQLAlchemy provides. Just for completeness, this is the more-likely (for me) real-world situation where you have a session instance and a User class that is ORM mapped to the users table.
for user in session.query(User).filter_by(name='jack'):
print(user)
# ...
And this does an explicit select on all columns.

The following selection works for me in the core expression language (returning a RowProxy object):
foo_col = sqlalchemy.sql.column('foo')
s = sqlalchemy.sql.select(['*']).where(foo_col == 1)

If you don't list any columns, you get all of them.
query = users.select()
query = query.where(users.c.name=='jack')
result = conn.execute(query)
for row in result:
print row
Should work.

You can always use a raw SQL too:
str_sql = sql.text("YOUR STRING SQL")
#if you have some args:
args = {
'myarg1': yourarg1
'myarg2': yourarg2}
#then call the execute method from your connection
results = conn.execute(str_sql,args).fetchall()

Where Bar is the class mapped to your table and session is your sa session:
bars = session.query(Bar).filter(Bar.foo == 1)

Turns out you can do:
sa.select('*', ...)

I had the same issue, I was trying to get all columns from a table as a list instead of getting ORM objects back. So that I can convert that list to pandas dataframe and display.
What works is to use .c on a subquery or cte as follows:
U = select(User).cte('U')
stmt = select(*U.c)
rows = session.execute(stmt)
Then you get a list of tuples with each column.
Another option is to use __table__.columns in the same way:
stmt = select(*User.__table__.columns)
rows = session.execute(stmt)
In case you want to convert the results to dataframe here is the one liner:
pd.DataFrame.from_records(rows, columns=rows.keys())

For joins if columns are not defined manually, only columns of target table are returned. To get all columns for joins(User table joined with Group Table:
sql = User.select(from_obj(Group, User.c.group_id == Group.c.id))
# Add all coumns of Group table to select
sql = sql.column(Group)
session.connection().execute(sql)

I had the same issue, I was trying to get all columns from a table as a list instead of getting ORM objects back. So that I can convert that list to pandas dataframe and display.
What works is to use .c on a subquery or cte as follows:
U = select(User).cte('U')
stmt = select(*U.c)
rows = session.execute(stmt)
Then you get a list of tuples with each column.
Another option is to use __table__.columns in the same way:
stmt = select(*User.__table__.columns)
rows = session.execute(stmt)
In case you want to convert the results to dataframe here is the one liner:
pd.DataFrame.from_records(dict(zip(r.keys(), r)) for r in rows)

If you're using the ORM, you can build a query using the normal ORM constructs and then execute it directly to get raw column values:
query = session.query(User).filter_by(name='jack')
for cols in session.connection().execute(query):
print cols

every_column = User.__table__.columns
records = session.query(*every_column).filter(User.foo==1).all()
When a ORM class is passed to the query function, e.g. query(User), the result will be composed of ORM instances. In the majority of cases, this is what the dev wants and will be easiest to deal with--demonstrated by the popularity of the answer above that corresponds to this approach.
In some cases, devs may instead want an iterable sequence of values. In these cases, one can pass the list of desired column objects to query(). This answer shows how to pass the entire list of columns without hardcoding them, while still working with SQLAlchemy at the ORM layer.