I need to execute a query which compares only the year and month value from TIMESTAMP column where the records look like this:
2015-01-01 08:33:06
The SQL Query is very simple (the interesting part is the year(timestamp) and month(timestamp) which extracts the year and the month so I can use them for comparison:
SELECT model, COUNT(model) AS count
FROM log.logs
WHERE SOURCE = "WEB"
AND year(timestamp) = 2015
AND month(timestamp) = 01
AND account = "TEST"
AND brand = "Nokia"
GROUP BY model
ORDER BY count DESC limit 10
Now the problem:
This is my SQLAlchemy Query:
devices = (db.session.query(Logs.model, Logs.timestamp,
func.count(Logs.model).label('count'))
.filter_by(source=str(source))
.filter_by(account=str(acc))
.filter_by(brand=str(brand))
.filter_by(year=year)
.filter_by(month=month)
.group_by(Logs.model)
.order_by(func.count(Logs.model).desc()).all())
The part:
.filter_by(year=year)
.filter_by(month=month)
is not the same as
AND year(timestamp) = 2015
AND month(timestamp) = 01
and my SQLAchemy query is not working. It seems like year and month are MySQL functions that extract the values from a timestamp column.
My DB Model looks like this:
class Logs(db.Model):
id = db.Column(db.Integer, primary_key=True)
timestamp = db.Column(db.TIMESTAMP, primary_key=False)
.... other attributes
It is interesting to mention that when I select and print Logs.timestamp it is in the following format:
(datetime.datetime(2013, 7, 11, 12, 47, 28))
How should this part be written in SQLAlchemy if I want my SQLAlchemy query to compare by the DB Timestamp year and month ?
.filter_by(year=year) #MySQL - year(timestamp)
.filter_by(month=month) #MySQL- month(timestamp)
I tried .filter(Logs.timestamp == year(timestamp) and similar variations but no luck. Any help will be greatly appreciated.
Simply replace:
.filter_by(year=year)
.filter_by(month=month)
with:
from sqlalchemy.sql.expression import func
# ...
.filter(func.year(Logs.timestamp) == year)
.filter(func.month(Logs.timestamp) == month)
Read more on this in SQL and Generic Functions section of documentation.
You can use custom constructs if you want to use functions that are specific to your database, like the year function you mention for MySQL. However I don't use MySQL and cannot give you some tested code (I did not even know about this function, by the way).
This would be an simple and useless example for Oracle (which is tested). I hope from this one you can quite easily deduce yours.
from sqlalchemy.sql import expression
from sqlalchemy.ext.compiler import compiles
from sqlalchemy import Date
class get_current_date(expression.FunctionElement):
type = Date()
#compiles(get_current_date, 'oracle')
def ora_get_current_date(element, compiler, **kw):
return "CURRENT_DATE"
session = schema_mgr.get_session()
q = session.query(sch.Tweet).filter(sch.Tweet.created_at == get_current_date())
tweets_today = pd.read_sql(q.statement, session.bind)
However I don't need to mention that this way you make your highly portable SQLAlchemy code a bit less portable.
Hope it helps.
Related
Using the following SQL expression but I'm getting an error.
select
CampaignCustomer.CampaignCustomerID,
convert(varchar, CampaignCustomer.ModifiedDate, 111) as startdate,
CampaignCustomer.CampaignID,
CampaignCustomer.CampaignCallStatusID,
CampaignCustomer.UserID,
CampaignCustomerSale.Value,
Users.Name
from CampaignCustomer
inner join CampaignCustomerSale
on CampaignCustomer.CampaignCustomerID = CampaignCustomerSale.CampaignCustomerID
inner join Users
on CampaignCustomer.UserID = Users.UserID
where
CampaignCustomer.CampaignCallStatusID = 21
and CampaignCustomer.startdate = '2011/11/22' <------- THIS
order by
startdate desc,
Users.Name asc
Error:
Msg 207, Level 16, State 1, Line 1
Invalid column name 'startdate'.
I can't recognize my alias name startdate in the WHERE clause, but it can in my ORDER BY clause. What's wrong?
Edit:
And no, it is not possible for me to change the datatype to date instead of datetime. The time is needed elsewhere. But in this case, I need only to get all posts on a specific date and I really don't care about what time of the date the modifieddate is :)
Maybe another method is needed instead of convert()?
You can't use column alias in WHERE clause.
Change it to:
where
CampaignCustomer.CampaignCallStatusID = 21
and convert(varchar, CampaignCustomer.ModifiedDate, 111) = '2011/11/22'
Do this:
select
CampaignCustomer.CampaignCustomerID,
convert(varchar, CampaignCustomer.ModifiedDate, 111) as startdate,
CampaignCustomer.CampaignID,
CampaignCustomer.CampaignCallStatusID,
CampaignCustomer.UserID,
CampaignCustomerSale.Value,
Users.Name
from CampaignCustomer
inner join CampaignCustomerSale
on CampaignCustomer.CampaignCustomerID = CampaignCustomerSale.CampaignCustomerID
inner join Users
on CampaignCustomer.UserID = Users.UserID
where
CampaignCustomer.CampaignCallStatusID = 21
and convert(varchar, CampaignCustomer.ModifiedDate, 111) = '2011/11/22'
order by
startdate desc,
Users.Name asc
You need to put in your where clause no aliases, and in the above query I replaced your alias with what it represents.
You didn't mention what version of SQL Server you're using - but if you're on 2008 or newer, you could use:
where
CampaignCustomer.CampaignCallStatusID = 21
and CAST(CampaignCustomer.ModifiedDate AS DATE) = '20111122'
You could cast it to a DATE - just for this comparison.
Also: I would recommend to always use the ISO-8601 standard format of representing a date if you need to compare a date to string - ISO-8601 defines a date as YYYYMMDD and is the only format in SQL Server that will always work - no matter what language/regional settings you have. Any other string representation of a date is always subject to settings in your SQL Server - it might work for you, but I bet for someone else, it will break....
I'm trying to query a database using Python/Pandas. This will be a recurring request where I'd like to look back into a window of time that changes over time, so I'd like to use some smarts in how I do this.
In my SQLite query, if I say
WHERE table.date BETWEEN DATETIME('now', '-6 month') AND DATETIME('now')
I get the result I expect. But if I try to move those to variables, the resulting table comes up empty. I found out that the endDate variable does work but the startDate does not. Presumably I'm doing something wrong with the escapes around the apostrophes? Since the result is coming up empty it's like it's looking at DATETIME(\'now\') and not seeing the '-6 month' bit (comparing now vs. now which would be empty). Any ideas how I can pass this through to the query correctly using Python?
startDate = 'DATETIME(\'now\', \'-6 month\')'
endDate = 'DATETIME(\'now\')'
query = '''
SELECT some stuff
FROM table
WHERE table.date BETWEEN ? AND ?
'''
df = pd.read_sql_query(query, db, params=[startDate, endDate])
You can try with the string format as shown below,
startDate = "DATETIME('now', '-6 month')"
endDate = "DATETIME('now')"
query = '''
SELECT some stuff
FROM table
WHERE table.date BETWEEN {start_date} AND {end_data}
'''
df = pd.read_sql_query(query.format(start_date=startDate, end_data=endDate), db)
When you provide parameters to a query, they're treated as literals, not expressions that SQL should evaluate.
You can pass the function arguments rather than the function as a string.
startDate = 'now'
startOffset = '-6 month'
endDate = 'now'
endOffset = '+0 seconds'
query = '''
SELECT some stuff
FROM table
WHERE table.date BETWEEN DATETIME(?, ?) AND DATETIME(?, ?)
'''
df = pd.read_sql_query(query, db, params=[startDate, startOffset, endDate, endOffset])
I'm trying to create a simple daily time recording app, that updates the table row upon submitting.
Here's what I mean, suppose a staff timed-in in the morning, then my table row would be like this:
id
time_in_am
time_out_am
time_in_pm
time_out_pm
staff_id
1
2021-05-09 08:17:07.27
NULL
NULL
NULL
223-8881
and upon submitting or scanning an id again, then it would update time_out_am, until the end of the day which is time_out_pm.
My problem then starts here, how would I know if the staff with an id no. of 223-8881 already clocked in today?
I've tried this:
today_dt = datetime(datetime.today().year, datetime.today().month, datetime.today().day)
# check if staff clocked in today
dtr_log = DailyTimeRecord.query.filter(DailyTimeRecord.time_in_am==today_dt, staff_id=staff.id).first()
# end check
using the above code, I get the error: TypeError: filter() got an unexpected keyword argument 'staff_id'
and if I use filter_by(), I get this: filter_by() takes 1 positional argument but 2 were given
heres my model if it helps:
class DailyTimeRecord(db.Model):
id = db.Column(db.Integer, primary_key=True)
time_in_am = db.Column(db.DateTime(timezone=True))
time_out_am = db.Column(db.DateTime(timezone=True))
time_in_pm = db.Column(db.DateTime(timezone=True))
time_out_pm = db.Column(db.DateTime(timezone=True))
staff_id = db.Column(db.Integer, db.ForeignKey('staff.id'))
When you're using filter you need to specify the model and use 'proper' comparisons e.g: the staff_id=staff.id should be DailyTimeRecord.staff_id==staff.id so would look like:
dtr_log = DailyTimeRecord.query.filter(
DailyTimeRecord.time_in_am==today_dt,
DailyTimeRecord.staff_id==staff.id
).first()
If you were using the filter_by helper it would look like:
dtr_log = DailyTimeRecord.query.filter(
time_in_am=today_dt,
staff_id=staff.id
).first()
But you'll also run into a problem with the date comparison. time_in_am is a datetime, and you're building a datetime to compare with, but essentially your today_dt is a datetime at midnight (the hour, min, sec will default to zero because you didn't give them a value).
You really want to just deal with a date to date comparison-- so would want to cast the database value to also be a date to do what you mention, so with filter:
from sqlalchemy import func
from datetime import date
today_dt = date.today()
dtr_log = DailyTimeRecord.query.filter(
func.date(DailyTimeRecord.time_in_am)==today_dt,
DailyTimeRecord.staff_id==staff.id
).first()
I would like to do a SUM on rows in a database and group by date.
I am trying to run this SQL query using Django aggregates and annotations:
select strftime('%m/%d/%Y', time_stamp) as the_date, sum(numbers_data)
from my_model
group by the_date;
I tried the following:
data = My_Model.objects.values("strftime('%m/%d/%Y',
time_stamp)").annotate(Sum("numbers_data")).order_by()
but it seems like you can only use column names in the values() function; it doesn't like the use of strftime().
How should I go about this?
This works for me:
select_data = {"d": """strftime('%%m/%%d/%%Y', time_stamp)"""}
data = My_Model.objects.extra(select=select_data).values('d').annotate(Sum("numbers_data")).order_by()
Took a bit to figure out I had to escape the % signs.
As of v1.8, you can use Func() expressions.
For example, if you happen to be targeting AWS Redshift's date and time functions:
from django.db.models import F, Func, Value
def TimezoneConvertedDateF(field_name, tz_name):
tz_fn = Func(Value(tz_name), F(field_name), function='CONVERT_TIMEZONE')
dt_fn = Func(tz_fn, function='TRUNC')
return dt_fn
Then you can use it like this:
SomeDbModel.objects \
.annotate(the_date=TimezoneConvertedDateF('some_timestamp_col_name',
'America/New_York')) \
.filter(the_date=...)
or like this:
SomeDbModel.objects \
.annotate(the_date=TimezoneConvertedDateF('some_timestamp_col_name',
'America/New_York')) \
.values('the_date') \
.annotate(...)
Any reason not to just do this in the database, by running the following query against the database:
select date, sum(numbers_data)
from my_model
group by date;
If your answer is, the date is a datetime with non-zero hours, minutes, seconds, or milliseconds, my answer is to use a date function to truncate the datetime, but I can't tell you exactly what that is without knowing what RBDMS you're using.
I'm not sure about strftime, my solution below is using sql postgres trunc...
select_data = {"date": "date_trunc('day', creationtime)"}
ttl = ReportWebclick.objects.using('cms')\
.extra(select=select_data)\
.filter(**filters)\
.values('date', 'tone_name', 'singer', 'parthner', 'price', 'period')\
.annotate(loadcount=Sum('loadcount'), buycount=Sum('buycount'), cancelcount=Sum('cancelcount'))\
.order_by('date', 'parthner')
-- equal to sql query execution:
select date_trunc('month', creationtime) as date, tone_name, sum(loadcount), sum(buycount), sum(cancelcount)
from webclickstat
group by tone_name, date;
my solution like this when my db is mysql:
select_data = {"date":"""FROM_UNIXTIME( action_time,'%%Y-%%m-%%d')"""}
qs = ViewLogs.objects.filter().extra(select=select_data).values('mall_id', 'date').annotate(pv=Count('id'), uv=Count('visitor_id', distinct=True))
to use which function, you can read mysql datetime processor docs like DATE_FORMAT,FROM_UNIXTIME...
Im trying to search for some values within a date range for a specific type, but content for dates that exist in the database are not being returned by the query.
Here is an extract of the python code:
deltaDays = timedelta(days= 20)
endDate = datetime.date.today()
startDate = endDate - deltaDays
result = db.GqlQuery(
"SELECT * FROM myData WHERE mytype = :1 AND pubdate >= :2 and pubdate <= :3", type, startDate, endDate
)
class myData(db.Model):
mytype = db.StringProperty(required=True)
value = db.FloatProperty(required=True)
pubdate = db.DateTimeProperty(required=True)
The GQL returns data, but some rows that I am expecting are missing:
2009-03-18 00:00:00
(missing date in results: 2009-03-20 data exists in database)
2009-03-23 00:00:00
2009-03-24 00:00:00
2009-03-25 00:00:00
2009-03-26 00:00:00
(missing date in results: 2009-03-27 data exists in database)
2009-03-30 00:00:00
(missing date in results: 2009-03-31. data exists in database)
2009-04-01 00:00:00
2009-04-02 00:00:00
2009-04-03 00:00:00
2009-04-06 00:00:00
I uploaded the data via de bulkload script. I just can think of the indexes being corrupted or something similar. This same query used to work for another table i had. But i had to replace it with new content from another source, and this new content is not responding to the query in the same way. The table has around 700.000 rows if that makes any difference.
I have done more research ant it appears that its a bug in the appEngine DataStore.
For more information about the bug check this link:
http://code.google.com/p/googleappengine/issues/detail?id=901
I have tried droping the index and recreating it with no luck.
thanks
nothing looks wrong to me. are you sure that the missing dates also have mytype == type?
i have observed some funny behaviour with indexes in the past. I recommend writing a handler to iterate through all of your records and just put() them back in the database. maybe something with the bulk uploader isn't working properly.
Here's the type of handler I use to iterate through all the entities in a model class:
class PPIterator(BaseRequestHandler):
def get(self):
query = Model.gql('ORDER BY __key__')
last_key_str = self.request.get('last')
if last_key_str:
last_key = db.Key(last_key_str)
query = Model.gql('WHERE __key__ > :1 ORDER BY __key__', last_key)
entities = query.fetch(11)
new_last_key_str = None
if len(entities) == 11:
new_last_key_str = str(entities[9].key())
for e in entities:
e.put()
if new_last_key_str:
self.response.out.write(json.write(new_last_key_str))
else:
self.response.out.write(json.write('done'))
You can use whatever you want to iterate through the entities. I used to use Javascript in a browser window, but found that was a pig when making hundreds of thousands of requests. These days I find it more convenient to use a ruby script like this one:
require 'net/http'
require 'json'
last=nil
while last != 'done'
url = 'your_url'
path = '/your_path'
path += "?/last=#{last}" if last
last = Net::HTTP.get(url,path)
puts last
end
Ben
UPDATE: now that remote api is working and reliable, I rarely write this type of handler anymore. The same ideas apply to the code you'd use there to iterate through the entities in the remote api console.