I am using Python to query a MySQL table and getting one datetime in string format, which is stored in row[3]. I need to convert this string timestamp to epoch seconds.
import MySQLdb
import os
import datetime
try:
db = MySQLdb.connect("localhost","root","","test" )
except MySQLdb.Error, e:
print "Error %d: %s" % (e.args[0], e.args[1])
sys.exit (1)
cursor = db.cursor()
cursor.execute ("SELECT * from main_tbl WHERE login_user_name='kumar'")
data = cursor.fetchall()
for row in data :
print row[3] ###printing 2014-09-26 12:24:23
date = datetime.datetime.strptime(row[3], "%Y-%m-%d %H:%M:%S")
print date
On execution it throws this error:
2014-09-26 12:24:23
Traceback (most recent call last):
File "test1.py", line 22, in <module>
date = datetime.datetime.strptime(row[3], "%Y-%m-%d %H:%M:%S")
TypeError: must be string, not datetime.datetime
What am I doing wrong?
I have also tried the following:
epoch_start = time.mktime(time.strptime(row[3], "%Y-%m-%d %H:%M:%S"));
But I get this error:
Traceback (most recent call last):
File "test1.py", line 29, in <module>
epoch_start = time.mktime(time.strptime(row[3], "%Y-%m-%d %H:%M:%S"));
File "C:\Python27\lib\_strptime.py", line 467, in _strptime_time
return _strptime(data_string, format)[0]
File "C:\Python27\lib\_strptime.py", line 322, in _strptime
found = format_regex.match(data_string)
TypeError: expected string or buffer
The value in row[3] is already in the datetime.datetime format as it is being clearly pointed out by the traceback. So there is no need for creating the variable date. You can use the row[3] directly as a datetime.datetime object.
Just try printing:
print type(row[3])
That should give the type as datetime.datetime
row[3] is already in datetime.datetime format.
From your question its sounds like you want convert to EPOCH
so do something like:
import time
epochTime = time.mktime(row[3].timetuple())
print epochTime
Then check if the converted epoch is correct or not:
print time.strftime('%Y-%m-%d %H:%M:%S', time.localtime(epochTime))
print row[3]
Verify last two statements have the same output.
Related
I want to check if my table is loaded correctly or not. If it is not loaded correctly then the number of records will be zero. I am using SQLCheckOperator to do this task.
This is the code
from airflow.operators.sql import SQLCheckOperator
from datetime import date, timedelta
CURRENT_DATE = str(date.today() - timedelta(2))
TABLE_NAME = "foo"
search_monolith_post_sanity = SQLCheckOperator(
task_id="search_monolith_post_sanity",
sql=f"SELECT COUNT(*) FROM `{TABLE_NAME}` WHERE feed_date = DATE_SUB('{CURRENT_DATE}', INTERVAL 1 DAY)",
bigquery_conn_id='bigquery_default',
use_legacy_sql=False,
dag=dag
)
I got the below error:
Executing SQL check: SELECT COUNT(*) FROM `foo` WHERE feed_date = DATE_SUB('2021-01-31', INTERVAL 1 DAY)
[2021-02-02 07:16:43,664] {taskinstance.py:1153} ERROR - 'NoneType' object has no attribute 'upper'
Traceback (most recent call last)
File "/usr/local/lib/airflow/airflow/models/taskinstance.py", line 986, in _run_raw_tas
result = task_copy.execute(context=context
File "/usr/local/lib/airflow/airflow/operators/sql.py", line 95, in execut
records = self.get_db_hook().get_first(self.sql
File "/usr/local/lib/airflow/airflow/operators/sql.py", line 116, in get_db_hoo
return BaseHook.get_hook(conn_id=self.conn_id
File "/usr/local/lib/airflow/airflow/hooks/base_hook.py", line 94, in get_hoo
connection = cls.get_connection(conn_id
File "/usr/local/lib/airflow/airflow/hooks/base_hook.py", line 87, in get_connectio
conn = random.choice(list(cls.get_connections(conn_id))
File "/usr/local/lib/airflow/airflow/hooks/base_hook.py", line 83, in get_connection
return secrets.get_connections(conn_id
File "/usr/local/lib/airflow/airflow/secrets/__init__.py", line 55, in get_connection
conn_list = secrets_backend.get_connections(conn_id=conn_id
File "/usr/local/lib/airflow/airflow/secrets/base_secrets.py", line 64, in get_connection
conn_uri = self.get_conn_uri(conn_id=conn_id
File "/usr/local/lib/airflow/airflow/secrets/environment_variables.py", line 39, in get_conn_ur
environment_uri = os.environ.get(CONN_ENV_PREFIX + conn_id.upper()
AttributeError: 'NoneType' object has no attribute 'upper
I have tried using BigQueryCheckOperator and CheckOperator instead of SQLCheckOperator but ran into error. If I replace BigQueryCheckOperator with BigQueryOperator the code works fine and I get zero as output.
I am new to airflow. Any help is much appreciated. Thanks !!
If you look at the line before the error message in the stacktrace.
environment_uri = os.environ.get(CONN_ENV_PREFIX + conn_id.upper()
AttributeError: 'NoneType' object has no attribute 'upper'
In this case the NoneType object that is having upper() called on it is conn_id.
If you're using Airflow 1.10.15 the documentation for this operator has a rather important Note buried at the bottom
Note that this is an abstract class and get_db_hook needs to be defined. Whereas a get_db_hook is hook that gets a single record from an external source.
Also note that the definition of the function appears to expect a conn_id parameter.
Assumption: You are using Airflow >= 2.0.0
Use the following code, notice usage of BigQueryCheckOperator and that I used gcp_conn_id instead of bigquery_conn_id.
from airflow.providers.google.cloud.operators.bigquery import BigQueryCheckOperator
from datetime import date, timedelta
CURRENT_DATE = str(date.today() - timedelta(2))
TABLE_NAME = "foo"
search_monolith_post_sanity = BigQueryCheckOperator(
task_id="search_monolith_post_sanity",
sql=f"SELECT COUNT(*) FROM `{TABLE_NAME}` WHERE feed_date = DATE_SUB('{CURRENT_DATE}', INTERVAL 1 DAY)",
gcp_conn_id='bigquery_default',
use_legacy_sql=False,
dag=dag
)
Hi i am using the library win32com.client to read emails from an outlook sharedmailbox and i get a value error:Microsecond must be in 0..999999. I tried formatting the "ReceivedDate" as "%Y-%D-%M %H:%M:%S" with no luck. do you know what else i can try?
Question 2: im trying to count how many emails have not been replied to but i dont see a property for that in the documentation. SO i went with reading all that have a "FlagRequest" marked as completed. I would have the representatives go with this process as a way to tell if the emails have been complete.
import win32com.client
def readEmail():
outlook = win32com.client.Dispatch("Outlook.Application").GetNamespace("MAPI")
folder = outlook.Folders.Item('SharedMailbox, TEST')
inbox = folder.Folders.Item('Inbox')
messages = inbox.Items
counter = 0
for message in messages:
rcvd_dt = message.ReceivedTime
if message.FlagRequest != 'Follow up' and str(rcvd_dt) >= '2020-06-01 00:00:00':
counter +=1
print(counter)
print(received_dt)
traceback:
Traceback (most recent call last):
File "C:/Users/TEST/PycharmProjects/TEST/Metrics.py", line 422, in <module>
main()
File "C:/Users/TEST/PycharmProjects/TEST/Metrics.py", line 409, in main
readEmail()
File "C:/Users/TEST/PycharmProjects/TEST/Metrics.py", line 89, in readEmail
rcvd_dt = message.ReceivedTime
File "C:\Program Files (x86)\Python37-32\lib\site-
packages\win32com\client\dynamic.py", line 516, in __getattr__
ret = self._oleobj_.Invoke(retEntry.dispid,0,invoke_type,1)
ValueError: microsecond must be in 0..999999
What i tried:
rcvd_dt= datetime.strptime(str(rcvd_dt.split('.')[0], '%Y-%m-%d %H:%M:%S')
but get error:
valueerror: time data '2' does not match format %Y-%m-%d %H:%M:%S.%f'
if i try:
rcvd_dt= datetime.strptime(str(rcvd_dt.split('.')[0], '%Y-%m-%d %H:%M:%S.%f')
i get :
valueerror: time data '2020-06-16 08:53:56' does not match format %Y-%m-%d.%f'
This is my code:
now = datetime.datetime.now().replace(microsecond=0)
curs.execute("SELECT name, msgDate, FROM test where msgDate=%s",(now))
I got these msgs:
File "C:\Python\lib\site-packages\mysql\connector\cursor.py", line 220, in _process_params
res = list(map(to_mysql,res))
TypeError: 'datetime.datetime' object is not iterable
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\projectse\Email\src\a.py", line 65, in <module>
curs.execute("SELECT name, msgDate FROM messages where msgDate=%s",(now))
File "C:\Python\lib\site-packages\mysql\connector\cursor.py", line 300, in execute
stmt = operation % self._process_params(params)
File "C:\Python\lib\site-packages\mysql\connector\cursor.py", line 225, in _process_params
"Failed processing format-parameters; %s" % e)
mysql.connector.errors.ProgrammingError: -1: Failed processing format-parameters;
'datetime.datetime' object is not iterable
Any tips?
I think you should replace:
curs.execute("SELECT name, msgDate, FROM test where msgDate=%s",(now))
with:
curs.execute("SELECT name, msgDate, FROM test where msgDate=%s",%(now))
and use this for your time variable:
import time
now=time.strftime('%Y-%m-%d %H:%M:%S')
Try seeing what now = str(datetime.datetime.now().replace(microsecond=0)) looks like in the python interpreter, if you're curious why that's a problem.
now = datetime.datetime(2012, 2, 23, 0, 0)
now.strftime('%m/%d/%Y')
curs.execute("SELECT name, msgDate, FROM test where msgDate=:now",dict(now=str(now))
please try converting date to string format and execute
This is my code for formatting a date:
def updateUserDBDates():
global userDB, currentDate, previousDate, changeInDate
index = 0
index2 = 0
userDB[1] = datetime.strptime("%d-%m-%Y", userDB[0])
userDB[0] = datetime.today().strftime("%d-%m-%Y")
saveData()
currentDate = userDB[0]
previousDate = userDB[1]
changeInDate = currentDate - previousDate
and I get this error:
File "/home/nathan/Documents/project001/programFiles/Project 001.py", line 170, in updateUserDBDates
userDB[1] = datetime.strptime("%d-%m-%Y", userDB[0])
File "/usr/lib/python2.7/_strptime.py", line 325, in _strptime
(data_string, format))
ValueError: time data '%d-%m-%Y' does not match format '28-09-2013'
From what I can see everything should work fine, what is causing this error and how can I fix it easily?
datetime.datetime.strptime receive date_str, format as arguments (not format, date_str):
>>> import datetime
>>> datetime.datetime.strptime('28-09-2013', '%d-%m-%Y')
datetime.datetime(2013, 9, 28, 0, 0)
The argument ordering for strptime() is wrong.
http://docs.python.org/2/library/datetime.html#datetime.datetime.strptime
I am trying to read date from excel file using xlrd module. Below is my code for this :
# Variables
myfile = '/home/mobaxterm/.git/Operation_Documentation/docs/Servicing Portal User & Certificate Inventory.xlsx'
mydate = 'Expiration Date'
row_head = 0
# Import required modules
import xlrd
import datetime
today = datetime.date.today()
book = xlrd.open_workbook(myfile)
sheet = book.sheet_by_index(1)
for col_index in range(sheet.ncols):
print xlrd.cellname(row_head,col_index),"-",
print sheet.cell(row_head,col_index).value
if sheet.cell(row_head,col_index).value == mydate:
for raw_index in range(sheet.nrows):
expire = sheet.cell(raw_index,col_index).value
print expire
expire_date = datetime.datetime(*xlrd.xldate_as_tuple(expire, book.datemode))
print 'datetime: %s' % expire_date
break
While running the code i am getting following error :
Traceback (most recent call last):
File "cert_monitor.py", line 31, in <module>
expire_date = datetime.datetime(*xlrd.xldate_as_tuple(expire, book.datemode))
File "/usr/lib/python2.6/site-packages/xlrd/xldate.py", line 61, in xldate_as_tuple
xldays = int(xldate)
ValueError: invalid literal for int() with base 10: 'Expiration Date'
Can anyone suggest what could be the issue here?
Thanks for your time.
I believe that you should only skip the header:
for raw_index in range(1, sheet.nrows):
...
You are checking that sheet.cell(row_head,col_index).value == mydate, and then you want to iterate over the rows, but you should skip row_head first - it is ==mydate, which is not a date but a simple 'Expiration Date' string.