Develop Reference

Iterating a list in AWS Lambda Function for Email Template using Python HTML

I'm building an email body using AWS lambda_function in Python. I've created html template that is provided below. I'm calling to MySQL Data base to get the data using python. I need to pass the data to the template to populate them in email body - iterating that list in a table. How can I pass it from Python Lambda_function so that I can display as shown below.
QA: I'm new to Python and AWS Lambda_function integration. Can you please show me how to pass record to sendemail and dynamically populate data in html table rows. Looking for code level solution, it would be great if you could help. Please feel free to modify the HTML template and Python code as necessary. Thanks.
This is my What I need to build -
Here is my HTML Template for Table
<tr>
<td style="padding:35px 70px 30px;" class="em_padd" valign="top" align="center" bgcolor="#f6f7f8">
<table width="100%%" cellspacing="0" cellpadding="10" border="1" align="center">
<tbody>
<tr style="font-weight: bold">
<td>Ticker</td>
<td>Order Type</td>
<td>Creation Units</td>
<td>Authorized Participant</td>
<td>Payment Type</td>
<td>Settlement Date</td>
<td>Restricted Securities</td>
</tr>
<% var value; for value in list{ %>
<tr>
<td>%(value.ticker)s</td>
<td>%(value.orderType)s</td>
<td>%(value.creationUnits)d</td>
<td>%(value.ap)s</td>
<td>%(value.paymentType)s</td>
<td>%(value.settlementDate)s</td>
<td>%(value.rs)s</td>
</tr>
}%>
</tbody>
</table>
</td>
</tr>
Here is lambda_function.py
import json
import fund_contacts
import client_agreement_contacts
import boto3
import os
import mysql.connector
from mysql.connector import Error
import json
from datetime import datetime
def sendemail(template_name, toAddr, subject, data):
sns_client = boto3.client(
'sns',
region_name=os.environ['REGION'],
aws_access_key_id=os.environ['ACCESS_KEY_ID'],
aws_secret_access_key=os.environ['SECRET_ACCESS_KEY'],
)
message = {
"tmpl_id": template_name,
"to_email": toAddr,
"subject": subject,
"data": data
}
# Send Event
sns_client.publish(
TargetArn=os.environ['EMAIL_ARN'], Message=json.dumps(message))
def getOrderSummary():
try:
mySQLConnection = mysql.connector.connect(host='asbc.ueue.rds.amazonaws.com',
database='QA',
user='abcd',
password='random#1')
cursor = mySQLConnection.cursor(buffered=True)
sql_select_query = """SELECT ord.ticker AS 'Ticker',
ord.transaction_type AS 'Order Type',
ord.unit_quantity AS 'Creation Units',
apf.legal_name AS 'Authorized Participant',
ord.payment_method AS 'Payment Type',
ord.settlement 'Settlement Date',
'No' AS 'Restricted Securities'
FROM ETP_QA.order ord
LEFT JOIN ETP_QA.apfirm apf
ON ord.apfirm = apf.id
WHERE order_time > Curdate() - INTERVAL 1 DAY """
cursor.execute(sql_select_query)
record = cursor.fetchall()
#my_json_string = json.dumps(record)
# print(my_json_string)
return record
except mysql.connector.Error as error:
print("Failed to get record from MySQL table: {}".format(error))
finally:
if (mySQLConnection.is_connected()):
cursor.close()
mySQLConnection.close()
print("MySQL connection is closed")
def lambda_handler(event, context):
if 'Records' in event:
message = json.loads(event['Records'][0]['Sns']['Message'])
else:
message = event
records = getOrderSummary()
if not records:
templateName = 'ORDER_SUMMARY_NO_DATA'
else:
templateName = 'ORDER_SUMMARY_DATA'
for row in records:
print("Ticker = ", row[0], )
print("Order Type = ", row[1])
print("Creation Units = ", row[2])
print("Authorized Participant = ", row[3])
print("Settlement Date = ", row[4], )
print("Restricted Securities = ", row[5], "\n")
recipient_list = 'SomeEmail#gmail.com'
current_time = datetime.now()
subject = 'ABC Investment Series Trust Trade Recap -' + current_time.strftime('%m/%d/%Y')
# How to set the message so I can itegrate them as per Email Template?
# message = records
sendemail(templateName, recipient_list, subject, message)
This is what data set that I'm getting from database looks like -
Ticker = BTAL
Order Type = Create
Creation Units = 1
Authorized Participant = Hamlin AP Firm
Settlement Date = Cash
Restricted Securities = T+1
Ticker = CFA
Order Type = Create
Creation Units = 1
Authorized Participant = Hamlin AP Firm
Settlement Date = Cash
Restricted Securities = T+1
Ticker = SSUS
Order Type = Create
Creation Units = 1
Authorized Participant = Hamlin AP Firm
Settlement Date = Cash
Restricted Securities = T+1
Ticker = TPHDE
Order Type = Create
Creation Units = 2
Authorized Participant = Hamlin AP Firm
Settlement Date = Inkind
Restricted Securities = T+1

Sqlite format output in python

I've got a simple flask web-app set up where the user can add and delete tasks from a database. All the entries in the database are displayed in a template, sorted by the type they're assigned. I can't format the output to be at least somewhat readable, though. How do i do that?
The actual database uses different values and whatnot, so not all of them may make sense right now.
This is the function that gets all the entries from my sqlite database:
def get_tsk_by_type(type):
c.execute("SELECT * FROM tasks WHERE type=:type", {'type': type})
result = c.fetchall()
return result
The database:
c.execute("""CREATE TABLE IF NOT EXISTS tasks (
type text,
description text,
amount integer,
id integer
)""")
And here's how i'm returning all the entries that are then displayed in in a template. There is also a function to delete tasks if you input their id.
#app.route('/', methods = ['GET', 'POST'])
def index():
form = DeleteForm()
curr_homework = str(get_tsk_by_type("homework"))
curr_cleaning = str(get_tsk_by_type("cleaning"))
curr_cooking = str(get_tsk_by_type("cooking"))
if form.validate_on_submit():
try:
conn = sqlite3.connect('tasks.db', check_same_thread=False)
c = conn.cursor()
delete = request.form['delete']
if (delete):
remove_tsk(delete)
return redirect('/')
conn.close()
except:
return "Something went wrong while submitting the form"
return render_template('index.html', curr_homework = curr_homwork, curr_cleaning = curr_cleaning, curr_cooking = curr_cooking, form = form)
The relevant parts of my index.html look like this:
{% block content %}
<div>
<p>
<span>Currently registered homework: {{ curr_homework }}</span><br />
<span>Currently registered cleaning tasks: {{ curr_cleaning }}</span><br />
<span>Currently registered cooking tasks {{ curr_cooking }}</span>
</p>
</div>
{% endblock content %}
However, the output i'm getting looks like this:
Currently registered homework: [('homework', 'math', 1, 'df19c0b1-a2128-431274-2e32-3a2f901b1b26')]
Currently registered cleaning tasks: [('cleaning', 'kitchen', 1, 'df19c0b1-aa18-4874-9e32-3a2f901b1b26')]
Currently registered cooking tasks: [('cooking', 'lunch', 1, '0697c139-0299-4c93-88ac-c07d77377796')]
I've tried for-loops and that kinda thing but it only ever returns the first tuple inside the list that get_tsk_by_type() returns. I also tried panda but i couldn't get that to output the way i want it to, either. How do i prettify it to the point that it's easily readable? Without brackets etc.? I later want to display each individual task separately, preferably in divs.

I recommend using a dict cursor so you can access the result elements by name.
You can do that like this (from: https://stackoverflow.com/a/3300514/42346):
def dict_factory(cursor, row):
d = {}
for idx, col in enumerate(cursor.description):
d[col[0]] = row[idx]
return d
conn.row_factory = dict_factory
Then you will get this kind of result:
result = c.fetchall()
result
# [{'type':'homework','description':'math',
# 'amount':1,'id':'df19c0b1-a2128-431274-2e32-3a2f901b1b26'}]
Then in your template you can do something like:
{% for homework in curr_homework %}
<div>
<h6>{{ homework['type'] }}</h6>
<div>{{ homework['description'] }}</div>
</div>
{% if not loop.last %}
<hr>
{% endif %}
{% endfor %}
You may benefit from a little bit of re-organization of the database-specific code.
You can do this:
from flask import g
def dict_factory(cursor, row):
d = {}
for idx, col in enumerate(cursor.description):
d[col[0]] = row[idx]
return d
def get_db():
if 'db' not in g:
conn = sqlite3.connect('tasks.db', check_same_thread=False)
conn.row_factory = dict_factory
g.db = conn.cursor()
return g.db
And then in your view do this:
db = get_db()
db.execute('your query here')

Calling function from request function in views Django

I have simple html template with link, that starts script, in views that retrieves data to the page,in views file I have two functions: render function def output(request):(it retrieves data to the page) and another function def summoner(): that makes postgres quires in cycle and appends results to the list. Separately each of them work fine, but I have to call second function from render function and retrieve the data to the page, but now when I do that all I am getting is empty list.
enter image description here
template:
<html>
<head>
<title>
Python script
</title>
</head>
<body>
Execute Script <hr>
{% if data %}
{{ data }}
{% endif %}
</body>
</html>
views:
from django.shortcuts import render
import pandas as pd
import psycopg2
import os, glob
conn = psycopg2.connect(host='127.0.0.1', database='db',
user='user', password='pass')
cur = conn.cursor()
def insert_data_as_is(file_name):
cur.execute('truncate table test_inv.start_tbl')
with open(file_name, 'r') as file:
cur.execute("insert into test_inv.start_tbl values {0}".format(file.read()))
conn.commit()
def insert_data_to_be(file_name):
cur.execute('truncate table test_inv.res_calc_ratios_t')
with open(file_name, 'r') as file:
cur.execute('insert into test_inv.res_calc_ratios_t (test_no, test_name, hcode_id, '
'hcode_name, hcode_'
'unit_name,'
' org_id, dor_kod, duch_id, nod_id, date_type_id, metric_type_id, cargo_type_id, val_type_id,'
' unit_id, dt, value, ss, dir_id, kato_id, vids_id) values {0}'.format(file.read()))
conn.commit()
path_start = 'files/csv_s/as_is/'
start_list = []
path_finish = 'files/csv_s/to_be'
finish_list = []
for infile in glob.glob(os.path.join(path_start, '*.*')):
start_list.append(infile)
for infile in glob.glob(os.path.join(path_finish, '*.*')):
finish_list.append(infile)
def summoner():
fun_sql_set = []
fun_query = """select * from test_inv.test_ratios('1','15')"""
for i in range(len(finish_list)):
insert_data_as_is(start_list[i])
insert_data_to_be(finish_list[i])
results = pd.read_sql_query(fun_query, conn)
fun_sql_set.append(results)
return fun_sql_set
def index(request):
return render(request,'calculus/index.html')
def output(request):
data = summoner()
print(data)
return render(request,'calculus/index.html',{'data':data})

The answer is creating absolute file path:
cur_dir = os.path.dirname(os.path.abspath(__file__))
path_start = '{}/static/csv_s/as_is/'.format(cur_dir)
path_finish = '{}/static/csv_s/to_be/'.format(cur_dir)

Display more posts on same page with Flask?

I am screwing around with an image uploader build with python and flask. I have it up and running on my VPS here
http://107.170.119.38/
The application displays the first 25 images just fine, but anything after the 25th image does not get displayed anywhere. I am trying to add a button that would allow me to display the next 25 images from. Here is a sample of my code
#app.route('/', methods=['GET', 'POST'])
def get_more_pics():
g.db = connect_db()
cur = g.db.execute('select filename, label from pics order by id desc limit 25')
more_pics = [dict("filename": row[0], "label": row[1]) for row in cur.fetchall()]
g.db.close()
return render_template('upload.html', more_pics=more_pics)
def upload_pic():
if request.method == 'POST':
file = request.files['file']
label = request.form['label']
try:
extension = file.filename.rsplit('.', 1)[1].lower()
except IndexError, e:
abort(404)
if file and check_extension(extension):
# Salt and hash the file contents
filename = md5(file.read() + str(round(time.time() * 1000))).hexdigest() + '.' + extension
file.seek(0) # Move cursor back to beginning so we can write to disk
file.save(os.path.join(app.config['UPLOAD_DIR'], filename))
add_pic(filename, label)
gen_thumbnail(filename)
return redirect(url_for('show_pic', filename=filename))
else:
# Bad file extension
abort(404)
else:
return render_template('upload.html', pics=get_last_pics())
#app.route('/show')
def show_pic():
filename = request.args.get('filename', '')
t = (filename,)
cur = g.db.execute('select label from pics where filename=?', t)
label = cur.fetchone()[0]
return render_template('upload.html', filename=filename, label=label)
# Return a list of the last 25 uploaded images
def get_last_pics():
try:
cur = g.db.execute('select filename, label from pics order by id desc limit 25')
filenames = []
for row in cur.fetchall():
filenames.append({"filename": row[0], "label": row[1] or ''})
return filenames
except:
return []
and in the jinja2 template file:
<input type="button" value="Get More Pics" onclick="{{ more_pics }}">
<ul>
{% for pic in more_pics %}
<li class="thumb">
<img class="thumb" src="{{ pic_path('thumb2_'+pic.filename) }}"></li>
{% endfor %}
</ul>
all in all I can't figure out how to get the get_more_pics() function to display the next 25 images. Any help would be appreciated

You need to add an offset parameter to your SQL:
select filename, label from pics order by id desc limit 25 offset 0
You'll have to pass it into your SQL, as it will be 0 for the first page, then 25, then 50, etc. Probably better to break that 25 out into a parameter rather than hard-coding it, and calculate offset as the product of page and limit.
Also, bear in mind that LIMIT and OFFSET don't necessarily work with all databases, although they do work with SQLite and MySQL, the syntax can vary. E.g. for Oracle: http://www.oracle-base.com/articles/12c/row-limiting-clause-for-top-n-queries-12cr1.php.
An alternative is to use a slightly higher-level approach, such as an ORM.

How to use nested Python dictionaries and D3.js?

I have a Python/Flask app that gathers data from third-party APIs and stores them in a JSON-like structure (nested Python dictionaries called 'results').
I'm sending this to my template using:
def format_results():
item_data = {'name':name, 'age':age, 'address':address}
results = {'title':item_title, 'item_data':item_data}
return jsonify(results)
#app.route('/')
def display_results():
data = format_results()
return render_template('index.html', data = data)
I would like to use d3.js in my template to plot the results on a graph.
What's the recommended way for doing so? (disclaimer: this is my first time using D3.js)

Figured out a way to make this work!
def format_results():
item_data = {'name':name, 'age':age, 'address':address}
results = {'title':item_title, 'item_data':item_data}
return results
#app.route('/')
def display_results():
data = format_results()
return render_template('index.html', data = data)
Removed 'jsonify' and disabled escaping for {{data}}
de = {% autoescape false %} {{data}} {% endautoescape %}