I have a database I created with a migration. One of my tables looks like this
def create_customer_table(self):
with self.schema.create("customer") as table:
table.char("name",120).unique()
table.integer("transmitting_hours").default(24) #how many hours after transmission vehicle is considered transmitting
table.boolean("is_tpms").default(False)
table.boolean("is_dor").default(False)
table.boolean("is_otr").default(False)
table.boolean("is_track_and_trace").default(False)
table.char("contact_person",25)
table.char("created_by",25)
table.enum("temperature_unit",TEMP_UNITS)
table.enum("pressure_unit",PRESSURE_UNITS)
table.enum("distance_unit",DISTANCE_UNITS)
table.char("time_zone",25)
table.char("language",2)
table.timestamps()
I have a very simplistic ORM model on top
class Customer(Model):
__table__ = "customer"
__timestamps__ = False
__primary_key__ = "name"
__fillable__ = ['*']
I then try to do a basic insert with the following code
def add_sample_customer():
sample_customer = {}
sample_customer["name"] = "customer_2"
sample_customer["contact_person"] = "Abradolf"
sample_customer["created_by"] = "Frodo"
sample_customer["time_zone"] = "GMT-5"
sample_customer["language"] = "EN"
sample_customer["temperature_unit"] = "FAHRENHEIT"
sample_customer["pressure_unit"] = "PSI"
sample_customer["distance_unit"] = "MI"
customer_model = Customer.create(_attributes = sample_customer)
The exception I get from this code looks like
orator.exceptions.query.QueryException: syntax error at or near ")"
LINE 1: INSERT INTO "customer" () VALUES () RETURNING "name"
(SQL: INSERT INTO "customer" () VALUES () RETURNING "name" ([]))
it looks like orator just isn't filling in the cols and vals here. I have also tried it with a few different syntactic ways of dropping the dict in there, using **sample_customer and also just putting the dict in directly and none of them work, all with the same exception. I started debugging by printing stuff out of the orator libraries but haven't gotten anywhere yet.
my inserts work if I do the model attribute assignment individually and use the model.save() method like this
def add_sample_customer():
sample_customer = {}
sample_customer["name"] = "customer_2"
sample_customer["contact_person"] = "Abradolf"
sample_customer["created_by"] = "Frodo"
sample_customer["time_zone"] = "GMT-5"
sample_customer["language"] = "EN"
sample_customer["temperature_unit"] = "FAHRENHEIT"
sample_customer["pressure_unit"] = "PSI"
sample_customer["distance_unit"] = "MI"
customer_model = Customer()
for k,v in sample_customer.items():
setattr(customer_model,k,v)
customer_model.save()
Does anyone understand why the model.create() syntax fails?
I would think the answer would be:
Simply passing the dictionary instead of using keyword notation with attributes:
Customer.create(sample_customer)
or
Customer.create(attribute=value,attribute2=value2,..etc)
Which are the valid notations
I have a SOAP request that takes below XML body
<x:Body>
<ser:CreateExportJobRequest>
<ser:ExportJobTypeName>Products</ser:ExportJobTypeName>
<ser:ExportColumns>
<ser:ExportColumn>Id</ser:ExportColumn>
<ser:ExportColumn>itemName</ser:ExportColumn>
</ser:ExportColumns>
<ser:ExportFilters>
<ser:ExportFilter id="updatedSince">
<ser:Text>2.0</ser:Text>
</ser:ExportFilter>
</ser:ExportFilters>
<ser:Frequency>ONETIME</ser:Frequency>
</ser:CreateExportJobRequest>
</x:Body>
I can make a successful request using Boomerang.
Now I actually want to use it in my python code. So I tried,
inputElement = client.factory.create('CreateExportJobRequest')
inputElement.ExportJobTypeName = "Products"
inputElement.ExportColumns.ExportColumn = ["Id", "itemName"]
inputElement.Frequency = 'ONETIME'
if updatedSince:
inputElement.ExportFilters.ExportFilter = ['updatedSince']
t = client.service.CreateExportJob(inputElement.ExportJobTypeName, inputElement.ExportColumns, inputElement.ExportFilters, None, None, inputElement.Frequency)
I get an error,
'list' object has no attribute 'id'
Because a somewhat wrong XML request gets created
<ns1:ExportFilters>
<ns1:ExportFilter>updatedSince</ns1:ExportFilter>
</ns1:ExportFilters>
So I tried few other things for ExportFilter like
inputElement.ExportFilters.ExportFilter = [{'id': 'updatedSince', 'text': updatedSince}]
and
inputElement.ExportFilters.ExportFilter = [('updatedSince', updatedSince)]
and
inputElement.ExportFilters.ExportFilter = [{'updatedSince': updatedSince}]
# says, Type not found: 'updatedSince'
and
inputElement.ExportFilters.ExportFilter = [
{'key': 'updatedSince', 'value': {'key': 'eq', 'value': updatedSince}}
]
# says, Type not found: 'value'
but nothing is working.
Before setting ExportFilter, it's value is in the form of
ExportFilters: (ExportFilters){
ExportFilter[] = <empty>
}
Please help.
After debugging and going through some suds code, I have found the fix.
The complete code snippet of the fix:
inputElement = client.factory.create('CreateExportJobRequest')
inputElement.ExportJobTypeName = "Products"
inputElement.ExportColumns.ExportColumn = ["Id", "itemName"]
inputElement.Frequency = 'ONETIME'
if updatedSince:
efilter = client.factory.create("ExportFilter")
efilter._id = 'updatedSince'
efilter.Text = updatedSince
inputElement.ExportFilters.ExportFilter.append(efilter)
t = client.service.CreateExportJob(inputElement.ExportJobTypeName, inputElement.ExportColumns, inputElement.ExportFilters, None, None, inputElement.Frequency)
Debugging:
Because suds was raising TypeNotFound exception, I looked for all the places that raise TypeNotFound inside suds. I put debug points in my PyCharm.
I found that the start method from Typed class inside suds/mx/literal.py was raising the error I was getting.
def start(self, content):
#
# Start marshalling the 'content' by ensuring that both the
# 'content' _and_ the resolver are primed with the XSD type
# information. The 'content' value is both translated and
# sorted based on the XSD type. Only values that are objects
# have their attributes sorted.
#
log.debug('starting content:\n%s', content)
if content.type is None:
name = content.tag
if name.startswith('_'):
name = '#'+name[1:]
content.type = self.resolver.find(name, content.value)
if content.type is None:
raise TypeNotFound(content.tag)
else:
known = None
if isinstance(content.value, Object):
known = self.resolver.known(content.value)
if known is None:
log.debug('object has no type information', content.value)
known = content.type
frame = Frame(content.type, resolved=known)
self.resolver.push(frame)
frame = self.resolver.top()
content.real = frame.resolved
content.ancestry = frame.ancestry
self.translate(content)
self.sort(content)
if self.skip(content):
log.debug('skipping (optional) content:\n%s', content)
self.resolver.pop()
return False
else:
return True
So from this logic, I came to the fix.
But, It would be really great if somebody suggests a standard procedure for this.
Currently working on a project with TurboGears2 and ToscaWidgets2. I have a form setup with a few static fields, name, date, and point of contact information. Inside this form I have added a sub form where the user can dynamically add numerous entries in a GrowingGridLayout. The form, its layout, and submitting information is all well and good but I'm having a hard time figuring out how to capture the information from the GrowingGridLayout once it's passed on for saving. Guess the main points are, how do I know how many entries were included in the form?
Included the code for the form:
class OnrampForm(twf.Form):
title = "Onramp Submission Form"
class child(twd.CustomisedTableForm):
onramp_name = twf.TextField(validator=twc.Required)
class Destinations (twd.GrowingGridLayout):
environment = twf.SingleSelectField(label='Environment', validator=twc.Validator(required=True), options=[<OPTIONS>])
location = twf.SingleSelectField(validator=twc.Required, label='Location', options=[<OPTIONS>])
jms_type = twf.SingleSelectField(label='JMS Type', validator=twc.Validator(required=True), options=[<OPTIONS>])
subscription_type = twf.SingleSelectField(label='Subscription Type', validator=twc.Validator(required=True), options=[<OPTIONS>])
onramp_status = twf.SingleSelectField(prompt_text='Status', options=['Initial Release', 'Update'], validator=twc.Required)
current_date = datetime.date.today()
need_by_date = twd.CalendarDatePicker(validators=[twc.Required, twc.DateTimeValidator])
need_by_date.default = current_date + datetime.timedelta(days=30)
organization = twf.TextField(validator=twc.Required)
poc_name = twf.TextField(validator=twc.Required)
poc_email = twf.EmailField(validator=twc.EmailValidator)
poc_phone = twf.TextField(validator=twc.Required)
poc_address = twf.TextField()
poc_city = twf.TextField()
poc_state = twf.TextField()
onramp_form = twf.FileField()
submit = twf.SubmitButton(value="Submit")
action = "/print_args"
submit = ""
If you controller #validates against the form you should get the data into the Destination parameter which should be a list of dictionaries.
Also I just noticed you have two nested forms, that's something that might confuse TW2 pretty much. What you wanted to do is probably have OnrampForm inherit CustomisedForm and then have child inherit TableLayout. See http://turbogears.readthedocs.org/en/latest/cookbook/TwForms.html#displaying-forms
PS: note that need_by_date.default = current_date + datetime.timedelta(days=30) will always return 30 days from when the server started as you are actually storing a current_date = datetime.date.today() class variable that gets computed when the module is imported and no more.
You should use default = Deferred(lambda: datetime.date.today() + datetime.timedelta(days=30)) to achieve that
One feature I have been struggling to implement in flask-admin is when the user edits a form, to constrain the value of Field 2 once Field 1 has been set.
Let me give a simplified example in words (the actual use case is more convoluted). Then I will show a full gist that implements that example, minus the "constrain" feature.
Let's say we have a database that tracks some software "recipes" to output reports in various formats. The recipe table of our sample database has two recipes: "Serious Report", "ASCII Art".
To implement each recipe, we choose one among several methods. The method table of our database has two methods: "tabulate_results", "pretty_print".
Each method has parameters. The methodarg table has two parameter names for "tabulate_results" ("rows", "display_total") and two parameters for "pretty_print" ("embellishment_character", "lines_to_jump").
Now for each of the recipes ("Serious Report", "ASCII Art") we need to provide the value of the arguments of their respective methods ("tabulate_results", "pretty_print").
For each record, the recipearg table lets us select a recipe (that's Field 1, for instance "Serious Report") and an argument name (that's Field 2). The problem is that all possible argument names are shown, whereas they need to be constrained based on the value of Field 1.
What filtering / constraining mechanism can we implement such that once we select "Serious Report", we know we will be using the "tabulate_results" method, so that only the "rows" and "display_total" arguments are available?
I'm thinking some AJAX wizardry that checks Field 1 and sets a query for Field 2 values, but have no idea how to proceed.
You can see this by playing with the gist: click on the Recipe Arg tab. In the first row ("Serious Report"), if you try to edit the "Methodarg" value by clicking on it, all four argument names are available, instead of just two.
# full gist: please run this
from flask import Flask
from flask_admin import Admin
from flask_admin.contrib import sqla
from flask_sqlalchemy import SQLAlchemy
from sqlalchemy import Column, ForeignKey, Integer, String
from sqlalchemy.orm import relationship
# Create application
app = Flask(__name__)
# Create dummy secrey key so we can use sessions
app.config['SECRET_KEY'] = '123456790'
app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:///a_sample_database.sqlite'
app.config['SQLALCHEMY_ECHO'] = True
db = SQLAlchemy(app)
# Create admin app
admin = Admin(app, name="Constrain Values", template_mode='bootstrap3')
# Flask views
#app.route('/')
def index():
return 'Click me to get to Admin!'
class Method(db.Model):
__tablename__ = 'method'
mid = Column(Integer, primary_key=True)
method = Column(String(20), nullable=False, unique=True)
methodarg = relationship('MethodArg', backref='method')
recipe = relationship('Recipe', backref='method')
def __str__(self):
return self.method
class MethodArg(db.Model):
__tablename__ = 'methodarg'
maid = Column(Integer, primary_key=True)
mid = Column(ForeignKey('method.mid', ondelete='CASCADE', onupdate='CASCADE'), nullable=False)
methodarg = Column(String(20), nullable=False, unique=True)
recipearg = relationship('RecipeArg', backref='methodarg')
inline_models = (Method,)
def __str__(self):
return self.methodarg
class Recipe(db.Model):
__tablename__ = 'recipe'
rid = Column(Integer, primary_key=True)
mid = Column(ForeignKey('method.mid', ondelete='CASCADE', onupdate='CASCADE'), nullable=False)
recipe = Column(String(20), nullable=False, index=True)
recipearg = relationship('RecipeArg', backref='recipe')
inline_models = (Method,)
def __str__(self):
return self.recipe
class RecipeArg(db.Model):
__tablename__ = 'recipearg'
raid = Column(Integer, primary_key=True)
rid = Column(ForeignKey('recipe.rid', ondelete='CASCADE', onupdate='CASCADE'), nullable=False)
maid = Column(ForeignKey('methodarg.maid', ondelete='CASCADE', onupdate='CASCADE'), nullable=False)
strvalue = Column(String(80), nullable=False)
inline_models = (Recipe, MethodArg)
def __str__(self):
return self.strvalue
class MethodArgAdmin(sqla.ModelView):
column_list = ('method', 'methodarg')
column_editable_list = column_list
class RecipeAdmin(sqla.ModelView):
column_list = ('recipe', 'method')
column_editable_list = column_list
class RecipeArgAdmin(sqla.ModelView):
column_list = ('recipe', 'methodarg', 'strvalue')
column_editable_list = column_list
admin.add_view(RecipeArgAdmin(RecipeArg, db.session))
# More submenu
admin.add_view(sqla.ModelView(Method, db.session, category='See Other Tables'))
admin.add_view(MethodArgAdmin(MethodArg, db.session, category='See Other Tables'))
admin.add_view(RecipeAdmin(Recipe, db.session, category='See Other Tables'))
if __name__ == '__main__':
db.drop_all()
db.create_all()
db.session.add(Method(mid=1, method='tabulate_results'))
db.session.add(Method(mid=2, method='pretty_print'))
db.session.commit()
db.session.add(MethodArg(maid=1, mid=1, methodarg='rows'))
db.session.add(MethodArg(maid=2, mid=1, methodarg='display_total'))
db.session.add(MethodArg(maid=3, mid=2, methodarg='embellishment_character'))
db.session.add(MethodArg(maid=4, mid=2, methodarg='lines_to_jump'))
db.session.add(Recipe(rid=1, mid=1, recipe='Serious Report'))
db.session.add(Recipe(rid=2, mid=2, recipe='ASCII Art'))
db.session.commit()
db.session.add(RecipeArg(raid=1, rid=1, maid=2, strvalue='true' ))
db.session.add(RecipeArg(raid=2, rid=1, maid=1, strvalue='12' ))
db.session.add(RecipeArg(raid=3, rid=2, maid=4, strvalue='3' ))
db.session.commit()
# Start app
app.run(debug=True)
I see two ways of tacking this problem:
1- When Flask-Admin generate the form, add data attributes with the mid of each methodArg on each option tag in the methodArg select. Then have some JS code filter the option tags based on the recipe selected.
EDIT
Here is a tentative try at putting a data-mid attribute on each option:
def monkeypatched_call(self, field, **kwargs):
kwargs.setdefault('id', field.id)
if self.multiple:
kwargs['multiple'] = True
html = ['<select %s>' % html_params(name=field.name, **kwargs)]
for (val, label, selected), (_, methodarg) in zip(field.iter_choices(), field._get_object_list()):
html.append(self.render_option(val, label, selected, **{'data-mid': methodarg.mid}))
html.append('</select>')
return HTMLString(''.join(html))
Select.__call__ = monkeypatched_call
The blocker is in the fact that those render calls are triggered from the jinja templates, so you are pretty much stuck updating a widget (Select being the most low-level one in WTForms, and is used as a base for Flask-Admin's Select2Field).
After getting those data-mid on each of your options, you can proceed with just binding an change on your recipe's select and display the methodarg's option that have a matching data-mid. Considering Flask-Admin uses select2, you might have to do some JS tweaking (easiest ugly solution would be to clean up the widget and re-create it for each change event triggered)
Overall, I find this one less robust than the second solution. I kept the monkeypatch to make it clear this should not be used in production imho. (the second solution is slightly less intrusive)
2- Use the supported ajax-completion in Flask-Admin to hack your way into getting the options that you want based on the selected recipe:
First, create a custom AjaxModelLoader that will be responsible for executing the right selection query to the DB:
class MethodArgAjaxModelLoader(sqla.ajax.QueryAjaxModelLoader):
def get_list(self, term, offset=0, limit=10):
query = self.session.query(self.model).filter_by(mid=term)
return query.offset(offset).limit(limit).all()
class RecipeArgAdmin(sqla.ModelView):
column_list = ('recipe', 'methodarg', 'strvalue')
form_ajax_refs = {
'methodarg': MethodArgAjaxModelLoader('methodarg', db.session, MethodArg, fields=['methodarg'])
}
column_editable_list = column_list
Then, update Flask-Admin's form.js to get the browser to send you the recipe information instead of the methodArg name that needs to be autocompleted. (or you could send both in query and do some arg parsing in your AjaxLoader since Flask-Admin does no parsing whatsoever on query, expecting it to be a string I suppose [0]. That way, you would keep the auto-completion)
data: function(term, page) {
return {
query: $('#recipe').val(),
offset: (page - 1) * 10,
limit: 10
};
},
This snippet is taken from Flask-Admin's form.js [1]
Obviously, this needs some tweaking and parametrising (because doing such a hacky solution would block you from using other ajax-populated select in the rest of your app admin + the update on form.js directly like that would make upgrading Flask-Admin extremely cumbersome)
Overall, I am unsatisfied with both solutions and this showcase that whenever you want to go out of the tracks of a framework / tool, you can end up in complex dead ends. This might be an interesting feature request / project for someone willing to contribute a real solution upstream to Flask-Admin though.
There is another easy solution that I made and it works
1- Create your first select option normally with data loaded on it and add a hook to it which will add js event listener when it selects change like this.
from wtforms import SelectField
form_extra_fields = {
'streetname': SelectField(
'streetname',
coerce=str,
choices=([street.streetname for street in StreetsMetadata.query.all()]),
render_kw={'onchange': "myFunction()"}
)
}
**2- Add a JavaScript URL file to the view you want to use this function in, for example.
def render(self, template, **kwargs):
#using extra js in render method allow use url_for that itself requires an app context
self.extra_js = [url_for("static", filename="admin/js/users.js")]
response = render_miror(self, template,**kwargs)
return response
3- Create a role-protected endpoint that you used for this view that will accept a GET request from JS based on the first value specified for the entry, for example this route returns a list of house numbers by querying the street name that came from the first entry
#super_admin_permission.require(http_exception=403)
#adminapp.route('/get_houses_numbers')
def gethouses():
request_data = request.args
if request_data and 'street' in request_data:
street = StreetsMetadata.query.filter(StreetsMetadata.streetname == request_data['street']).one_or_none()
street_houses = lambda:giveMeAllHousesList(street.excluded, street.min, street.max)
if street_houses:
return jsonify({'code': 200, 'houses': street_houses()})
else:
return jsonify({'code': 404, 'houses': []})
else:
return jsonify({'code': 400, 'street': []})
now python part completed time for JavaScript
4- We have to define three functions, the first of which will be called when the form build page is loaded and which do two things first,
A dummy select entry will be created using JS and append that entry to the same string input container
Make string entry read-only to improve user experience
Second, it will send a GET request to the specified route to get a list of house numbers using the specified street input value
Then get the result and create the option elements and append these options to the dummy selection, you can also select the first option while appending the options.
5- The second function "myFunction" is the hook defined in Python in this part
render_kw={'onchange': "myFunction()"}
This function will do nothing new, it will only send a GET request when the first specified input value is changed, send a GET request to get a list of new house numbers based on the given street name input value by doing a query on the database, then dump the inner HTML of the dummy selection entry , then create and append new options to it.
6- The last function is the callback function which listens for the change on the dummy select entry created with JS when the user chooses the house number which will be reflected in the main string entry, finally you can click save and you will see it working
Note that this whole idea I created is not as good as the built in flask admin, but if you are looking for the end goal and without any problems you can use it
My JS code
/*
This Function when run when a form included it will create JS select input with the
default loaded streetname and add house number on that select this select will used
to guide creator of the house number or to select the house number
*/
async function onFlaskFormLoad(){
const streetSelect = document.querySelector("#streetname");
const checkIfForm = document.querySelector("form.admin-form");
if (checkIfForm){
let checkSelect = document.querySelector("#realSelect");
if (!checkSelect){
const mySelectBox = document.createElement("select");
const houseString = document.querySelector("#housenumber");
const houseStringCont = houseString.parentElement;
mySelectBox.classList.add("form-control")
mySelectBox.id = "realSelect";
houseStringCont.appendChild(mySelectBox);
mySelectBox.addEventListener("change", customFlaskAdminUnpredefinedSelect);
houseString.setAttribute("readonly", "readonly");
const res = await fetch(`/get_houses_numbers?street=${streetSelect.value}`);
const data = await res.json();
console.log(data);
if (data.code == 200 && mySelectBox){
data.houses.forEach( (houseOption, index)=>{
if (index == 0){
houseString.value = houseOption;
}
let newHouse = document.createElement("option");
newHouse.setAttribute("value", houseOption);
newHouse.innerText = houseOption;
mySelectBox.appendChild(newHouse);
});
}
}
}
}
onFlaskFormLoad();
/*
this function will called to change the string input value to my custom js select
value and then use that string to house number which required by flask-admin
*/
function customFlaskAdminUnpredefinedSelect(){
const theSelect = document.querySelector("#realSelect");
const houseString = document.querySelector("#housenumber");
houseString.value = theSelect.value;
return true;
}
/*
flask admin hook that will listen on street input change and then it will send
get request to secured endpoint with role superadmin required and get the housenumbers
using the streetname selected and then create options and add to my select input
*/
async function myFunction(){
const streetSelect = document.querySelector("#streetname");
const houseString = document.querySelector("#housenumber");
const houseStringCont = houseString.parentElement;
const theSelect = document.querySelector("#realSelect");
const res = await fetch(`/get_houses_numbers?street=${streetSelect.value}`);
const data = await res.json();
console.log(data);
if (data.code == 200 && theSelect){
theSelect.innerHTML = "";
data.houses.forEach( (houseOption, index)=>{
if (index == 0){
houseString.value = houseOption;
}
let newHouse = document.createElement("option");
newHouse.setAttribute("value", houseOption);
newHouse.innerText = houseOption;
theSelect.appendChild(newHouse);
});
}
}
Now if I change the street name of the first specified input I will get a new list containing the numbers based on the first input value, note if you have a way to create a python field that accepts the non-predefined options then there is no need to create dummy input you can create and append the new options Directly to second select input
final result
I have a very simple "guestbook" script on GAE/Python. It often happens however, that entries which I put() into the datastore are not showing right away - I almost always need to refresh.
def post(self):
t = NewsBase(
date = datetime.now(),
text = self.request.get('text'),
title = self.request.get('title'),
link = self.request.get('link'),
upvotes = [],
downvotes = [],
)
t.put()
q = db.GqlQuery('SELECT * FROM NewsBase ORDER BY date DESC')
template_values = {
'q' : q,
'user' : user,
'search' : search
}
template = jinja_environment.get_template('finaggnews.html')
self.response.out.write(template.render(template_values))
I'm sure there is a solution to this?
Best,
Oliver
This is due to the eventual consistency model of HRD.
You should really read some of the intro docs, Structuring Data for Strong Consistency - https://developers.google.com/appengine/docs/python/datastore/structuring_for_strong_consistency and do some searching of SO. This question has been asked many times before.