I am passing a csv file to Mako via csv.DictReader, and the dictionary uses row headers for keys. Some of these have names like 'Entry Id'. Mako throws out errors when I try to reference these multi-word keys in a template. Specifically, the error message is mako.exceptions.SyntaxException: (SyntaxError) invalid syntax (<unknown>, line 1).
The following code demonstrates the issue I am experiencing:
from mako.template import Template
mydata = {'foo': 'bar', 'better foo': 'beach bar'}
working_template = Template("Let's go to the ${foo}")
fail_template = Template("Let's go to the ${better foo}")
# this works
print working_template.render(**mydata)
# this generates an Exception
print fail_template.render(**mydata)
Is there a way to escape spaces for multi-word keys?
I have found one way that works, but I'd rather send the dictionary as **kwargs if possible. Here is the working solution:
from mako.template import Template
mydata = {'foo': 'bar', 'better foo': 'beach bar'}
working_template = Template("Let's go to the ${foo}")
fail_template = Template("Let's go to the ${mydata['better foo']}")
print working_template.render(**mydata)
print fail_template.render(mydata=mydata)
Related
I tried to use python+jinja2 to deal with file, but jinja2 does not seem to be friendly to dict and list support. When using list and dict, an exception will be thrown:'xxx is undefined'. I tried to use the DebugUndefined parameter in jinja2, but it didn't work.
Example code:
from jinja2 import Template,DebugUndefined
template = Template("{{var1}}-{{ var2[0] }}",undefined = DebugUndefined)
rendered1 = template.render(var1=5)
print(rendered1)
# expect 5-{{ var2[0]}}
rendered2 = Template(rendered1).render(var2=['a', 'b', 'c', 'd'])
print(rendered2)
# expect 5-a
the code can't run and thrown exceptions "jinja2.exceptions.UndefinedError: 'var2' is undefined".The code about dict is the same.
from jinja2 import Template,DebugUndefined
template = Template("{{var1}}-{{ var2[Alice] }}",undefined = DebugUndefined)
rendered1 = template.render(var1=5)
print(rendered1)
# expect 5-{{ var2[Alice]}}
rendered2 = Template(rendered1).render(var2 = {'Alice': '2341', 'Beth': '9102', 'Cecil': '3258'})
print(rendered2)
# expect 5-2341
I noticed that flask+jinja2 can deal with this problem. Does flask changed some jinja2's code?
Ideas please? Newbie - be gentle. Thanks.
Here is the code snippet I have initialized jinja2 template
from jinja2 import Template
templ = Template("{{foo}} to {{bar}}")
And I am willing to extract the template string variable keys from the template obj as below.
templ.keys() == ["foo", "bar"]
Is there any API to make it work? I have searched for a long while but got nothing work.
using jinja2.meta you could:
from jinja2 import Environment, meta
templ_str = "{{foo}} to {{bar}}"
env = Environment()
ast = env.parse(templ_str)
print(meta.find_undeclared_variables(ast)) # {'bar', 'foo'}
which returns the set of the undeclared variables.
you could also use the regex module to find the variable names in your template string:
from jinja2 import Template
import re
rgx = re.compile('{{(?P<name>[^{}]+)}}')
templ_str = "{{foo}} to {{bar}}"
templ = Template(templ_str)
variable_names = {match.group('name') for match in rgx.finditer(templ_str)}
print(variable_names) # {'foo', 'bar'}
the regex (could probably be better...) matches {{ followed by anything that is not a curly bracket until }} is encountered.
I'm using Python 2.7.11 on windows to get JSON data from API (data on trees in Warsaw, Poland, but nevermind that). I want to generate output csv file with all the data provided by the api, for further analysis. I started with a script I used for another project (also discussed here on Stackoverflow and corrected for me by #Martin Taylor).That script didn't work so I tried to modify it using my very basic understanding, googling around and applying pdb debugger. At the moment, the result looks like this:
import pdb
import json
import urllib2
import csv
pdb.set_trace()
url = "https://api.um.warszawa.pl/api/action/datastore_search/?resource_id=ed6217dd-c8d0-4f7b-8bed-3b7eb81a95ba"
myfile = 'C:/dane/drzewa.csv'
csv_myfile = csv.writer(open(myfile, 'wb'))
cols = ['numer_adres', 'stan_zdrowia', 'y_wgs84', 'dzielnica', 'adres', 'lokalizacja', 'wiek_w_dni', 'srednica_k', 'pnie_obwod', 'miasto', 'jednostka', 'x_pl2000', 'wysokosc', 'y_pl2000', 'numer_inw', 'x_wgs84', '_id', 'gatunek_1', 'gatunek', 'data_wyk_pom']
csv_myfile.writerow(cols)
def api_iterate(myfile):
while True:
global url
print url
json_page = urllib2.urlopen(url)
data = json.load(json_page)
json_page.close()
for data_object in data ['result']['records']:
csv_myfile.writerow([data_object[col] for col in cols])
try:
url = data['_links']['next']
except KeyError as e:
break
with open(myfile, 'wb'):
api_iterate(myfile)
I'm a very fresh Python user so I get confused all the time. Now I got to the point when, while reading the objects in json dictionary, I get a Keyerror message associated with the 'x_wgs84' element. I suppose it has something to do with the fact that in the source url this element is preceded by a U+FEFF unicode character. I tried to get around this but I got stuck and would appreciate assistance.
I suspect the code may be corrupt in several other ways - as I mentioned, I'm a very unskilled programmer (yet).
You need to put the key with the unicode character:
To know how to do it, one easy way is to print the keys:
>>> import requests
>>> res = requests.get('https://api.um.warszawa.pl/api/action/datastore_search/?resource_id=ed6217dd-c8d0-4f7b-8bed-3b7eb81a95ba')
>>> data = res.json()
>>> records = data['result']['records']
>>> records[0]
{u'numer_adres': u'', u'stan_zdrowia': u'dobry', u'y_wgs84': u'52.21865', u'y_pl2000': u'5787241.04475524', u'adres': u'ul. ALPEJSKA', u'x_pl2000': u'7511793.96937063', u'lokalizacja': u'Ulica ALPEJSKA', u'wiek_w_dni': u'60', u'miasto': u'Warszawa', u'jednostka': u'Dzielnica Wawer', u'pnie_obwod': u'73', u'wysokosc': u'14', u'data_wyk_pom': u'20130709', u'dzielnica': u'Wawer', u'\ufeffx_wgs84': u'21.172584', u'numer_inw': u'D386200', u'_id': 125435, u'gatunek_1': u'Quercus robur', u'gatunek': u'd\u0105b szypu\u0142kowy', u'srednica_k': u'7'}
>>> records[0].keys()
[u'numer_adres', u'stan_zdrowia', u'y_wgs84', u'y_pl2000', u'adres', u'x_pl2000', u'lokalizacja', u'wiek_w_dni', u'miasto', u'jednostka', u'pnie_obwod', u'wysokosc', u'data_wyk_pom', u'dzielnica', u'\ufeffx_wgs84', u'numer_inw', u'_id', u'gatunek_1', u'gatunek', u'srednica_k']
>>> records[0][u'\ufeffx_wgs84']
u'21.172584'
As you can see, to get your key, you need to write it as '\ufeffx_wgs84' with the unicode character that is causing trouble.
Note: I don't know if you are using python2 or 3, but you might need to put a u before your string declaration in python2 to declare it as unicode string.
Why am I getting this error when trying to print keys from dictionary?
import urllib
import urllib2
import json
ret = urllib2.urlopen(urllib2.Request('https://poloniex.com/public?command=returnTicker'))
a = json.loads(ret.read())
print a.keys()
ret = urllib2.urlopen(urllib2.Request('https://poloniex.com/public?command=return24Volume'))
b = json.loads(ret.read())
print b.keys()
The error is produced by the website - your code as such is ok, it produces a json-object which has apparently the structure '{ "error" : ""}'. Try printing it out & figure out what's wrong, you probably need some authentication tokens or similar stuff.
There seems to be a API wrapper available, you should consider using or at least understanding it: http://pastebin.com/8fBVpjaj
It's directly featured on polniex website, and it clearly shows the need of API secret and key.
The error is originating from the site. The dict is loaded, and expresses the error as it's only key.
Try this:
ret = urllib2.urlopen(urllib2.Request('https://poloniex.com/public?command=return24hVolume'))
b = json.loads(ret.read())
print b.keys()
Notice the 'h' in return24hVolume.
The second URL returns:
{"error":"Invalid command."}
So as Reut Sharabani points out, you have to use 'h' in the URL:
https://poloniex.com/public?command=return24hVolume
I've been trying to unpickle some dictionaries from the database. I've reverted to using the marshal module, but was still wondering why pickle is having such a difficult time unserializing some data. Here is a command line python session showing essentially what I am trying to do:
>>> a = {'service': 'amazon', 'protocol': 'stream', 'key': 'lajdfoau09424jojf.flv'}
>>> import pickle; import base64
>>> pickled = base64.b64encode(pickle.dumps(a))
>>> pickled
'KGRwMApTJ3Byb3RvY29sJwpwMQpTJ3N0cmVhbScKcDIKc1Mna2V5JwpwMwpTJ2xhamRmb2F1MDk0MjRqb2pmLmZsdicKcDQKc1Mnc2VydmljZScKcDUKUydhbWF6b24nCnA2CnMu'
>>> unpickled = pickle.loads(base64.b64decode(pickled))
>>> unpickled
{'protocol': 'stream', 'service': 'amazon', 'key': 'lajdfoau09424jojf.flv'}
>>> unpickled['service']
'amazon'
This works all fine, but when I try this inside of a factory method for a class, it seems like the pickle.loads part errors out. The strings I am trying to load are pickled the same way as above. I've even tried copying the exact string that is pickled in the command line session above and just trying to unpickle that, but with no success. Here is the code for this latter attempt:
class Resource:
_service = 'unknown'
_protocol = 'unknown'
_key = 'unknown'
'''
Factory method that creates an appropriate instance of one of Resource’s subclasses based on
the type of data provided (the data being a serialized dictionary with at least the keys 'service',
'protocol', and 'key').
#param resource_data (string) -- the data used to create the new Resource instance.
'''
#staticmethod
def resource_factory(resource_data):
# Unpack the raw resource data and then create the appropriate Resource instance and return.
resource_data = "KGRwMApTJ3Byb3RvY29sJwpwMQpTJ3N0cmVhbScKcDIKc1Mna2V5JwpwMwpTJ2xhamRmb2F1MDk0MjRqb2pmLmZsdicKcDQKc1Mnc2VydmljZScKcDUKUydhbWF6b24nCnA2CnMu" #hack to just see if we can unpickle this string
logging.debug("Creating resource: " + resource_data)
unencoded = base64.b64decode(resource_data)
logging.debug("Unencoded is: " + unencoded)
unpacked = pickle.loads(unencoded)
logging.debug("Unpacked: " + unpacked)
service = unpacked['service']
protocol = unpacked['protocol']
key = unpacked['key']
if (service == 'amazon'):
return AmazonResource(service=service, protocol=protocol, key=key)
elif (service == 'fs'):
return FSResource(service=service, protocol=protocol, key=key)
Your code works. How are you testing it?
import logging
import base64
import pickle
class Resource:
#staticmethod
def resource_factory(resource_data):
resource_data = "KGRwMApTJ3Byb3RvY29sJwpwMQpTJ3N0cmVhbScKcDIKc1Mna2V5JwpwMwpTJ2xhamRmb2F1MDk0MjRqb2pmLmZsdicKcDQKc1Mnc2VydmljZScKcDUKUydhbWF6b24nCnA2CnMu" #hack to just see if we can unpickle this string
# logging.debug("Creating resource: " + resource_data)
unencoded = base64.b64decode(resource_data)
# logging.debug("Unencoded is: " + unencoded)
unpacked = pickle.loads(unencoded)
logging.debug("Unpacked: " + repr(unpacked))
service = unpacked['service']
protocol = unpacked['protocol']
key = unpacked['key']
logging.basicConfig(level=logging.DEBUG)
Resource.resource_factory('')
yields
# DEBUG:root:Unpacked: {'protocol': 'stream', 'service': 'amazon', 'key': 'lajdfoau09424jojf.flv'}
I was able to solve this after making some simplifications and then debugging in django. The main issue was that there were some errors in the Resource class itself that were preventing the correct completion of the resource_factory method. First, I was trying to concatenate a string and dictionary, which was throwing an error. I also had some errors elsewhere in the class where I was referring to the instance variables _service, _protocol, and key withouth the '' (typos).
In any case, the interesting thing was that when I used this code within Django's custom field infrastructure, the errors would get caught and I did not see any actual message indicating the problem. The debug statements suggested it was a problem with loads, but actually it was a problem with the debug statement itself and some code that came later. When I tried to implement this behavior using model properties instead of custom model fields for the data I was saving, the errors actually got printed out correctly and I was able to quickly debug.