I'd like to parse a JSON string into an object under Google App Engine (python). What do you recommend? Something to encode/stringify would be nice too. Is what you recommend built in, or a library that I have to include in my app? Is it secure? Thanks.
Consider using Django's json lib, which is included with GAE.
from django.utils import simplejson as json
# load the object from a string
obj = json.loads( string )
The link above has examples of Django's serializer, and here's the link for simplejson's documentation.
If you're looking at storing Python class instances or objects (as opposed to compositions of lists, strings, numbers, and dictionaries), you probably want to look at pickle.
Incidentally, to get Django 1.0 (instead of Django 0.96) running on GAE, you can use the following call in your main.py, per this article:
from google.appengine.dist import use_library
use_library('django', '1.0')
Edit: Native JSON support in Google App Engine 1.6.0 with Python 2.7
As of Google App Engine 1.6.0, you can use the Python 2.7 runtime by adding runtime: python27 in app.yaml, and then you can import the native JSON library with import json.
Google App Engine now supports python 2.7. If using python 2.7, you can do the following:
import json
structured_dictionary = json.loads(string_received)
Include the simplejson library with your app?
This is an old question, but I thought I'd give an updated, more detailed answer. For those landing here now, you are almost certainly using python 2.6 or greater, so you can use the built-in json module for Python 2 (or for Python 3, since Google recently added support for Python 3 on GAE). Importing is as easy as import json. Here are some examples of how to use the json module:
import json
# parse json_string into a dict
json_string = '{"key_one": "value_one", "key_two": 1234}'
json_dict = json.loads(json_string)
# json_dict: {u'key_two': 1234, u'key_one': u'value_one'}
# generate json from a dict
json_dict = {'key': 'value', 'key_two': 1234, 'key_three': True}
json_string = json.dumps(json_dict)
# json_string: '{"key_two": 1234, "key": "value", "key_three": true}'
If you are using an older version of python, stick to #Brian M. Hunt's answer.
Again, here is the doc page for the json module for Python 2, and here it is for Python 3.
If you're using Python2.6 or greater, I've used with success the built-in json.load function. Otherwise, simplejson works on 2.4 without dependencies.
Look at the python section of json.org. The standard library support for JSON started at python 2.6, which I believe is newer than what the app engine provides. Maybe one of the other options listed?
Related
I have a Python3 Flask app using Flask-Session (which adds server-side session support) and configured to use the filesystem type.
This type underlying uses the Werkzeug class werkzeug.contrib.cache.FileSystemCache (Werkzeug cache documentation).
The raw cache files look like this if opened:
J¬».].Äï;}î(å
_permanentîàå
respondentîåuuidîåUUIDîìî)Åî}î(åintîät˙ò∑flŒºçLÃ/∆6jhåis_safeîhåSafeUUIDîìîNÖîRîubåSECTIONS_VISITEDî]îåcurrent_sectionîKåSURVEY_CONTENTî}î(å0î}î(ås_idîås0îånameîåWelcomeîådescriptionîåîå questionsî]î}î(ås_idîhåq_idîhåq_constructîhåq_textîhå
q_descriptionîhåq_typeîhårequiredîhåoptions_rowîhåoptions_row_alpha_sortîhåreplace_rowîhåoptions_colîhåoptions_col_codesîhåoptions_col_alpha_sortîhåcond_continue_rules_rowîhåq_meta_notesîhuauå1î}î(hås1îhå Screeningîhå[This section determines if you fit into the target group.îh]î(}î(hh/håq1îh hh!å9Have you worked on a product in this field before?
The items stored in the session can be seen a bit above:
- current_section should be an integer, e.g., 0
- SECTIONS_VISITED should be an array of integers, e.g., [0,1,2]
- SURVEY_CONTENT format should be an object with structure like below
{
'item1': {
'label': string,
'questions': [{}]
},
'item2': {
'label': string,
'questions': [{}]
}
}
What you can see in the excerpt above, for example the text This section determines if you fit into the target group is the value of one label. The stuff after questions are keys that can be found in each questions object, e.g., q_text as well as their values, e.g., Have you worked on a product in this field before? is the value of q_text.
I need to retrieve data from the stored cache files in a way that I can read them without all the extra characters like å.
I tried using Werkzeug like this, where the item 9c3c48a94198f61aa02a744b16666317 is the name of the cache file I want to read. However, it was not found in the cache directory.
from werkzeug.contrib.cache import FileSystemCache
cache_dir="flask_session"
mode=0600
threshold=20000
cache = FileSystemCache(cache_dir, threshold=threshold, mode=mode)
item = "9c3c48a94198f61aa02a744b16666317"
print(cache.has(item))
data = cache.get(item)
print(data)
What ways are there to read the cache files?
I opened a GitHub issue in Flask-Session, but that's not really been actively maintained in years.
For context, I had an instance where for my web app writing to the database was briefly not working - but the data I need was also being saved in the session. So right now the only way to retrieve that data is to get it from these files.
EDIT:
Thanks to Tim's answer I solved it using the following:
import pickle
obj = []
with open(file_name,"rb") as fileOpener:
while True:
try:
obj.append(pickle.load(fileOpener))
except EOFError:
break
print(obj)
I needed to load all pickled objects in the file, so I combined Tim's solution with the one here for loading multiple objects: https://stackoverflow.com/a/49261333/11805662
Without this, I was just seeing the first pickled item.
Also, in case anyone has the same problem, I needed to use the same python version as my Flask app (related post). If I didn't, then I would get the following error:
ValueError: unsupported pickle protocol: 4
You can decode the data with pickle. Pickle is part of the Python standard library.
import pickle
with open("PATH/TO/SESSION/FILE") as f:
data = pickle.load(f)
I am using markdown2 module (found here: https://github.com/trentm/python-markdown2) The module is great but documentation is lacking.
I attempted something simple like converting a file using a Python script. I wanted to then use this to convert a batch of files. But even the first part is hitting a dead end. My code:
from markdown2 import Markdown
markdowner = Markdown()
markdowner.convert("file.md", "file.htm")
Could someone show me how?
NVM, I got the answer (and not from the repository mentioned above). Here's a simple Pythonic two-liner:
from markdown2 import Markdown
with open("file.htm", 'w') as output_file: output_file.write(Markdown().convert(open("file.md").read()))
I need to pull information on a long list of JIRA issues that live in a CSV file. I'm using the JIRA REST API in Python in a small script to see what kind of data I can expect to retrieve:
#!/usr/bin/python
import csv
import sys
from jira.client import JIRA
*...redacted*
csvfile = list(csv.reader(open(sys.argv[1])))
for row in csvfile:
r = str(row).strip("'[]'")
i = jira.issue(r)
print i.id,i.fields.summary,i.fields.fixVersions,i.fields.resolution,i.fields.resolutiondate
The ID (Key), Summary, and Resolution dates are human-readable as expected. The fixVersions and Resolution fields are resources as follows:
[<jira.resources.Version object at 0x105096b11>], <jira.resources.Resolution object at 0x105096d91>
How do I use the API to get the set of available fixVersions and Resolutions, so that I can populate this correctly in my output CSV?
I understand how JIRA stores these values, but the documentation on the jira-python code doesn't explain how to harness it to grab those base values. I'd be happy to just snag the available fixVersion and Resolution values globally, but the resource info I receive doesn't map to them in an obvious way.
You can use fixVersion.name and resolution.name to get the string versions of those values.
User mdoar answered this question in his comment:
How about using version.name and resolution.name?
I use pisa to create pdf documents to render to the user:
response = HttpResponse()
pisa.CreatePDF(src=html, dest=response, show_error_as_pdf=True)
return response
response.content contains the pdf. I've used the dropbox-python sdk to do this:
dropbox_client.put_file(folder_path, response.content)
It seems to understand response.content as a pdf and uploads the file correctly
I need to do the same thing with the google-drive-python-api. This reference (https://developers.google.com/drive/v2/reference/files/insert) shows a basic method, but MediaFileUpload seems to look for a physical file. There is MediaIoBaseUpload also but it doesn't seem to accept response.content. I'm not very familiar with file/i/o stuff, so I listed everything from django to dropbox to G-Drive here, hoping it would clarify my use; hopefully I didn't confuse matters.
The apiclient.http file from the python Google API toolkit, contains the MediaIoBaseUpload object that does exactly what you need.
Only that it expects a file handle or something that behaves like file handle (the fh parameter). You're in luck: that's exactly what the StringIO module is for:
import StringIO # You could try importing cStringIO which gives better performance
fh = StringIO.StringIO(response.content)
media = MediaIoBaseUpload(fh, mimetype='some/mimetype')
# See https://developers.google.com/drive/v2/reference/files/insert for the rest
MediaInMemoryUpload would do the trick too, but it's deprecated now.
This question already has answers here:
Closed 11 years ago.
Possible Duplicate:
How can I parse JSON in Google App Engine?
json module was added in python 2.6 but GAE supports 2.5 only.
How can I use it there?
Well you wotn be able to use the standard python library, you will have to use some third party lib such as
jyson or
simplejson
At least simplejson works on python 2.4
You can try using yaml module (which is included in GAE) as JSON parser. YAML in recent version is superset of JSON.
>>> import yaml
>>> yaml.load('[false, null, {"x": 1}]')
[False, None, {'x': 1}]
You can try using django.utils.simplejson