Using django trunk r13359 and django piston, I created a small restful service that stores string values.
This is the model I am using to store strings:
class DataStore(models.Model):
data = models.CharField(max_length=200)
url = models.URLField(default = '', verify_exists=False, blank = True)
I used curl to post following data:
curl -d "data=somedata" http://localhost:8000/api/datastorage/
This is the code that handles storage as part of the django-piston handler
store = DataStore()
store.url = request.POST.get('url',""),
store.data = request.POST['data'],
store.save()
return {'data':store}
When I post the data with curl I get the following response body, which is expected:
{
"result": {
"url": [
""
],
"data": [
"somedata"
],
"id": 1
}
}
Whats not expected however is when I look at the stored instance from django admin, the value stored in the data field looks something like this:
(u'somedata',)
and the following is stored in the url:
('',)
Whats even more interesting is when I query the service with curl to see what is stored, I get the following:
{
"result": {
"url": [
"('',)"
],
"data": [
"(u'somedata',)"
],
"id": 1
}
}
I'm stumped .. any ideas what could be going on?
Actually your response is also not what should be expected, note the [] around your strings, those shouldn't be there.
Your error is adding the comma after these two lines:
store.url = request.POST.get('url',""),
store.data = request.POST['data'],
Python will interprete you want to store a tuple in url and data, and django will convert those tuples to strings implicitly, resulting in the behaviour you see. Just remove the two commas and you'll be fine.
Related
I am trying to write a program that transfers users drive and docs files from one user to another. It looks like I can do it using this documentation Documentation.
I created the data transfer object, which looks like this:
datatransfer = {
'kind': 'admin#datatransfer#DataTransfer',
'oldOwnerUserId': 'somenumberhere',
'newOwnderUserId': 'adifferentnumberhere',
'applicationDataTransfers':[
{
'applicationId': '55656082996', #the app id for drive and docs
'applicationTransferParams': [
{
'key': 'PRIVACY_LEVEL',
'value': [
{
'PRIVATE',
'SHARED'
}
]
}
]
}
]
}
I have some code here for handling Oauth and then I bind the service with:
service = build('admin', 'datatransfer_v1', credentials=creds)
Then I attempt to call insert() with
results = service.transfers().insert(body=datatransfer).execute()
and I get back an error saying that it 'missing required field: resource'.
I tried nesting all of this inside a field called resource and I get the same message.
I tried passing in JUST a json structure that looked like this {'resource': 'test'} and I get the same message.
So I tried using the "Try this method" live tool on the documentation website,
If I pass in no arguments at all, or just pass in the old and new user, I get the same message 'missing required nested field: resource'.
If I put in 'id':'55656082996' with ANY other arguments it just says error code 500 backend error.
I tried manually adding a field named "resource" to the live tool and it says 'property 'resource' does not exist in object specification"
I finally got this to work. If anyone else is struggling with this and stumbles on this, "applicationId" is a number, not a string. Also, the error message is misleading - there is no nested field called "resource." This is what worked for me:
datatransfer = {
"newOwnerUserId": "SomeNumber",
"oldOwnerUserId": "SomeOtherNumber",
"kind": "admin#datatransfer#DataTransfer",
"applicationDataTransfers": [
{
"applicationId": 55656082996,
"applicationTransferParams": [
{
"key": "PRIVACY_LEVEL"
},
{
"value": [
"{PRIVATE, SHARED}"
]
}
]
}
]
}
service = build('admin', 'datatransfer_v1', credentials=creds)
results = service.transfers().insert(body=datatransfer).execute()
print(results)
To get the user's Id's I'm first using the Directory API to query all users who are suspended, and getting their ID from that. Then passing their ID into this to transfer their files to another user before deleting them.
I am using Twitter's streaming API code (found here). I am able to get my desired output which is a series of filtered results. However, I specifically need to assign the 'text' field from the JSON result to a variable and I am unable to come up with the right way to do it.
I have isolated the part of the code that returns the streaming data and display it in the terminal when I run it:
for response_line in response.iter_lines():
if response_line:
json_response = json.loads(response_line)
print(json.dumps(json_response, indent=4, sort_keys=True))
What I need is to just get the text part of the tweet that is returned. Here's an output example, noting I only need to set a variable - twitterVariable to the "text" result:
{
"data": {
"id": "125855555555",
"text": "hello this is a test"
},
"matching_rules": [
{
"id": 1234567890,
"tag": ""
}
]
}
As you have already loaded the response into dict object of python, you can use key to get the text field as below:
twitter_variable = json_response['data']['text']
It's when I send a PUT request to my API endpoint from python with a JSON request body I receive empty request body, because sometimes It's containing special characters which is not supported by JSON.
How can I sanitize my JSON before sending my request?
I've tried with stringify and parsing json before I sent my request!
profile = json.loads(json.dumps(profile))
My example invalid json is:
{
"url": "https://www.example.com/edmund-chand/",
"name": "Edmund Chand",
"current_location": "FrankfurtAmMainArea, Germany",
"education": [],
"skills": []
}
and My expected validated json should be:
{
"url": "https://www.example.com/edmund-chand/",
"name": "Edmund Chand",
"current_location": "Frankfurt Am Main Area, Germany",
"education": [],
"skills": []
}
If you're looking for something quick to sanitize json data for limited fields i.e. current_location, you can try something like the following below:
def sanitize(profile):
profile['current_location'] = ', '.join([val.strip() for val in profile['current_location'].split(',')])
return profile
profile = sanitize(profile)
The idea here is that you would write code to sanitize each bits in that function and send it your api or throw exception if invalid etc.
For more robust validation, you can consider using jsonschema package. More details here.
With that package you can validate strings and json schema more flexibly.
Example taken from the package readme:
from jsonschema import validate
# A sample schema, like what we'd get from json.load()
schema = {
"type" : "object",
"properties" : {
"url" : {"type" : "string", "format":"uri"},
"current_location" : {"type" : "string", "maxLength":25, "pattern": "your_regex_pattern"},
},
}
# If no exception is raised by validate(), the instance is valid.
validate(instance=profile, schema=schema)
You can find more infor and types of available validation for strings here.
Thank you #Rithin for your solution but that one seems more coupled with one field of the whole JSON.
I found a solution to replace it with below example code which works for any field:
profile = json.loads(json.dumps(profile).replace("\t", " "))
I am trying to write some test cases for some code I've developed using Elasticsearch and Django. The concept is straightforward - I just want to test a get request, which will be an Elasticsearch query. However, I am constructing the query as a nested dict. When I pass the nested dict to the Client object in the test script it gets passed through Django until it ends up at the urlencode function which doesn't look like it can handle nested dicts only MultivalueDicts. Any suggestions or solutions? I don't want to use any additional packages as I don't want to depend on potentially non-supported packages for this application.
Generic Code:
class MyViewTest(TestCase):
es_connection = elasticsearch.Elasticsearch("localhost:9200")
def test_es_query(self):
client = Client()
query = {
"query": {
"term": {
"city": "some city"
}
}
}
response = client.get("", query)
print(response)
Link for urlencode function: urlencode Django
The issue is clearly at the conditional statement when the urlencode function checks if the dictionary value is a str or bytes object. If it isn't it creates a generator object which can never access the nested portions of the dictionary.
EDIT: 07/25/2018
So I was able to come up with a temporary work around to at least run the test. However, it is ugly and I feel like there must be a better way. The first thing I tried was specifying the content_type and converting the dict to a json string first. However, Django still kicked back and error in the urlencode function.
class MyViewTest(TestCase):
es_connection = elasticsearch.Elasticsearch("localhost:9200")
def test_es_query(self):
client = Client()
query = {
"query": {
"term": {
"city": "some city"
}
}
}
response = client.get("", data=json.dumps(query), content_type="application/json")
print(response)
So instead I had to do:
class MyViewTest(TestCase):
es_connection = elasticsearch.Elasticsearch("localhost:9200")
def test_es_query(self):
client = Client()
query = {
"query": {
"term": {
"city": "some city"
}
}
}
query = json.dumps(query)
response = client.get("", data={"q": query}, content_type="application/json")
print(response)
This let me send the HttpRequest to my View and parse it back out using:
json.loads(request.GET["q"])
Then I was able to successfully get the requested data from Elasticsearch and return it as an HttpResponse. I feel like in Django though there has to be a way to just pass a json formatted string directly to the Client object's get function. I thought specifying the content_type as application/json would work but it still calls the urlencode function. Any ideas? I really don't want to implement this current system into production.
Very new to json so please forgive if I am using the wrong terms here. Any ways I am trying to create a json file every x minutes with updated twitter info. Now I know i could just use the twitter api and grab what I need but I am wanting to mess around a bit. My problem is getting a new key/dict? or what ever it is called for each new item added in a for statement.
What im trying to get
[
{
"name": "thename",
"date": "thedate",
"text": "thetext"
}, <--- trying to get that little comma
{
"name": "thename",
"date": "thedate",
"text": "thetext"
}
]
Now i am getting the data that i want and all but not that comma. It outputs it all but not like it should be with that little one character thing that makes it valid.
[
{
"name": "thename",
"date": "thedate",
"text": "thetext"
} <--- I get this instead for each new object
{
"name": "thename",
"date": "thedate",
"text": "thetext"
}
]
Here is just a snippet of the code as the rest should be explanatory since its just Twitter oauth stuff.
for player in users:
statuses = api.GetUserTimeline(screen_name=player, count=10)
print "["
for s in statuses:
print json.dumps('timestamp': s.created_at,'username': s.user.name,'status': s.text)
print "]"
Also is there a better way to do the [ ] at the start and end because I know that is ugly and way un-proper i bet XD . Like I said newbie on json/python stuff but its a good learning experience.
Instead of trying to build the JSON array yourself, you should just let the JSON encoder do that for you too. To do that, you just need to pass a list of objects to json.dumps. So instead of printing each JSON object on its own, just collect them in a list, and dump that:
allstatuses = []
for player in users:
statuses = api.GetUserTimeline(screen_name=player, count=10)
for s in statuses:
allstatuses.append({'timestamp': s.created_at, 'username': s.user.name, 'status': s.text})
print(json.dumps(allstatuses))