Develop Reference

Python: Printing all Protocol Buffer field options

I have proto message that I need to attach a string containing information about the data (units, scaling factor etc).
I am using custom options as per the proto2 guide. I also have attempted to follow a previous question where they printed custom options for 1 message here. I would like to print out all fields from the message along side the string. I have been struggling to find a solution, so far what i have tried is:
In my .proto file: The Service message contains fields that are themselves messages such as My_Message.
import "google/protobuf/descriptor.proto";
extend google.protobuf.FieldOptions {
optional string formatting_stuff = 50000;
}
message My_Message
{
optional uint32 base_frequency_hz = 1 [(formatting_stuff) = "test"];
optional float trigger_frequency_hz = 2 [(formatting_stuff) = "test2"];
...
}
message Service
{
optional My_Message x = 13;
...
}
Then in the python file,how would iterate over each field in Service and the nested message My_Message to retrieve both its value and also then extract the 'formatting_stuff'.
This is my attempt, but I can not get the field options to print the custom options:
messanger.py
...
rsp = self.proto.Service().FromString(self.data)
desc = protobuf_file.Service.My_Message.DESCRIPTOR
for rsp_field in response.DESCRIPTOR.fields:
print("Response Field name: ", rsp_field.name)
print("Response Field value: ", getattr(rsp, rsp_field.name))
options = desc.GetOptions()
formatting = options.Extensions[protobuf_file.formatting_stuff]
print("Response Field formatting: ", formatting)
The terminal output is:
Response Field name: base_frequency_hz
Response Field value: 720000
Response Field formatting:
Response Field name: trigger_frequency_hz
Response Field value: 3000.0
Response Field formatting:
As you can see, my custom fields are not being printed. Please could you assist me with printing out the custom options.
Kind regards

How to retrieve well formatted JSON from AWS Lambda using Python

I have a function in AWS Lambda that connects to the Twitter API and returns the tweets which match a specific search query I provided via the event. A simplified version of the function is below. There's a few helper functions I use like get_secret to manage API keys and process_tweet which limits what data gets sent back and does things like convert the created at date to a string. The net result is that I should get back a list of dictionaries.
def lambda_handler(event, context):
twitter_secret = get_secret("twitter")
auth = tweepy.OAuthHandler(twitter_secret['api-key'],
twitter_secret['api-secret'])
auth.set_access_token(twitter_secret['access-key'],
twitter_secret['access-secret'])
api = tweepy.API(auth)
cursor = tweepy.Cursor(api.search,
q=event['search'],
include_entities=True,
tweet_mode='extended',
lang='en')
tweets = list(cursor.items())
tweets = [process_tweet(t) for t in tweets if not t.retweeted]
return json.dumps({"tweets": tweets})
From my desktop then, I have code which invokes the lambda function.
aws_lambda = boto3.client('lambda', region_name="us-east-1")
payload = {"search": "paint%20protection%20film filter:safe"}
lambda_response = aws_lambda.invoke(FunctionName="twitter-searcher",
InvocationType="RequestResponse",
Payload=json.dumps(payload))
results = lambda_response['Payload'].read()
tweets = results.decode('utf-8')
The problem is that somewhere between json.dumpsing the output in lambda and reading the payload in Python, the data has gotten screwy. For example, a line break which should be \n becomes \\\\n, all of the double quotes are stored as \\" and Unicode characters are all prefixed by \\. So, everything that was escaped, when it was received by Python on my desktop with the escaping character being escaped. Consider this element of the list that was returned (with manual formatting).
'{\\"userid\\": 190764134,
\\"username\\": \\"CapitalGMC\\",
\\"created\\": \\"2018-09-02 15:00:00\\",
\\"tweetid\\": 1036267504673337344,
\\"text\\": \\"Protect your vehicle\'s paint! Find out how on this week\'s blog.
\\\\ud83d\\\\udc47\\\\n\\\\nhttps://url/XYMxPhVhdH https://url/mFL2Zv8nWW\\"}'
I can use regex to fix some problems (\\" and \\\\n) but the Unicode is tricky because even if I match it, how do I replace it with a properly escaped character? When I do this in R, using the aws.lambda package, everything is fine, no weird escaped escapes.
What am I doing wrong on my desktop with the response from AWS Lambda that's garbling the data?
Update
The process tweet function is below. It literally just pulls out the bits I care to keep, formats the datetime object to be a string and returns a dictionary.
def process_tweet(tweet):
bundle = {
"userid": tweet.user.id,
"username": tweet.user.screen_name,
"created": str(tweet.created_at),
"tweetid": tweet.id,
"text": tweet.full_text
}
return bundle
Just for reference, in R the code looks like this.
payload = list(search="paint%20protection%20film filter:safe")
results = aws.lambda::invoke_function("twitter-searcher"
,payload = jsonlite::toJSON(payload
,auto_unbox=TRUE)
,type = "RequestResponse"
,key = creds$key
,secret = creds$secret
,session_token = creds$session_token
,region = creds$region)
tweets = jsonlite::fromJSON(results)
str(tweets)
#> 'data.frame': 133 obs. of 5 variables:
#> $ userid : num 2231994854 407106716 33553091 7778772 782310 ...
#> $ username: chr "adaniel_080213" "Prestige_AdamL" "exclusivedetail" "tedhu" ...
#> $ created : chr "2018-09-12 14:07:09" "2018-09-12 11:31:56" "2018-09-12 10:46:55" "2018-09-12 07:27:49" ...
#> $ tweetid : num 1039878080968323072 1039839019989983232 1039827690151444480 1039777586975526912 1039699310382931968 ...
#> $ text : chr "I liked a #YouTube video https://url/97sRShN4pM Tesla Model 3 - Front End Package - Suntek Ultra Paint Protection Film" "Another #Corvette #ZO6 full body clearbra wrap completed using #xpeltech ultimate plus PPF ... Paint protection"| __truncated__ "We recently protected this Tesla Model 3 with Paint Protection Film and Ceramic Coating.#teslamodel3 #charlotte"| __truncated__ "Tesla Model 3 - Front End Package - Suntek Ultra Paint Protection Film https://url/AD1cl5dNX3" ...
tweets[131,]
#> userid username created tweetid
#> 131 190764134 CapitalGMC 2018-09-02 15:00:00 1036267504673337344
#> text
#> 131 Protect your vehicle's paint! Find out how on this week's blog.👇\n\nhttps://url/XYMxPhVhdH https://url/mFL2Zv8nWW

In your lambda function you should return a response object with a JSON object in the response body.
# Lambda Function
def get_json(event, context):
"""Retrieve JSON from server."""
# Business Logic Goes Here.
response = {
"statusCode": 200,
"headers": {},
"body": json.dumps({
"message": "This is the message in a JSON object."
})
}
return response

Don't use json.dumps()
I had a similar issue, and when I just returned "body": content instead of "body": json.dumps(content) I could easily access and manipulate my data. Before that, I got that weird form that looks like JSON, but it's not.

Django and JSON/AJAX testing

I tried looking around for an answer and gave it a great many tries, but there's something strange going on here. I got some functions in my view that operate on JSON data that comes in via AJAX. Currently I'm trying to do some unit testing on these.
In my test case I have:
kwargs = {'HTTP_X_REQUESTED_WITH': 'XMLHttpRequest'}
url = '/<correct_url>/upload/'
data = {
"id" : p.id
}
c = Client()
response = c.delete(url, data, **kwargs)
content_unicode = response.content.decode('utf-8')
content = json.loads(content_unicode)
p.id is just an integer that comes from a model I'm using.
I then have a function that is being tested, parts of which looks like follows:
def delete_ajax(self, request, *args, **kwargs):
print (request.body)
body_unicode = request.body.decode('utf-8')
print (body_unicode)
body_json = json.loads(body_unicode)
The first print statement yields:
.....b"{'id': 1}"
The other one:
{'id': 1}
and finally I get an error for fourth line as follows:
json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)
What's going wrong here? I understand that correct JSON format should be {"id": 1} and that's what I'm sending from my test case. But somewhere along the way single-quotes are introduced into the mix causing me head ache.
Any thoughts?

You need to pass a json string to Client.delete(), not a Python dict:
kwargs = {'HTTP_X_REQUESTED_WITH': 'XMLHttpRequest'}
url = '/<correct_url>/upload/'
data = json.dumps({
"id" : p.id
})
c = Client()
response = c.delete(url, data, **kwargs)
You should also set the content-type header to "application/json" and check the content-type header in your view but that's another topic.

Sending automatic SMS alerts Python/Django

I'm building a website using Django. And I want that users can be able to receive alerts via SMS when new topics are posted.
I tested textlocal but I had an issue when trying to send SMS to multiple numbers (numbers = ['xxxxx','xxxxx']) .(I don't want to use group_id).
Generally I want to be able to do something like this:
numbers = (SELECT number FROM users WHERE SMS_subscribe=1)
sender = 'mywebsite'
message = 'Hey, a new topic was posted'
send_sms(numbers, message, sender)
My textlocal test code:
#!/user/bin/python
# -*- coding: utf-8 -*-
from urllib2 import Request, urlopen
from urllib import urlencode
def send_sms(uname, hash_code, numbers, message, sender):
data = urlencode({
'username' : uname,
'hash' : hash_code,
'numbers' : numbers,
'message' : message,
'sender' : sender,
'test' : True
})
#data = data.encode('utf-8')
request = Request('https://api.txtlocal.com/send/?')
response = urlopen(request, data)
return response.read()
def just_one_sms_message(message, annonce_link, sender):
links_len=len(annonce_link) + len(sender) + 1
sms_max_len = 160 - links_len
if len(message)>sms_max_len:
message = message[:sms_max_len-6]+'... : '
else:
message += ' : '
return message + annonce_link + '\n' + sender
username = 'xxxxxxx#gmail.com'
hash_code = '3b5xxxxxxxxxxxxxxxxxxxxxxxxxxx8d83818'
numbers = ('2126xxxxx096','2126xxxxx888')
annonce_link = 'http://example.com/'
sender = 'sender'
message = 'New topics..'
message = just_one_sms_message(message, annonce_link, sender)
resp = send_sms(username, hash_code, numbers, message, sender)
print resp
Executing this code I get this error :
{"warnings":[{"code":3,"message":"Invalid number"}],"errors":[{"code":4,"message":"No recipients specified"}],"status":"failure"}
But if I change: numbers=('2126xxxxx096')it works.
What is the best way or web service to do this ?

There are a couple issues you're running into. The first is how tuple literals are defined.
('somenumber') is equivalent to 'somenumber' in python. It's just a string. The parentheses alone do not define a tuple literal. To define a single-element tuple literal, you need a trailing comma of the first element. E.G. ('somenumber',).
The second issue is how urlencode works. For each value in the data dictionary, it asks for the string representation.
In the case of ('2126xxxxx096','2126xxxxx888'), since it's evaluated as a tuple, it's encoded as ('2126xxxxx096','2126xxxxx888'), resulting in %28%272126xxxxx096%27%2C+%272126xxxxx888%27%29.
In the case of ('2126xxxxx096'), since it's evaluated as a string, it's encoded as 2126xxxxx096. Notice the lack of junk characters like %28 and %29.
So, in short, since the value of numbers in the urlencode dictionary is a tuple when you have multiple numbers, you need to convert the tuple into a comma-separated string. This can be accomplished via ",".join(numbers), which in the case of ('2126xxxxx096','2126xxxxx888') produces 2126xxxxx096%2C2126xxxxx888. With the fixed encoding, your message should now send to multiple numbers.

my post method returns (u'') and django saves includes the (u'') string when saving it

This is how I retrieve the post data from the webpage. The person models can be saved but it includes the "(u'')" string. For example if change the firstname to "Alex", it gets the raw value u('Alex') and saves it.
def submit_e(req, person_id=None):
if(req.POST):
try:
person_id = req.POST['driver']
person = Person.objects.get(pk=person_id)
person.firstname = req.POST['firstname'],
person.midname = req.POST['middleinitial'],
person.lastname = req.POST['lastname'],
person.full_clean()
person.save()
except Exception as e:
print e
return HttpResponseRedirect(reverse('users:user_main'))

NB: the following is my best guess at what you are seeing based on your question. If I have guessed wrongly, the please update your post with more details - putting print statements throughout your code and adding the output to your post would be a good start.
The u prefix on a string indicates a Unicode string. It is not actually part of the contents of the string. If we create a string in the interpreter:
>>> name = u'Me'
and then request details of the string,
>>> name
u'Me'
then the u is shown as it is part of the information about the string, which is what we have requested. If we print the contents of the string
>>> print name
Me
then the u is not shown (just like the quotes aren't shown).
Using the interpreter to try and reproduce your problem, I created a new user with a Unicode string for a username:
>>> from django.contrib.auth.models import User
>>> new_user = User()
>>> new_user.username = u'Me'
>>> new_user.save()
And as before, if we request the details about the string we see the u and the quotes, but if we print the contents of the string we don't:
>>> new_user.username
u'Me'
>>> print new_user.username
>>> Me
To further confirm the u was not stored, we can explore the database directly:
sqlite> select username from auth_user;
Me

you need to delete the "," at the end of each line
so, before:
person.firstname = req.POST['firstname'],
person.midname = req.POST['middleinitial'],
person.lastname = req.POST['lastname'],
after
person.firstname = req.POST['firstname']
person.midname = req.POST['middleinitial']
person.lastname = req.POST['lastname']