I am currently creating a callback URL in Django for a webhook in Mailchimp where Mailchimp will send a POST request with urlencoded data in the form of application/x-www-form-urlencoded.
The issue I have run into is that the data returned contains nested data. Some of the data in this urlencoded string looks like its defining nested JSON, which I believe is non-standard (I could be mistaken, though).
For example, one POST request from Mailchimp, which is sent when a user changes their name, would look like:
type=profile&fired_at=2021-05-25+18%3A03%3A23&data%5Bid%5D=abcd1234&data%5Bemail%5D=test%40domain.com&data%5Bemail_type%5D=html&data%5Bip_opt%5D=0.0.0.0&data%5Bweb_id%5D=1234&data%5Bmerges%5D%5BEMAIL%5D=test%40domain.com&data%5Bmerges%5D%5BFNAME%5D=first_name&data%5Bmerges%5D%5BLNAME%5D=last_name&data%5Blist_id%5D=5678
Using Django's request.POST, the data is decoded into:
{
'type': 'profile',
'fired_at': '2021-05-25 18:03:23',
'data[id]': 'abcd1234',
'data[email]': 'test#domain.com',
'data[email_type]': 'html',
'data[ip_opt]': '0.0.0.0',
'data[web_id]': '1234',
'data[merges][EMAIL]': 'test#domain.com',
'data[merges][FNAME]': 'first_name',
'data[merges][LNAME]': 'last_name',
'data[list_id]': '5678'
}
This looks really ugly in practice, since to access the first name of the user from request.POST we would have to do
request.POST.get("data['merges']['FNAME']", None)
The data is obviously intended to look like
{
'type': 'profile',
'fired_at': '2021-05-25 18:03:23',
'data': {
'id': 'abcd1234',
'email': 'test#domain.com',
'email_type': 'html',
'ip_opt': '0.0.0.0',
'web_id': '1234',
'merges':{
'email': 'test#domain.com',
'fname': 'first_name',
'lname': 'last_name',
},
'list_id': '5678'
},
}
and be accessed like
data = request.POST.get('data', None)
first_name = data['merges']['FNAME']
I have looked for a Django/Python specific way to decode this nested URL-encoded data into more appropriate formats to work with it in Python, but have been unable to find anything. Python's urllib library provides methods such as urllib.parse.parse_qs() to decode urlencoded strings, but these methods do not handle this nested type data.
Is there a way to properly decode this nested urlencoded data using Django/Python?
There is no standard library nor Django utility function for this.
We can implement convert_form_dict_to_json_dict as such:
Initialise json_dict to an empty dict {}.
For each form_key, using the example 'data[merges][EMAIL]',
Use regex to obtain nested_keys, i.e. ('data', 'merges', 'EMAIL').
Determine last_nesting_level, i.e. 2 from nesting levels (0, 1, 2).
Initialise current_dict to json_dict.
For each nesting_level, current_key, i.e. 0, 'data', 1, 'merges', 2, 'EMAIL',
If it is before last_nesting_level, get next current_dict using current_key.
Else, set current_dict entry for current_key to value.
Return json_dict.
import re
def convert_form_dict_to_json_dict(form_dict):
json_dict = {}
for form_key, value in form_dict.items():
nested_keys = (re.match(r'\w+', form_key).group(0), *re.findall(r'\[(\w+)]', form_key))
last_nesting_level = len(nested_keys) - 1
current_dict = json_dict
for nesting_level, current_key in enumerate(nested_keys):
if nesting_level < last_nesting_level:
current_dict = current_dict.setdefault(current_key, {})
else:
current_dict[current_key] = value
return json_dict
Usage:
POST_dict = {
'type': 'profile',
'fired_at': '2021-05-25 18:03:23',
'data[id]': 'abcd1234',
'data[email]': 'test#domain.com',
'data[email_type]': 'html',
'data[ip_opt]': '0.0.0.0',
'data[web_id]': '1234',
'data[merges][EMAIL]': 'test#domain.com',
'data[merges][FNAME]': 'first_name',
'data[merges][LNAME]': 'last_name',
'data[list_id]': '5678'
}
from pprint import pprint
pprint(convert_form_dict_to_json_dict(POST_dict))
Related
I'am trying to make an API request where I add some optional values but if I don't add them I don't want them to be in the request.
In this case, I would like the parameters to be 'startblock' & 'endblock'
def get_transactions_by_address_(self, address, action='txlist'):
"""
:param address:
:param action: [txlist, txlistinternal]
:return:
"""
token = self.etherscan_api_key
return requests.get('https://api.etherscan.io/api'
'?module=account'
f'&action={action}'
f'&address={address}'
# '&startblock=0'
# '&endblock=92702578'
'&page=1'
'&offset=1000'
'&sort=desc'
f'&apikey={token}'
)
I was thinking on adding some conditionals like
request_url = 'https://api.etherscan.io/api...'
if startblock:
request_url = request_url + f'&startblock={startblock}'
if endblock:
request_url = request_url + f'&endblock={endblock}'
But I don't know if it is the most pythonic way to do it and I would like to get other options on how to do it
Use the payload option instead of constructing the URL yourself. For example, create a dict containing all the required options, then add additional parameters to the dict as necessary. requests.get will build the required URL from the base URL and the values found in your dict.
options = {
'module': 'account',
'action': action,
'address': address,
'apikey': token,
'sort': sort,
'page': page,
'offset': offset
}
if startblock:
options['startblock'] = startblock
if endblock:
options['endblock'] = endblock
return requests.get('https://api.etherscan.io/api', params=options)
The correct way to implement is:
def get_transactions_by_address_(self, address,
action='txlist',
sort='desc',
page=1,
offset=1000,
startblock=None,
endblock=None):
token = self.etherscan_api_key
options = {
'module': 'account',
'action': action,
'address': address,
'apikey': token,
'sort': sort,
'page': page,
'offset': offset
}
if startblock:
options['startblock'] = startblock
if endblock:
options['endblock'] = endblock
return requests.get('https://api.etherscan.io/api',
params=options
)
I have a rather basic bit of code. Basically what it does is sends an API request to a locally hosted Server and returns a JSON string. I'm taking that string and cracking it apart. Then I take what I need from it, make a Dictionary, and export it as an XML file with an nfo extension.
The issue is sometimes there are missing bits to the source data. Season is missing fairly frequently for example. It breaks the Data Mapping. I need a way to handle that. For somethings I may want to exclude the data and for others I need a sane default value.
#!/bin/env python
import os
import requests
import re
import json
import dicttoxml
import xml.dom.minidom
from xml.dom.minidom import parseString
# Grab Shoko Auth Key
apiheaders = {
'Content-Type': 'application/json',
'Accept': 'application/json',
}
apidata = '{"user": "Default", "pass": "", "device": "CLI"}'
r = requests.post('http://192.168.254.100:8111/api/auth',
headers=apiheaders, data=apidata)
key = json.loads(r.text)['apikey']
# Grabbing Episode Data
EpisodeHeaders = {
'accept': 'text/plain',
'apikey': key
}
EpisodeParams = (
('filename',
"FILE HERE"),
('pic', '1'),
)
fileinfo = requests.get(
'http://192.168.254.100:8111/api/ep/getbyfilename', headers=EpisodeHeaders, params=EpisodeParams)
# Mapping Data from Shoko to Jellyfin NFO
string = json.loads(fileinfo.text)
print(string)
eplot = json.loads(fileinfo.text)['summary']
etitle = json.loads(fileinfo.text)['name']
eyear = json.loads(fileinfo.text)['year']
episode = json.loads(fileinfo.text)['epnumber']
season = json.loads(fileinfo.text)['season']
aid = json.loads(fileinfo.text)['aid']
seasonnum = season.split('x')
# Create Dictionary From Mapped Data
show = {
"plot": eplot,
"title": etitle,
"year": eyear,
"episode": episode,
"season": seasonnum[0],
}
Here is some example output when the code crashes
{'type': 'ep', 'eptype': 'Credits', 'epnumber': 1, 'aid': 10713, 'eid': 167848,
'id': 95272, 'name': 'Opening', 'summary': 'Episode Overview not Available',
'year': '2014', 'air': '2014-11-23', 'rating': '10.00', 'votes': '1',
'art': {'fanart': [{'url': '/api/v2/image/support/plex_404.png'}],
'thumb': [{'url': '/api/v2/image/support/plex_404.png'}]}}
Traceback (most recent call last):
File "/home/fletcher/Documents/Shoko-Jellyfin-NFO/Xml3.py", line 48, in <module>
season = json.loads(fileinfo.text)['season']
KeyError: 'season'
The solution based on what Mahori suggested. Worked perfectly.
eplot = json.loads(fileinfo.text).get('summary', None)
etitle = json.loads(fileinfo.text).get('name', None)
eyear = json.loads(fileinfo.text).get('year', None)
episode = json.loads(fileinfo.text).get('epnumber', None)
season = json.loads(fileinfo.text).get('season', '1x1')
aid = json.loads(fileinfo.text).get('aid', None)
This is fairly common scenario with web development, where you cannot always assume other party will send all keys.
The standard way to get around this is by using get instead of named fetch.
season = json.loads(fileinfo.text).get('season', None)
#you can change None to any default value here
I have a pretty big dictionary which looks like this:
{
'startIndex': 1,
'username': 'myemail#gmail.com',
'items': [{
'id': '67022006',
'name': 'Adopt-a-Hydrant',
'kind': 'analytics#accountSummary',
'webProperties': [{
'id': 'UA-67522226-1',
'name': 'Adopt-a-Hydrant',
'websiteUrl': 'https://www.udemy.com/,
'internalWebPropertyId': '104343473',
'profiles': [{
'id': '108333146',
'name': 'Adopt a Hydrant (Udemy)',
'type': 'WEB',
'kind': 'analytics#profileSummary'
}, {
'id': '132099908',
'name': 'Unfiltered view',
'type': 'WEB',
'kind': 'analytics#profileSummary'
}],
'level': 'STANDARD',
'kind': 'analytics#webPropertySummary'
}]
}, {
'id': '44222959',
'name': 'A223n',
'kind': 'analytics#accountSummary',
And so on....
When I copy this dictionary on my Jupyter notebook and I run the exact same function I run on my django code it runs as expected, everything is literarily the same, in my django code I'm even printing the dictionary out then I copy it to the notebook and run it and I get what I'm expecting.
Just for more info this is the function:
google_profile = gp.google_profile # Get google_profile from DB
print(google_profile)
all_properties = []
for properties in google_profile['items']:
all_properties.append(properties)
site_selection=[]
for single_property in all_properties:
single_propery_name=single_property['name']
for single_view in single_property['webProperties'][0]['profiles']:
single_view_id = single_view['id']
single_view_name = (single_view['name'])
selections = single_propery_name + ' (View: '+single_view_name+' ID: '+single_view_id+')'
site_selection.append(selections)
print (site_selection)
So my guess is that my notebook has some sort of json parser installed or something like that? Is that possible? Why in django I can't access dictionaries the same way I can on my ipython notebooks?
EDITS
More info:
The error is at the line: for properties in google_profile['items']:
Django debug is: TypeError at /gconnect/ string indices must be integers
Local Vars are:
all_properties =[]
current_user = 'myemail#gmail.com'
google_profile = `the above dictionary`
So just to make it clear for who finds this question:
If you save a dictionary in a database django will save it as a string, so you won't be able to access it after.
To solve this you can re-convert it to a dictionary:
The answer from this post worked perfectly for me, in other words:
import json
s = "{'muffin' : 'lolz', 'foo' : 'kitty'}"
json_acceptable_string = s.replace("'", "\"")
d = json.loads(json_acceptable_string)
# d = {u'muffin': u'lolz', u'foo': u'kitty'}
There are many ways to convert a string to a dictionary, this is only one. If you stumbled in this problem you can quickly check if it's a string instead of a dictionary with:
print(type(var))
In my case I had:
<class 'str'>
before converting it with the above method and then I got
<class 'dict'>
and everything worked as supposed to
I have clien-server app.
I localized trouble and there logic of this:
Client:
# -*- coding: utf-8 -*-
import requests
def fixing:
response = requests.post('http://url_for_auth/', data={'client_id': 'client_id',
'client_secret':'its_secret', 'grant_type': 'password',
'username': 'user', 'password': 'password'})
f = response.json()
data = {'coordinate_x': 12.3, 'coordinate_y': 8.4, 'address': u'\u041c, 12',
'products': [{'count': 1, 'id': 's123'},{'count': 2, 'id': 's124'}]}
data.update(f)
response = requests.post('http://url_for_working/, data=data)
response.text #There I have an Error about which I will say later
oAuth2 working well. But in server-side I have no products in request.data
<QueryDict: {u'token_type': [u'type_is_ok'], u'access_token': [u'token_is_ok'],
u'expires_in': [u'36000'], u'coordinate_y': [u'8.4'],
u'coordinate_x': [u'12.3'], u'products': [u'count', u'id', u'count',
u'id'], u'address': [u'\u041c, 12'], u'scope': [u'read write'],
u'refresh_token': [u'token_is_ok']}>
This part of QueryDict make me sad...
'products': [u'count', u'id', u'count', u'id']
And when I tried to make python dict:
request.data.dict()
... u'products': u'id', ...
And for sure other fields working well with Django serializer's validation. But not that, because there I have wrong values.
Looks like request (because it have x-www-encoded-form default) cant include list of dicts as value for key in dict so... I should use json in this case.
Finally I maked this func:
import requests
import json
def fixing:
response = requests.post('http://url_for_auth/', data={'client_id': 'client_id',
'client_secret':'its_secret', 'grant_type': 'password',
'username': 'user', 'password': 'password'})
f = response.json()
headers = {'authorization': f['token_type'].encode('utf-8')+' '+f['access_token'].encode('utf-8'),
'Content-Type': 'application/json'}
data = {'coordinate_x': 12.3, 'coordinate_y': 8.4, 'address': u'\u041c, 12',
'products': [{'count': 1, 'id': 's123'},{'count': 2, 'id': 's124'}]}
response = requests.post('http://url_for_working/', data=json.dumps(data),
headers=headers)
response.text
There I got right response.
Solved!
Hello i would like to refresh this topic, cause i have similar problem to this and above solution doesn`t work for me.
import requests
import urllib.request
import pprint
import json
from requests import auth
from requests.models import HTTPBasicAuth
payload = {
'description': 'zxcy',
'tags':[{
'id': 22,
'label': 'Card'}]
}
files = {'file': open('JAM5.pdf','rb')}
client_id = 32590
response = requests.post('https://system...+str(client_id)' , files=files ,data=payload, auth=HTTPBasicAuth(...)
Above code succesfully add file to CRM system and description to added file, but i have to add label to this too, and its seems doesnt work at all
When i try it with data=json.dumps(payload) i got this:
raise ValueError("Data must not be a string.")
ValueError: Data must not be a string.
I have been reading through this document. Most of the document is based on finding an email's uid. From the article:
"The way this works is pretty simple: use the uid function, and pass in the string of the command in as the first argument. The rest behaves exactly the same.
result, data = mail.uid('search', None, "ALL") # search and return uids instead
latest_email_uid = data[0].split()[-1]
result, data = mail.uid('fetch', latest_email_uid, '(RFC822)')
raw_email = data[0][1]
I'm working with a django app called django-mailbox (http://django-mailbox.readthedocs.org/en/latest/index.html) the purpose of which is to consume emails.
The app creates a "Message" model that looks like:
u'django_mailbox.message': {
'Meta': {'object_name': 'Message'},
'body': ('django.db.models.fields.TextField', [], {}),
'encoded': ('django.db.models.fields.BooleanField', [], {'default': 'False'}),
'from_header': ('django.db.models.fields.CharField', [], {'max_length': '255'}),
u'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
'in_reply_to': ('django.db.models.fields.related.ForeignKey', [], {'blank': 'True', 'related_name': "'replies'", 'null': 'True', 'to': u"orm['django_mailbox.Message']"}),
'mailbox': ('django.db.models.fields.related.ForeignKey', [], {'related_name': "'messages'", 'to': u"orm['django_mailbox.Mailbox']"}),
'message_id': ('django.db.models.fields.CharField', [], {'max_length': '255'}),
'outgoing': ('django.db.models.fields.BooleanField', [], {'default': 'False'}),
'processed': ('django.db.models.fields.DateTimeField', [], {'auto_now_add': 'True', 'blank': 'True'}),
'read': ('django.db.models.fields.DateTimeField', [], {'default': 'None', 'null': 'True', 'blank': 'True'}),
'subject': ('django.db.models.fields.CharField', [], {'max_length': '255'}),
'to_header': ('django.db.models.fields.TextField', [], {})
using the python "email" library I can select a record from a django queryset and turn it into an email object:
qs = Message.objects.filter("my criteria")
first = qs[0]
one = first.get_email_object() // one is an email object
Does the existing data in the db have a uid, and if so how can I grab it.
The strict answer to your question is "no". The document you quote is about looping through an IMAP folder (in this case, a Gmail account), which will certainly get a unique ID (uid) from the server which tracks the unique message ID for each Email message.
Because you are constructing a mail message object using Django, you won't have such a UID. The "ID" field you do get from django.db.models.fields.AutoField is the sequential auto-increment ID that the Gmail/IMAP web page you quote says is "unacceptable".
You may want to look at the "uuid" library (http://docs.python.org/2/library/uuid.html) to generate unique ID values for your messages, but unless you also store those in your database, you'll be re-generating them over and over.
If you care to share more exact information about what you're trying to build (a web-based Email reader, perhaps?) then we as a community might have some better ideas for you.
you will get the uid of your mail in response
email_user = 'your gmail'
email_pass = 'your app password'
mail = imaplib.IMAP4_SSL('imap.gmail.com')
mail.login(email_user, email_pass)
mail.select('inbox')
status, response = mail.uid('search', None, r'(X-GM-RAW "subject:\"your latest mail subject\"")')
response = response[0].decode('utf-8').split()
response.reverse()
response = response[:min(10, len(response))]
print (response)