Develop Reference

Formatting string representation of structure to python dictionary

I need a little help processing a String to a Dict, considering that the String is not in a common format, but an output from a UDF function
The return from the PySpark UDF looks like the string below:
"{list=[{a=1}, {a=2}, {a=3}]}"
And I need to convert it to a python dictionary with the structure below:
{
"list": [
{"a": 1}
{"a": 2}
{"a": 3}
]
}
So I can access it's values, like
dict["list"][1]["a"]
I already tried using:
JSON.loads
ast_eval()
Could someone please help me?
As an example of how this unparsed string is generated:
#udf()
def execute_method():
return {"list": [{"a":1},{"b":1}{"c":1}]}
df_result = df_source.withColumn("result", execute_method())

By the very least you will need to replace = with : and surround keys with double quotes:
import json
import re
string = "{list=[{a=1}, {a=2}, {a=3}]}"
fixed_string = re.sub(r'(\w+)=', r'"\1":', string)
print(type(fixed_string), fixed_string)
parsed = json.loads(fixed_string)
print(type(parsed), parsed)
outputs
<class 'str'> {"list":[{"a":1}, {"a":2}, {"a":3}]}
<class 'dict'> {'list': [{'a': 1}, {'a': 2}, {'a': 3}]}

try this :
import re
import json
data="{list=[{a=1}, {a=2}, {a=3}]}"
data=data.replace('=',':')
pattern=[e.group() for e in re.finditer('[a-z]+', data, flags=re.IGNORECASE)]
for e in set(pattern):
data=data.replace(e,"\""+e+"\"")
print(json.loads(data))

Python JSON "column1=value1;column2=value2" to {"column1":"value1","column2":"value2"}

A really basic requirement.
I would like to convert from this format:
"column1=value1;column2=value2"
to this format (JSON):
{"column1":"value1","column2":"value2"}
Any best approach in Python would be appreciated.
Thanks in advance.

using regular expressions
import re
REGEX = r"([^=;]+)=([^=;]+)"
finder = re.compile(REGEX)
s = "column1=value1;column2=value2"
matches = re.finditer(finder, s)
d = {}
for match in matches:
key = match.group(1)
val = match.group(2)
d[key] = val
print(d)
Output:
{'column2': 'value2', 'column1': 'value1'}

If you really want to parse your string to JSON you should try something like this:
import json # simplejson if you use a python version below 2.6
string = u'{"column1":"value1", "column2": "value2"}'
json = json.loads(string)
If you want to parse your string to a dictionary you should try ast:
import ast
string = u'{"column1":"value1", "column2": "value2"}'
ast.literal_eval(string)=>{'column1': 'value1', 'column2': 'value2'}

Updating Query In URL With Urlib In Python

I have a url that is being parsed out of an XML file.
product_url = urlparse(item.find('product_url').text)
When I use urlib to break the url up I get this,
ParseResult(scheme='http', netloc='example.com', path='/dynamic', params='', query='t=MD5-YOUR-OAUTH-TOKEN&p=11111111', fragment='')
I need to update the
MD5-YOUR-OAUTH-TOKEN part of the query with a MD5 Hashed Oauth Key.
Which I have in this tokenHashed = encryptMd5Hash(token)
My goal is to after it is parsed and the hash has been inserted to the string in place of the MD5-YOUR-OAUTH-TOKEN, to have the whole url in a string I can use somewhere else. Originally I was trying to use regex to do this but found urlib. I cannot find where it says to do something like this?
Am I right to be using urlib for this? How do I achieve my goal of updating the url with the hashed token and having the whole url stored in a string?
So the string should look like this,
newString = 'http://example.com/dynamic?t='+tokenHashed+'&p=11112311312'

You'll first want to use the parse_qs function to parse the query string into a dictionary:
>>> import urlparse
>>> import urllib
>>> url = 'http://example.com/dynamic?t=MD5-YOUR-OAUTH-TOKEN&p=11111111'
>>> parsed = urlparse.urlparse(url)
>>> parsed
ParseResult(scheme='http', netloc='example.com', path='/dynamic', params='', query='t=MD5-YOUR-OAUTH-TOKEN&p=11111111', fragment='')
>>> qs = urlparse.parse_qs(parsed.query)
>>> qs
{'p': ['11111111'], 't': ['MD5-YOUR-OAUTH-TOKEN']}
>>>
Now you can modify the dictionary as desired:
>>> qs['t'] = ['tokenHashed']
Note here that because the parse_qs returned lists for each query
parameter, we need replace them with lists because we'll be calling
urlencode next with doseq=1 to handle those lists.
Next, rebuild the query string:
>>> newqs = urllib.urlencode(qs, doseq=1)
>>> newqs
'p=11111111&t=tokenHashed'
And then reassemble the URL:
>>> newurl = urlparse.urlunparse(
... [newqs if i == 4 else x for i,x in enumerate(parsed)])
>>> newurl
'http://example.com/dynamic?p=11111111&t=tokenHashed'
That list comprehension there is just using all the values from
parsed except for item 4, which we are replacing with our new query
string.

How to convert an string to json in Python

I'm trying to convert an string into json output from local Data or Those datas from BeautifulSoup output as Json.for example:
#! /usr/bin/python
data = ('Hello')
print data
and i need to convert this Hello as json output.
How can do that?
is this possible?

Check out the json module in Python https://docs.python.org/2/library/json.html
import json
json.dumps({"hello": 0}, sort_keys=True)

You can use the json module to encode Python objects as JSON, e.g.
>>> import json
>>> data = ('Hello')
>>> json.dumps(data)
'"Hello"'
>>> data = ('Hello', 'There')
>>> json.dumps(data)
'["Hello", "There"]'
>>> data = {'message': 'Hello'}
>>> json.dumps(data)
'{"message": "Hello"}'

How to escape dict from str in python?

Come straight to the point.
A str object below:
s = '{"key1":"value1", "key2":"value2", "key3":"value3"}'
As you see, dict is wrapped in str. Now how to escape dict from str?
In other words, is it possible d = operate(s), d["key1"] = "value1", and etc?

>>> ast.literal_eval('{"key1":"value1", "key2":"value2", "key3":"value3"}')
{'key3': 'value3', 'key2': 'value2', 'key1': 'value1'}

i'd use json:
try:
import json
except ImportError:
import simplejson as json
d = json.loads(s)

You're looking for eval.