Develop Reference

How to properly encode in Python?

I am new to python and I am struggling with encoding
I have a list of String like this:
keys = ["u'part-00000-6edc0ee4-de74-4f82-9f8c-b4c965896224-c000.csv'",
" u'part-00001-6edc0ee4-de74-4f82-9f8c-b4c965896224-c000.csv'"]
I do this to encode
keys = [x.encode('UTF-8') for x in keys]
However I am getting "b" appended, the result being
[b"u'part-00000-6edc0ee4-de74-4f82-9f8c-b4c965896224-c000.csv'",
b" u'part-00001-6edc0ee4-de74-4f82-9f8c-b4c965896224-c000.csv'"]
I thought it would be simpler to just encode with utf-8
What am I doing wrong?

You should first try fixing the method you use to obtain your original list of strings, but if you have no control on that, you can use the following:
>>> import ast
>>> [ast.literal_eval(i.strip()) for i in keys]
The result should be
[u'part-00000-6edc0ee4-de74-4f82-9f8c-b4c965896224-c000.csv',
u'part-00001-6edc0ee4-de74-4f82-9f8c-b4c965896224-c000.csv']
for Python 2, and
['part-00000-6edc0ee4-de74-4f82-9f8c-b4c965896224-c000.csv',
'part-00001-6edc0ee4-de74-4f82-9f8c-b4c965896224-c000.csv']
for Python 3.

Urlencode dictionary using Python - naming key and value in the url

I am attempting to generate a URL link in the following format using urllib and urlencode.
<img src=page.psp?KEY=%28SpecA%2CSpecB%29&VALUE=1&KEY=%28SpecA%2C%28SpecB%2CSpecC%29%29&VALUE=2>
I'm trying to use data from my dictionary to input into the urllib.urlencode() function however, I need to get it into a format where the keys and values have a variable name, like below. So the keys from my dictionary will = NODE and values will = VALUE.
wanted = urllib.urlencode( [("KEY",v1),("VALUE",v2)] )
req.write( "<a href=page.psp?%s>" % (s) );
The problem I am having is that I want the URL as above and instead I am getting what is below, rather than KEY=(SpecA,SpecB) NODE=1, KEY=(SpecA,SpecB,SpecC) NODE=2 which is what I want.
KEY=%28SpecA%2CSpecB%29%2C%28%28SpecA%2CSpecB%29%2CSpecC%29&VALUE=1%2C2
So far I have extracted keys and values from the dictionary, extracted into tuples, lists, strings and also tried dict.items() but it hasn't helped much as I still can't get it to go into the format I want. Also I am doing this using Python server pages which is why I keep having to print things as a string due to constant string errors. This is part of what I have so far:
k = (str(dict))
ver1 = dict.keys()
ver2 = dict.values()
new = urllib.urlencode(function)
f = urllib.urlopen("page.psp?%s" % new)
I am wondering what I need to change in terms of extracting values from the dictionary/converting them to different formats in order to get the output I want? Any help would be appreciated and I can add more of my code (as messy as it has become) if need be. Thanks.

This should give you the format you want:
data = {
'(SpecA,SpecB)': 1,
'(SpecA,SpecB,SpecC)': 2,
}
params = []
for k,v in data.iteritems():
params.append(('KEY', k))
params.append(('VALUE', v))
new = urllib.urlencode(params)
Note that the KEY/VALUE pairings may not be the order you want, given that dicts are unordered.

Parse json data

My data.json is
{"a":[{"b":{"c":{ "foo1":1, "foo2":2, "foo3":3, "foo4":4}}}],"d":[{"e":{"bar1":1, "bar2":2, "bar3":3, "bar4":4}}]}
I am able to list both key/pair values. My code is:
#! /usr/bin/python
import json
from pprint import pprint
with open('data2.json') as data_file:
data = json.load(data_file)
pprint(data["d"][0]["e"])
Which gives me:
{u'bar1': 1, u'bar2': 2, u'bar3': 3, u'bar4': 4}
But I want to display only the keys without any quotes and u like this:
bar1, bar2, bar3, bar4
Can anybody suggest anything? It need not be only in python, can be in shell script also.

The keys of this object are instances of the unicode string class. Given this, the default printing behavior of the dict instance for which they are the keys will print them as you show in your post.
This is because the dict implementation of representing its contents as a string (__repr__ and/or __str__) seeks to show you what objects reside in the dict, not what the string representation of those objects looks like. This is an important distinction, for example:
In [86]: print u'hi'
hi
In [87]: x = u'hi'
In [88]: x
Out[88]: u'hi'
In [89]: print x
hi
This should work for you, assuming that printing the keys together as a comma-separated unicode is fine:
print ", ".join(data["d"][0]["e"])
You can achieve this using the keys member function from dict too, but it's not strictly necessary.

print ', '.join((data["d"][0]["e"].keys()))

data["d"][0]["e"] returns a dict. In python2, You could use this to get the keys of that dict with something like this:
k = data["d"][0]["e"].keys()
print(", ".join(k))
In python3, wrap k in a list like this
k = list(data["d"][0]["e"].keys())
print(", ".join(k))
Even simpler, join will iterate over the keys of the dict.
print(", ".join(data["d"][0]["e"]))
Thanks to #thefourtheye for pointing this out.

Sending an array with POST request to Django [duplicate]

I'm trying to urlencode an dictionary in python with urllib.urlencode. The problem is, I have to encode an array.
The result needs to be:
criterias%5B%5D=member&criterias%5B%5D=issue
#unquoted: criterias[]=member&criterias[]=issue
But the result I get is:
criterias=%5B%27member%27%2C+%27issue%27%5D
#unquoted: criterias=['member',+'issue']
I have tried several things, but I can't seem to get the right result.
import urllib
criterias = ['member', 'issue']
params = {
'criterias[]': criterias,
}
print urllib.urlencode(params)
If I use cgi.parse_qs to decode a correct query string, I get this as result:
{'criterias[]': ['member', 'issue']}
But if I encode that result, I get a wrong result back. Is there a way to produce the expected result?

The solution is far simpler than the ones listed above.
>>> import urllib
>>> params = {'criterias[]': ['member', 'issue']}
>>>
>>> print urllib.urlencode(params, True)
criterias%5B%5D=member&criterias%5B%5D=issue
Note the True. See http://docs.python.org/library/urllib.html#urllib.urlencode the doseq variable.
As a side note, you do not need the [] for it to work as an array (which is why urllib does not include it). This means that you do not not need to add the [] to all your array keys.

You can use a list of key-value pairs (tuples):
>>> urllib.urlencode([('criterias[]', 'member'), ('criterias[]', 'issue')])
'criterias%5B%5D=member&criterias%5B%5D=issue'

To abstract this out to work for any parameter dictionary and convert it into a list of tuples:
import urllib
def url_encode_params(params={}):
if not isinstance(params, dict):
raise Exception("You must pass in a dictionary!")
params_list = []
for k,v in params.items():
if isinstance(v, list): params_list.extend([(k, x) for x in v])
else: params_list.append((k, v))
return urllib.urlencode(params_list)
Which should now work for both the above example as well as a dictionary with some strings and some arrays as values:
criterias = ['member', 'issue']
params = {
'criterias[]': criterias,
}
url_encode_params(params)
>>'criterias%5B%5D=member&criterias%5B%5D=issue'

Listcomp of values:
>>> criterias = ['member', 'issue']
>>> urllib.urlencode([('criterias[]', i) for i in criterias])
'criterias%5B%5D=member&criterias%5B%5D=issue'
>>>

as aws api defines its get url: params.0=foo&params.1=bar
however, the disadvantage is that you need to write code to encode and decode by your own, the result is: params=[foo, bar]

How can i convert the dictionary items into flat strings in python

I have the dictionary of items from which i am generating the URLs like this
request.build_absolute_uri("/myurl/" + urlencode(myparams))
The output i am getting is like this
number=['543543']&region=['5,36,37']
but i want the url to be
number=543543&region=5,36,37
all those items are in myparams dictionary

You'll probably find for ease of use that passing doseq=True will be useful (albeit not exactly what you want - but does mean that any url parsing library should be able to handle the input without custom coding...)
>>> from urllib import urlencode
>>> a = range(3)
>>> urlencode({'test': a})
'test=%5B0%2C+1%2C+2%5D'
>>> urlencode({'test': a}, True)
'test=0&test=1&test=2'
Otherwise, you'll have to write custom code to ','.join(str(el) for el in your_list) for values in myparams where it's a list/similar... (then .split(',') the other end)

looks like myparams is a dict that has lists as values.
new_params = dict(k, v[0] for k, v in dict.iteritems())
will construct a new dict.