I am trying to remove certain items from a query string, the best way doing this would be to parse the query string, iterate over and remove the particular key I dont want and join it all back together.
Following the python guide, it seems the urlencode function they say to use, doesn't work as one would expect.
Fee the following code, which simply parses the query string, and then joins it back together. I've set it to keep the empty value.
>>> f = 'name=John%20Doe&seq=123412412412&wer'
>>> q = urlparse.parse_qs(f, keep_blank_values=True)
>>> q
{'wer': [''], 'name': ['John Doe'], 'seq': ['123412412412']}
>>> urllib.urlencode(q)
'wer=%5B%27%27%5D&name=%5B%27John+Doe%27%5D&seq=%5B%27123412412412%27%5D'
I am expecting the result of the query code, to be the same as the f string.
http://docs.python.org/2/library/urlparse.html#urlparse.parse_qs
Use the urllib.urlencode() function to convert such dictionaries into query strings.
So I assume I have to loop over the q variable and build the string manually, calling urlencode on each item of the dictionary? Isn't there a better way...
Using python 2.7
Thanks
You should use the doseq argument to urlencode since you have sequences in your query dict:
>>> urllib.urlencode(q, doseq=1)
'wer=&name=John+Doe&seq=123412412412'
Related
I have a Python dictionary
original_dict={'body': '{"infra":["dev4","elk"],{"type":"file_integrity"}'}
I want to be able to parse original_dict keys and values as a normal dictionary which I am not able to do now because 'body' key has a a dictionary casted as string and therefore I am not refer to any of it's keys. So I should be able to say:
infra=original_dict['body]['infra']
Can anyone help me out with this.
First of all, you are missing a curly brace in the original_dict.
Here is an example of converting a string into a dictionary.
import json
original_dict={'body':'{"infra":["dev4","elk"],"type":"file_integrity"}'}
original_dict['body'] = json.loads(original_dict['body'])
infra=original_dict['body']['infra']
print(infra)
Output : ['dev4', 'elk']
You can use ast too:)
import ast
original_dict = {'body': '{"infra":["dev4","elk"],"type":"file_integrity"}'}
original_dict['body'] = ast.literal_eval(original_dict['body'])
Scrapy noob here. I am extracting an href 'rel'attribute which looks like the following:
rel=""prodimage":"image_link","intermediatezoomimage":"image_link","fullimage":"image_link""
This can be seen as a dict like structure within the attribute.
My main goal is to obtain the image url against 'fullimage'. Hence, I want to store the response as a python dictionary.
However, Xpath returns a unicode "list" ( Not just a string but a list!) with one item ( the whole rel contents as one item)
res = response.xpath('//*[#id="detail_product"]/div[1]/div[2]/ul/li[1]/a/#rel').extract()
print res
[u'"prodimage":"image_link", "intermediatezoomimage":"image_link", "fullimage":"image_link"']
type(res)
type 'list'
How do I convert the content of 'res' into something like a python dictionary ( with separated out items as list items, not just one whole item) so that I can grab individual components from the structure within 'rel'.
I hope I am clear. Thank you!
SOLVED
The XPATH response above is basically a list with ONE item in unicode.
Convert the respective items into strings ( using x.encode('ascii') )
and then form a string representation of a dict. In my case I had to append and prepend the string (the rel contents) with curly braces. Thats all!
Then convert that string representation of a dict into an actual dict using the method mentioned in the link below.
Convert a String representation of a Dictionary to a dictionary?
I have a problem that I would like to know how to efficiently tackle.
I have data that is JSON-formatted (used with dumps / loads) and contains unicode.
This is part of a protocol implemented with JSON to send messages. So messages will be sent as strings and then loaded into python dictionaries. This means that the representation, as a python dictionary, afterwards will look something like:
{u"mykey": u"myVal"}
It is no problem in itself for the system to handle such structures, but the thing happens when I'm going to make a database query to store this structure.
I'm using pyOrient towards OrientDB. The command ends up something like:
"CREATE VERTEX TestVertex SET data = {u'mykey': u'myVal'}"
Which will end up in the data field getting the following values in OrientDB:
{'_NOT_PARSED_': '_NOT_PARSED_'}
I'm assuming this problem relates to other cases as well when you wish to make a query or somehow represent a data object containing unicode.
How could I efficiently get a representation of this data, of arbitrary depth, to be able to use it in a query?
To clarify even more, this is the string the db expects:
"CREATE VERTEX TestVertex SET data = {'mykey': 'myVal'}"
If I'm simply stating the wrong problem/question and should handle it some other way, I'm very much open to suggestions. But what I want to achieve is to have an efficient way to use python2.7 to build a db-query towards orientdb (using pyorient) that specifies an arbitrary data structure. The data property being set is of the OrientDB type EMBEDDEDMAP.
Any help greatly appreciated.
EDIT1:
More explicitly stating that the first code block shows the object as a dict AFTER being dumped / loaded with json to avoid confusion.
Dargolith:
ok based on your last response it seems you are simply looking for code that will dump python expression in a way that you can control how unicode and other data types print. Here is a very simply function that provides this control. There are ways to make this function more efficient (for example, by using a string buffer rather than doing all of the recursive string concatenation happening here). Still this is a very simple function, and as it stands its execution is probably still dominated by your DB lookup.
As you can see in each of the 'if' statements, you have full control of how each data type prints.
def expr_to_str(thing):
if hasattr(thing, 'keys'):
pairs = ['%s:%s' % (expr_to_str(k),expr_to_str(v)) for k,v in thing.iteritems()]
return '{%s}' % ', '.join(pairs)
if hasattr(thing, '__setslice__'):
parts = [expr_to_str(ele) for ele in thing]
return '[%s]' % (', '.join(parts),)
if isinstance(thing, basestring):
return "'%s'" % (str(thing),)
return str(thing)
print "dumped: %s" % expr_to_str({'one': 33, 'two': [u'unicode', 'just a str', 44.44, {'hash': 'here'}]})
outputs:
dumped: {'two':['unicode', 'just a str', 44.44, {'hash':'here'}], 'one':33}
I went on to use json.dumps() as sobolevn suggested in the comment. I didn't think of that one at first since I wasn't really using json in the driver. It turned out however that json.dumps() provided exactly the formats I needed on all the data types I use. Some examples:
>>> json.dumps('test')
'"test"'
>>> json.dumps(['test1', 'test2'])
'["test1", "test2"]'
>>> json.dumps([u'test1', u'test2'])
'["test1", "test2"]'
>>> json.dumps({u'key1': u'val1', u'key2': [u'val21', 'val22', 1]})
'{"key2": ["val21", "val22", 1], "key1": "val1"}'
If you need to take more control of the format, quotes or other things regarding this conversion, see the reply by Dan Oblinger.
I want to know if it's possible to get values from this query string?
'?agencyID=1&agencyID=2&agencyID=3'
Why I have to use a query string like this?
I have a form with 10 check boxes. my user should send me ID of news agencies which he/she is interested in. so to query string contains of multiple values with the same name. total count of news agencies are variable and they are loaded from database.
I'm using Python Tornado to parse query strings.
Reading the tornado docs, this seems to do what you want
RequestHandler.get_arguments(name, strip=True)
Returns a list
of the arguments with the given name.
If the argument is not present, returns an empty list.
The returned values are always unicode.
So like this
ids = self.get_arguments('agencyID')
Note that here i used get_arguments, there is also a get_argument but that gets only a single argument.
You can get the whole query with
query = self.request.query
>>> from urlparse import urlparse, parse_qs
>>> url = '?agencyID=1&agencyID=2&agencyID=3'
>>> parse_qs(urlparse(url).query)
{'agencyID': ['1', '2', '3']}
tornado.web.RequestHandler has two methods to get arguments:
get_argument(id), which will get the last argument with name 'id'.
get_arguments(id), which will get all arguments with name 'id' into a list even if there is only one.
So, maybe get_arguments is what you need.
I'm finding this to be a difficult question to put into words, hence the examples. However, I'm basically being given arbitrary format strings, and I need to go (efficiently) fetch the appropriate values from a database, in order to build a relevant mapping object dynamically.
Given a format string expecting a mapping object, e.g.:
>>> 'Hello, %(first-name)s!' % {'first-name': 'Dolph'}
'Hello, Dolph!'
I'm looking for an implementation of 'infer_field_names()' below:
>>> infer_field_names('Hello, %(first-name)s! You are #%(customer-number)d.')
['first-name', 'customer-number']
I know I could write regex (or even try to parse exception messages!), but I'm hoping there's an existing API call I can use instead..?
Based on the string Formatter docs, I thought this would work:
>>> import string
>>> format_string = 'Hello, %(first-name)s! You are #%(customer-number)d.'
>>> [x[1] for x in string.Formatter().parse(format_string)]
[None]
But that doesn't quite return what I would expect (a list of field_names, per the docs).
When using the % operator to format strings, the right operand doesn't have to be a dictionary -- it only has to be some object mapping the required field names to the values that are supposed to be substistuted. So all you need to do is write a class with an redefined __getitem__() that retrieves the values from the database.
Here's a pointless example:
class Mapper(object):
def __getitem__(self, item):
return item
print 'Hello, %(first-name)s!' % Mapper()
prints
Hello, first-name!