Develop Reference

Python request containing parameters that have values and parameters that do not have values

I am writing a Python wrapper for an API that supports the query parameters that have values (e.g. param1 below) and query parameters that do not have values value (e.g. param2 below) i.e.
https://example.com/service?param1=value1&param2
The HTTP requests library famously supports parameters with values and parameters without values. However I need to be able to specify both types of parameters.

You have two options with Requests in terms of the query string: 1) provide key/value pairs by way of a dictionary, or 2) provide a string. If you provide #1, you'll always get a '=' for each key/value pair...not what you want. So you have to use #2, which will let you do whatever you want, since it will just include what you give it as the entire query string. The downside is that you have to construct the query string yourself. There are many ways of doing that, of course. Here's one way:
params = {'param1': 'value1', 'param2': None}
params = '&'.join([k if v is None else f"{k}={v}" for k, v in params.items()])
r = requests.get('https://example.com/service', params=params)
print(r.url)
This way lets you supply a dictionary, just like if you were letting Requests build the query string, but it allows you to specify a value of Null to indicate that you want just the key name, with no '='. Requests will normally not include the parameter at all if its value is None in the dictionary.
The result of this code is exactly what you gave as an example of what you want:
https://example.com/service?param1=value1&param2

Different behavior of json.dump() on Windows and Linux

I wrote a python script to retrieve data from a website in json format using the requests library, and then I dump it into a json file. I have written a lot of code utilizing this data and have tested it in Windows only. Recently I shifted to a Linux system, and when the same python script is executed, the order of the keys in the json file is completely different.
This is the code I'm using:
API_request = requests.get('https://www.abcd.com/datarequest')
alertJson_Data = API_request.json() # To convert returned data to json
json.dump(alertJson_Data, jsonDataFile) # for adding the json data for the alert to the file
jsonDataFile.write('\n')
jsonDataFile.close()
A lot of my other scripts depends on the ordering of the keys in this json file, so is there any way to maintain the same ordering that is used in Windows to be used in Linux as well?
For example in Windows the order is "id":, "src":, "dest":, whereas in Linux its completely different. If I directly go to the Web link on my browser, it has the same ordering as the one saved in Windows. How do I retain this ordering?

Can you use collections.OrderedDict when loading json?
e.g
from collections import OrderedDict
alertJson_Data = API_request.json(object_pairs_hook=OrderedDict)
should works, because json() method implemented on requests take the same optional arguments as json.loads
json(**kwargs)
Returns the json-encoded content of a response, if any.
Parameters **kwargs – Optional arguments that json.loads takes. Raises
ValueError – If the response body does not contain valid json.
And the doc of json.loads specify:
object_hook, if specified, will be called with the result of every
JSON object decoded and its return value will be used in place of the
given dict. This can be used to provide custom deserializations (e.g.
to support JSON-RPC class hinting).
object_pairs_hook, if specified will be called with the result of
every JSON object decoded with an ordered list of pairs. The return
value of object_pairs_hook will be used instead of the dict. This
feature can be used to implement custom decoders that rely on the
order that the key and value pairs are decoded (for example,
collections.OrderedDict() will remember the order of insertion). If
object_hook is also defined, the object_pairs_hook takes priority.

Using A Python List or String in Dictionary Lookup?

Use Case
I am making a factory type script in Python that consumes XML and based on that XML, returns information from a specific factory. I have created a file that I call FactoryMap.json that stores the mapping between the location an item can be found in XML and the appropriate factory.
Issue
The JSON in my mapping file looks like:
{
"path": "['project']['builders']['hudson.tasks.Shell']",
"class": "bin.classes.factories.step.ShellStep"
}
path is where the element can be found in the xml once its converted to a dict.
class is the corresponding path to the factory that can consume that elements information.
In order to do anything with this, I need to descend into the dictionaries structure, which would look like this if I didn't have to draw this information from a file(note the key reference = 'path' from my json'):
configDict={my xml config dict}
for k,v in configDict['project']['builders']['hudson.tasks.Shell'].iteritems():
#call the appropriate factory
The issue is that if I look up the path value as a string or a list, I can not use it in 'iteritems'():
path="['project']['builders']['hudson.tasks.Shell']" #this is taken from the JSON
for k,v in configDict[path].iteritems():
#call the appropriate factory
This returns a key error stating that I can't use a string as the key value. How can I used a variable as the key for that python dictionary?

You could use eval:
eval( "configDict"+path )

You can use the eval() function to evaluate your path into an actual dict object vs a string. Something like this is what I'm referring to:
path="['project']['builders']['hudson.tasks.Shell']" #this is taken from the JSON
d = eval("configDict%s" % path)
for k,v in d.iteritems():
#call the appropriate factory

Django interpreting dict values ambiguously [duplicate]

In a Django view you can access the request.GET['variablename'], so in your view you can do something like this:
myvar = request.GET['myvar']
The actual request.GET['myvar'] object type is:
<class 'django.http.QueryDict'>
Now, if you want to pass multiple variables with the same parameter name, i.e:
http://example.com/blah/?myvar=123&myvar=567
You would like a python list returned for the parameter myvar, then do something like this:
for var in request.GET['myvar']:
print(var)
However, when you try that you only get the last value passed in the url i.e in the example above you will get 567, and the result in the shell will be:
5
6
7
However, when you do a print of request.GET it seems like it has a list i.e:
<QueryDict: {u'myvar': [u'123', u'567']}>
Ok Update:
It's designed to return the last value, my use case is i need a list.
from django docs:
QueryDict.getitem(key)
Returns
the value for the given key. If the
key has more than one value,
getitem() returns the last value. Raises
django.utils.datastructures.MultiValueDictKeyError
if the key does not exist. (This is a
subclass of Python's standard
KeyError, so you can stick to catching
KeyError
QueryDict.getlist(key) Returns the
data with the requested key, as a
Python list. Returns an empty list if
the key doesn't exist. It's guaranteed
to return a list of some sort.
Update:
If anyone knows why django dev's have done this please let me know, seems counter-intuitive to show a list and it does not behave like one. Not very pythonic!

You want the getlist() function of the GET object:
request.GET.getlist('myvar')

Another solution is creating a copy of the request object... Normally, you can not iterate through a request.GET or request.POST object, but you can do such operations on the copy:
res_set = request.GET.copy()
for item in res_set['myvar']:
item
...

When creating a query string from a QueryDict object that contains multiple values for the same parameter (such as a set of checkboxes) use the urlencode() method:
For example, I needed to obtain the incoming query request, remove a parameter and return the updated query string to the resulting page.
# Obtain a mutable copy of the original string
original_query = request.GET.copy()
# remove an undesired parameter
if 'page' in original_query:
del original_query['page']
Now if the original query has multiple values for the same parameter like this:
{...'track_id': ['1', '2'],...} you will lose the first element in the query string when using code like:
new_query = urllib.parse.urlencode(original_query)
results in...
...&track_id=2&...
However, one can use the urlencode method of the QueryDict class in order to properly include multiple values:
new_query = original_query.urlencode()
which produces...
...&track_id=1&track_id=2&...

How to deal with params in URL schema in Django application

As we know most URL schemes base their URL syntax on this nine-part general format:
<scheme>://<user>:<password>#<host>:<port>/<path>;<params>?<query>#<frag>
I know I can get the query strings in Django request.GET dict. But how to get parameters? It seems HttpRequest in Django can not deal with params defined in URL.
For example: http://www.example.com/hello;param1=aaaa;param2=bbbb
Is there any way to extract params like this in Django without writing own regex in url pattern.
Any help will be appreciated.

The path segment parameters you're talking about were defined in RFC 2396. This RFC has been obsolete since 2005. The new standard, RFC 3986, does not define a specific way to define path segment parameters. The relevant section states:
Aside from dot-segments in hierarchical paths, a path segment is
considered opaque by the generic syntax. [...] For
example, the semicolon (";") and equals ("=") reserved characters are
often used to delimit parameters and parameter values applicable to
that segment. [...] Parameter types may be defined by scheme-specific
semantics, but in most cases the syntax of a parameter is specific to
the implementation of the URI's dereferencing algorithm.
Django uses the new standard and does not define any specific way to parse path segment parameters. However, Python defines the urlparse() function, which parses the url according to RFC2396. You can use this to extract the path parameters:
from urllib.parse import urlparse # urlparse.urlparse on Python 2
params = urlparse(request.path).params

to capture params you've to define url patterns as per that.
lets assume your url is http://myurl.com/mydashboard/250?sort=-1&q=this so here 250 is param which you want.
To capture the same you've to define url like below
url(r'^mydashboard/(?P<weeknum>\d+)/$',
mydashboard, name='mydashboard'),
and now the view definition of mydashboard will have another input param like below
def nagdashboard(request, weeknum):
And to get GET params or POST params you can use.
request.GET.get('sort')
For further reading on patterns refer to here