Develop Reference

API gives only the headers in Python but not the data

I am trying to access an API from this website. (https://www.eia.gov/opendata/qb.php?category=717234)
I am able to call the API but I am getting only headers. Not sure if I am doing correctly or any additions are needed.
Code:
import urllib
import requests
import urllib.request
locu_api = 'WebAPI'
def locu_search(query):
api_key = locu_api
url = 'https://api.eia.gov/category?api_key=' + api_key
locality = query.replace(' ', '%20')
response = urllib.request.urlopen(url).read()
json_obj = str(response, 'utf-8')
data = json.loads(json_obj)
When I try to print the results to see whats there in data:
data
I am getting only the headers in JSON output. Can any one help me figure out how to do extract the data instead of headers.

Avi!
Look, the data you posted seems to be an application/json response. I tried to reorganize your snippet a little bit so you could reuse it for other purposes later.
import requests
API_KEY = "insert_it_here"
def get_categories_data(api_key, category_id):
"""
Makes a request to gov API and returns its JSON response
as a python dict.
"""
host = "https://api.eia.gov/"
endpoint = "category"
url = f"{host}/{endpoint}"
qry_string_params = {"api_key": api_key, "category_id": category_id}
response = requests.post(url, params=qry_string_params)
return response.json()
print(get_categories_data(api_key=API_KEY, category_id="717234"))
As far as I can tell, the response contains some categories and their names. If that's not what you were expecting, maybe there's another endpoint that you should look for. I'm sure this snippet can help you if that's the case.
Side note: isn't your API key supposed to be private? Not sure if you should share that.
Update:
Thanks to Brad Solomon, I've changed the snippet to pass query string arguments to the requests.post function by using the params parameter which will take care of the URL encoding, if necessary.

You haven't presented all of the data. But what I see here is first a dict that associates category_id (a number) with a variable name. For example category_id 717252 is associated with variable name 'Import quantity'. Next I see a dict that associates category_id with a description, but you haven't presented the whole of that dict so 717252 does not appear. And after that I would expect to see a third dict, here entirely missing, associating a category_id with a value, something like {'category_id': 717252, 'value': 123.456}.
I think you are just unaccustomed to the way some APIs aggressively decompose their data into key/value pairs. Look more closely at the data. Can't help any further without being able to see the data for myself.

Request.args as filters from several path levels

I would know if it's possible to get request.args from different path levels.
Currently i can use request.args for only one level :
my_ip:my_port/modules/?name=git
or
my_ip:my_port/modules/?id=0
 
#module_blueprint.route('/', methods=['GET'])
def get_modules():
if len(request.args) == 0:
modules = Module.query.all()
return jsonify(modules=list_to_json(modules))
try:
module = Module.query.filter_by(**request.args.to_dict()).first()
except InvalidRequestError as i:
return json_response(400, "InvalidRequestError : {}".format(i.args))
if module is None:
return json_response(204, "")
return jsonify(module=module.serialize)
But in my Module model I have actions and I would like to get actions in the same way :
my_ip:my_port/modules/?name=git/actions/?id=0
But I don't know how to do this.
If I try with
#module_blueprint.route('/actions', methods=['GET'])
request.args just keep args after the action path
my_ip:my_port/modules/?name=git/actions
request.args is empty for this path
Thanks for helping.

Note: Request params are always appended at the end of the request url. Everything after a ? is considered URI Params. Thus, a url like
my_ip:my_port/modules/?name=git/actions/?id=0
is not possible since it violates the above mentioned guideline.
Solution if there is going to be only 1 param per level:
In this case, passing the info as params does not provide any advantage. Thus, the following would be a better way of doing things:
For fetching module by name:
#module_blueprint.route('/<module_name>', methods=['GET'])
For fetching actions for that module:
#module_blueprint.route('/<module_name>/action/<int:action_id>', methods=['GET'])
Solution if there are going to be more than 1 filter param per level:
In this case, it makes more sense to use request params.
For fetching module by name:
#module_blueprint.route('/', methods=['GET'])
For fetching actions for that module:
#module_blueprint.route('/actions', methods=['GET'])

What is the best way to force a keyword while using **kwargs?

I'm not sure if I have used the correct terminology in the question.
Currently, I am trying to make a wrapper/interface around Google's Blogger API (Blog service).
[I know it has been done already, but I am using this as a project to learn OOP/python.]
I have made a method that gets a set of 25 posts from a blog:
def get_posts(self, **kwargs):
""" Makes an API request. Returns list of posts. """
api_url = '/blogs/{id}/posts'.format(id=self.id)
return self._send_request(api_url, kwargs)
def _send_request(self, url, parameters={}):
""" Sends an HTTP GET request to the Blogger API.
Returns JSON decoded response as a dict. """
url = '{base}{url}?'.format(base=self.base, url=url)
# Requests formats the parameters into the URL for me
try:
r = requests.get(url, params=parameters)
except:
print "** Could not reach url:\n", url
return
api_response = r.text
return self._jload(api_response)
The problem is, I have to specify the API key every time I call the get_posts function:
someblog = BloggerClient(url='http://someblog.blogger.com', key='0123')
someblog.get_posts(key=self.key)
Every API call requires that the key be sent as a parameter on the URL.
Then, what is the best way to do that?
I'm thinking a possible way (but probably not the best way?), is to add the key to the kwargs dictionary in the _send_request():
def _send_request(self, url, parameters={}):
""" Sends an HTTP get request to Blogger API.
Returns JSON decoded response. """
# Format the full API URL:
url = '{base}{url}?'.format(base=self.base, url=url)
# The api key will be always be added:
parameters['key']= self.key
try:
r = requests.get(url, params=parameters)
except:
print "** Could not reach url:\n", url
return
api_response = r.text
return self._jload(api_response)
I can't really get my head around what is the best way (or most pythonic way) to do it.

You could store it in a named constant.
If this code doesn't need to be secure, simply
API_KEY = '1ih3f2ihf2f'
If it's going to be hosted on a server somewhere or needs to be more secure, you could store the value in an environment variable
In your terminal:
export API_KEY='1ih3f2ihf2f'
then in your python script:
import os
API_KEY = os.environ.get('API_KEY')

The problem is, I have to specify the API key every time I call the get_posts function:
If it really is just this one method, the obvious idea is to write a wrapper:
def get_posts(blog, *args, **kwargs):
returns blog.get_posts(*args, key=key, **kwargs)
Or, better, wrap up the class to do it for you:
class KeyRememberingBloggerClient(BloggerClient):
def __init__(self, *args, **kwargs):
self.key = kwargs.pop('key')
super(KeyRememberingBloggerClient, self).__init__(*args, **kwargs)
def get_posts(self, *args, **kwargs):
return super(KeyRememberingBloggerClient, self).get_posts(
*args, key=self.key, **kwargs)
So now:
someblog = KeyRememberingBloggerClient(url='http://someblog.blogger.com', key='0123')
someblog.get_posts()
Yes, you can override or monkeypatch the _send_request method that all of the other methods use, but if there's only 1 or 2 methods that need to be fixed, why delve into the undocumented internals of the class, and fork the body of one of those methods just so you can change it in a way you clearly weren't expected to, instead of doing it cleanly?
Of course if there are 90 different methods scattered across 4 different classes, you might want to consider building these wrappers programmatically (and/or monkeypatching the classes)… or just patching the one private method, as you're doing. That seems reasonable.

How to POST multiple FILES using Flask test client?

In order to test a Flask application, I got a flask test client POSTing request with files as attachment
def make_tst_client_service_call1(service_path, method, **kwargs):
_content_type = kwargs.get('content-type','multipart/form-data')
with app.test_client() as client:
return client.open(service_path, method=method,
content_type=_content_type, buffered=True,
follow_redirects=True,**kwargs)
def _publish_a_model(model_name, pom_env):
service_url = u'/publish/'
scc.data['modelname'] = model_name
scc.data['username'] = "BDD Script"
scc.data['instance'] = "BDD Stub Simulation"
scc.data['timestamp'] = datetime.now().strftime('%d-%m-%YT%H:%M')
scc.data['file'] = (open(file_path, 'rb'),file_name)
scc.response = make_tst_client_service_call1(service_url, method, data=scc.data)
Flask Server end point code which handles the above POST request is something like this
#app.route("/publish/", methods=['GET', 'POST'])
def publish():
if request.method == 'POST':
LOG.debug("Publish POST Service is called...")
upload_files = request.files.getlist("file[]")
print "Files :\n",request.files
print "Upload Files:\n",upload_files
return render_response_template()
I get this Output
Files:
ImmutableMultiDict([('file', <FileStorage: u'Single_XML.xml' ('application/xml')>)])
Upload Files:
[]
If I change
scc.data['file'] = (open(file_path, 'rb'),file_name)
into (thinking that it would handle multiple files)
scc.data['file'] = [(open(file_path, 'rb'),file_name),(open(file_path, 'rb'),file_name1)]
I still get similar Output:
Files:
ImmutableMultiDict([('file', <FileStorage: u'Single_XML.xml' ('application/xml')>), ('file', <FileStorage: u'Second_XML.xml' ('application/xml')>)])
Upload Files:
[]
Question:
Why request.files.getlist("file[]") is returning an empty list?
How can I post multiple files using flask test client, so that it can be retrieved using request.files.getlist("file[]") at flask server side ?
Note:
I would like to have flask client I dont want curl or any other client based solutions.
I dont want to post single file in multiple requests
Thanks
Referred these links already:
Flask and Werkzeug: Testing a post request with custom headers
Python - What type is flask.request.files.stream supposed to be?

You send the files as the parameter named file, so you can't look them up with the name file[]. If you want to get all the files named file as a list, you should use this:
upload_files = request.files.getlist("file")
On the other hand, if you really want to read them from file[], then you need to send them like that:
scc.data['file[]'] = # ...
(The file[] syntax is from PHP and it's used only on the client side. When you send the parameters named like that to the server, you still access them using $_FILES['file'].)

Lukas already addressed this,just providing these info as it may help someone
Werkzeug client is doing some clever stuff by storing requests data in MultiDict
#native_itermethods(['keys', 'values', 'items', 'lists', 'listvalues'])
class MultiDict(TypeConversionDict):
"""A :class:`MultiDict` is a dictionary subclass customized to deal with
multiple values for the same key which is for example used by the parsing
functions in the wrappers. This is necessary because some HTML form
elements pass multiple values for the same key.
:class:`MultiDict` implements all standard dictionary methods.
Internally, it saves all values for a key as a list, but the standard dict
access methods will only return the first value for a key. If you want to
gain access to the other values, too, you have to use the `list` methods as
explained below.
getList call looks for a given key in the "requests" dictionary. If the key doesn't exist, it returns empty list.
def getlist(self, key, type=None):
"""Return the list of items for a given key. If that key is not in the
`MultiDict`, the return value will be an empty list. Just as `get`
`getlist` accepts a `type` parameter. All items will be converted
with the callable defined there.
:param key: The key to be looked up.
:param type: A callable that is used to cast the value in the
:class:`MultiDict`. If a :exc:`ValueError` is raised
by this callable the value will be removed from the list.
:return: a :class:`list` of all the values for the key.
"""
try:
rv = dict.__getitem__(self, key)
except KeyError:
return []
if type is None:
return list(rv)
result = []
for item in rv:
try:
result.append(type(item))
except ValueError:
pass
return result

Django Middleware - How to edit the HTML of a Django Response object?

I'm creating a custom middleware to django edit response object to act as a censor. I would like to find a way to do a kind of search and replace, replacing all instances of some word with one that I choose.
I've created my middleware object, added it to my MIDDLEWARE_CLASSES in settings and have it set up to process the response. But so far, I've only found methods to add/edit cookies, set/delete dictionary items, or write to the end of the html:
class CensorWare(object):
def process_response(self, request, response):
"""
Directly edit response object here, searching for and replacing terms
in the html.
"""
return response
Thanks in advance.

You can simply modify the response.content string:
response.content = response.content.replace("BAD", "GOOD")

Perhaps my reply little better. When you try to make response.content.replace("BAD", "GOOD"), you will get error, that you cannot do it with strings, because response.content is byte array. I've added syntactic strings 'gen_duration_time_777' and 'server_time_777' to base template. And this works for me.
import time
from datetime import datetime
class StatsMiddleware(object):
duration = 0
def process_request(self, request):
# Store the start time when the request comes in.
request.start_time = time.time()
def process_response(self, request, response):
# Calculate and output the page generation duration
# Get the start time from the request and calculate how long
# the response took.
self.duration = time.time() - request.start_time
response["x-server-time"] = datetime.now().strftime("%d/%m/%Y %H:%M")
response.content = response.content.replace(b"server_time_777", str.encode(response["x-server-time"]))
response["x-page-generation-duration-ms"] = '{:.3f}'.format(self.duration)
response.content = response.content.replace(b"gen_duration_time_777", str.encode(response["x-page-generation-duration-ms"]))
return response