Develop Reference

How to post request to API using only code?

I am developing a DAG to be scheduled on Apache Airflow which main porpuse will be to post survey data (on json format) to an API and then getting a response (the answers to the surveys). Since this whole process is going to be automated, every part of it has to be programmed in the DAG, so I can´t use Postman or any similar app (unless there is a way to automate their usage, but I don't know if this is possible).
I was thinking of using the requests library for Python, and the function I've written for posting the json to the API looks like this:
def postFileToAPI(**context):
print('uploadFileToAPI() ------ ')
json_file = context['ti'].xcom_pull(task_ids='toJson') ## this pulls the json file from a previous task
print('--------------- Posting survey request to API')
r = requests.post('https://[request]', data = json_file)
(I haven't finished defining the http link for the request because my source data is incomplete.)
However, since this is my frst time working with APIs and the requests library, I don't know if this is enough. For example, I'm unsure if I need to provide a token from the API to perform the request.
I also don't know if there are other libraries that are better suited for this or that could be a good support.
In short: I don't know if what I'm doing will work as intended, what other information I need t provide my DAG or if there are any libraries to make my work easier.

The Python requests package that you're using is all you need, except if you're making a request that needs extra authorisation - then you should also import for example requests_jwt (then from requests_jwt import JWTAuth) if you're using JSON web tokens, or whatever relevant requests package corresponds for your authorisation style.
You make POST and GET requests and all individual requests separately.
Include the URL and data arguments as you have done and that should work!
You may also need headers and/or auth arguments to get through security,
eg for the GitLab api for a private repository you would include these extra arguments, where GITLAB_TOKEN is a GitLab web token.
```headers={'PRIVATE-TOKEN': GITLAB_TOKEN},
auth=JWTAuth(GITLAB_TOKEN)```
If you just try it it should work, if it doesn't work then test the API with curl requests directly in the Terminal, or let us know :)

How to Create Python Code with Google API Client

I have code below that was given to me to list Google Cloud Service Accounts for a specific Project.
import os
from googleapiclient import discovery
from gcp import get_key
"""gets all Service Accounts from the Service Account page"""
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = get_key()
service = discovery.build('iam', 'v1')
project_id = 'projects/<google cloud project>'
request = service.projects().serviceAccounts().list(name=project_id)
response = request.execute()
accounts = response['accounts']
for account in accounts:
print(account['email'])
This code works perfectly and prints the accounts as I need them. What I'm trying to figure out is:
Where can I go to see how to construct code like this? I found a site that has references to the Python API Client, but I can't seem to figure out how to make the code above from it. I can see the Method to list the Service Accounts, but it's still not giving me enough information.
Is there somewhere else I should be going to educate myself. Any information you have is appreciated so I don't pull out the rest of my hair.
Thanks, Eric

Let me share with you this documentation page, where there is a detailed explanation on how to build a script such as the one you shared, and what does each line of code mean. It is extracted from the documentation of ML Engine, not IAM, but it is using the same Python Google API Client Libary, so just ignore the references to ML and the rest will be useful for you.
In any case, here it is a commented version of your code, so that you understand it better:
# Imports for the Client API Libraries and the key management
import os
from googleapiclient import discovery
from gcp import get_key
# Look for an environment variable containing the credentials for Google Cloud Platform
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = get_key()
# Build a Python representation of the REST API
service = discovery.build('iam', 'v1')
# Define the Project ID of your project
project_id = 'projects/<google cloud project>'
"""Until this point, the code is general to any API
From this point on, it is specific to the IAM API"""
# Create the request using the appropriate 'serviceAccounts' API
# You can substitute serviceAccounts by any other available API
request = service.projects().serviceAccounts().list(name=project_id)
# Execute the request that was built in the previous step
response = request.execute()
# Process the data from the response obtained with the request execution
accounts = response['accounts']
for account in accounts:
print(account['email'])
Once you understand the first part of the code, the last lines are specific to the API you are using, which in this case is the Google IAM API. In this link, you can find detailed information on the methods available and what they do.
Then, you can follow the Python API Client Library documentation that you shared in order to see how to call the methods. For instance, in the code you shared, the method used depends on service, which is the Python representation of the API, and then goes down the tree of methods in the last link as in projects(), then serviceAccounts() and finally the specificlist() method, which ends up in request = service.projects().serviceAccounts().list(name=project_id).
Finally, just in case you are interested in the other available APIs, please refer to this page for more information.
I hope the comments I made on your code were of help, and that the documentation shared makes it easier for you to understand how a code like that one could be scripted.

You can use ipython having googleapiclient installed - with something like:
sudo pip install --upgrade google-api-python-client
You can go to interactive python console and do:
from googleapiclient import discovery
dir(discovery)
help(discovery)
dir - gives all entries that object has - so:
a = ''
dir(a)
Will tell what you can do with string object. Doing help(a) will give help for string object. You can do dipper:
dir(discovery)
# and then for instance
help(discovery.re)
You can call your script in steps, and see what is result print it, do some research, having something - do %history to printout your session, and have solution that can be triggered as a script.

Authenticate to Google Drive and download spreadsheet with Python urllib2/requests

I would like to download a document I have in my Google Drive authenticating to Google (I only want certain users to be able to access it and do not want to publish it on the web).
I have tried using requests but apparently I am doing something wrong.
From a browser I can download my document going to the address
https://docs.google.com/spreadsheets/d/<document key>/export?format=xls.
So in my python script I do the following:
import os
import requests
import shutil
from requests.auth import HTTPBasicAuth
remote = "https://docs.google.com/spreadsheets/d/<document key>/export?format=xls"
username = os.environ['GOOGLEUSERNAME']
password = os.environ['GOOGLEPASSWORD']
r = requests.get(remote, auth=HTTPBasicAuth(username,password))
if r.status_code == 200:
with open("document.xls","wb") as f:
shutil.copyfileobj(r.raw, f)
however the resulting document.xls is empty.
What am I doing wrong?

It might actually be possible what you are trying to do, but here are some reasons why it will be non-trivial(by no means a complete list):
Google is usually blocking user-agents that are non-browsers(like your Python script) for browser intended content (for security reasons); you would have to spoof it, which is actually easy
Multi-factor authentication - you would have to turn that off (easy, but you open yourself up for being hacked...)
Session-cookie - aka security cookie; (not so easy to get ahold of)
What you should do instead
Use the official google-drive API. Also, the Python client library has a nice tutorial and this page describes how to download files from google-drive.
If you want to write even less code, then libraries like PyDrive will make your live even easier.
Hope this helps!

I might have a simple solution for you, depending on what exactly the auth requirements are. You are saying
I only want certain users to be able to access it and do not want to
publish it on the web
From this statement alone, it may be sufficient for you to create a "secret" link for your document, and share this among your users. You can then easily retrieve this document automatically, for instance with wget, and specify the format, e.g. csv:
wget -O data.csv "https://docs.google.com/spreadsheets/d/***SHARED-SECRET***/export?format=csv"
Or, in Python (2):
import urllib2
from cookielib import CookieJar
spreadsheet_url = "https://docs.google.com/spreadsheets/d/***SHARED-SECRET***/export?format=csv"
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(CookieJar()))
response = opener.open(spreadsheet_url)
with open("data.csv", "wb") as f:
f.write(response.read())
I am actually using that in production, it works reliably, without faking the user agent.

puppet cert list all using api/python?

I need to get the list of all puppet nodes (basically output of puppet cert list --all). What is the best way to do the same using python (without using exec or similar things on the command itself) in puppet 2.6.18
puppet 2.7.0 onwards has HTTP API to achieve the same.
http://docs.puppetlabs.com/guides/rest_api.html#certificate-request
GET /{environment}/certificate_statuses/no_key
puppetdb also one api but am not sure if the env am working with has puppetdb. (checking on that).
Is there anything like ansible.runner for puppet?
Any other thoughts?

You first need to configure access to the REST API in auth.conf. Then you can use the built-in urllib2 or external requests library to query the API with the appropriate SSL client certificate for authentication.
If you don't want to deal with SSL client certificates, you can use allow_ip in auth.conf. I'd only do that if you're not interested in the more sensitive areas of the API (like requesting a catalog).
I wrote a Python wrapper around the Puppet REST API and posted it on GitHub: pypuppet.
For example,
>>> import puppet
>>> p = puppet.Puppet()
>>> print p.certificates()
See the README and example auth.conf for more info. Let me know how it works out for you.

Windows live api for python

I have read documentation about Windows Live API: http://msdn.microsoft.com/en-us/library/bb463989.aspx
But how can I retrieve contacts from hotmail with python ?
Is there any example ?

Your program will first need to obtain "delegated authentication", for which the Python samples are here.
After that, the interface is REST-like: you only need to HTTP GET the appropriate URI (per the docs, that's '/LiveContacts/contacts' to retrieve all contacts. The REST Schema is documented here. You can make an HTTP GET request in Python with such standard library modules as urllib and urllib2, though the lower-level httplib module is also fine.

For those who are searching for the download link for the library
http://download.microsoft.com/download/6/2/a/62adfe67-6fee-487f-9c3e-911ce5d0bc9d/webauth-python-1.2.tar.gz