I'm setting up the Google Contacts CardDAV API client.
OAuth 2.0 using oauth2client.
Request using requests.
from oauth2client import file, client, tools
import requests
SCOPES = 'https://www.googleapis.com/auth/carddav'
store = file.Storage('credentials.json')
creds = store.get()
if not creds or creds.invalid:
flow = client.flow_from_clientsecrets('client_secret.json', SCOPES)
creds = tools.run_flow(flow, store)
print(creds.access_token)
hed = {'Authorization': 'Bearer ' + creds.access_token}
response = requests.request('PROPFIND', 'https://www.googleapis.com/.well-known/carddav', headers=hed, allow_redirects=False)
if response.status_code == 301:
location = response.headers['location']
response = requests.request('PROPFIND', 'https://www.googleapis.com' + location, headers=hed)
print(response.text)
But when I request url for get the address book (I get it from Location header of first request), it returns error:
{
"error": {
"code": 400,
"message": "Request contains an invalid argument.",
"status": "INVALID_ARGUMENT"
}
}
Full requests information
First request
requests.request('PROPFIND', 'https://www.googleapis.com/.well-known/carddav', headers=hed, allow_redirects=False)
REQUEST
=======
endpoint: PROPFIND https://www.googleapis.com/.well-known/carddav
headers:
User-Agent: python-requests/2.22.0
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive
Authorization: Bearer ya29.***********************************************
Content-Length: 0
=======
RESPONSE
========
status_code: 301
headers:
Content-Type: text/plain; charset=UTF-8
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Expires: Mon, 01 Jan 1990 00:00:00 GMT
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
X-Frame-Options: SAMEORIGIN
Location: /carddav/v1/principals/<my_email>/lists/default/
Pragma: no-cache
Vary: Origin, X-Origin, Referer
Date: Fri, 21 Jun 2019 11:43:23 GMT
Server: ESF
Content-Length: 0
Alt-Svc: quic=":443"; ma=2592000; v="46,44,43,39"
========
Second request
response = requests.request('PROPFIND', 'https://www.googleapis.com' + location, headers=hed)
REQUEST
=======
endpoint: PROPFIND https://www.googleapis.com/carddav/v1/principals/<my_email>/lists/default/
headers:
User-Agent: python-requests/2.22.0
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive
Authorization: Bearer ya29.***********************************************
Content-Length: 0
=======
RESPONSE
========
status_code: 400
headers:
Vary: Origin, X-Origin, Referer
Content-Type: application/json; charset=UTF-8
Date: Fri, 21 Jun 2019 11:43:23 GMT
Server: ESF
Content-Length: 127
X-XSS-Protection: 0
X-Frame-Options: SAMEORIGIN
X-Content-Type-Options: nosniff
Alt-Svc: quic=":443"; ma=2592000; v="46,44,43,39"
body:
{
"error": {
"code": 400,
"message": "Request contains an invalid argument.",
"status": "INVALID_ARGUMENT"
}
}
========
The short answer: The PROPFIND method is generic and must include a body that specifies what information the server should return. You must pass an XML payload in the body of the request that identifies the properties that you are requesting.
Getting the addressbooks URIs
According to Google's CardDav API docs, your first request is perfect and will redirect you to the Address Book Resource for the current user. Here is Google's description of the next step:
Your client program can then discover the principal address book by performing a PROPFIND on the addressbook-home-set and looking for the addressbook and collection resources.
Let me unpack this: your second request should query a list of the user's resources found at the location you got from the first request. To do this query properly, you'll need to pass an XML body with the PROPFIND request, like so:
PROPFIND https://www.googleapis.com/carddav/v1/principals/<my_email>/lists/default/
Authorization: Bearer ya29.***********************************************
Depth: 1
Content-Type: application/xml; charset=utf-8
<D:propfind xmlns:D="DAV:">
<D:prop>
<D:resourcetype />
<D:displayname />
</D:prop>
</D:propfind>
Here you specify the properties that you want the server to respond. You specify the resourcetype property because you are only interested in addressbook or collection resources which contain contacts.
This request will return a list of URIs to resources from which you can select any that have a resource type of addressbook or collection.
At this point, you don't have any contacts or even URIs of contacts. You have a list of URIs for the user's address books or collections of contacts. (Typically, there's just one of these, but there could be many.)
You didn't ask how to get the user's contacts, but I'm assuming it is your end goal and will continue with the subsequent steps.
Getting the contacts URIs
Your next set of requests will query each of the address book URIs for the URIs of their contacts. Loop over each result from the previous query and issue another PROPFIND request on the URI with this payload:
REPORT <addressbook_uri>
Authorization: Bearer ya29.***********************************************
Content-Type: application/xml; charset=utf-8
<D:propfind xmlns:D="DAV:">
<D:prop>
<D:getetag />
<D:getcontenttype />
</D:prop>
</D:propfind>
Here we query the content type of each item so we can determine if it is a VCard type. VCards are legit contact records.
Now you can filter this set of results by contenttype == 'text/vcard' to get a new list of URIs pointing to each contact in the user's address book.
Oh man, we are getting so close.
Get the contact VCards
Finally, assemble your list of URIs to actual contact data and query the data from the server.
Here you will make an addressbook-multiget REPORT request to retrieve a batch of the contacts in your list. Google doesn't say how many contact URIs you may include in your request. I've typically limited my request to a few hundred at a time.
E.g.
REPORT <addressbook_uri>
Authorization: Bearer ya29.***********************************************
Content-Type: application/xml; charset=utf-8
<C:addressbook-multiget xmlns:D="DAV:" xmlns:C="urn:ietf:params:xml:ns:carddav">
<D:prop>
<D:getetag/>
<C:address-data>
<C:allprop/>
</C:address-data>
</D:prop>
<D:href>/carddav/v1/principals/<my_email>/lists/default/contact1.vcf</D:href>
<D:href>/carddav/v1/principals/<my_email>/lists/default/contact2.vcf</D:href>
...
</C:addressbook-multiget>
The response will contain the VCard data for each of the contacts, packed inside XML. Parse out the XML text and then parse the VCard data to finally retrieve your contact details.
Done!
Resources:
Google's limited CardDav docs
The best resource I've found is this description of building a CardDav client by Sabre
When others fail, go to the source: RFC6352
Related
Is there a way to form batched requests using python requests module ? I want to send multiple homogenous http API requests in a single POST request. I am trying to use the GCP documentation https://cloud.google.com/dns/docs/reference/batch?hl=en_US&_ga=2.138235123.-2126794010.1660759555 to create multiple DNS records in a single POST request. Can anyone help with a simple example of how this could be achieved using python requests module ?
This is how the sample POST request will look like:
POST /batch/farm/v1 HTTP/1.1
Authorization: Bearer your_auth_token
Host: www.googleapis.com
Content-Type: multipart/mixed; boundary=batch_foobarbaz
Content-Length: total_content_length
--batch_foobarbaz
Content-Type: application/http
Content-ID: <item1:12930812#barnyard.example.com>
GET /farm/v1/animals/pony
--batch_foobarbaz
Content-Type: application/http
Content-ID: <item2:12930812#barnyard.example.com>
PUT /farm/v1/animals/sheep
Content-Type: application/json
Content-Length: part_content_length
If-Match: "etag/sheep"
{
"animalName": "sheep",
"animalAge": "5"
"peltColor": "green",
}
--batch_foobarbaz
Content-Type: application/http
Content-ID: <item3:12930812#barnyard.example.com>
GET /farm/v1/animals
If-None-Match: "etag/animals"
--batch_foobarbaz--
Basically; the main intention here is to not overload the remote API with multiple http requests causing the rate limit throttling but instead use batched http requests so that the remote API gets only a single batched request embedded with multiple requests in the form of parts.
No. HTTP doesn't work that way. You can send multiple requests simultaneously using threads, but you can't send multiple POSTs through a single request.
According to the doc, the individual batch HTTP requests are supposed to go in the body of the request. You can try building the requests up manually. Like so:
import requests
body = """
--batch_foobarbaz
Content-Type: application/http
Content-ID: <item1:12930812#barnyard.example.com>
GET /farm/v1/animals/pony
--batch_foobarbaz
Content-Type: application/http
Content-ID: <item2:12930812#barnyard.example.com>
PUT /farm/v1/animals/sheep
Content-Type: application/json
Content-Length: part_content_length
If-Match: "etag/sheep"
{
"animalName": "sheep",
"animalAge": "5"
"peltColor": "green",
}
--batch_foobarbaz
Content-Type: application/http
Content-ID: <item3:12930812#barnyard.example.com>
GET /farm/v1/animals
If-None-Match: "etag/animals"
--batch_foobarbaz--
"""
response = requests.post(
"https://www.googleapis.com/batch/API/VERSION/batch/form/v1",
data=body,
headers={
'Authorization': 'Bearer your_auth_token',
'Host': 'www.googleapis.com',
'Content-Type': 'multipart/mixed; boundary=batch_foobarbaz'
}
)
Of course, you'd have to build the individual requests manually:
Content-Type: application/http
Content-ID: <item1:12930812#barnyard.example.com>
GET /farm/v1/animals/pony
Unless you can find a library that can construct HTTP requests according to the HTTP/1.1 RFC. I can't think of one off the top of my head.
I'm trying to understand how Etag works in Django. I added middleware in settings ('django.middleware.http.ConditionalGetMiddleware') and this seems to work as it generates the Etag:
HTTP/1.0 200 OK
Date: Mon, 15 Jan 2018 16:58:30 GMT
Server: WSGIServer/0.2 CPython/3.6.0
Content-Type: application/json
Vary: Accept
Allow: GET, HEAD, OPTIONS
X-Frame-Options: SAMEORIGIN
Content-Length: 1210
ETag: "060e28ac5f08d82ba0cd876a8af64e6d"
Access-Control-Allow-Origin: *
However, when I put If-None-Match: '*' in the request header, I get the following error:
Request header field If-None-Match is not allowed by Access-Control-Allow-Headers in preflight response.
And I notice the request method sent back in the response is OPTIONS and the rest of the headers look like this:
HTTP/1.0 200 OK
Date: Mon, 15 Jan 2018 17:00:26 GMT
Server: WSGIServer/0.2 CPython/3.6.0
Content-Type: text/html; charset=utf-8
Access-Control-Allow-Origin: *
Access-Control-Allow-Headers: accept, accept-encoding, authorization, content-type, dnt, origin, user-agent, x-csrftoken, x-requested-with
Access-Control-Allow-Methods: DELETE, GET, OPTIONS, PATCH, POST, PUT
Access-Control-Max-Age: 86400
So the question is how do I get If-None-Match to be an allowed header? I'm not sure if this is a server or client issue. I'm using Django/DRF/Vue for my stack and Axios for making http requests.
As the response contains various CORS headers, I believe you have already used django-cors-headers, you could adjust Access-Control-Allow-Headers with CORS_ALLOW_HEADERS config option, get more detail on its doc.
I'm trying to write a python script that would help me install a theme remotely. Unfortunately, the upload part doesn't play nice, trying to do it with requests' POST helpers.
The HTTP headers of a successful upload look like this:
http://127.0.0.1/wordpress/wp-admin/update.php?action=upload-theme
POST /wordpress/wp-admin/update.php?action=upload-theme HTTP/1.1
Host: 127.0.0.1
User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:53.0) Gecko/20100101 Firefox/53.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Content-Type: multipart/form-data; boundary=---------------------------2455316848522
Content-Length: 2580849
Referer: http://127.0.0.1/wordpress/wp-admin/theme-install.php
Cookie: wordpress_5bd7a9c61cda6e66fc921a05bc80ee93=admin%7C1497659497%7C4a1VklpOs93uqpjylWqckQs80PccH1QMbZqn15lovQu%7Cee7366eea9b5bc9a9d492a664a04cb0916b97b0d211e892875cec86cf43e2f9d; wordpress_test_cookie=WP+Cookie+check; wordpress_logged_in_5bd7a9c61cda6e66fc921a05bc80ee93=admin%7C1497659497%7C4a1VklpOs93uqpjylWqckQs80PccH1QMbZqn15lovQu%7C9949f19ef5d900daf1b859c0bb4e2129cf86d6a970718a1b63e3b9e56dc5e710; wp-settings-1=libraryContent%3Dbrowse; wp-settings-time-1=1497486698
Connection: keep-alive
Upgrade-Insecure-Requests: 1
-----------------------------2455316848522: undefined
Content-Disposition: form-data; name="_wpnonce"
b1467671e0
-----------------------------2455316848522
Content-Disposition: form-data; name="_wp_http_referer"
/wordpress/wp-admin/theme-install.php
-----------------------------2455316848522
Content-Disposition: form-data; name="themezip"; filename="oedipus_theme.zip"
Content-Type: application/octet-stream
PK
HTTP/1.1 200 OK
Date: Thu, 15 Jun 2017 01:33:25 GMT
Server: Apache/2.4.25 (Win32) OpenSSL/1.0.2j PHP/7.1.1
X-Powered-By: PHP/7.1.1
Expires: Wed, 11 Jan 1984 05:00:00 GMT
Cache-Control: no-cache, must-revalidate, max-age=0
X-Frame-Options: SAMEORIGIN
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: text/html; charset=UTF-8
----------------------------------------------------------
To create a simple session for WP, in order to use later for uploads:
global wp_session
def wpCreateSession(uname, upassword, site_link):
"""
:param uname: Username for the login.
:param upaswword: Password for the login.
:param site_link: Site to login on.
:return: Returns a sessions for the said website.
"""
global wp_session
wp_session = requests.session()
wp_session.post(site_link, data={'log' : uname, 'pwd' : upassword})
To upload the said file to WP, using the wp_session global:
def wpUploadTheme(file_name):
global wp_session
try:
with open(file_name, 'rb') as up_file:
r = wp_session.post('http://127.0.0.1/wordpress/wp-admin/update.php', files = {file_name: up_file})
print "Got after try."
finally:
up_file.close()
And this last bit is where it doesn't work, the upload is not successful and I get returned to WordPress' basic 404.
I have also tried requests_toolbelt MultiPart_Encoder to no avail.
Question: 'requests' POST file fails when trying to upload
Check your files dict, your dict is invalid
files = {file_name: up_file}
Maybe you need a full blown files dict, for instance:
files = {'themezip': ('oedipus_theme.zip',
open('oedipus_theme.zip', 'rb'),
'application/octet-stream', {'Expires': '0'})}
From docs.python-requests.org
files = {'file': open('test.jpg', 'rb')}
requests.post(url, files=files)
From SO Answer Upload Image using POST form data in Python-requests
I'm getting "urllib.error.HTTPError: HTTP Error 401: Unauthorized" after trying this code:
_HTTPHandler = urllib.request.HTTPBasicAuthHandler()
_HTTPHandler.add_password(None,'http://192.168.1.205','admin','password')
opener = urllib.request.build_opener(_HTTPHandler)
opener.open('http://192.168.1.205/api/swis/resource')
I'm sure that the user/password is correct. I've tested it with Google's postman app setting a Basic Auth header and i receive the correct response.
My question is how can a see the headers that are being used by the "opener" so i can check if they are being generated correctly or not.
For manual debugging, you can set the debuglevel of your HTTPHandler:
handler = HTTPHandler(debuglevel=1)
This will produce rich output in stdout, where you can see the full request/response dump:
send: GET / HTTP/1.1
Accept-Encoding: identity
Host: www.example.com
Connection: close
reply: HTTP/1.1 200 OK
header: Date: Mon May 22 2017 12:21:31 GMT
header: Cache-Control: private
header: Content-Type: text/html; charset=ISO-8859-1
Relevant docs: https://docs.python.org/3/library/urllib.request.html
I am using the python urllib2 library for opening URL, and what I want is to get the complete header info of the request. When I use response.info I only get this:
Date: Mon, 15 Aug 2011 12:00:42 GMT
Server: Apache/2.2.0 (Unix)
Last-Modified: Tue, 01 May 2001 18:40:33 GMT
ETag: "13ef600-141-897e4a40"
Accept-Ranges: bytes
Content-Length: 321
Connection: close
Content-Type: text/html
I am expecting the complete info as given by live_http_headers (add-on for firefox), e.g:
http://www.yellowpages.com.mt/Malta-Web/127151.aspx
GET /Malta-Web/127151.aspx HTTP/1.1
Host: www.yellowpages.com.mt
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0.1) Gecko/20100101 Firefox/4.0.1
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-gb,en;q=0.5
Accept-Encoding: gzip, deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 115
Connection: keep-alive
Cookie: __utma=156587571.1883941323.1313405289.1313405289.1313405289.1; __utmz=156587571.1313405289.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)
HTTP/1.1 302 Found
Connection: Keep-Alive
Content-Length: 141
Date: Mon, 15 Aug 2011 12:17:25 GMT
Location: http://www.trucks.com.mt
Content-Type: text/html; charset=utf-8
Server: Microsoft-IIS/6.0
X-Powered-By: ASP.NET, UrlRewriter.NET 2.0.0
X-AspNet-Version: 2.0.50727
Set-Cookie: ASP.NET_SessionId=zhnqh5554omyti55dxbvmf55; path=/; HttpOnly
Cache-Control: private
My request function is:
def dorequest(url, post=None, headers={}):
cOpener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cookielib.CookieJar()))
urllib2.install_opener( cOpener )
if post:
post = urllib.urlencode(post)
req = urllib2.Request(url, post, headers)
response = cOpener.open(req)
print response.info() // this does not give complete header info, how can i get complete header info??
return response.read()
url = 'http://www.yellowpages.com.mt/Malta-Web/127151.aspx'
html = dorequest(url)
Is it possible to achieve the desired header info details by using urllib2? I don't want to switch to httplib.
Those are all of the headers the server is sending when you do the request with urllib2.
Firefox is showing you the headers it's sending to the server as well.
When the server gets those headers from Firefox, some of them may trigger it to send back additional headers, so you end up with more response headers as well.
Duplicate the exact headers Firefox sends, and you'll get back an identical response.
Edit: That location header is sent by the page that does the redirect, not the page you're redirected to. Just use response.url to get the location of the page you've been sent to.
That first URL uses a 302 redirect. If you don't want to follow the redirect, but see the headers from the first page instead, use a URLOpener instead of a FancyURLOpener, which automatically follows redirects.
I see that server returns HTTP/1.1 302 Found - HTTP redirect.
urllib automatically follow redirects, so headers returned by urllib is headers from http://www.trucks.com.mt, not http://www.yellowpages.com.mt/Malta-Web/127151.aspx