I want to send a URL request equivalent to using to json objects in the post data, separated by newline.
This is for indexing two items in bulk for Elasticsearch.
This works fine:
curl -XPOST 'localhost:9200/myindex/mydoc?pretty=true' --data-binary #myfile.json
where myfile.json:
{"index": {"_parent": "btaCovzjQhqrP4s3iPjZKQ"}}
{"title": "hello"}
{"index": {"_parent": "btaCovzjQhqrP4s3iPjZKQ"}}
{"title": "world"}
When I try it using:
req = urllib2.Request(url,data=
json.dumps({"index": {"_parent": "btaCovzjQhqrP4s3iPjZKQ"}}) + "\n" +
json.dumps({"title":"hello"}) + "\n" +
json.dumps({"index": {"_parent": "btaCovzjQhqrP4s3iPjZKQ"}}) + "\n" +
json.dumps({"title":"world"})
I get:
HTTP Error 500: Internal Server Error
The "HTTP Error 500" may be from forgetting to include an index name or index type.
Also: for bulk inserts, elasticsearch requires a trailing "\n" character after the last record, or it won't insert that record.
Try:
import urllib2
import json
url = 'http://localhost:9200/myindex/mydoc/_bulk?pretty=true'
data = json.dumps({"index": {"_parent": "btaCovzjQhqrP4s3iPjZKQ"}}) + "\n" + json.dumps({"title":"hello"}) + "\n" + json.dumps({"index": {"_parent": "btaCovzjQhqrP4s3iPjZKQ"}}) + "\n" + json.dumps({"title":"world"})
req = urllib2.Request(url,data=data+"\n")
f = urllib2.urlopen(req)
print f.read()
Or, with some refactoring:
import urllib2
import json
url = 'http://localhost:9200/myindex/mydoc/_bulk?pretty=true'
data = [
{"index": {"_parent": "btaCovzjQhqrP4s3iPjZKQ"}},
{"title":"hello"},
{"index": {"_parent": "btaCovzjQhqrP4s3iPjZKQ"}},
{"title":"world"}
]
encoded_data = "\n".join(map(json.dumps,data)) + "\n"
req = urllib2.Request(url,data=encoded_data)
f = urllib2.urlopen(req)
print f.read()
The usecase for me is actually a bulk_index request for ElasticSearch.
This is made a lot easier with rawes:
import rawes
es = rawes.Elastic('localhost:9200')
with open('myfile.json') as f:
lines = f.readlines()
es.post('someindex/sometype/_bulk', data=lines)
Related
I have been trying to export offer file using Python for Bol Retailer API
According to the official docs on request an offer file export
I have include all the headers and formats but it throws a 400 Bad Request
400
Bad Request
b'{\n "type" : "https://api.bol.com/problems",\n "title" : "Bad Request",\n "status" : 400,\n "detail" : "The supplied content-type media type is not supported.",\n "host" : "Instance-111",\n "instance" : "https://api.bol.com/retailer/offers/export"\n}'
Here is a minimal example from my code
import base64
import requests
import json
import time
class BolService:
def __init__(self, _id, secret):
self.host = "https://api.bol.com"
self.__header = {
"Accept": "application/vnd.retailer.v7+json",
"Content-Type": "N/A",
"Authorization": "Bearer " + self.__get_token(_id, secret)
}
def __get_token(self, _id, secret) -> str:
creds = (_id + ":" + secret).encode('ascii') #creds : ascii bytes
creds_b64_b = base64.b64encode(creds) #creds : base64 bytes
creds_b64 = creds_b64_b.decode('ascii') #creds : base64 string
header = {
"Authorization":"Basic " + creds_b64
}
link = "https://login.bol.com/token?grant_type=client_credentials"
response = requests.post(link, headers=header)
response_data = json.loads(response.content.decode())
return response_data['access_token']
def get_offer_file(self):
path = f"/retailer/offers/export"
new_header = self.__header.copy()
new_header["format"] = "CSV"
response = requests.post(self.host + path, headers=new_header)
return response
Note: I have also tried changing the "Content-Type" in self.__header to "application/vnd.retailer.v7+json", I have also changed the same to add csv using "application/vnd.retailer.v7+json+csv" or "application/vnd.retailer.v7+csv". I have also tried adding self.__header['Content-Type'] = 'text/csv' but nothing seems to work it keeps on throwing the same Bad Request. I have also tried using the v6 of the API instead of v7 but same issue.
I know this is something that should be dealt with the customer service of Bol but they their service is too pathetic to even give a simple reply. Also as of August 2022 their site which details API issues is down. Maybe if someone with experience can help here.
I don't think I am missing anything here. Please let me know.
So I was able to sucessfully make the POST request.
1st what I did was change the "Content-Type" in self.__header to "application/vnd.retailer.v7+json"
so the header now looks like this
self.__header = {
"Accept": "application/vnd.retailer.v7+json",
"Content-Type": "application/vnd.retailer.v7+json",
"Authorization": "Bearer " + self.__get_token(_id, secret)
}
Since we require the content type in JSON format so we have to include a JSON body by dumping our dictionary content using json.dumps
So the get_offer_file method now looks like with {"format":"CSV"} as the body
def get_offer_file(self):
path = f"/retailer/offers/export"
response = requests.post(self.host + path, headers=self.__header, data=json.dumps({"format":"CSV"}))
return response
Here is the full code:
import base64
import requests
import json
class BolService:
def __init__(self, _id, secret):
self.host = "https://api.bol.com"
self.__header = {
"Accept": "application/vnd.retailer.v7+json",
"Content-Type": "application/vnd.retailer.v7+json",
"Authorization": "Bearer " + self.__get_token(_id, secret)
}
def __get_token(self, _id, secret) -> str:
creds = (_id + ":" + secret).encode('ascii') #creds : ascii bytes
creds_b64_b = base64.b64encode(creds) #creds : base64 bytes
creds_b64 = creds_b64_b.decode('ascii') #creds : base64 string
header = {
"Authorization":"Basic " + creds_b64
}
link = "https://login.bol.com/token?grant_type=client_credentials"
response = requests.post(link, headers=header)
response_data = json.loads(response.content.decode())
return response_data['access_token']
def get_offer_file(self):
path = f"/retailer/offers/export"
response = requests.post(self.host + path, headers=self.__header, data=json.dumps({"format":"CSV"}))
return response
This is what i have try:
import time
import json
import hashlib
import hmac
from urllib.parse import urlparse
import requests
from datetime import datetime, timedelta
from pprint import pprint
def generate_signature(secret, verb, url, expires, data):
"""Generate a request signature compatible with BitMEX."""
# Parse the url so we can remove the base and extract just the path.
parsedURL = urlparse(url)
path = parsedURL.path
if parsedURL.query:
path = path + '?' + parsedURL.query
if isinstance(data, (bytes, bytearray)):
data = data.decode('utf8')
print("Computing HMAC: %s" % verb + path + str(expires) + data)
message = verb + path + str(expires) + data
signature = hmac.new(bytes(secret, 'utf8'), bytes(message, 'utf8'), digestmod=hashlib.sha256).hexdigest()
return signature
# Or you might generate it like so:
# expires = int(round(time.time()) + 5)
data = {
'filter': {"symbol": "XBTUSD"},
'columns': [],
'count': 100
'start': 1,
'reverse': "true",
'startTime': str(datetime.now() - timedelta(minutes=50)),
'endTime': str(datetime.now())
}
verb = 'GET'
path = '/api/v1/position'
time_in_epoch = int(datetime.timestamp(datetime.now() + timedelta(hours=.5)))
print("========",time_in_epoch, int(round(time.time()) + 1800))
expires = time_in_epoch
gen_signat = generate_signature('api-sceret', verb, path, expires, json.dumps(data))
URL_get_data = "http://www.bitmex.com/api/v1/position"
headers = {
'api-expires': str(expires),
'api-key': "api-key",
'api-signature': str(gen_signat),
}
response = requests.get(URL_get_data, headers=headers, data= data).json()
Still showing error:
{'error': {'message': 'Signature not valid.', 'name': 'HTTPError'}}
If I pass this gen_signat = generate_signature('api-sceret', verb, path, expires, data) the showing reponse<200> but returns empty list.
Looks like you're basing on this Bitmex sample. Please note that data must be JSON without whitespace between keys. The default separators value for json.dumps is (', ', ': '), so you need to change your code to use (',', ':'). It should be gen_signat = generate_signature('api-sceret', verb, path, expires, json.dumps(data,separators=(',', ':'))).
I am trying to save the status of the response of requests.post() to a file and also print it in terminal. i have a list of hostnames in file, so one exception is caught it comes out and doesn't continue the loop. How t achieve he intended action.
with open(file_input,'r') as f:
content = f.readlines()
try:
for x in content:
url='https://url/a/b/c/{}'.format(x.strip())
payload=('{{"ip-address": "x.x.x.x","user-name": "john","password": str(Pass),"db-name": str(x.strip()),"service-name": "y","port": "y","connection-string": "y"}}')
response = requests.post(url,json=payload,header=add_cookie)
print(response.text)
response.raise_for_status()
except requests.exceptions.HTTPError as e:
# Whoops it wasn't a 200
print ("Error: " + str(e))
# Must have been a 200 status code
json_obj = response.json()
print (json_obj)
with open('response.txt',mode='wb') as localfile:
localfile.write(response.text)
It can be achieved a little bit easier:
import requests
with open(file_input, "r") as f:
file_urls = f.readlines() # read all urls into a list
for url_suffix in file_urls:
url = 'https://url/a/b/c/{}'.format(url_suffix.strip())
payload = { # your previous payload in not a JSON representation
"ip-address": "x.x.x.x",
"user-name": "john",
"password": str(Pass),
"db-name": str(x.strip()),
"service-name": "y",
"port": "y",
"connection-string": "y",
}
response = requests.post(url, json=payload, header=add_cookie)
if response.ok:
print("JSON received:")
print(response.json()) # NOTE: verify that response is a JSON
with open('response.txt', 'a') as localfile: # NOTE: append mode!
localfile.write(response.text)
else: # status code 4XX/5XX
print("Error getting url", url, r.status_code, r.text)
Note that you should pass dict to json= argument of requests.post(). Your payload is a strangely formatted string.
with open(file_input,'r') as f:
content = f.readlines()
for x in content:
url='https://url/a/b/c/{}'.format(x.strip())
payload=('{{"ip-address": "x.x.x.x","user-name": "john","password": str(Pass),"db-name": str(x.strip()),"service-name": "y","port": "y","connection-string": "y"}}')
try:
response = requests.post(url,json=payload,header=add_cookie)
print(response.text)
print(response.status_code)
except Exception as err:
print("Error: " + str(err))
with open('errors.txt', mode='a') as f:
f.write(str(err))
continue
json_obj = response.json()
print(json_obj)
with open('response.txt',mode='wb') as localfile:
localfile.write(response.text)
I am working with an API that doesn't have all the information I need in a single call, and I need to the project code it came from into the call that I am making. Right now it appends the project data to the list, but I really need it to be part of the original call. Here is my output now:
[{"committer_email": "justin.m.boucher#example.com", "short_id": "981147b9", "title": "Added .gitignore", "author_email": "justin.m.boucher#example.com", "authored_date": "2017-08-29T08:31:11.000-07:00", "created_at": "2017-08-29T08:31:11.000-07:00", "author_name": "Justin Boucher", "parent_ids": [], "committed_date": "2017-08-29T08:31:11.000-07:00", "message": "Added .gitignore\n", "committer_name": "Justin Boucher", "id": "981147b905913a60796283ce10f915c53679df49"}, {"project_id": "2"}]
Here is the output I want to achieve:
[{"project_id": "2", "committer_email": "justin.m.boucher#example.com", "short_id": "981147b9", "title": "Added .gitignore", "author_email": "justin.m.boucher#example.com", "authored_date": "2017-08-29T08:31:11.000-07:00", "created_at": "2017-08-29T08:31:11.000-07:00", "author_name": "Justin Boucher", "parent_ids": [], "committed_date": "2017-08-29T08:31:11.000-07:00", "message": "Added .gitignore\n", "committer_name": "Justin Boucher", "id": "981147b905913a60796283ce10f915c53679df49"}]
Here is my code so far:
get_commits.py:
import gitlab
import json
gitlab = gitlab.Gitlab()
projects = gitlab.getProjectID()
for i in projects:
api_str = '/projects/' + str(i) + '/repository/commits'
connect = gitlab.connectAPI(apiCall=api_str)
data = json.dumps(connect)
# Append project id to json, since it isn't created
# in the commits from Gitlab
commit = json.loads(data)
commit.append({'project_id': str(i)})
# make it pretty again for Splunk to read
commit = json.dumps(commit)
print commit
gitlab.py
import os
import ConfigParser
import requests
import json
# Setup Splunk Environment
APPNAME = 'splunk_gitlab'
CONFIG = 'appconfig.conf'
SPLUNK_HOME = os.environ['SPLUNK_HOME']
parser = ConfigParser.SafeConfigParser()
class Gitlab():
# # Load Settings
# parser.read(SPLUNK_HOME + '/etc/apps/' + APPNAME + '/local/' + CONFIG)
# if parser.has_section('Authentication'):
# pass
# else:
# parser.read(SPLUNK_HOME + '/etc/apps/' + APPNAME + '/default/' + CONFIG)
#
# GITLAB_URL = parser.get('Authentication', 'GITLAB_URL')
# API_KEY = parser.get('Authentication', 'API_KEY')
# Used for testing only
GITLAB_URL = 'http://<my_address>'
API_KEY = '<my_key>'
API_SERVER = GITLAB_URL + '/api/v4'
# Place api call to retrieve data
def connectAPI(self, apiCall='/projects'):
headers = {
'PRIVATE-TOKEN': self.API_KEY
}
final_url = self.API_SERVER + apiCall
resp = requests.get(final_url, headers=headers)
status_code = resp.status_code
resp = resp.json()
if status_code == 200:
return resp
else:
raise Exception("Something went wrong requesting (%s): %s" % (
resp['errors'][0]['errorType'], resp['errors'][0]['message']))
def getProjectID(self):
connect = self.connectAPI(apiCall='/projects')
data = json.dumps(connect)
projects = json.loads(data)
project_list = []
for i in projects:
project_list.append(i['id'])
return project_list
If you want to add a new element to the first dictionary in the list instead of appending a new dictionary to the list, try using assignment instead of append.
commit[0]['project_id'] = str(i)
Need this:
POST&https%3A%2F%2Fsecure.trademe.co.nz%2FOauth%2FRequestToken&oauth_callback%3Dhttp%253A%252F%252Fwww.website-tm-access.co.nz%252Ftrademe-callback%26oauth_consumer_key%3DC74CD73FDBE37D29BDD21BAB54BC70E422%26oauth_nonce%3D7O3kEe%26oauth_signature_method%3DHMAC-SHA1%26oauth_timestamp%3D1285532322%26oauth_version%3D1.0%26scope%3DMyTradeMeRead%252CMyTradeMeWrite
Myattempt:
New_base_string ="POST&https%3A%2F%2Fsecure.trademe.co.nz%2FOauth%2FRequestToken&oauth_callback%3Dhttp%253A%252F%252Fwww.website-tm-access.co.nz%252Ftrademe-callback%26oauth_consumer_key%" + str(consumer_key) +"3DC74CD73FDBE37D29BDD21BAB54BC70E422%26oauth_nonce%3" + str(nonce) + "%26oauth_signature_method%3DHMAC-SHA1%26oauth_timestamp%3" + str(time) + "%26oauth_version%3D1.0%26scope%3DMyTradeMeRead%252CMyTradeMeWrite"
I just tried to append it to the end, will this work or will i need to append to a list and then encode?
so like this:
headers = { my_variable + other_variable }
authorization = '5C82CC6BC7C6472154FBC9CAB24A29A2 ' + ', '.join([key + '="' + urllib.parse.quote_plus(str(value)) + '"' for key, value in headers.items()])
General
If you want to URL encode parameters to your POST request the best way is:
import urllib
f = { 'eventName' : 'myEvent',
'eventDescription' : 'cool event',
'url' : 'http://www.google.com'}
print 'POST&%s' % urllib.urlencode(f)
Output:
POST&eventName=myEvent&url=http%3A%2F%2Fwww.google.com&eventDescription=cool+event
with Dictionary its not ordered if you want to order it just use a list
import urllib
f = [ ('eventName', 'myEvent'),
('eventDescription', 'cool event'),
('url', 'http://www.google.com')]
print 'POST&%s' % urllib.urlencode(f)
Output
POST&eventName=myEvent&eventDescription=cool+event&url=http%3A%2F%2Fwww.google.com
How to get your need this string (Python 3.5)
While the general example is tested in python 2.7, I wrote your example with python 3.5 code.
import urllib.parse
method = "POST"
url = "https://secure.trademe.co.nz/Oauth/RequestToken"
params = [('oauth_callback', 'http://www.website-tm-access.co.nz/trademe-callback'),
('oauth_consumer_key', 'C74CD73FDBE37D29BDD21BAB54BC70E422'),
('oauth_nonce', '7O3kEe'),
('oauth_signature_method', 'HMAC-SHA1'),
('oauth_timestamp', 1285532322),
('oauth_version', 1.0),
('scope', "MyTradeMeRead,MyTradeMeWrite")]
print('POST&%(url)s&%(params)s' % { 'url' : urllib.parse.quote_plus(url), 'params' : urllib.parse.quote_plus(urllib.parse.urlencode(params)) })
Output
POST&https%3A%2F%2Fsecure.trademe.co.nz%2FOauth%2FRequestToken&oauth_callback%3Dhttp%253A%252F%252Fwww.website-tm-access.co.nz%252Ftrademe-callback%26oauth_consumer_key%3DC74CD73FDBE37D29BDD21BAB54BC70E422%26oauth_nonce%3D7O3kEe%26oauth_signature_method%3DHMAC-SHA1%26oauth_timestamp%3D1285532322%26oauth_version%3D1.0%26scope%3DMyTradeMeRead%252CMyTradeMeWrite