I am trying to write some test cases for some code I've developed using Elasticsearch and Django. The concept is straightforward - I just want to test a get request, which will be an Elasticsearch query. However, I am constructing the query as a nested dict. When I pass the nested dict to the Client object in the test script it gets passed through Django until it ends up at the urlencode function which doesn't look like it can handle nested dicts only MultivalueDicts. Any suggestions or solutions? I don't want to use any additional packages as I don't want to depend on potentially non-supported packages for this application.
Generic Code:
class MyViewTest(TestCase):
es_connection = elasticsearch.Elasticsearch("localhost:9200")
def test_es_query(self):
client = Client()
query = {
"query": {
"term": {
"city": "some city"
}
}
}
response = client.get("", query)
print(response)
Link for urlencode function: urlencode Django
The issue is clearly at the conditional statement when the urlencode function checks if the dictionary value is a str or bytes object. If it isn't it creates a generator object which can never access the nested portions of the dictionary.
EDIT: 07/25/2018
So I was able to come up with a temporary work around to at least run the test. However, it is ugly and I feel like there must be a better way. The first thing I tried was specifying the content_type and converting the dict to a json string first. However, Django still kicked back and error in the urlencode function.
class MyViewTest(TestCase):
es_connection = elasticsearch.Elasticsearch("localhost:9200")
def test_es_query(self):
client = Client()
query = {
"query": {
"term": {
"city": "some city"
}
}
}
response = client.get("", data=json.dumps(query), content_type="application/json")
print(response)
So instead I had to do:
class MyViewTest(TestCase):
es_connection = elasticsearch.Elasticsearch("localhost:9200")
def test_es_query(self):
client = Client()
query = {
"query": {
"term": {
"city": "some city"
}
}
}
query = json.dumps(query)
response = client.get("", data={"q": query}, content_type="application/json")
print(response)
This let me send the HttpRequest to my View and parse it back out using:
json.loads(request.GET["q"])
Then I was able to successfully get the requested data from Elasticsearch and return it as an HttpResponse. I feel like in Django though there has to be a way to just pass a json formatted string directly to the Client object's get function. I thought specifying the content_type as application/json would work but it still calls the urlencode function. Any ideas? I really don't want to implement this current system into production.
Related
I would like to load a JSON object from mongo query, access part of the object and then have this serialised and sent to the web client. I can get the web client to work not the python side. Any examples would be great..
The request is a serialised JSON object like this
{"page":"home"}
the mongo query results would be like this, but each "home", "hello" etc has many more keys and values in them.
{ "_id" : ObjectId("5a8838782430d5ae7526e9eb"), "home" : { "next_page" : "/stats"}, "hello" : { "next_page" : "/home" }, "stats" : { "next_page" : "/hello" } }
The views.py code includes this, which I provide as pseudocode.
#view_config(route_name='nav_route', renderer='json')
def content(request):
page_name=request.page #that is, I want page_name="home"
mongo_json=mongo_query(woof) #load and parse the above query
page_nav=mongo_json[page_name] #get the part for "home"
return {'nav': page_nav}
UPDATE: With feedback this now works - cheers!
#view_config(route_name='rachel_content', renderer='json')
def rachel_content(request):
page=request.json_body['page']
result=db.rachel.find_one()
return { 'rachel' : json.dumps(result[page])}
(Rachel is named after the replicant)
I would suggest you simply need to dump the JSON data. I tend to default to using the simplejson package so would add this to the top of the file.
import simplejson as json
Then simply tell it to dump your dictionary object to a JSON string by using the dumps function:
return {'nav': json.dumps(page_nav)}
I believe you can do the same with the built-in JSON library if you don't want/can't install simplejson
I am using the following filters in Postman to make a POST request in a Web API but I am unable to make a simple POST request in Python with the requests library.
First, I am sending a POST request to this URL (http://10.61.202.98:8081/T/a/api/rows/cat/ect/tickets) with the following filters in Postman applied to the Body, with the raw and JSON(application/json) options selected.
Filters in Postman
{
"filter": {
"filters": [
{
"field": "RCA_Assigned_Date",
"operator": "gte",
"value": "2017-05-31 00:00:00"
},
{
"field": "RCA_Assigned_Date",
"operator": "lte",
"value": "2017-06-04 00:00:00"
},
{
"field": "T_Subcategory",
"operator": "neq",
"value": "Temporary Degradation"
},
{
"field": "Issue_Status",
"operator": "neq",
"value": "Queued"
}],
"logic": "and"
}
}
The database where the data is stored is Cassandra and according to the following links Cassandra not equal operator, Cassandra OR operator,
Cassandra Between order by operators, Cassandra does not support the NOT EQUAL TO, OR, BETWEEN operators, so there is no way I can filter the URL with these operators except with AND.
Second, I am using the following code to apply a simple filter with the requests library.
import requests
payload = {'field':'T_Subcategory','operator':'neq','value':'Temporary Degradation'}
url = requests.post("http://10.61.202.98:8081/T/a/api/rows/cat/ect/tickets",data=payload)
But what I've got is the complete data of tickets instead of only those that are not temporary degradation.
Third, the system is actually working but we are experiencing a delay of 2-3 mins to see the data. The logic goes as follows: We have 8 users and we want to see all the tickets per user that are not temporary degradation, then we do:
def get_json():
if user_name == "user 001":
with urllib.request.urlopen(
"http://10.61.202.98:8081/T/a/api/rows/cat/ect/tickets?user_name=user&001",timeout=15) as url:
complete_data = json.loads(url.read().decode())
elif user_name == "user 002":
with urllib.request.urlopen(
"http://10.61.202.98:8081/T/a/api/rows/cat/ect/tickets?user_name=user&002",timeout=15) as url:
complete_data = json.loads(url.read().decode())
return complete_data
def get_tickets_not_temp_degradation(start_date,end_date,complete_):
return Counter([k['user_name'] for k in complete_data if start_date < dateutil.parser.parse(k.get('DateTime')) < end_date and k['T_subcategory'] != 'Temporary Degradation'])
Basically, we get the whole set of tickets from the current and last year, then we let Python to filter the complete set by user and so far there are only 10 users which means that this process is repeated 10 times and makes me no surprise to discover why we get the delay...
My questions is how can I fix this problem of the requests library? I am using the following link Requests library documentation as a tutorial to make it working but it just seems that my payload is not being read.
Your Postman request is a JSON body. Just reproduce that same body in Python. Your Python code is not sending JSON, nor is it sending the same data as your Postman sample.
For starters, sending a dictionary via the data arguments encodes that dictionary to application/x-www-form-urlencoded form, not JSON. Secondly, you appear to be sending a single filter.
The following code replicates your Postman post exactly:
import requests
filters = {"filter": {
"filters": [{
"field": "RCA_Assigned_Date",
"operator": "gte",
"value": "2017-05-31 00:00:00"
}, {
"field": "RCA_Assigned_Date",
"operator": "lte",
"value": "2017-06-04 00:00:00"
}, {
"field": "T_Subcategory",
"operator": "neq",
"value": "Temporary Degradation"
}, {
"field": "Issue_Status",
"operator": "neq",
"value": "Queued"
}],
"logic": "and"
}}
url = "http://10.61.202.98:8081/T/a/api/rows/cat/ect/tickets"
response = requests.post(url, json=filters)
Note that filters is a Python data structure here, and that it is passed to the json keyword argument. Using the latter does two things:
Encode the Python data structure to JSON (producing the exact same JSON value as your raw Postman body value).
Set the Content-Type header to application/json (as you did in your Postman configuration by picking the JSON option in the dropdown menu after picking raw for the body).
requests is otherwise just an HTTP API, it can't make Cassandra do any more than any other HTTP library. The urllib.request.urlopen code sends GET requests, and are trivially translated to requests with:
def get_json():
url = "http://10.61.202.98:8081/T/a/api/rows/cat/ect/tickets"
response = requests.get(url, params={'user_name': user}, timeout=15)
return response.json()
I removed the if branching and replaced that with using the params argument, which translates a dictionary of key-value pairs to a correctly encoded URL query (passing in the user name as the user_name key).
Note the json() call on the response; this takes care of decoding JSON data coming back from the server. This still takes long, you are not filtering the Cassandra data much here.
I would recommend using the json attribute instead of data. It handles the dumping for you.
import requests
data = {'user_name':'user&001'}
headers = {'Content-Type': 'application/json', 'Accept': 'application/json'}
url = "http://10.61.202.98:8081/T/a/api/rows/cat/ect/tickets/"
r = requests.post(url, headers=headers, json=data)
Update, answer for question 3. Is there a reason you are using urllib? I’d use python requests as well for this request.
import requests
def get_json():
r = requests.get("http://10.61.202.98:8081/T/a/api/rows/cat/ect/tickets”, params={"user_name": user_name.replace(" ", "&")})
return r.json
# not sure what you’re doing here, more context/code example would help
def get_tickets_not_temp_degradation(start_date, end_date, complete_):
return Counter([k['user_name'] for k in complete_data if start_date < dateutil.parser.parse(k.get('DateTime')) < end_date and k['T_subcategory'] != 'Temporary Degradation'])
Also, is the username really supposed to be user+001 and not user&001 or user 001?
I think, you can use requests library as follows:
import requests
import json
payload = {'field':'T_Subcategory','operator':'neq','value':'Temporary Degradation'}
url = requests.post("http://10.61.202.98:8081/T/a/api/rows/cat/ect/tickets",data=json.dumps(payload))
You are sending user in url, use it through post, but its depend upon how end points are implemented. You can try the below code :
import requests
from json import dumps
data = {'user_name':'user&001'}
headers = {'Content-Type': 'application/json', 'Accept': 'application/json'}
url = "http://10.61.202.98:8081/T/a/api/rows/cat/ect/tickets/"
r = requests.post(url, headers=headers, data=dumps(data))
I have a json file i need to retrieve data from it and then insert it into another API.
WorkFlow: External Feed -> Parsing -> Insert into to Another API
Coding Part:
Function Defined in a Parsing class.
def parsed_items(self):
self.get_response()
items = self.soup.find_all('item')
self.payload = []
for item in items:
self.payload.append({'title': item.find('title').text,
'description': item.find('description').text,
'status': '3'
}
)
return self.payload
Function Defined in main class to get values of this function.
for items in parser.parsed_items():
response2 = requests.request('POST', settings.BASE_URL,
json= (items['title'], items['description'], items['status']),
headers=headers())
Sample of JSON:
{ Data:
{
"title": "ipsum",
"description": "lorem"
}
{
"title": "ipsum1",
"description": "lorem1"
}
{
"title": "ipsum2",
"description": "lorem2"
}
{
"title": "ipsum3",
"description": "lorem3"
}
}
Error:
{"errors":[{"status":"400","source":"non_field_errors","detail":"Invalid data. Expected a dictionary, but got list."}]}
I need to know ?
Q1: What is the best way to handle such scenarios? Please refer any tutorial which can be helpful in this scenario.
Q2: How to retrieve list of values from payload ? Any example that you can refer ?
Q3: How can the list which is returned back by parse_item() be converted into dictionary and passed into request for value of json parameter.
I need to fetch these values of "title" and "description" from JSON and POST them in local API. (Note: Local API is authenticated successfully)
First post here, so ya know, be nice?
I'm setting up a dashboard in Dashing (http://dashing.io/) using a JSON feed on a server, which looks like:
{
"error":0,
"message_of_the_day":"Welcome!",
"message_of_the_day_hash":"a1234567890123456789012345678901",
"metrics":{
"daily":{
"metric1":"1m 30s",
"metric2":160
},
"monthly":{
"metric1":"10m 30s",
"metric2":"3803"
}
},
I have been experimenting with grabbing the data from the feed, and have managed to do so by Python with no issues:
import json
import urllib2
data = {
'region': "Europe"
}
req = urllib2.Request('http://192.168.1.2/info/handlers/handler.php')
req.add_header('Content-Type', 'application/json')
response = urllib2.urlopen(req, json.dumps(data))
print response.read()
However I haven't yet been successful, and get numerous errors in Ruby.
Would anyone be able to point me in the right direction in parsing this in Ruby?
My attempts to write a basic script, (keeping it simple and outside of Dashing) don't pull through any data.
#!/usr/bin/ruby
require 'httparty'
require 'json'
response = HTTParty.get("http://192.168.1.2/info/handlers/handler.php?region=Europe")
json = JSON.parse(response.body)
puts json
In python code you are sending a JSON and adding a header. I bet it makes sense to do that in ruby as well. The code below is untested, since I can’t test it, but it should lead you into the right direction:
#!/usr/bin/ruby
require 'httparty'
require 'json'
response = HTTParty.post(
"http://192.168.1.2/info/handlers/handler.php",
headers: {'Content-Type', 'application/json'},
query: { data: { 'region' => "Europe" } }
# or maybe query: { 'region' => "Europe" }
)
puts response.inspect
I am trying to set the 'transition' property in a JIRA issue from whatever it is, to completed(which according to the doc is 10000). According to the documentation, this error is 'If there is no transition specified.'
Also I have used ?expand=transitions.fields to verify that 10000 is for complete.
using these docs
https://docs.atlassian.com/jira/REST/latest/#api/2/issue-doTransition
https://jira.atlassian.com/plugins/servlet/restbrowser#/resource/api-2-issue-issueidorkey-transitions/POST
Here is my request
url = 'http://MYURL/rest/api/2/issue/ISSUE-ID/transitions'
payload1 = open('data3.json', 'r').read()
payload = json.loads(payload1)
textFile = requests.post(url, auth=('username', 'password'), json=payload)
The contents on my data3.json file are
{
"transition": 10000
}
edit: I also changed my JSON to this and I get a 500 error
{
"transition": {
"id": "10000"
}
}
The error I get
{"errorMessages":["Can not instantiate value of type [simple type,classcom.atlassian.jira.rest.v2.issue.TransitionBean] from JSON integral number;no single-int-arg constructor/factory method (through reference chain:com.atlassian.jira.rest.v2.issue.IssueUpdateBean[\"transition\"])"]}400
I'm pretty confident that my issue is in my json file since I have used GET in the code above this snippit multiple times, but I could be wrong.
Possible cause - https://jira.atlassian.com/browse/JRA-32132
I believe the issue I was having was a process flow one. I cannot jump right from my issue being opened, to 'completed'. However, I can go from the issue being created to 'Done'.
{
"transition": {
"name": "Done",
"id": "151"
}
}
As this does what I need, I will use it. If I find how to make ticket complete I will post back.
Also, I think the fact we customize our JIRA lead to my getting 'Completed' as a valid transition even though it wasn't.
Yes, you're right that the JSON is wrong, it's not even a valid json since the value is not a number, string, object, or array. The doc says:
The fields that can be set on transtion, in either the fields
parameter or the update parameter can be determined using the
/rest/api/2/issue/{issueIdOrKey}/transitions?expand=transitions.fields
resource.
So you need to do a get request on /rest/api/2/issue/{issueIdOrKey}/transitions?expand=transitions.fields to get the list of possible values and then set that in the json
{
"transition": {
"id" : "an_id_from_response"
}
}