Develop Reference

Using Solr-Docker with python return a wrong results

I have a flask app which runs in a Docker container and I wanted to use Solr with it for indexing and searching, so I built a container for Solr using the Solr official image and used it with my app using docker-compose.
In the app I have multiple types of objects that I want to index for example type1 and type2 and each type has specific fields, so I got in Solr, documents that have different fields, such as doc1 could have field1 and field2, and doc2 could have field3, field4 and field5, and each document has a field called type to specify its type.
I have two types of search first one is searching for documents of a specific type and this is an example URL of it which is used with requests Python package:
response = requests.get("http://solr:8983/solr/myCollection/select?q=*val*&defType=edismax&fq=type:type1&qf=field1^2&qf=field2^1")
, and the other is overall search so I search for documents of all types, and here is its URL example:
response = requests.get("http://solr:8983/solr/myCollection/select?q=*val*&defType=edismax&fq=type:type1||type2&qf=field1^1&qf=field2^1&qf=field3^1&qf=field4^1&qf=field1^1")
I have two problems with my work:
I don't get the result that I expected when I run some queries.
some fields have values with special characters like (z=x+y*f) and when I try to escape these special characters by '\' it doesn't work.
So, is the queries that I wrote have something wrong and is there any article or tutorial that could help me because I searched a lot in the documentation and the internet but I couldn't find I way to solve my problems.
Note: I didn't change the schema file I let it as default.

I've solved the problems by using the tokenizers and filters in indexing and querying.
You can use them by the Client API that Solr provide.
Here is an example of JSON data to add tokenizers and filters to a field type:
{
"replace-field-type": {
"name": "field_name",
"class": "solr.TextField",
"multiValued": True,
"indexAnalyzer": {
"tokenizer": {
"class": "solr.LowerCaseTokenizerFactory"
},
"filters": [
{
"class": "solr.LowerCaseFilterFactory"
}
]
},
"queryAnalyzer": {
"tokenizer": {
"class": "solr.WhitespaceTokenizerFactory",
"rule": "java"
},
"filters": [
{
"class": "solr.LowerCaseFilterFactory"
}
]
}
}
}

How to validate JSON request body before sending PUT request in python

It's when I send a PUT request to my API endpoint from python with a JSON request body I receive empty request body, because sometimes It's containing special characters which is not supported by JSON.
How can I sanitize my JSON before sending my request?
I've tried with stringify and parsing json before I sent my request!
profile = json.loads(json.dumps(profile))
My example invalid json is:
{
"url": "https://www.example.com/edmund-chand/",
"name": "Edmund Chand",
"current_location": "FrankfurtAmMainArea, Germany",
"education": [],
"skills": []
}
and My expected validated json should be:
{
"url": "https://www.example.com/edmund-chand/",
"name": "Edmund Chand",
"current_location": "Frankfurt Am Main Area, Germany",
"education": [],
"skills": []
}

If you're looking for something quick to sanitize json data for limited fields i.e. current_location, you can try something like the following below:
def sanitize(profile):
profile['current_location'] = ', '.join([val.strip() for val in profile['current_location'].split(',')])
return profile
profile = sanitize(profile)
The idea here is that you would write code to sanitize each bits in that function and send it your api or throw exception if invalid etc.
For more robust validation, you can consider using jsonschema package. More details here.
With that package you can validate strings and json schema more flexibly.
Example taken from the package readme:
from jsonschema import validate
# A sample schema, like what we'd get from json.load()
schema = {
"type" : "object",
"properties" : {
"url" : {"type" : "string", "format":"uri"},
"current_location" : {"type" : "string", "maxLength":25, "pattern": "your_regex_pattern"},
},
}
# If no exception is raised by validate(), the instance is valid.
validate(instance=profile, schema=schema)
You can find more infor and types of available validation for strings here.

Thank you #Rithin for your solution but that one seems more coupled with one field of the whole JSON.
I found a solution to replace it with below example code which works for any field:
profile = json.loads(json.dumps(profile).replace("\t", " "))

How do you implement custom response?

I found that the DRF's return value could be various upon different occasions. So I want to make sure all my JSON return have "code" "message" or other values nested inside in order to keep consistency of my APIs.
For example:
Success
{"code": 1, "status": "success", "message": "", "data": [{"id": 1, "name": "John Doe", "email": "johndoe#gmail.com"}]}
Error
{"code": -1, "status": "error", "message":"Something went wrong","data": [] }
The return will always have "code" "status" "message" "data" inside whatever the result would become.
After looked up on Google but couldn't find any work-around over DRF. So I suppose everybody is redefining the APIViews or Mixins (get put post etc. ) to control the response. But I am not very sure if the return should be that widely different without a certain pattern. Or is it cool that DRF's JSON response could be directly adopted as a production case?
Hope to have some advice from you guys.
Thanks.

You can use the status code of the HTTP to indicate the errors and success and you dont have to explicitly specify the error codes and response codes. In DRF status codes are defined in the module rest_framework.status
and you should use them and form the response accordingly.

400 Error while trying to POST to JIRA issue

I am trying to set the 'transition' property in a JIRA issue from whatever it is, to completed(which according to the doc is 10000). According to the documentation, this error is 'If there is no transition specified.'
Also I have used ?expand=transitions.fields to verify that 10000 is for complete.
using these docs
https://docs.atlassian.com/jira/REST/latest/#api/2/issue-doTransition
https://jira.atlassian.com/plugins/servlet/restbrowser#/resource/api-2-issue-issueidorkey-transitions/POST
Here is my request
url = 'http://MYURL/rest/api/2/issue/ISSUE-ID/transitions'
payload1 = open('data3.json', 'r').read()
payload = json.loads(payload1)
textFile = requests.post(url, auth=('username', 'password'), json=payload)
The contents on my data3.json file are
{
"transition": 10000
}
edit: I also changed my JSON to this and I get a 500 error
{
"transition": {
"id": "10000"
}
}
The error I get
{"errorMessages":["Can not instantiate value of type [simple type,classcom.atlassian.jira.rest.v2.issue.TransitionBean] from JSON integral number;no single-int-arg constructor/factory method (through reference chain:com.atlassian.jira.rest.v2.issue.IssueUpdateBean[\"transition\"])"]}400
I'm pretty confident that my issue is in my json file since I have used GET in the code above this snippit multiple times, but I could be wrong.
Possible cause - https://jira.atlassian.com/browse/JRA-32132

I believe the issue I was having was a process flow one. I cannot jump right from my issue being opened, to 'completed'. However, I can go from the issue being created to 'Done'.
{
"transition": {
"name": "Done",
"id": "151"
}
}
As this does what I need, I will use it. If I find how to make ticket complete I will post back.
Also, I think the fact we customize our JIRA lead to my getting 'Completed' as a valid transition even though it wasn't.

Yes, you're right that the JSON is wrong, it's not even a valid json since the value is not a number, string, object, or array. The doc says:
The fields that can be set on transtion, in either the fields
parameter or the update parameter can be determined using the
/rest/api/2/issue/{issueIdOrKey}/transitions?expand=transitions.fields
resource.
So you need to do a get request on /rest/api/2/issue/{issueIdOrKey}/transitions?expand=transitions.fields to get the list of possible values and then set that in the json
{
"transition": {
"id" : "an_id_from_response"
}
}

adding new dict to json output python

Very new to json so please forgive if I am using the wrong terms here. Any ways I am trying to create a json file every x minutes with updated twitter info. Now I know i could just use the twitter api and grab what I need but I am wanting to mess around a bit. My problem is getting a new key/dict? or what ever it is called for each new item added in a for statement.
What im trying to get
[
{
"name": "thename",
"date": "thedate",
"text": "thetext"
}, <--- trying to get that little comma
{
"name": "thename",
"date": "thedate",
"text": "thetext"
}
]
Now i am getting the data that i want and all but not that comma. It outputs it all but not like it should be with that little one character thing that makes it valid.
[
{
"name": "thename",
"date": "thedate",
"text": "thetext"
} <--- I get this instead for each new object
{
"name": "thename",
"date": "thedate",
"text": "thetext"
}
]
Here is just a snippet of the code as the rest should be explanatory since its just Twitter oauth stuff.
for player in users:
statuses = api.GetUserTimeline(screen_name=player, count=10)
print "["
for s in statuses:
print json.dumps('timestamp': s.created_at,'username': s.user.name,'status': s.text)
print "]"
Also is there a better way to do the [ ] at the start and end because I know that is ugly and way un-proper i bet XD . Like I said newbie on json/python stuff but its a good learning experience.

Instead of trying to build the JSON array yourself, you should just let the JSON encoder do that for you too. To do that, you just need to pass a list of objects to json.dumps. So instead of printing each JSON object on its own, just collect them in a list, and dump that:
allstatuses = []
for player in users:
statuses = api.GetUserTimeline(screen_name=player, count=10)
for s in statuses:
allstatuses.append({'timestamp': s.created_at, 'username': s.user.name, 'status': s.text})
print(json.dumps(allstatuses))