Related
I kind of have two real questions. Both relate to this code:
import urllib
import requests
def query(q):
base_url = "https://api.duckduckgo.com/?q={}&format=json"
resp = requests.get(base_url.format(urllib.parse.quote(q)))
json = resp.json()
return json
One is this: When I query something like this: "US Presidents", I get back something like this:
{'Abstract': '', 'AbstractSource': '', 'AbstractText': '', 'AbstractURL': '', 'Answer': '', 'AnswerType': '', 'Definition': '', 'DefinitionSource': '', 'DefinitionURL': '', 'Entity': '', 'Heading': '', 'Image': '', 'ImageHeight': '', 'ImageIsLogo': '', 'ImageWidth': '', 'Infobox': '', 'Redirect': '', 'RelatedTopics': [], 'Results': [], 'Type': '', 'meta': {'attribution': None, 'blockgroup': None, 'created_date': '2021-03-24', 'description': 'testing', 'designer': None, 'dev_date': '2021-03-24', 'dev_milestone': 'development', 'developer': [{'name': 'zt', 'type': 'duck.co', 'url': 'https://duck.co/user/zt'}], 'example_query': '', 'id': 'just_another_test', 'is_stackexchange': 0, 'js_callback_name': 'another_test', 'live_date': None, 'maintainer': {'github': ''}, 'name': 'Just Another Test', 'perl_module': 'DDG::Lontail::AnotherTest', 'producer': None, 'production_state': 'offline', 'repo': 'fathead', 'signal_from': 'just_another_test', 'src_domain': 'how about there', 'src_id': None, 'src_name': 'hi there', 'src_options': {'directory': '', 'is_fanon': 0, 'is_mediawiki': 0, 'is_wikipedia': 0, 'language': '', 'min_abstract_length': None, 'skip_abstract': 0, 'skip_abstract_paren': 0, 'skip_icon': 0, 'skip_image_name': 0, 'skip_qr': '', 'src_info': '', 'src_skip': ''}, 'src_url': 'Hello there', 'status': None, 'tab': 'is this source', 'topic': [], 'unsafe': None}}
Basically, everything is empty. Even the Heading key, which I know was sent as "US Presidents" encoded into url form. This issue seems to affect all queries I send with a space in them. Even when I go to this url: "https://api.duckduckgo.com/?q=US%20Presidents&format=json&pretty=1" in a browser, all I get is a bunch of blank json keys.
My next question is this. When I send in something like this: "1+1", the json response's "Answer" key is this:
{'from': 'calculator', 'id': 'calculator', 'name': 'Calculator', 'result': '', 'signal': 'high', 'templates': {'group': 'base', 'options': {'content': 'DDH.calculator.content'}}}
Everything else seems to be correct, but the 'result' should be '2', should it not? The entire rest of the json seems to be correct, including all 'RelatedTopics'
Any help with this would be greatly appreciated.
Basically duckduckgo api is not a real search engine. It is just a dictionary. So try US%20President instead of US%20presidents and you'll get an answer. For encoding you can use blanks, but if it's not a fixed term I would prefer the plus-sign. You can do this by using urllib.parse.quote_plus()
With calculation you're right. But I see absolutely no use case to use a calculus-api within a python code. It is like using trampoline to travel to the moon if there is a rocket available. And maybe they see it the same and do not offer calc services in their api?!
I have a web aplication where I am able to create extra proprities (keys and values) and pass them in a JSON output. Here is the final format of what is passed from the web application.
{
'abendCode': '',
'action': 'CREATE',
'assignGroup': '[Team]',
'component': '',
'description': 'test alert',
'entered': '',
'logicalName': '',
'pageGroups': '["[Team]"]',
'priority': '1',
'RACF': '',
'sendPage': 'false',
'sysProdName': 'MANUAL',
'transactionId': '',
'type': 'RESPONDER'
}
I have created a python method to build out a json payload to create a ticket leveraging an api call to POST to our tickeiting system. In the method I do some formatting to strip unnessary characters like ["[Team]"] and replace it with ['Team'] (this is in the pageGroups value). I than remove the square brackets from 'assignGroup': '[Team]' to show 'assignGroup': 'Team'
When I run the code it get a 500, which I am 90% sure it is because my JSON is not properly formatted.
When I run the following in Postman it will create a ticket.
{
"type": "RESPONDER",
"action": "CREATE",
"logicalName": "",
"component": "",
"entered": "",
"racfId": "",
"description": "incident creation",
"priority": "1",
"assignGroup": "TEAM",
"transactionId": "",
"abendCode": "",
"sysProdName": "MANUAL",
"sendPage": "false",
"pageGroups": ["TEAM"]
}
Here is my code with some things changed for removed for sensitivity.
import requests
import json
import ast
from datetime import datetime
import re
# Date and time stamp for logging
now = datetime.now()
# Format: month day year day of the week HH:SS:00
# 03/18/2021 Tuesday 07:18:25
dt_string = now.strftime("%m/%d/%Y %A %H:%M:%S ")
ogArray = {
'abendCode': '',
'action': 'CREATE',
'assignGroup': '[Team]',
'component': '',
'description': 'test alert',
'entered': '',
'logicalName': '',
'pageGroups': '["[Team]"]',
'priority': '1',
'RACF': '',
'sendPage': 'false',
'sysProdName': 'MANUAL',
'transactionId': '',
'type': 'RESPONDER'
}
action = "create"
alertId = "ab40f5e4-867c-b80b-a11aeea57731-1617302963378"
Team = "Team"
tcisApiToken = 'api token'
tcisApi_url_base = 'https://api.com/events/v1/'
tcisHeaders = {'Content-Type': 'applicarion/json', 'Authorization': 'Basic {0}'.format(tcisApiToken)}
def createTicket(ogArray, action, alertId, Team):
api_url_base = Api_url_base
headers = Headers
time = dt_string
api_url = '{0}events'.format(api_url_base)
print(time + " Alert Id: " + str(alertId))
data = ogArray
# do some string processing
print("---Pre process ogArray" + str(data))
data['assignGroup'] = (data['assignGroup'][1:-1])
data = re.sub("\'\[\"\[", '[\'', str(data))
data = re.sub("\]\"\]\'", '\']', str(data))
# Having an Issue here, with data[description] with the key ValueError commented it out for now
# data['description'] = data['description'] + str(" [alertId:" + alertId + "]")
print("---Print after format" +str(data))
data = json.dumps(data)
print("---After dumps" + str(data))
data = json.loads(data)
print("---After loads" + str(data))
print(time + " Action: " + str(action))
print(time + "JSON payload: " + str(data))
# Build JSON POST request
response = requests.post(url=api_url, data=data, headers=headers)
print(time + 'API response = ' + str(response.status_code))
print(time + "Formatted JSON data :" + str(data))
print(response)
response_url = json.loads(response.content.decode('utf-8'))
if response.status_code == 200 and 'returnCode' in response_url:
eventId = response_url["eventId"]
incidentId = response_url["incidentId"]
print(time + str(response_url))
elif response.status_code == 200 and 'returnCode' not in response_url:
print(time + str(response_url))
print(time + "Failed to create ticket. It is possible the team does not exist or payload is formatted incorrectly. Contact admin for support. See logs for more info.")
else:
note = 'Create Ticket Failed.'
print(time + str(note))
return None
if __name__ == '__main__':
createTicket = createTicket(ogArray, action, alertId, Team)
Here is my output from the original code I am running:
04/02/2021 Friday 16:31:57 Alert Id: ab40f5e4-867c-46e4-b80b-a11aeea57731-1617302963378
---Pre process ogArray{'abendCode': '', 'action': 'CREATE', 'assignGroup': '[APP-SUPPORT]', 'component': '', 'description': 'test alert', 'entered': '', 'logicalName': '', 'pageGroups': '["[APP-SUPPORT]"]', 'priority': '1', 'RACF': '', 'sendPage': 'false', 'sysProdName': 'MANUAL', 'transactionId': '', 'type': 'RESPONDER'}
---Print after format{'abendCode': '', 'action': 'CREATE', 'assignGroup': 'APP-SUPPORT', 'component': '', 'description': 'test alert', 'entered': '', 'logicalName': '', 'pageGroups': ['APP-SUPPORT'], 'priority': '1', 'RACF': '', 'sendPage': 'false', 'sysProdName': 'MANUAL', 'transactionId': '', 'type': 'RESPONDER'}
---After dumps"{'abendCode': '', 'action': 'CREATE', 'assignGroup': 'APP-SUPPORT', 'component': '', 'description': 'test alert', 'entered': '', 'logicalName': '', 'pageGroups': ['APP-SUPPORT'], 'priority': '1', 'RACF': '', 'sendPage': 'false', 'sysProdName': 'MANUAL', 'transactionId': '', 'type': 'RESPONDER'}"
---After loads{'abendCode': '', 'action': 'CREATE', 'assignGroup': 'APP-SUPPORT', 'component': '', 'description': 'test alert', 'entered': '', 'logicalName': '', 'pageGroups': ['APP-SUPPORT'], 'priority': '1', 'RACF': '', 'sendPage': 'false', 'sysProdName': 'MANUAL', 'transactionId': '', 'type': 'RESPONDER'}
04/02/2021 Friday 16:31:57 Action: create
04/02/2021 Friday 16:31:57 JSON payload: {'abendCode': '', 'action': 'CREATE', 'assignGroup': 'APP-SUPPORT', 'component': '', 'description': 'test alert', 'entered': '', 'logicalName': '', 'pageGroups': ['APP-SUPPORT'], 'priority': '1', 'RACF': '', 'sendPage': 'false', 'sysProdName': 'MANUAL', 'transactionId': '', 'type': 'RESPONDER'}
04/02/2021 Friday 16:31:57 API response = 500
04/02/2021 Friday 16:31:57 Formatted JSON data :{'abendCode': '', 'action': 'CREATE', 'assignGroup': 'TCIS-APP-SUPPORT', 'component': '', 'description': 'test alert', 'entered': '', 'logicalName': '', 'pageGroups': ['APP-SUPPORT'], 'priority': '1', 'RACF': '', 'sendPage': 'false', 'sysProdName': 'MANUAL', 'transactionId': '', 'type': 'RESPONDER'}
<Response [500]>
createTicket = createTicket(ogArray, action, alertId, Team)
File "D:\AtomProjects\pingtest\manualTicketCreate.py", line 70, in createTicket
response_url = json.loads(response.content.decode('utf-8'))
File "D:\Python39\lib\json\__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "D:\Python39\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "D:\Python39\lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
[Finished in 0.931s]
I am not sure if there needs to be a dumps, since I am formatting the JSON strings before the dumps. I have been all over the net to solve this but I am stumped at this point. I have also looked at the error outputs and I do not understand them, even when searching for them on the web. Any help is appreciated.
Here's what's going wrong -
You are calling json.loads right after json.dumps . This now deserializes your payload back into a regular python dictionary.
You want to send serialized JSON object as your payload.
Here are your options:
OPTION 1:
data = json.dumps(data)
print("---After dumps" + str(data))
response = requests.post(url=api_url, data=data, headers=headers)
OPTION 2:
requests.post supports a json parameter, where you can use your python dict directly and it will serialize it internally.
# Note that json.dumps is commented out in this option.
# data = json.dumps(data)
response = requests.post(url=api_url, json=data, headers=headers)
Problem:
I am having trouble pulling some info off ups. The city and state in the shipToAddress section.
Below is the data in a easy to read format that i am pulling from ups website with requests:
Data:
data = {
'statusCode': '200',
'statusText': 'Successful',
'isLoggedInUser': False,
'trackedDateTime': '04/16/2019 1:33 P.M. EST',
'isBcdnMultiView': False,
'trackDetails': [{
'errorCode': None,
'errorText': None,
'requestedTrackingNumber': '1Z3774E8YN99957400',
'trackingNumber': '1Z3774E8YN99957400',
'isMobileDevice': False,
'packageStatus': 'Loaded on Delivery Vehicle',
'packageStatusType': 'I',
'packageStatusCode': '072',
'progressBarType': 'InTransit',
'progressBarPercentage': '90',
'simplifiedText': '',
'scheduledDeliveryDayCMSKey': 'cms.stapp.tue',
'scheduledDeliveryDate': '04/16/2019',
'noEstimatedDeliveryDateLabel': None,
'scheduledDeliveryTime': 'cms.stapp.eod',
'scheduledDeliveryTimeEODLabel': 'cms.stapp.eod',
'packageCommitedTime': '',
'endOfDayResCMSKey': None,
'deliveredDayCMSKey': '',
'deliveredDate': '',
'deliveredTime': '',
'receivedBy': '',
'leaveAt': None,
'leftAt': '',
'shipToAddress': {
'streetAddress1': '',
'streetAddress2': '',
'streetAddress3': '',
'city': 'OCEAN',
'state': 'NJ',
'province': None,
'country': 'US',
'zipCode': '',
'companyName': '',
'attentionName': '',
'isAddressCorrected': False,
'isReturnAddress': False,
'isHoldAddress': False,
}}]}
Code:
data = response.text
addressinfo =json.loads(data)['trackDetails']['shipToAddress']
for entry in addressinfo:
city = (entry['city'])
state = (entry['state'])
country = (entry['country'])
My Expected Results:
city = 'Ocean'
state = 'NJ'
etc
this is error:
addressinfo =json.loads(data2)['trackDetails']['shipToAddress']
TypeError: list indices must be integers or slices, not str
Note the format of your JSON:
'trackDetails': [{
...
'shipToAddress': {...}
}]
The dict you're trying to index into is actually contained inside of a list (note the square brackets). The proper way to access the shipToAddress field would be to do this:
addressinfo = json.loads(data2)['trackDetails'][0]['shipToAddress']
^^^
instead of what you were doing.
When you return data = response.text you should instead do data = response.json() since it is a json. This will allow you to access it like a json. Instead you are converting it to a string with .text and then attempting to load it back in which is not necessary.
Then access city:
city = data['trackDetails'][0]['shipToAddress']['city']
state = data['trackDetails'][0]['shipToAddress']['state']
Context:
We have a requirement where I need to capture a specific transaction using KafkaConsumer library.I use a unique session as key which I obtain from a different tool(let's call as Feeder tool) to capture the transaction.
I run my code immediately once the session is obtained from Feeder.
Problem
I am able to fetch multiple records from Kafka but I don't see the record which I'm trying to filter using the session.
Code
from kafka import KafkaConsumer
import json
SESSION = 'sessionID'
def consumeRecords(topic,group,bootstrapserver,auto_offset_reset,mySession,auto_commit_boolean):
consumer = KafkaConsumer(topic,group_id=group,bootstrap_servers=bootstrapserver,auto_offset_reset=auto_offset_reset,enable_auto_commit=auto_commit_boolean)
consumer.topics()
consumer.seek_to_beginning()
try:
while True:
print("CALLING POLL")
records = consumer.poll(timeout_ms=1000)
print("RETURNED FROM POLL")
if records:
for consumedRecord in records.values():
for msg in consumedRecord:
json_data = json.loads(msg.value.decode('utf-8'))
#print(json_data)
if 'alias' in json_data.keys() and json_data['alias']=='myServer':
current_session = json_data[SESSION]
print("SESSION is :" , current_session)
if mySession == current_session :
print('My record is ', json_data)
except Exception as e:
print("Unable to find any related sessions")
print(e)
if __name__ == '__main__':
KAFKA_TOPIC = 'e-commerce.request'
KAFKA_GROUP = 'test'
KAFKA_BROKERS = ['ABC.net:9092', 'DEF:9092']
auto_commit = False
consumeRecords(KAFKA_TOPIC,KAFKA_GROUP,KAFKA_BROKERS,'earliest','38l87jondkvnefNW886QMTWVcN6S4my5Y-No167ZzqF',auto_commit)
I'm supposed to print the following json data consumed from Kafka , but my code doesn't fetch this record and hence prints nothing and runs for infinite time
{'Type': 'request', 'requestID': '2018100819564659-5', 'payload': {'timing': {'startTime': '20181008195624322', 'total': '0.063', 'totalActions': '0', 'jsp': '0.063'}, 'user': {'orgID': '', 'userID': '', 'newComer': 'FALSE', 'helpdeskUserID': '', 'helpdeskUserOrgID': '', 'travelerID': ''}, 'version': '1.0.0', 'client': {'referrer': '', 'ip': ''}, 'url': {'parameters': {'JSESSIONID': '38l87jondkvnefNW886QMTWVcN6S4my5Y-No167ZzqF!430553578!-1652153437'}, 'baseUrl': 'http://server_url', 'path': 'DUMMY', 'method': 'POST'}, 'actions': [{'cumulTiming': '0', 'name': 'OverrideServlet', 'isChained': 'FALSE', 'features': '[GlobalTimeSpent = 0][PRE_RULES = 0][POST_RULES = 0]', 'chainParent': ''}], 'context': {'sessionSize': '12|12', 'fatalError': 'FALSE', 'requestType': 'XML', 'error': [], 'requestedSessionID': '', 'templateName': ''}}, 'Hostname': 'DummyAgain', 'sessionID': '38l87jondkvnefNW886QMTWVcN6S4my5Y-No167ZzqF', 'Site': 'ABCDEFGH', 'ClientId': 1234551515439, 'Epoch': 1539028584353, 'IP': 'A.B.C.D', 'alias': 'myServer', 'SeqNb': 21845, 'Source': 'eCommerce'}
I'm unable to parse JSON. My JSON snippet returned from requests.post response :-
{'result': {'parent': '', 'reason': '', 'made_sla': 'true', 'backout_plan': '', 'watch_list': '', 'upon_reject': 'cancel', 'sys_updated_on': '2018-08-22 11:16:09', 'type': 'Comprehensive', 'conflict_status': 'Not Run', 'approval_history': '', 'number': 'CHG0030006', 'test_plan': '', 'cab_delegate': '', 'sys_updated_by': 'admin', 'opened_by': {'link': 'https://dev65345.service-now.com/api/now/table/sys_user/6816f79cc0a8016401c5a33be04be441', 'value': '6816f79cc0a8016401c5a33be04be441'}, 'user_input': '', 'requested_by_date': '', 'sys_created_on': '2018-08-22 11:16:09', 'sys_domain': {'link': 'https://dev65345.service-now.com/api/now/table/sys_user_group/global', 'value': 'global'}, 'state': '-5', 'sys_created_by': 'admin', 'knowledge': 'false', 'order': '', 'phase': 'requested', 'closed_at': '', 'cmdb_ci': '', 'delivery_plan': '', 'impact': '3', 'active': 'true', 'review_comments': '', 'work_notes_list': '', 'business_service': '', 'priority': '4', 'sys_domain_path': '/', 'time_worked': '', 'cab_recommendation': '', 'expected_start': '', 'production_system': 'false', 'opened_at': '2018-08-22 11:16:09', 'review_date': '', 'business_duration': '', 'group_list': '', 'requested_by': {'link': 'https://dev6345.service-now.com/api/now/table/sys_user/user1', 'value': 'user1'}, 'work_end': '', 'change_plan': '', 'phase_state': 'open', 'approval_set': '', 'cab_date': '', 'work_notes': '', 'implementation_plan': '', 'end_date': '', 'short_description': '', 'close_code': '', 'correlation_display': '', 'delivery_task': '', 'work_start': '', 'assignment_group': {'link': 'https://dev65345.service-now.com/api/now/table/sys_user_group/testgroup', 'value': 'testgroup'}, 'additional_assignee_list': '', 'outside_maintenance_schedule': 'false', 'description': '', 'on_hold_reason': '', 'calendar_duration': '', 'std_change_producer_version': '', 'close_notes': '', 'sys_class_name': 'change_request', 'closed_by': '', 'follow_up': '', 'sys_id': '436eda82db4023008e357a61399619ee', 'contact_type': '', 'cab_required': 'false', 'urgency': '3', 'scope': '3', 'company': '', 'justification': '', 'reassignment_count': '0', 'review_status': '', 'activity_due': '', 'assigned_to': '', 'start_date': '', 'comments': '', 'approval': 'requested', 'sla_due': '', 'comments_and_work_notes': '', 'due_date': '', 'sys_mod_count': '0', 'on_hold': 'false', 'sys_tags': '', 'conflict_last_run': '', 'escalation': '0', 'upon_approval': 'proceed', 'correlation_id': '', 'location': '', 'risk': '3', 'category': 'Other', 'risk_impact_analysis': ''}}
I searched on the net. It is showing as as it is single quotes it's not parsing.
So I tried to convert the single quotes into double quotes.
with open ('output.json','r') as handle:
handle=open('output.json')
str="123"
str=handle.stringify() #also with .str()
str = str.replace("\'", "\"")
jsonobj=json.load(json.dumps(handle))
But it shows me No attribute stringify or str as it is an json object and these are string object function. So, can you please help me with what is the correct way of parsing the json object with single quotes in a file.
The code:-
import requests
import json
from pprint import pprint
print("hello world")
url="********"
user="****"
password="*****"
headers={"Content-Type":"application/xml","Accept":"application/json"}
#response=requests.get(url,auth=(user,password),headers=headers)
response = requests.post(url, auth=(user, password), headers=headers ,data="******in xml****")
print(response.status_code)
print(response.json())
jsonobj=json.load(json.dumps(response.json()))
pprint(jsonobj)
What you receive from requests.post is not JSON, it's a dictionary.
One that can be encoded in JSON, via json.dumps(result).
JSON is a text format to represent objects (the "ON" means "object notation"). You can convert a dictionary (or list or scalar) into a JSON-encoded string, or the other way around.
What requests.post does is taking the JSON response and already parsing it (with json.loads), so you don't have to think about JSON at all.
You haven't shown the code where you get the data from the post. However, you are almost certainly doing something like this:
response = requests.post('...')
data = response.json()
Here data is already parsed from JSON to a Python dict; that is what the requests json method does. There is no need to parse it again.
If you need raw JSON rather than Python data, then don't call the json method. Get the data direct from the response:
data = response.content
Now data will be a string containing JSON.