Develop Reference

TypeError: list indices must be integers or slices, not str Using Python

I am still learning stuff through python and would like your help.
{
"context": {
"authnAttemptId": "14fee7f5-2f81-448c-a0ca-41558eb68b8f",
"messageId": "e1d8e981-5b71-4434-b92e-182f1706dd95",
"inResponseTo": "42dd9ss"
},
"credentialValidationResults": [
{
"methodId": "APPROVE",
"methodResponseCode": "IN_PROCESS",
"methodReasonCode": "VERIFICATION_PENDING:",
"authnAttributes": []
}
],
"attemptResponseCode": "CHALLENGE",
"attemptReasonCode": "METHOD_VERIFY_IN_PROCESS",
"challengeMethods": {
"challenges": [
{
"methodSetId": "850e5638-54c5-4429-9fa8-e48da2594736",
"requiredMethods": [
{
"methodId": "APPROVE",
"displayName": "Approve",
"priority": 1,
"versions": [
{
"versionId": "1.0.0",
"methodAttributes": [
{
"name": "deviceName",
"value": "DESKTOP-P1S563M",
"dataType": "STRING"
}
],
"valueRequired": false,
"referenceId": "14fee7f5-2f81-448c-a0ca-41558eb68b8f:P6JQ",
"prompt": {
"promptResourceId": "IA.Resource.Prompt.Approve",
"defaultText": "Initiate Approve? ",
"formatRegex": null,
"defaultValue": null,
"valueBeingDefined": false,
"sensitive": true,
"minLength": null,
"maxLength": null,
"promptArgs": []
}
}
]
}
]
},
{
"methodSetId": "3dd02306-212e-47e2-b14f-cf0bb6f87e4d",
"requiredMethods": [
{
"methodId": "FINGERPRINT",
"displayName": "Device Biometrics",
"priority": 50,
"versions": [
{
"versionId": "1.0.0",
"methodAttributes": [
{
"name": "deviceName",
"value": "DESKTOP-P1S563M",
"dataType": "STRING"
}
],
"valueRequired": false,
"referenceId": null,
"prompt": {
"promptResourceId": "IA.Resource.Prompt.Fingerprint",
"defaultText": "Initiate Fingerprint? ",
"formatRegex": null,
"defaultValue": null,
"valueBeingDefined": false,
"sensitive": true,
"minLength": null,
"maxLength": null,
"promptArgs": []
}
}
]
}
]
},
{
"methodSetId": "649243db-928e-42ac-b009-42ca278c0a75",
"requiredMethods": [
{
"methodId": "FIDOTOKEN_INITIALIZE_CHALLENGE",
"displayName": "FIDO Challenge",
"priority": 50,
"versions": [
{
"versionId": "1.0.0",
"methodAttributes": [
{
"name": "METHOD_NOT_APPLICABLE",
"value": "DEVICE_NOT_CAPABLE",
"dataType": "STRING"
}
],
"valueRequired": false,
"referenceId": null,
"prompt": {
"promptResourceId": "IA.Resource.Prompt.Fido_Initialize_Challenge",
"defaultText": "Start FIDO Authentication? ",
"formatRegex": null,
"defaultValue": null,
"valueBeingDefined": false,
"sensitive": true,
"minLength": null,
"maxLength": null,
"promptArgs": []
}
}
]
}
]
},
{
"methodSetId": "d896e4e8-49ef-48fd-a232-8a3c3d7e97b0",
"requiredMethods": [
{
"methodId": "TOKEN",
"displayName": "Authenticate Tokencode",
"priority": 50,
"versions": [
{
"versionId": "1.0.0",
"methodAttributes": [
{
"name": "deviceName",
"value": "DESKTOP-P1S563M",
"dataType": "STRING"
}
],
"valueRequired": true,
"referenceId": null,
"prompt": {
"promptResourceId": "IA.Resource.Prompt.Token",
"defaultText": "Enter Tokencode: ",
"formatRegex": null,
"defaultValue": null,
"valueBeingDefined": false,
"sensitive": true,
"minLength": null,
"maxLength": null,
"promptArgs": []
}
}
]
}
]
},
{
"methodSetId": "5324cd43-62eb-4b55-acdb-93d0079a79a9",
"requiredMethods": [
{
"methodId": "EMERGENCY_TOKENCODE",
"displayName": "Emergency Tokencode",
"priority": 100,
"versions": [
{
"versionId": "1.0.0",
"methodAttributes": [
{
"name": "METHOD_NOT_APPLICABLE",
"value": "METHOD_NOT_ENROLLED",
"dataType": "STRING"
}
],
"valueRequired": true,
"referenceId": null,
"prompt": {
"promptResourceId": "IA.Resource.Prompt.Emergency_Tokencode",
"defaultText": "Enter your Emergency Tokencode: ",
"formatRegex": "^[a-zA-Z0-9]{8}$",
"defaultValue": null,
"valueBeingDefined": false,
"sensitive": true,
"minLength": 8,
"maxLength": 12,
"promptArgs": []
}
}
]
}
]
},
{
"methodSetId": "1e2046ad-1c4e-4045-b30c-d39c1a48db86",
"requiredMethods": [
{
"methodId": "SECURID",
"displayName": "RSA SecurID",
"priority": 50,
"versions": [
{
"versionId": "1.0.0",
"methodAttributes": [],
"valueRequired": true,
"referenceId": null,
"prompt": {
"promptResourceId": "IA.Resource.Prompt.SecurId_Passcode",
"defaultText": "Enter PASSCODE: ",
"formatRegex": null,
"defaultValue": null,
"valueBeingDefined": false,
"sensitive": true,
"minLength": null,
"maxLength": null,
"promptArgs": []
}
}
]
}
]
}
]
}
}
So what I am doing is I want to get the value present in this JSON response which is the reference: 14fee7f5-2f81-448c-a0ca-41558eb68b8f:P6JQ
so I am using the following command in python:
result2 = response2.json()
reference_id_approve = result2['challengeMethods']['challenges']['methodSetId']['requiredMethods']['versions']['referenceId']
However, I am getting the error TypeError: list indices must be integers or slices.
Appreciate your help :)

If result is set to that whole dictionary, the correct path to the value '14fee7f5-2f81-448c-a0ca-41558eb68b8f:P6JQ' is:
result["challengeMethods"]["challenges"][0]["requiredMethods"][0]["versions"][0]["referenceId"]
When you are addressing to an element of a dictionary (class dict), you use a its key, that can be any hashable object (a string (class str), for example). When you are addressing to an element of a list (class list) you use its index, that is an integer (class int). (The index of the first element is 0)
`type(d)`-->`dict`
`type(d["challengeMethods"])`-->`dict`
`type(d["challengeMethods"]["challenges"])`-->`list`
`type(d["challengeMethods"]["challenges"][0])`-->`dict`
`type(d["challengeMethods"]["challenges"][0]["requiredMethods"][0]["versions"])`-->`list`
`type(d["challengeMethods"]["challenges"][0]["requiredMethods"][0]["versions"][0]["referenceId"])`-->`str`

How to query the AWS CostExplorer API to get a specific tag's costs in Python?

I am trying to set up some simple billing based on AWS usage so thought that I'd automate it by using AWS's CostExplorer API with the goal of querying the last months AWS bill by client_id and then creating the client invoice automatically based on the total value.
I've set up a tag on a number of AWS products I use for a specific client_id (Lambda, LightSail, SES, API Gateway and SQS).
Then created this simple example:
import os
import json
import boto3
region_name = os.getenv('region_name')
aws_access_key_id = os.getenv('aws_access_key_id')
aws_secret_access_key = os.getenv('aws_secret_access_key') or
client = boto3.client('ce',
region_name=region_name,
aws_access_key_id=aws_access_key_id,
aws_secret_access_key=aws_secret_access_key)
response = client.get_cost_and_usage(
TimePeriod={
'Start': '2021-01-01',
'End': '2021-10-22'
},
Filter={
'Tags': {
'Key': 'client_id',
'Values': ['1234567890'],
'MatchOptions': ['EQUALS']
}
},
Granularity='MONTHLY',
Metrics=[
'NormalizedUsageAmount',
]
)
print(json.dumps(response, indent=4, sort_keys=True))
I got a result:
{
"DimensionValueAttributes": [],
"ResponseMetadata": {
"HTTPHeaders": {
"cache-control": "no-cache",
"connection": "keep-alive",
"content-length": "1538",
"content-type": "application/x-amz-json-1.1",
"date": "Thu, 21 Oct 2021 13:25:09 GMT",
"x-amzn-requestid": ""
},
"HTTPStatusCode": 200,
"RequestId": "",
"RetryAttempts": 0
},
"ResultsByTime": [
{
"Estimated": false,
"Groups": [],
"TimePeriod": {
"End": "2021-02-01",
"Start": "2021-01-01"
},
"Total": {
"NormalizedUsageAmount": {
"Amount": "0",
"Unit": "N/A"
}
}
},
{
"Estimated": false,
"Groups": [],
"TimePeriod": {
"End": "2021-03-01",
"Start": "2021-02-01"
},
"Total": {
"NormalizedUsageAmount": {
"Amount": "0",
"Unit": "N/A"
}
}
},
{
"Estimated": false,
"Groups": [],
"TimePeriod": {
"End": "2021-04-01",
"Start": "2021-03-01"
},
"Total": {
"NormalizedUsageAmount": {
"Amount": "0",
"Unit": "N/A"
}
}
},
{
"Estimated": false,
"Groups": [],
"TimePeriod": {
"End": "2021-05-01",
"Start": "2021-04-01"
},
"Total": {
"NormalizedUsageAmount": {
"Amount": "0",
"Unit": "N/A"
}
}
},
{
"Estimated": false,
"Groups": [],
"TimePeriod": {
"End": "2021-06-01",
"Start": "2021-05-01"
},
"Total": {
"NormalizedUsageAmount": {
"Amount": "0",
"Unit": "N/A"
}
}
},
{
"Estimated": false,
"Groups": [],
"TimePeriod": {
"End": "2021-07-01",
"Start": "2021-06-01"
},
"Total": {
"NormalizedUsageAmount": {
"Amount": "0",
"Unit": "N/A"
}
}
},
{
"Estimated": false,
"Groups": [],
"TimePeriod": {
"End": "2021-08-01",
"Start": "2021-07-01"
},
"Total": {
"NormalizedUsageAmount": {
"Amount": "0",
"Unit": "N/A"
}
}
},
{
"Estimated": false,
"Groups": [],
"TimePeriod": {
"End": "2021-09-01",
"Start": "2021-08-01"
},
"Total": {
"NormalizedUsageAmount": {
"Amount": "0",
"Unit": "N/A"
}
}
},
{
"Estimated": false,
"Groups": [],
"TimePeriod": {
"End": "2021-10-01",
"Start": "2021-09-01"
},
"Total": {
"NormalizedUsageAmount": {
"Amount": "0",
"Unit": "N/A"
}
}
},
{
"Estimated": true,
"Groups": [],
"TimePeriod": {
"End": "2021-10-22",
"Start": "2021-10-01"
},
"Total": {
"NormalizedUsageAmount": {
"Amount": "0",
"Unit": "N/A"
}
}
}
]
}
But there are no costs even though there are definitely invoiced bills across this time period. Is what I am trying to achieve not the use case for this or is there something trivial I have missed?

Loading variables into json string using python for MS teams

The 3rd party system I am using (vendor product) still uses Python 2.7 and doesn't support Python 3+ so bear with me, I'm fully aware Python 3 is out and this is a limitation of the system I have to use rather than a choice.
I am trying to do an integration between this third party product and MS teams - basically, the third party system provides data, I read this into my Python script and output a message to Teams using a webhook. It mostly works, but I'm struggling to load in some of the variables from the systems data.
For example, in my code, I use the following:
messageID='"{}"'.format(item["messageId"])
recipient='"{}"'.format(item["recipient"]["email"])
subject='"{}"'.format(item["subject"])
sender='"{}"'.format(item["sender"]["email"])
which has output like this:
messageId="34239482030783472#test.net"
recipient="testuser#domain.com"
subject="Email subject here"
sender="sender#domain2.com"
This is all fine, the trouble comes when I need to format my string to post to the Teams webhook.
It currently looks like:
teams_card='{"#type": "MessageCard","#context": "http://schema.org/extensions","themeColor": "0076D7","summary": “PTR”,”sections": [{"activityTitle": "PTR Incident Created","activitySubtitle": “End “User Exposed to Phishing Threat,”facts": [{"name": “Message” ID,”value": %s}, {"name": "Subject”,”value": %s},{“name": "End User","value": %s},{“name": “sender”,”value": %s}],”markdown": true}],"potentialAction": [{"#type": "OpenUri","name": "View Related Emails","targets": [{"os": "default","uri": "https://maskedurlhere.com”}]}]}’ % (messageId,subject,recipient,sender)
which throws an error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: not enough arguments for format string
I tried to use .format option also, but this fails with a different error:
teams_card='{"#type": "MessageCard","#context": "http://schema.org/extensions","themeColor": "0076D7","summary": “PTR”,”sections": [{"activityTitle": "PTR Incident Created","activitySubtitle": “End “User Exposed to Phishing Threat,”facts": [{"name": “Message” ID,”value": %s}, {"name": "Subject”,”value": %s},{“name": "End User","value": %s},{“name": “sender”,”value": %s}],”markdown": true}],"potentialAction": [{"#type": "OpenUri","name": "View Related Emails","targets": [{"os": "default","uri": "https://maskedurlhere.com”}]}]}’.format(messageId,subject,recipient,sender)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
KeyError: '"#type"'
The teams card variable is fine and posts to Teams successfully when it's just text, but trying to load in these variables doesn't seem to work at all.
Any ideas?

To pass dynamic values in Json you need to use format like ${value}
Please follow below example json format
Template JSON
{
"type": "AdaptiveCard",
"body": [
{
"type": "Container",
"style": "emphasis",
"items": [
{
"type": "ColumnSet",
"columns": [
{
"type": "Column",
"items": [
{
"type": "TextBlock",
"size": "Large",
"weight": "Bolder",
"text": "**EXPENSE APPROVAL**",
"wrap": true
}
],
"width": "stretch"
},
{
"type": "Column",
"items": [
{
"type": "Image",
"url": "${status_url}",
"altText": "${status}",
"height": "30px"
}
],
"width": "auto"
}
]
}
],
"bleed": true
},
{
"type": "Container",
"items": [
{
"type": "ColumnSet",
"columns": [
{
"type": "Column",
"items": [
{
"type": "TextBlock",
"size": "ExtraLarge",
"text": "${purpose}",
"wrap": true
}
],
"width": "stretch"
},
{
"type": "Column",
"items": [
{
"type": "ActionSet",
"actions": [
{
"type": "Action.OpenUrl",
"title": "EXPORT AS PDF",
"url": "https://adaptivecards.io"
}
]
}
],
"width": "auto"
}
]
},
{
"type": "TextBlock",
"spacing": "Small",
"size": "Small",
"weight": "Bolder",
"color": "Accent",
"text": "[${code}](https://adaptivecards.io)",
"wrap": true
},
{
"type": "FactSet",
"spacing": "Large",
"facts": [
{
"title": "Submitted By",
"value": "**${created_by_name}** ${creater_email}"
},
{
"title": "Duration",
"value": "${formatTicks(min(select(expenses, x, int(x.created_by))), 'yyyy-MM-dd')} - ${formatTicks(max(select(expenses, x, int(x.created_by))), 'yyyy-MM-dd')}"
},
{
"title": "Submitted On",
"value": "${formatDateTime(submitted_date, 'yyyy-MM-dd')}"
},
{
"title": "Reimbursable Amount",
"value": "$${formatNumber(sum(select(expenses, x, if(x.is_reimbursable, x.total, 0))), 2)}"
},
{
"title": "Awaiting approval from",
"value": "**${approver}** ${approver_email}"
},
{
"title": "Submitted to",
"value": "**${other_submitter}** ${other_submitter_email}"
}
]
}
]
},
{
"type": "Container",
"spacing": "Large",
"style": "emphasis",
"items": [
{
"type": "ColumnSet",
"columns": [
{
"type": "Column",
"items": [
{
"type": "TextBlock",
"weight": "Bolder",
"text": "DATE",
"wrap": true
}
],
"width": "auto"
},
{
"type": "Column",
"spacing": "Large",
"items": [
{
"type": "TextBlock",
"weight": "Bolder",
"text": "CATEGORY",
"wrap": true
}
],
"width": "stretch"
},
{
"type": "Column",
"items": [
{
"type": "TextBlock",
"weight": "Bolder",
"text": "AMOUNT",
"wrap": true
}
],
"width": "auto"
}
]
}
],
"bleed": true
},
{
"$data": "${expenses}",
"type": "Container",
"items": [
{
"type": "ColumnSet",
"columns": [
{
"type": "Column",
"items": [
{
"type": "TextBlock",
"text": "${formatDateTime(created_time, 'MM-dd')}",
"wrap": true
}
],
"width": "auto"
},
{
"type": "Column",
"spacing": "Medium",
"items": [
{
"type": "TextBlock",
"text": "${description}",
"wrap": true
}
],
"width": "stretch"
},
{
"type": "Column",
"items": [
{
"type": "TextBlock",
"text": "$${formatNumber(total, 2)}",
"wrap": true
}
],
"width": "auto"
},
{
"type": "Column",
"spacing": "Small",
"selectAction": {
"type": "Action.ToggleVisibility",
"targetElements": [
"cardContent${$index}",
"chevronDown${$index}",
"chevronUp${$index}"
]
},
"verticalContentAlignment": "Center",
"items": [
{
"type": "Image",
"id": "chevronDown${$index}",
"url": "https://adaptivecards.io/content/down.png",
"width": "20px",
"altText": "${description} $${total} collapsed"
},
{
"type": "Image",
"id": "chevronUp${$index}",
"url": "https://adaptivecards.io/content/up.png",
"width": "20px",
"altText": "${description} $${total} expanded",
"isVisible": false
}
],
"width": "auto"
}
]
},
{
"type": "Container",
"id": "cardContent${$index}",
"isVisible": false,
"items": [
{
"type": "Container",
"items": [
{
"$data": "${custom_fields}",
"type": "TextBlock",
"text": "* ${value}",
"isSubtle": true,
"wrap": true
},
{
"type": "Container",
"items": [
{
"type": "Input.Text",
"id": "comment${$index}",
"placeholder": "Add your comment here."
}
]
}
]
},
{
"type": "Container",
"items": [
{
"type": "ColumnSet",
"columns": [
{
"type": "Column",
"items": [
{
"type": "ActionSet",
"actions": [
{
"type": "Action.Submit",
"title": "Send",
"data": {
"id": "_qkQW8dJlUeLVi7ZMEzYVw",
"action": "comment",
"lineItem": 1
}
}
]
}
],
"width": "auto"
}
]
}
]
}
]
}
]
},
{
"type": "ColumnSet",
"spacing": "Large",
"separator": true,
"columns": [
{
"type": "Column",
"items": [
{
"type": "TextBlock",
"horizontalAlignment": "Right",
"text": "Total Expense Amount \t",
"wrap": true
},
{
"type": "TextBlock",
"horizontalAlignment": "Right",
"text": "Non-reimbursable Amount",
"wrap": true
},
{
"type": "TextBlock",
"horizontalAlignment": "Right",
"text": "Advance Amount",
"wrap": true
}
],
"width": "stretch"
},
{
"type": "Column",
"items": [
{
"type": "TextBlock",
"text": "$${formatNumber(sum(select(expenses, x, x.total)), 2)}",
"wrap": true
},
{
"type": "TextBlock",
"text": "(-) $${formatNumber(sum(select(expenses, x, if(x.is_reimbursable, 0, x.total))), 2)} \t",
"wrap": true
},
{
"type": "TextBlock",
"text": "(-) 0.00 \t",
"wrap": true
}
],
"width": "auto"
},
{
"type": "Column",
"width": "auto"
}
]
},
{
"type": "Container",
"style": "emphasis",
"items": [
{
"type": "ColumnSet",
"columns": [
{
"type": "Column",
"items": [
{
"type": "TextBlock",
"horizontalAlignment": "Right",
"text": "Amount to be Reimbursed",
"wrap": true
}
],
"width": "stretch"
},
{
"type": "Column",
"items": [
{
"type": "TextBlock",
"weight": "Bolder",
"text": "$${formatNumber(sum(select(expenses, x, if(x.is_reimbursable, x.total, 0))), 2)}",
"wrap": true
}
],
"width": "auto"
},
{
"type": "Column",
"width": "auto"
}
]
}
],
"bleed": true
},
{
"type": "ColumnSet",
"columns": [
{
"type": "Column",
"selectAction": {
"type": "Action.ToggleVisibility",
"targetElements": [
"cardContent4",
"showHistory",
"hideHistory"
]
},
"verticalContentAlignment": "Center",
"items": [
{
"type": "TextBlock",
"id": "showHistory",
"horizontalAlignment": "Right",
"color": "Accent",
"text": "Show history",
"wrap": true
},
{
"type": "TextBlock",
"id": "hideHistory",
"horizontalAlignment": "Right",
"color": "Accent",
"text": "Hide history",
"wrap": true,
"isVisible": false
}
],
"width": 1
}
]
},
{
"type": "Container",
"id": "cardContent4",
"isVisible": false,
"items": [
{
"type": "Container",
"items": [
{
"type": "TextBlock",
"text": "* Expense submitted by **${created_by_name}** on {{DATE(${formatDateTime(created_date, 'yyyy-MM-ddTHH:mm:ssZ')}, SHORT)}}",
"isSubtle": true,
"wrap": true
},
{
"type": "TextBlock",
"text": "* Expense ${expenses[0].status} by **${expenses[0].approver}** on {{DATE(${formatDateTime(approval_date, 'yyyy-MM-ddTHH:mm:ssZ')}, SHORT)}}",
"isSubtle": true,
"wrap": true
}
]
}
]
},
{
"type": "Container",
"items": [
{
"type": "ActionSet",
"actions": [
{
"type": "Action.Submit",
"title": "Approve",
"style": "positive",
"data": {
"id": "_qkQW8dJlUeLVi7ZMEzYVw",
"action": "approve"
}
},
{
"type": "Action.ShowCard",
"title": "Reject",
"style": "destructive",
"card": {
"type": "AdaptiveCard",
"body": [
{
"type": "Input.Text",
"id": "RejectCommentID",
"placeholder": "Please specify an appropriate reason for rejection.",
"isMultiline": true
}
],
"actions": [
{
"type": "Action.Submit",
"title": "Send",
"data": {
"id": "_qkQW8dJlUeLVi7ZMEzYVw",
"action": "reject"
}
}
],
"$schema": "http://adaptivecards.io/schemas/adaptive-card.json"
}
}
]
}
]
}
],
"$schema": "http://adaptivecards.io/schemas/adaptive-card.json",
"version": "1.2",
"fallbackText": "This card requires Adaptive Cards v1.2 support to be rendered properly."
}
Data Json
{
"code": "ER-13052",
"message": "success",
"created_by_name" : "Matt Hidinger",
"created_date" : "2019-07-15T18:33:12+0800",
"submitted_date": "2019-04-14T18:33:12+0800",
"creater_email" : "matt#contoso.com",
"status" : "Pending",
"status_url" : "https://adaptivecards.io/content/pending.png",
"approver": "Thomas",
"purpose" : "Trip to UAE",
"approval_date" : "2019-07-15T22:33:12+0800",
"approver" : "Thomas",
"approver_email" : "thomas#contoso.com",
"other_submitter" : "David",
"other_submitter_email" : "david#contoso.com",
"expenses": [
{
"expense_id": "16367000000083065",
"approver" : "Thomas",
"date": "2017-02-21",
"description": "Air Travel Expense",
"created_by": "636965431200000000",
"created_by_name": "PATRICIA",
"employee_number": "E001",
"currency_id": "16367000000000097",
"currency_code": "USD",
"paid_through_account_id": "16367000000036003",
"paid_through_account_name": "Employee Reimbursements",
"bcy_total": 13900.79,
"bcy_subtotal": 13900.79,
"total": 300,
"total_without_tax": 300,
"is_billable": true,
"is_reimbursable": true,
"reference_number": "DD145",
"due_days": "Due in 15 days",
"merchant_id": "16367000000074027",
"merchant_name": "ABS Solutions",
"status": "approved",
"created_time": "2019-06-19T18:33:12+0800",
"last_modified_time": "2017-02-21T18:42:46+0530",
"receipt_name": "receipt1.jpg",
"report_id": "16367000000083075",
"mileage_type": "non_mileage",
"report_name": "Purchase",
"is_receipt_only": false,
"distance": 0,
"per_diem_rate": 0,
"per_diem_days": 0,
"per_diem_id": "",
"per_diem_name": "",
"expense_type": "non_mileage",
"location": "Washington",
"receipt_type": "jpg",
"policy_violated": false,
"comments_count": 0,
"report_status": "submitted",
"price_precision": 2,
"mileage_rate": 0,
"mileage_unit": "km",
"receipt_status": "processed",
"is_uncategorized": false,
"is_expired": false,
"gl_code": "LG001",
"exchange_rate": 66.943366,
"start_reading": "",
"end_reading": "",
"payment_mode": "Check",
"customer_id": "27927000000075081",
"customer_name": "ACME Corp.",
"custom_fields": [
{
"customfield_id": "16367000000277001",
"label": "Other Name",
"value": "Leg 1 on Tue, Jun 19th, 2019 at 6:00 AM."
},
{
"customfield_id": "16367000000277001",
"label": "Other Name",
"value": "Leg 2 on Tue, Jun 19th, 2019 at 7:15 PM."
}
],
"project_id": "27927000001243001",
"project_name": "Coffee Research",
"transaction_description": "",
"tax_id": "16367000000086001",
"tax_name": "Sales Tax",
"tax_percentage": 2,
"amount": 207.65,
"is_inclusive_tax": false,
"vehicle_type": "Bike",
"vehicle_id": "17456000000078029",
"fuel_type": "lpg",
"engine_capacity_range": "between_1401cc_and_1600cc",
"is_personal": false,
"policy_id": "16367000000092011",
"policy_name": "LIC",
"documents": [
{
"file_name": "receipt1.jpg",
"file_size_formatted": "71.8 KB",
"attachment_order": 1,
"document_id": "16367000000083071"
}
],
"reimbursement_reference": "",
"reimbursement_date": "",
"reimbursement_paid_through_account_id": "",
"reimbursement_paid_through_account_name": "",
"reimbursement_currency_id": "",
"reimbursement_currency_code": ""
},
{
"expense_id": "16367000000083065",
"date": "2019-06-19",
"description": "Auto Mobile Expense",
"created_by": "636965431200000000",
"created_by_name": "PATRICIA",
"employee_number": "E001",
"currency_id": "16367000000000097",
"currency_code": "USD",
"paid_through_account_id": "16367000000036003",
"paid_through_account_name": "Employee Reimbursements",
"bcy_total": 13900.79,
"bcy_subtotal": 13900.79,
"total": 100,
"total_without_tax": 100,
"is_billable": true,
"is_reimbursable": true,
"reference_number": "DD145",
"due_days": "Due in 15 days",
"merchant_id": "16367000000074027",
"merchant_name": "ABS Solutions",
"status": "submitted",
"created_time": "2019-06-19T18:33:12+0800",
"last_modified_time": "2017-02-21T18:42:46+0530",
"receipt_name": "receipt1.jpg",
"report_id": "16367000000083075",
"mileage_type": "non_mileage",
"report_name": "Purchase",
"is_receipt_only": false,
"distance": 0,
"per_diem_rate": 0,
"per_diem_days": 0,
"per_diem_id": "",
"per_diem_name": "",
"expense_type": "non_mileage",
"location": "Washington",
"receipt_type": "jpg",
"policy_violated": false,
"comments_count": 0,
"report_status": "submitted",
"price_precision": 2,
"mileage_rate": 0,
"mileage_unit": "km",
"receipt_status": "processed",
"is_uncategorized": false,
"is_expired": false,
"gl_code": "LG001",
"exchange_rate": 66.943366,
"start_reading": "",
"end_reading": "",
"payment_mode": "Check",
"customer_id": "27927000000075081",
"customer_name": "ACME Corp.",
"custom_fields": [
{
"customfield_id": "16367000000277001",
"label": "Other Name",
"value": " Contoso Car Rentrals, Tues 6/19 at 7:00 AM"
}
],
"project_id": "27927000001243001",
"project_name": "Coffee Research",
"transaction_description": "",
"tax_id": "16367000000086001",
"tax_name": "Sales Tax",
"tax_percentage": 2,
"amount": 207.65,
"is_inclusive_tax": false,
"vehicle_type": "Bike",
"vehicle_id": "17456000000078029",
"fuel_type": "lpg",
"engine_capacity_range": "between_1401cc_and_1600cc",
"is_personal": false,
"policy_id": "16367000000092011",
"policy_name": "LIC",
"documents": [
{
"file_name": "receipt1.jpg",
"file_size_formatted": "71.8 KB",
"attachment_order": 1,
"document_id": "16367000000083071"
}
],
"reimbursement_reference": "",
"reimbursement_date": "",
"reimbursement_paid_through_account_id": "",
"reimbursement_paid_through_account_name": "",
"reimbursement_currency_id": "",
"reimbursement_currency_code": ""
},
{
"expense_id": "16367000000083065",
"date": "2019-06-21",
"description": "Excess Baggage Cost",
"created_by": "636967159200000000",
"created_by_name": "PATRICIA",
"employee_number": "E001",
"currency_id": "16367000000000097",
"currency_code": "USD",
"paid_through_account_id": "16367000000036003",
"paid_through_account_name": "Employee Reimbursements",
"bcy_total": 13900.79,
"bcy_subtotal": 13900.79,
"total": 50.38,
"total_without_tax": 4.3,
"is_billable": true,
"is_reimbursable": false,
"reference_number": "DD145",
"due_days": "Due in 15 days",
"merchant_id": "16367000000074027",
"merchant_name": "ABS Solutions",
"status": "submitted",
"created_time": "2019-06-21T18:33:12+0800",
"last_modified_time": "2017-02-21T18:42:46+0530",
"receipt_name": "receipt1.jpg",
"report_id": "16367000000083075",
"mileage_type": "non_mileage",
"report_name": "Purchase",
"is_receipt_only": false,
"distance": 0,
"per_diem_rate": 0,
"per_diem_days": 0,
"per_diem_id": "",
"per_diem_name": "",
"expense_type": "non_mileage",
"location": "Washington",
"receipt_type": "jpg",
"policy_violated": false,
"comments_count": 0,
"report_status": "submitted",
"price_precision": 2,
"mileage_rate": 0,
"mileage_unit": "km",
"receipt_status": "processed",
"is_uncategorized": false,
"is_expired": false,
"gl_code": "LG001",
"exchange_rate": 66.943366,
"start_reading": "",
"end_reading": "",
"payment_mode": "Check",
"customer_id": "27927000000075081",
"customer_name": "ACME Corp.",
"custom_fields": [
],
"project_id": "27927000001243001",
"project_name": "Coffee Research",
"transaction_description": "",
"tax_id": "16367000000086001",
"tax_name": "Sales Tax",
"tax_percentage": 2,
"amount": 207.65,
"is_inclusive_tax": false,
"vehicle_type": "Bike",
"vehicle_id": "17456000000078029",
"fuel_type": "lpg",
"engine_capacity_range": "between_1401cc_and_1600cc",
"is_personal": false,
"policy_id": "16367000000092011",
"policy_name": "LIC",
"documents": [
{
"file_name": "receipt1.jpg",
"file_size_formatted": "71.8 KB",
"attachment_order": 1,
"document_id": "16367000000083071"
}
],
"reimbursement_reference": "",
"reimbursement_date": "",
"reimbursement_paid_through_account_id": "",
"reimbursement_paid_through_account_name": "",
"reimbursement_currency_id": "",
"reimbursement_currency_code": ""
}
]
}
Please go through this for more info.

Best way to build denormilazed dataframe with pandas from spotify API

I just downloaded some json from spotify and took a look into the pd.normalize_json().
But if I normalise the data i still have dictionaries within my dataframe. Also setting the level doesnt help.
DATA I want to have in my dataframe:
{
"collaborative": false,
"description": "",
"external_urls": {
"spotify": "https://open.spotify.com/playlist/5"
},
"followers": {
"href": null,
"total": 0
},
"href": "https://api.spotify.com/v1/playlists/5?additional_types=track",
"id": "5",
"images": [
{
"height": 640,
"url": "https://i.scdn.co/image/a",
"width": 640
}
],
"name": "Another",
"owner": {
"display_name": "user",
"external_urls": {
"spotify": "https://open.spotify.com/user/user"
},
"href": "https://api.spotify.com/v1/users/user",
"id": "user",
"type": "user",
"uri": "spotify:user:user"
},
"primary_color": null,
"public": true,
"snapshot_id": "M2QxNTcyYTkMDc2",
"tracks": {
"href": "https://api.spotify.com/v1/playlists/100&additional_types=track",
"items": [
{
"added_at": "2020-12-13T18:34:09Z",
"added_by": {
"external_urls": {
"spotify": "https://open.spotify.com/user/user"
},
"href": "https://api.spotify.com/v1/users/user",
"id": "user",
"type": "user",
"uri": "spotify:user:user"
},
"is_local": false,
"primary_color": null,
"track": {
"album": {
"album_type": "album",
"artists": [
{
"external_urls": {
"spotify": "https://open.spotify.com/artist/1dfeR4Had"
},
"href": "https://api.spotify.com/v1/artists/1dfDbWqFHLkxsg1d",
"id": "1dfeR4HaWDbWqFHLkxsg1d",
"name": "Q",
"type": "artist",
"uri": "spotify:artist:1dfeRqFHLkxsg1d"
}
],
"available_markets": [
"CA",
"US"
],
"external_urls": {
"spotify": "https://open.spotify.com/album/6wPXmlLzZ5cCa"
},
"href": "https://api.spotify.com/v1/albums/6wPXUJ9LzZ5cCa",
"id": "6wPXUmYJ9zZ5cCa",
"images": [
{
"height": 640,
"url": "https://i.scdn.co/image/ab676620a47",
"width": 640
},
{
"height": 300,
"url": "https://i.scdn.co/image/ab67616d0620a47",
"width": 300
},
{
"height": 64,
"url": "https://i.scdn.co/image/ab603e6620a47",
"width": 64
}
],
"name": "The (Deluxe ",
"release_date": "1920-07-17",
"release_date_precision": "day",
"total_tracks": 15,
"type": "album",
"uri": "spotify:album:6m5cCa"
},
"artists": [
{
"external_urls": {
"spotify": "https://open.spotify.com/artist/1dg1d"
},
"href": "https://api.spotify.com/v1/artists/1dsg1d",
"id": "1dfeR4HaWDbWqFHLkxsg1d",
"name": "Q",
"type": "artist",
"uri": "spotify:artist:1dxsg1d"
}
],
"available_markets": [
"CA",
"US"
],
"disc_number": 1,
"duration_ms": 21453,
"episode": false,
"explicit": false,
"external_ids": {
"isrc": "GBU6015"
},
"external_urls": {
"spotify": "https://open.spotify.com/track/5716J"
},
"href": "https://api.spotify.com/v1/tracks/5716J",
"id": "5716J",
"is_local": false,
"name": "Another",
"popularity": 73,
"preview_url": null,
"track": true,
"track_number": 3,
"type": "track",
"uri": "spotify:track:516J"
},
"video_thumbnail": {
"url": null
}
}
],
"limit": 100,
"next": null,
"offset": 0,
"previous": null,
"total": 1
},
"type": "playlist",
"uri": "spotify:playlist:fek"
}
So what are best practices to read nested data like this into one dataframe in pandas?
I'm glad for any advice.
EDIT:
so basically I want all keys as columns in my dataframe. But with normalise it stops at "tracks.items" and if I normalise this again i have the recursive problem again.

It depends on the information you are looking for. Take a look at pandas.read_json() to see if that can work. Also you can select data as such
json_output = {"collaborative": 'false',"description": "", "external_urls": {"spotify": "https://open.spotify.com/playlist/5"}}
df['collaborative'] = json_output['collaborative'] #set value of your df to value of returned json values

tap-mysql returns no data

I have been using tap-mysql from singer.io
Below is my config file:
{
"host": "localhost",
"port": "3306",
"user": "root",
"password": "password"}
This successfully returns schema on --discover
and my properties file is:
{
"streams": [
{
"key_properties": [
"id"
],
"tap_stream_id": "example_db-animals",
"schema": {
"selected": "true",
"properties": {
"likes_getting_petted": {
"selected": "true",
"inclusion": "available",
"type": [
"null",
"boolean"
]
},
"name": {
"selected": "true",
"maxLength": 255,
"inclusion": "available",
"type": [
"null",
"string"
]
},
"id": {
"selected": "true",
"minimum": -2147483648,
"inclusion": "automatic",
"maximum": 2147483647,
"type": [
"null",
"integer"
]
}
},
"type": "object"
},
"table_name": "animals",
"metadata": [
{
"metadata": {
"selected-by-default": true,
"sql-datatype": "int(11)"
},
"breadcrumb": [
"properties",
"id"
]
},
{
"metadata": {
"database-name": "example_db",
"selected-by-default": false,
"is-view": false,
"row-count": 3
},
"breadcrumb": []
},
{
"metadata": {
"selected-by-default": true,
"sql-datatype": "varchar(255)"
},
"breadcrumb": [
"properties",
"name"
]
},
{
"metadata": {
"selected-by-default": true,
"sql-datatype": "tinyint(1)"
},
"breadcrumb": [
"properties",
"likes_getting_petted"
]
}
],
"stream": "animals"
}
]
}
I have added selected flags.
On following command
$ tap-mysql -c config.json --properties properties.json
I have received the following response
{"type": "STATE", "value": {"currently_syncing": null}}
Though my table has rows

You need to make sure that you are marking that table as 'selected' in your properties.json file. Also make sure you specify the replication-method type.
this section below needs to be changed from
"metadata": {
"database-name": "example_db",
"selected-by-default": false,
"is-view": false,
"row-count": 3
},
"breadcrumb": []
to
"metadata": {
"database-name": "example_db",
"selected-by-default": false,
"is-view": false,
"row-count": 3,
"selected": true,
"replication-method": "FULL_TABLE"
},
"breadcrumb": []
The two pieces I believe you are missing are the two lines below:
"selected": true,
"replication-method": "FULL_TABLE"
See the GitHub documentation example for further clarification: https://github.com/singer-io/tap-mysql#replication-methods-and-state-file