How to write "required" for "pattern properties" of json schema

How to write "required" for "pattern properties" of json schema - python

I have this json output and I want to make sure that top_properties is not empty.
In top_properties the key value is dynamic and they are not static. That is where I'm stuck at.
{
"id": "test",
"name": "name",
"cake_name": "test",
"metric": 0.5,
"anticipations": [
{
"time": "2018-01-01 00:00:00",
"points": 0.49128797804879504,
"top_properties": {
"LA:TB2341": 0.23,
"LA:TB2342": 0.23,
"LA:TB2343": 0.23
},
"status": 0,
"alert": false
}
I have below schema but It wont fail when top_properties is empty. I want to make sure it fails when its empty.
{
"definitions": {},
"$schema": "http://json-schema.org/draft-07/schema#",
"$id": "http://example.com/root.json",
"type": "object",
"title": "The Root Schema",
"required": [
"id",
"name",
"cake_name",
"metric",
"anticipations"
],
"properties": {
"id": {
"$id": "#/properties/id",
"type": "string",
"title": "The Id Schema",
"default": "",
"examples": [
"test"
],
"pattern": "^(.*)$"
},
"name": {
"$id": "#/properties/name",
"type": "string",
"title": "The Name Schema",
"default": "",
"examples": [
"name"
],
"pattern": "^(.*)$"
},
"cake_name": {
"$id": "#/properties/cake_name",
"type": "string",
"title": "The Cake_name Schema",
"default": "",
"examples": [
"test"
],
"pattern": "^(.*)$"
},
"metric": {
"$id": "#/properties/metric",
"type": "number",
"title": "The Metric Schema",
"default": 0.0,
"examples": [
0.5
]
},
"anticipations": {
"$id": "#/properties/anticipations",
"type": "array",
"title": "The Anticipations Schema",
"items": {
"$id": "#/properties/anticipations/items",
"type": "object",
"title": "The Items Schema",
"required": [
"time",
"points",
"top_properties",
"status",
"alert"
],
"properties": {
"time": {
"$id": "#/properties/anticipations/items/properties/time",
"type": "string",
"title": "The Time Schema",
"default": "",
"examples": [
"2018-01-01 00:00:00"
],
"pattern": "^(.*)$"
},
"points": {
"$id": "#/properties/anticipations/items/properties/points",
"type": "number",
"title": "The Points Schema",
"default": 0.0,
"examples": [
0.49128797804879504
]
},
"top_properties": {
"$id": "#/properties/anticipations/items/properties/top_properties",
"type": "object",
"title": "The Top_properties Schema",
"patternProperties": {
"[A-Za-z:0-9]": {
"type": "number"
}
},
"additionalProperties": false
},
"status": {
"$id": "#/properties/anticipations/items/properties/status",
"type": "integer",
"title": "The Status Schema",
"default": 0,
"examples": [
0
]
},
"alert": {
"$id": "#/properties/anticipations/items/properties/alert",
"type": "boolean",
"title": "The Alert Schema",
"default": false,
"examples": [
false
]
}
}
}
}
}
}
How do I use required for pattern properties since I don't have static value, how is it implemented if you have come across this situation.

You want the minProperties keyword https://json-schema.org/understanding-json-schema/reference/object.html#size
For example,
{
"type": "object",
"patternProperties": {
"[A-Za-z:0-9]": { "type": "number" }
},
"minProperties": 1
}

Related

BigQuery - use_avro_logical_types doesn´t work in a python script

The doubt is similar to the question: BigQuery use_avro_logical_types ignored in Python script however I have already updated the libraries I use google without success. So I would like to understand what is happening in my case. I will send the intake script part of the avro file. Note: The issue occurs in more than one avro file ingestion pipeline for BigQuery.
EDIT: The mentioned solution of changing schema type didn't work, it gave another error.
Schema in BQ:
Data in BQ:
import csv
import base64
import json
import io
import avro.schema
import avro.io
from avro.datafile import DataFileReader, DataFileWriter
import math
import os
import gcloud
from gcloud import storage
from google.cloud import bigquery
from oauth2client.client import GoogleCredentials
from datetime import datetime, timedelta, date
import numpy as np
try:
script_path = os.path.dirname(os.path.abspath(__file__)) + "/"
except:
script_path = "C:\\Users\\me\\key.json"
#Bigquery Credentials and settings
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = script_path
folder = str((datetime.now() - timedelta(days=1)).strftime('%Y-%m-%d'))
data_folder = str((datetime.now() - timedelta(days=1)).strftime('%Y%m%d'))
bucket_name = 'gs://bucket/*.csv'
dataset = 'dataset'
tabela = 'table_ids'
new_file = 'C:\\Users\\me\\register_' + data_folder + '.avro'
file_schema = 'C:\\Users\\me\\schema.avsc'
new_filename = 'register_' + data_folder + '.avro'
bq1 = bigquery.Client()
#Deleta IDs
query1 = """DELETE FROM dataset.table_ids WHERE ID IS NOT NULL"""
query_job1 = bq1.query(query1)
def insert_bigquery(target_uri, dataset_id, table_id):
bigquery_client = bigquery.Client()
dataset_ref = bigquery_client.dataset(dataset_id)
job_config = bigquery.LoadJobConfig()
job_config.schema = [
bigquery.SchemaField('id','STRING',mode='REQUIRED')
]
job_config.source_format = bigquery.SourceFormat.CSV
job_config.field_delimiter = ";"
uri = target_uri
load_job = bigquery_client.load_table_from_uri(
uri,
dataset_ref.table(table_id),
job_config=job_config
)
print('Starting job {}'.format(load_job.job_id))
load_job.result()
print('Job finished.')
insert_bigquery(bucket_name, dataset, tabela)
def get_data_from_bigquery():
"""query bigquery to get data to import to PSQL"""
bq = bigquery.Client()
#Busca IDs
query = """SELECT id FROM dataset.table_ids"""
query_job = bq.query(query)
data = query_job.result()
rows = list(data)
return rows
a = get_data_from_bigquery()
length = len(a)
line_count = 0
schema = avro.schema.Parse(open(file_schema, "rb").read()) # need to know the schema to write. According to 1.8.2 of Apache Avro
writer = DataFileWriter(open(new_file, "wb"), avro.io.DatumWriter(), schema)
for row in range(length):
bytes = base64.b64decode(str(a[row][0]))
bytes = bytes[5:]
buf = io.BytesIO(bytes)
decoder = avro.io.BinaryDecoder(buf)
rec_reader = avro.io.DatumReader(avro.schema.Parse(open(file_schema).read()))
out=rec_reader.read(decoder)
writer.append(out)
writer.close()
def upload_blob(bucket_name, source_file_name, destination_blob_name):
storage_client = storage.Client()
bucket = storage_client.get_bucket(bucket_name)
blob = bucket.blob("insert/" + destination_blob_name)
blob.upload_from_filename(source_file_name)
print('File {} uploaded to {}'.format(
source_file_name,
destination_blob_name
))
upload_blob('bucket', new_file, new_filename)
def insert_bigquery_avro(target_uri, dataset_id, table_id):
bigquery_client = bigquery.Client()
dataset_ref = bigquery_client.dataset(dataset_id)
job_config = bigquery.LoadJobConfig()
job_config.autodetect = True
job_config.source_format = bigquery.SourceFormat.AVRO
job_config.write_disposition = bigquery.WriteDisposition.WRITE_APPEND
job_config.use_avro_logical_types = True
time_partitioning = bigquery.table.TimePartitioning()
job_config.time_partitioning = time_partitioning
uri = target_uri
load_job = bigquery_client.load_table_from_uri(
uri,
dataset_ref.table(table_id),
job_config=job_config
)
print('Starting job {}'.format(load_job.job_id))
load_job.result()
print('Job finished.')
Avro Schema:
"fields": [
{
"name": "id",
"type": {
"type": "string",
"avro.java.string": "String"
},
"doc": "the payment id"
},
{
"name": "merchant",
"type": {
"type": "string",
"avro.java.string": "String"
},
"doc": "the merchant who owns the payment"
},
{
"name": "date",
"type": {
"type": "long",
"logicalType": "timestamp-millis"
},
"doc": "the date where the transaction happend"
},
{
"name": "amount",
"type": {
"type": "record",
"name": "amount",
"fields": [
{
"name": "amount",
"type": [
"null",
{
"type": "bytes",
"logicalType": "decimal",
"precision": 5,
"scale": 5
}
],
"doc": "the original currency amount",
"default": null
},
{
"name": "foreignAmount",
"type": [
"null",
{
"type": "bytes",
"logicalType": "decimal",
"precision": 5,
"scale": 5
}
],
"doc": "the foreign amount for the payment",
"default": null
},
{
"name": "code",
"type": {
"type": "string",
"avro.java.string": "String"
},
"doc": "the destination currency code"
}
],
"default": null
}
},
{
"name": "exchange_rate",
"type": {
"type": "record",
"name": "code",
"fields": [
{
"name": "currency_code",
"type": [
"null",
{
"type": "string",
"avro.java.string": "String"
}
],
"doc": "the exchange rate currency code",
"default": null
},
{
"name": "rate",
"type": [
"null",
{
"type": "bytes",
"logicalType": "decimal",
"precision": 5,
"scale": 5
}
],
"default": null
},
{
"name": "online_rate",
"type": [
"null",
{
"type": "bytes",
"logicalType": "decimal",
"precision": 5,
"scale": 5
}
],
"default": null
}
]
},
"doc": "The transaction exchange rate"
},
{
"name": "consumer",
"type": {
"type": "record",
"name": "Consumer",
"fields": [
{
"name": "name",
"type": [
"null",
{
"type": "string",
"avro.java.string": "String"
}
],
"doc": "the consumer's name",
"default": null
},
{
"name": "email",
"type": [
"null",
{
"type": "string",
"avro.java.string": "String"
}
],
"doc": "the consumer's email address",
"default": null
},
{
"name": "external_id",
"type": [
"null",
{
"type": "string",
"avro.java.string": "String"
}
],
"doc": "the consumer's external id when needed",
"default": null
},
{
"name": "national_id",
"type": {
"type": "string",
"avro.java.string": "String"
},
"doc": "the national id"
},
{
"name": "phone",
"type": {
"type": "string",
"avro.java.string": "String"
},
"doc": "the consumer's phone number",
"default": ""
}
]
}
},
{
"name": "soft_descriptor",
"type": [
"null",
{
"type": "string",
"avro.java.string": "String"
}
],
"doc": "the description as it will be shown at the customer's invoice",
"default": null
},
{
"name": "merchant_contract",
"type": {
"type": "enum",
"name": "merchant_contract_type",
"symbols": [
"PAY",
"BANK"
]
},
"default": "PAY"
},
{
"name": "type",
"type": {
"type": "enum",
"name": "payment_type",
"symbols": [
"INITIAL",
"CREDIT_CARD",
"DEBIT_CARD",
"ONLINE_DEBIT",
"BANK_SLIP",
"DIGITAL_WALLET",
"ELECTRONIC_BANK_TRANSFER"
]
},
"default": "INITIAL"
},
{
"name": "card",
"type": {
"type": "record",
"name": "card",
"fields": [
{
"name": "type",
"type": [
"null",
{
"type": "enum",
"name": "card_type",
"symbols": [
"CARD",
"TOKEN"
]
}
],
"default": null
},
{
"name": "mask_number",
"type": [
"null",
{
"type": "string",
"avro.java.string": "String"
}
],
"default": null
},
{
"name": "card_holder",
"type": [
"null",
{
"type": "string",
"avro.java.string": "String"
}
],
"default": null
},
{
"name": "brand",
"type": [
"null",
{
"type": "string",
"avro.java.string": "String"
}
],
"default": null
}
]
}
},
{
"name": "confirm",
"type": "boolean",
"doc": "indicates whether is self confirmed",
"default": false
},
{
"name": "installments",
"type": "int",
"doc": "Number of installments for the payment",
"default": 1
},
{
"name": "due_date",
"type": [
"null",
{
"type": "int",
"logicalType": "date"
}
],
"default": null
},
{
"name": "correlation_id",
"type": [
"null",
{
"type": "string",
"avro.java.string": "String"
}
],
"doc": "the external customer correlationid",
"default": null
},
{
"name": "billing",
"type": {
"type": "record",
"name": "Billing",
"fields": [
{
"name": "national_id",
"type": [
"null",
{
"type": "string",
"avro.java.string": "String"
}
],
"doc": "billing info",
"default": null
},
{
"name": "name",
"type": [
"null",
{
"type": "string",
"avro.java.string": "String"
}
],
"doc": "the consumer address name",
"default": null
}
]
}
},
{
"name": "status",
"type": {
"type": "enum",
"name": "payment_status",
"symbols": [
"INITIAL",
"CONSUMER",
"AUTHORIZED",
"WAITING_CONFIRMATION",
"CANCELED",
"WAITING_CLEARING",
"CLEARED",
"TRANSFERENCE",
"DECLINED_BY_ISSUER",
"DECLINED_BY_BUSINESS_RULES",
"CONFIRMED",
"WAITING_CANCELING",
"WAITING_CONSUMER",
"TRANSFER_REQUESTED"
],
"default": "INITIAL"
}
},
{
"name": "metadata",
"type": {
"type": "map",
"values": {
"type": "string",
"avro.java.string": "String"
},
"avro.java.string": "String"
}
},
{
"name": "events",
"type": {
"type": "array",
"items": {
"type": "record",
"name": "event",
"fields": [
{
"name": "id",
"type": {
"type": "string",
"avro.java.string": "String"
},
"default": "0"
},
{
"name": "type",
"type": {
"type": "enum",
"name": "event_type",
"symbols": [
"AUTHORIZATION",
"AUTHENTICATION",
"CONFIRMATION",
"CANCELATION",
"CHECKOUT_CREATION",
"SETTLEMENT",
"TRANSFER_VALIDATION",
"TRANSFER_SCHEDULE",
"TRANSFERRED"
]
}
},
{
"name": "gateway",
"type": [
"null",
{
"type": "string",
"avro.java.string": "String"
}
],
"default": null
},
{
"name": "breadcrumb_id",
"type": [
"null",
{
"type": "string",
"avro.java.string": "String"
}
],
"default": null
},
{
"name": "request_time",
"type": {
"type": "long",
"logicalType": "timestamp-millis"
},
"doc": "the moment where this request was received by the platform"
},
{
"name": "response_time",
"type": {
"type": "long",
"logicalType": "timestamp-millis"
},
"doc": "the moment where this request was returned by the platform"
},
{
"name": "status",
"type": {
"type": "enum",
"name": "event_status",
"symbols": [
"SUCCESS",
"DENIED",
"ERROR",
"TIMEOUT",
"PENDING"
]
}
},
{
"name": "actor",
"type": {
"type": "enum",
"name": "actor",
"symbols": [
"AQ",
"GTW",
"CONCIL"
],
"default": "GTW"
},
"default": "GTW"
},
{
"name": "amount",
"type": [
"null",
{
"type": "bytes",
"logicalType": "decimal",
"precision": 5,
"scale": 5
}
],
"doc": "the original currency amount",
"default": null
},
{
"name": "foreign_amount",
"type": [
"null",
{
"type": "bytes",
"logicalType": "decimal",
"precision": 5,
"scale": 5
}
],
"doc": "the foreign amount for the payment",
"default": null
},
{
"name": "error",
"type": {
"type": "record",
"name": "Error",
"fields": [
{
"name": "code",
"type": [
"null",
{
"type": "string",
"avro.java.string": "String"
}
],
"default": null
},
{
"name": "message",
"type": [
"null",
{
"type": "string",
"avro.java.string": "String"
}
],
"default": null
}
]
}
},
{
"name": "message",
"type": [
"null",
{
"type": "string",
"avro.java.string": "String"
}
],
"default": null
},
{
"name": "fee_amount",
"type": [
"null",
{
"type": "bytes",
"logicalType": "decimal",
"precision": 5,
"scale": 5
}
],
"doc": "the fee amount",
"default": null
},
{
"name": "net_amount",
"type": [
"null",
{
"type": "bytes",
"logicalType": "decimal",
"precision": 5,
"scale": 5
}
],
"doc": "the net amount",
"default": null
},
{
"name": "metadata",
"type": {
"type": "map",
"values": {
"type": "string",
"avro.java.string": "String"
},
"avro.java.string": "String"
}
},
{
"name": "internal_metadata",
"type": {
"type": "map",
"values": {
"type": "string",
"avro.java.string": "String"
},
"avro.java.string": "String"
}
}
]
}
}
},
{
"name": "bank_account",
"type": [
"null",
{
"type": "record",
"name": "bank_account",
"fields": [
{
"name": "name",
"type": {
"type": "string",
"avro.java.string": "String"
},
"doc": "the bank name",
"default": ""
},
{
"name": "code",
"type": {
"type": "string",
"avro.java.string": "String"
},
"doc": "the bank code",
"default": ""
},
{
"name": "agency",
"type": {
"type": "string",
"avro.java.string": "String"
},
"doc": "the bank agency",
"default": ""
},
{
"name": "account",
"type": {
"type": "string",
"avro.java.string": "String"
},
"doc": "the bank account",
"default": ""
},
{
"name": "document_number",
"type": {
"type": "string",
"avro.java.string": "String"
},
"doc": "the bank document number (CNPJ)",
"default": ""
}
]
}
],
"doc": "The bank account values",
"default": null
}
]

Try setting this as "type" for all timestamps columns on your avro schema:
"type": ["null", {"type": "long", "logicalType": "timestamp-millis"}]

How to extract a a specific object from JSON file using python json

Using Python I need to extract all the API (just the API endpoint names) endpoints by reading from a JSON file.
Below is the sample JSON code,
{
"swagger": "2.0",
"info": {
"title": "None",
"description": "some thing over here sample content\n",
"version": "0.1"
},
"produces": [
"application/json"
],
"basePath": "/busrouting",
"schemes": [
"https"
],
"definitions": {
"OAuth2": {
"description": "some thing over here sample content\n",
"type": "coauthor",
"flow": "implicit",
"authorization": "NONE",
"scopes": {
"Zero": "three",
"One": "two"
}
}
},
"paths": {
"/service-provider/details": {
"get": {
"description": "some thing over here sample content",
"security": [
{
"OAuth2": [
"None",
"None"
]
}
],
"parameters": [
{
"name": "service provider",
"in": "query",
"description": "List of service-provider ID's",
"required": false,
"type": "array",
"items": {
"type": "string"
}
},
{
"name": "limit",
"in": "query",
"description": "Record limit. Default is 20",
"required": false,
"type": "integer"
},
{
"name": "offset",
"in": "query",
"description": "Record offset. Default is 0.",
"required": false,
"type": "integer"
}
],
"tags": [
"Service Providers"
],
"responses": {
"200": {
"description": "OK.",
"headers": {
"Link": {
"description": "some thing over here sample content\n",
"type": "array",
"items": {
"type": "string"
}
}
},
"schema": {
"type": "object",
"properties": {
"names": {
"type": "array",
"items": {
"$ref": "#/definitions/service-provider"
}
}
} #Sample add
}
},
"400": {
"description": "Bad Request"
},
"401": {
"description": "Unauthorized"
},
"403": {
"description": "Forbidden"
},
"404": {
"description": "Zero"
},
"500": {
"description": "Internal Server Error",
"schema": {
"$ref": "#/definitions/Error"
}
}
}
}
},
"/Bus/{bus-id}/names": {
"get": {
"description": "some thing over here sample content\n",
"security": [
{
"OAuth2": [
"None",
"None"
]
}
],
"parameters": [
{
"name": "Bus-id",
"in": "path",
"required": true,
"type": "string"
}
],
"tags": [
"BusNames"
],
"responses": {
"200": {
"description": "OK.",
"schema": {
"$ref": "#/definitions/busnames"
}
},
"400": {
"description": "Bad Request"
},
"401": {
"description": "Unauthorized"
},
"403": {
"description": "Forbidden"
},
"404": {
"description": "Balances not found"
},
"500": {
"description": "Internal Server Error",
"schema": {
"$ref": "#/definitions/Error"
}
}
}
}
},
"/busoperator/{bus-id}/names/routes": {
"get": {
"description": "some thing over here sample content.\n",
"security": [
{
"OAuth2": [
"None",
"None"
]
}
],
"parameters": [
{
"name": "bus-id",
"in": "path",
"required": true,
"type": "string"
},
{
"name": "limit",
"in": "query",
"description": "Record limit. Default is 10",
"required": false,
"type": "integer"
},
{
"name": "offset",
"in": "query",
"description": "Record offset. Default is 0.",
"required": false,
"type": "integer"
}
],
"tags": [
"bus route mapping"
],
"responses": {
"200": {
"description": "Recent Route",
"headers": {
"Link": {
"description": "some thing over here sample content.\n",
"type": "array",
"items": {
"type": "string"
}
}
},
"schema": {
"type": "object",
"properties": {
"Route details": {
"type": "array",
"items": {
"$ref": "#/definitions/Route"
}
}
}
}
},
"400": {
"description": "Bad Request"
},
"401": {
"description": "Unauthorized"
},
"403": {
"description": "Not entitled to this account and its transactions"
},
"404": {
"description": "User has no recent transactions"
},
"500": {
"description": "Internal Server Error",
"schema": {
"$ref": "#/definitions/Error"
}
}
}
}
},
"/operator/{id}/route/path": {
"get": {
"description": "some thing over here sample content.\n",
"security": [
{
"OAuth2": [
"None",
"None"
]
}
],
"parameters": [
{
"name": "bus-id",
"in": "path",
"required": true,
"type": "string"
},
{
"name": "limit",
"in": "query",
"description": "Record limit. Default is 10",
"required": false,
"type": "integer"
},
{
"name": "offset",
"in": "query",
"description": "Record offset. Default is 0.",
"required": false,
"type": "integer"
}
],
"tags": [
"Sample Reporting"
],
"responses": {
"200": {
"description": "sample actions",
"headers": {
"Link": {
"description": "some thing over here sample content\n",
"type": "array",
"items": {
"type": "string"
}
}
},
"schema": {
"type": "object",
"properties": {
"Boisterous": {
"type": "array",
"items": {
"$ref": "#/definitions/salesperson"
}
}
}
}
},
"400": {
"description": "Bad Request"
},
"401": {
"description": "Unauthorized"
},
"403": {
"description": "Not entitled to this account and its transactions"
},
"404": {
"description": "User has no recent transactions"
},
"500": {
"description": "Internal Server Error",
"schema": {
"$ref": "#/definitions/Error"
}
}
}
}
}
},
"definitions": {
"sample": {
"required": [
"demo",
"request"
],
"description": "Holds identifying attributes for an sample\n",
"properties": {
"request": {
"description": "some thing over here sample content\n",
"type": "string"
},
"sample one": {
"type": "string"
},
"sample two": {
"type": "string"
},
"number": {
"description": "some thing over here sample content",
"type": "string"
},
"name": {
"type": "string",
"description": "some thing over here sample content"
}
}
},
"dummy": {
"description": "some thing over here sample content\n",
"properties": {
"dummy": {
"$ref": "#/definitions/samples"
},
"data": {
"type": "string",
"description": "sample data "
},
"country": {
"type": "string",
"description": "some thing over here sample content"
},
"dataone": {
"$ref": "#/definitions/samples"
},
"error": {
"$ref": "#/definitions/Error"
},
"links": {
"description": "some thing over here sample content",
"type": "array",
"items": {
"$ref": "#/definitions/Links"
}
}
}
},
"reference": {
"description": "only description\n",
"properties": {
"available": {
"type": "number",
"format": "double",
"description": "some thing over here sample content"
},
"availableFormatted": {
"type": "string",
"description": "some thing over here sample content"
},
"held": {
"type": "number",
"format": "double",
"description": "some thing over here sample content."
},
"formatted": {
"type": "string",
"description": "some thing over here sample content"
},
"asst": {
"type": "string",
"format": "date-time",
"description": "Timestamp of , in UTC"
},
"sampler": {
"type": "string",
"format": "date-time",
"description": "Timestamp of , in users preferred TZ"
}
}
},
"dummy": {
"description": "Transaction done on an account",
"required": [
"because",
"versioned",
"postdates"
],
"properties": {
"tintype": {
"description": "sample content",
"type": "string"
},
"cringed": {
"description": "some thing over here sample content",
"type": "string",
"menu": [
"D",
"C"
]
},
"reference": {
"description": "Reference information",
"type": "string"
},
"amount": {
"description": "some thing over here sample content\n",
"type": "number",
"format": "double"
},
"formatted": {
"description": "formatted according to user's preferences\n"
},
"post Date": {
"type": "string",
"format": "date-time",
"description": "Date-time in UTC"
},
"postdate": {
"type": "string",
"format": "date-time",
"description": "Post Date-time in user-preferred TZ"
},
"narrative": {
"type": "string"
}
}
},
"Links": {
"description": "Related Links for the resource\n",
"properties": {
"rel": {
"description": "relationship to the resource",
"type": "string"
},
"ref": {
"description": "URL of the related link",
"type": "string"
}
}
},
"Error": { Sample
"description": "Error descriptor",
"properties": {
"code": {
"type": "integer",
"format": "intent"
},
"message": {
"type": "string"
},
"description": {
"type": "string"
},
"severity": {
"type": "string"
},
"location": {
"type": "string"
}
}
}
}
}
I have tried the below code:
import json
with open("example.json", "r") as reading:
data = json.load(reading)
print(data["paths"])
From here i need to go further the code to capture all the API endpoint name only.
In the sample json file, Under paths i need to capture all the endpoints and method type of an API as below,
In addition I would also like to capture the values for the below keys from the sample JSON code.
Under "parameters" i also need to capture the below keys,
name:
in:
required:
Expected output (just an example):
/service-provider/details
get
bus (comment - refers to the key - name under 'parameters')
query (comment - refers to the key - in under 'parameters')
false (comment - refers to the key - required under 'parameters')
/Bus/{bus-id}/names
get
bus-id (comment - refers to the key - name under 'parameters')
query (comment - refers to the key - in under 'parameters')
false (comment - refers to the key - required under 'parameters')
/busoperator/{bus-id}/names/routes
get
routes (comment - refers to the key - name under 'parameters')
query (comment - refers to the key - in under 'parameters')
false (comment - refers to the key - required under 'parameters')
/operator/{id}/route/path
get
id (comment - refers to the key - name under 'parameters')
query (comment - refers to the key - in under 'parameters')
false (comment - refers to the key - required under 'parameters')

Try this:
with open('example.json', 'r') as f:
data = json.load(f)
for path, values in data['paths'].items():
print(path)
for value in values:
print(value)
This should also get all the endpoints functions in a path if there are multiple.

To get a list of two-tuples (endpoint, type) use:
[(endpoint, value.keys[0]) for endpoint, value in data["paths"].items()]

Script to append JSON structure

I have a JSON structure which needs some code to be appended. I tried with SED and bash, that only appends at the end of a string or file, not the end of the structure.
{
"$schema": "http://json-schema.org/draft-04/schema#",
"required": [
"accounts"
],
"accounts": {
"required": "account",
"properties": {
"account": {
"type": "array",
"minItems": 1,
"maxItems": 999,
"required": [
"scheme",
"accountType",
"accountSubType"
],
"items": {
"type": "object",
"properties": {
"scheme": {
"description": "scheme",
"type": "object",
"required": [
"schemeName",
"identification"
],
"properties": {
"schemeName": {
"type": "string",
"maxLength": 40
},
"identification": {
"type": "string",
"maxLength": 256
},
"name": {
"type": "string",
"maxLength": 70
},
"secondaryIdentification": {
"type": "string",
"maxLength": 35
}
}
},
"currency": {
"type": "string",
"format": "iso-4217",
"pattern": "^[A-Z]{3,3}$",
"maxLength": 3,
"example": "EUR"
},
"accountType": {
"type": "string"
},
"accountSubType": {
"type": "string",
"maxLength": 35
}
}
}
}
}
}
}
I would like to update the above as
{
"$schema": "http://json-schema.org/draft-04/schema#",
"required": [
"accounts"
],
"accounts": {
"required": "account",
"properties": {
"account": {
"type": "array",
"minItems": 1,
"maxItems": 999,
"required": [
"scheme",
"accountType",
"accountSubType"
],
"items": {
"type": "object",
"properties": {
"scheme": {
"description": "scheme",
"type": "object",
"required": [
"schemeName",
"identification"
],
"properties": {
"schemeName": {
"type": "string",
"maxLength": 40
},
"identification": {
"type": "string",
"maxLength": 256
},
"name": {
"type": "string",
"maxLength": 70
},
"secondaryIdentification": {
"type": "string",
"maxLength": 35
}
},
"additionalProperties": false
},
"currency": {
"type": "string",
"format": "iso-4217",
"pattern": "^[A-Z]{3,3}$",
"maxLength": 3,
"example": "EUR"
},
"accountType": {
"type": "string"
},
"accountSubType": {
"type": "string",
"maxLength": 35
}
},
"additionalProperties": false
}
}
},
"additionalProperties": false
}
}
The difference is at the end of every "properties" section. I have appened it with "additionalProperties": false
Is there a way to do this through a script I can check and append all properties with that?

You can do this with jq (Requires jq 1.6 because it uses the walk() function to traverse the entire structure):
$ jq 'walk(if type == "object" and has("properties") then . + { additionalProperties: false } else . end)' your.json

Does it matter if "additionalProperties" comes after or before "properties"?
If not, you could use sed to add "additionalProperties" before the object "properties" like this:
sed -E 's/([[:space:]]*)"properties": {/\1"additionalProperties": false,|\1"properties": {/g'| tr '|' '\n'
With you you will get
{
"$schema": "http://json-schema.org/draft-04/schema#",
"required": [
"accounts"
],
"accounts": {
"required": "account",
"additionalProperties": false,
"properties": {
"account": {
"type": "array",
"minItems": 1,
"maxItems": 999,
"required": [
"scheme",
"accountType",
"accountSubType"
],
"items": {
"type": "object",
"additionalProperties": false,
"properties": {
"scheme": {
"description": "scheme",
"type": "object",
"required": [
"schemeName",
"identification"
],
"additionalProperties": false,
"properties": {
"schemeName": {
"type": "string",
"maxLength": 40
},
"identification": {
"type": "string",
"maxLength": 256
},
"name": {
"type": "string",
"maxLength": 70
},
"secondaryIdentification": {
"type": "string",
"maxLength": 35
}
}
},
"currency": {
"type": "string",
"format": "iso-4217",
"pattern": "^[A-Z]{3,3}$",
"maxLength": 3,
"example": "EUR"
},
"accountType": {
"type": "string"
},
"accountSubType": {
"type": "string",
"maxLength": 35
}
}
}
}
}
}
}

Adding an adaptive card to bot framework with python

I am playing a little bit with the samples of the bot framework in python from here https://github.com/Microsoft/botbuilder-python
Now I want to add a simple adaptive card to the response which I believe it is the part where it says await context.send_activity(response) but I can not attach the card. I grabbed the card from the docs sample:
{
"$schema": "http://adaptivecards.io/schemas/adaptive-card.json",
"type": "AdaptiveCard",
"version": "1.0",
"body": [
{
"type": "Container",
"items": [
{
"type": "TextBlock",
"text": "Publish Adaptive Card schema",
"weight": "bolder",
"size": "medium"
},
{
"type": "ColumnSet",
"columns": [
{
"type": "Column",
"width": "auto",
"items": [
{
"type": "Image",
"url": "https://pbs.twimg.com/profile_images/3647943215/d7f12830b3c17a5a9e4afcc370e3a37e_400x400.jpeg",
"size": "small",
"style": "person"
}
]
},
{
"type": "Column",
"width": "stretch",
"items": [
{
"type": "TextBlock",
"text": "Matt Hidinger",
"weight": "bolder",
"wrap": true
},
{
"type": "TextBlock",
"spacing": "none",
"text": "Created {{DATE(2017-02-14T06:08:39Z, SHORT)}}",
"isSubtle": true,
"wrap": true
}
]
}
]
}
]
},
{
"type": "Container",
"items": [
{
"type": "TextBlock",
"text": "Now that we have defined the main rules and features of the format, we need to produce a schema and publish it to GitHub. The schema will be the starting point of our reference documentation.",
"wrap": true
},
{
"type": "FactSet",
"facts": [
{
"title": "Board:",
"value": "Adaptive Card"
},
{
"title": "List:",
"value": "Backlog"
},
{
"title": "Assigned to:",
"value": "Matt Hidinger"
},
{
"title": "Due date:",
"value": "Not set"
}
]
}
]
}
],
"actions": [
{
"type": "Action.ShowCard",
"title": "Set due date",
"card": {
"type": "AdaptiveCard",
"body": [
{
"type": "Input.Date",
"id": "dueDate"
}
],
"actions": [
{
"type": "Action.Submit",
"title": "OK"
}
]
}
},
{
"type": "Action.ShowCard",
"title": "Comment",
"card": {
"type": "AdaptiveCard",
"body": [
{
"type": "Input.Text",
"id": "comment",
"isMultiline": true,
"placeholder": "Enter your comment"
}
],
"actions": [
{
"type": "Action.Submit",
"title": "OK"
}
]
}
}
]}
I can not find a way to attach the card to the python response.

You need to create the Attachment for the activity that is sent to the user:
ADAPTIVE_CARD_ATTACHMENT = Attachment(content_type='application/vnd.microsoft.card.adaptive',
content=ADAPTIVE_CARD)
After this, you can attach it to your response activity like this:
response.attachments = [ADAPTIVE_CARD_ATTACHMENT]
Or you could add it when you create the response:
response = Activity(type='message', attachments=[ADAPTIVE_CARD_ATTACHMENT])
Note: I left out the additional code needed to create a valid activity for brevity, you still need to add the fields such as channel_id, recipient and from_property, etc.

How to Remove Outer Layer of JSON in Python

I am trying to remove the outer (parent) layer of a JSON file so that I can process it, however I have no idea how.
As you will see by the code below, the outer 2 most layers are 2 dictionaries, however, python says the 2nd dictionary ("item") is just a string when I call its type. Am I incorrect in how I interpret the structure?
sample_object6 = {
"items":
{
"item":
[
{
"id": "0001",
"type": "donut",
"name": "Cake",
"ppu": 0.55,
"batters":
{
"batter":
[
{ "id": "1001", "type": "Regular" },
{ "id": "1002", "type": "Chocolate" },
{ "id": "1003", "type": "Blueberry" },
{ "id": "1004", "type": "Devil's Food" }
]
},
"topping":
[
{ "id": "5001", "type": "None" },
{ "id": "5002", "type": "Glazed" },
{ "id": "5005", "type": "Sugar" },
{ "id": "5007", "type": "Powdered Sugar" },
{ "id": "5006", "type": "Chocolate with Sprinkles" },
{ "id": "5003", "type": "Chocolate" },
{ "id": "5004", "type": "Maple" }
]
},
{
"id": "0002",
"type": "donut",
"name": "Raised",
"ppu": 0.55,
"batters":
{
"batter":
[
{ "id": "1001", "type": "Regular" }
]
},
"topping":
[
{ "id": "5001", "type": "None" },
{ "id": "5002", "type": "Glazed" },
{ "id": "5005", "type": "Sugar" },
{ "id": "5003", "type": "Chocolate" },
{ "id": "5004", "type": "Maple" }
]
},
{
"id": "0003",
"type": "donut",
"name": "Old Fashioned",
"ppu": 0.55,
"batters":
{
"batter":
[
{ "id": "1001", "type": "Regular" },
{ "id": "1002", "type": "Chocolate" }
]
},
"topping":
[
{ "id": "5001", "type": "None" },
{ "id": "5002", "type": "Glazed" },
{ "id": "5003", "type": "Chocolate" },
{ "id": "5004", "type": "Maple" }
]
},
{
"id": "0004",
"type": "bar",
"name": "Bar",
"ppu": 0.75,
"batters":
{
"batter":
[
{ "id": "1001", "type": "Regular" },
]
},
"topping":
[
{ "id": "5003", "type": "Chocolate" },
{ "id": "5004", "type": "Maple" }
],
"fillings":
{
"filling":
[
{ "id": "7001", "name": "None", "addcost": 0 },
{ "id": "7002", "name": "Custard", "addcost": 0.25 },
{ "id": "7003", "name": "Whipped Cream", "addcost": 0.25 }
]
}
},
{
"id": "0005",
"type": "twist",
"name": "Twist",
"ppu": 0.65,
"batters":
{
"batter":
[
{ "id": "1001", "type": "Regular" },
]
},
"topping":
[
{ "id": "5002", "type": "Glazed" },
{ "id": "5005", "type": "Sugar" },
]
},
{
"id": "0006",
"type": "filled",
"name": "Filled",
"ppu": 0.75,
"batters":
{
"batter":
[
{ "id": "1001", "type": "Regular" },
]
},
"topping":
[
{ "id": "5002", "type": "Glazed" },
{ "id": "5007", "type": "Powdered Sugar" },
{ "id": "5003", "type": "Chocolate" },
{ "id": "5004", "type": "Maple" }
],
"fillings":
{
"filling":
[
{ "id": "7002", "name": "Custard", "addcost": 0 },
{ "id": "7003", "name": "Whipped Cream", "addcost": 0 },
{ "id": "7004", "name": "Strawberry Jelly", "addcost": 0 },
{ "id": "7005", "name": "Rasberry Jelly", "addcost": 0 }
]
}
}
]
}
}
I thought that it might be possible to store the nested portion starting at the first list (right after 'item') in a variable and then work with this but if I can't get python to see that item is a dictionary inside the items dictionary, then I fear I am at a loss with how to proceed.
Does anyone know what I am doing wrong?
Thank you in advance!
As far as the processing goes, there has been none because I could not even get the string to read as a dictionary appropriately.
This is what I tried to test if it was a dictionary:
for i in sample_object6:
print(i + str(type(i)))
for n in i["item"]:
print(n + str(type(n)))

After submitting the same code that I thought I had already submitted, I noticed that python is interpreting the object correctly. I have some obvious fundamental gaps in how to work in python and I'm sorry I took it to the forum.
For the record (and for future python newbies out there like me), I used the following code which returned the proper class types:
#this returned a class type of dictionary
print(type(sample_object6["items"]))
#this returned a class type of list
print(type(sample_object6["items"]["item"]))
Thank you SungJin Steve Yoo & Pm2Ring for your help.

Develop Reference

Python is a programming language that lets you work quickly and integrate systems more effectively.

How to write "required" for "pattern properties" of json schema - python

You want the minProperties keyword https://json-schema.org/understanding-json-schema/reference/object.html#size For example, { "type": "object", "patternProperties": { "[A-Za-z:0-9]": { "type": "number" } }, "minProperties": 1 }

Related

BigQuery - use_avro_logical_types doesn´t work in a python script

How to extract a a specific object from JSON file using python json

Script to append JSON structure

Adding an adaptive card to bot framework with python

How to Remove Outer Layer of JSON in Python

Categories

Resources