Develop Reference

Issue in using Flask request.json.get method : converting the JSON object from url into string

I am trying to convert the below JSON into a pandas dataframe. The JSON is being captured using flask request method.
{
"json_str": [{
"TileName ": "Master",
"ImageLink ": "Link1",
"Report Details": [{
"ReportName": "Primary",
"ReportUrl": "link1",
"ADGroup": ["operations", "Sales"],
"IsActive": 1
}, {
"ReportName": "Secondry",
"ReportUrl": "link2",
"ADGroup": ["operations", "Sales"],
"IsActive": 1
}],
"OpsFlag": 1
}]
}
Now below are the code snippet that I am
Using `request.json.get() method to get the JSON
Converting into string using json.loads()
Normalizing using pd.json_normalize and finally
Using pyodbc to run a Stored Procedure to insert the data into Database
Below is the code snippets:
###Step 1 and 2###
json_strg = request.json.get("json_str",None) <----in flask app.py
json_strf = json.dumps(json_strg)
js_obj = json.loads(json_strf)
###Step 3###
df = pd.json_normalize(js_obj,record_path='Report Details',\
meta=
['TileName','ImageLink','OpsFlag'],errors='ignore').explode('ADGroup').apply(pd.Series)
Cols = ['TileName','ImageLink','ReportName','ReportUrl','ADGroup','OpsFlag','IsActive']
df= df[Cols]
###Step 4###
conn = pyodbc.connect(conn_string)
cur=conn.cursor()
for i,v in df.iterrows():
sql = """SET NOCOUNT ON;
EXEC [dbo].[mystored_proc] ?, ?, ?, ?, ?,?,?"""
value = tuple(v)
cur.execute(sql,value)
conn.commit()
The above code is giving the below error:
"('42000', '[42000] [Microsoft][ODBC Driver 18 for SQL Server][SQL Server]The incoming
tabular data stream (TDS) remote procedure call (RPC) protocol stream is incorrect.
Parameter 4 (\"\"): The supplied value is not a valid instance of data type float.
Check the source data for invalid values. An example of an invalid value is data of
numeric type with scale greater than precision. (8023) (SQLExecDirectW)')"
Strange thing is when I am running the above JSON string with the value of json_str part (i.e. string starting from '{"TileName:), no error is coming and it is actually inserting data into DB.
This means, there is no issue in Step 3 and 4. Issue is in step 1 and 2.
Any clue?

How to append JSON fields in sqlalchemy statements

How to update JSON field in sqlalchemy appending another json?
stmt = pg_insert(Users).values(
userid=user.id,
pricesjson=[{
"product1": user.product1,
"product2": user.product2,
"product3": user.product3
}],
tsins=datetime.now()
)
stmtUpsert = stmt.on_conflict_do_update(index_elements=[Users.userid],
set_={'pricesjson': cast({"product1": user.product1,
"product2": user.product2,
"product3": user.product3
} +
cast(stmt.excluded.pricesjson, JSONB),
JSON)
, 'tsvar': datetime.now()})
In that way i don't receive errors but overwrite json field without append.
Thank you ;)

Solved: After altered the field on table from json to jsonb, that's the working code:
stmtUpsert = stmt.on_conflict_do_update(index_elements=[Users.userid],
set_={'pricesjson': cast([{"product1": user.product1,
"product2": user.product2,
"product3": user.product3
], JSONB) + Users.pricesjson
, 'tsvar': datetime.now()})
That's the relative sample query:
insert into users (userid, pricesjson) values('1', '{"product1": "test1", product2": "test2"}')
on conflict (userid)
do update set pricesjson =cast('[{"productX": "testX"}]' as jsonb) || securitiesprices.pricesjson

I am trying to insert json into a postgres (pgadmin4) table using python 3 but I keep getting null

I have a Postgres table in pgadmin4 and it has 2 columns. ID (big serial, not null) and json (json). I keep getting the value null in my json column when trying to insert my json data from it which I got by doing a get request. Under the column settings for json I have not selected 'not null'.
I am still learning; my code might be a bit messy.
import os
import requests
import psycopg2
import json
def get_request():
token = os.environ['TOKEN']
url = os.environ['URL']
payload = {}
headers = {
'Accept': 'application/json',
'Authorization': 'Bearer ' + token
}
response = requests.request('GET', url, headers=headers, data=payload)
result = (response.text.encode('utf8'))
parsed = json.loads(result)
print(parsed)
print(type(parsed))
def db_connection(parsed):
con = psycopg2.connect(
host=os.environ['HOST'],
dbname=os.environ['DBNAME'],
user=os.environ['USER'],
password=os.environ['PASSWORD'],
port=os.environ['PORT'])
print("CONNECTED")
cur = con.cursor()
cur.execute("INSERT INTO ingest.occupations_group (json) VALUES (%s)", (parsed,))
con.commit()
cur.close()
con.close()
get_request()
db_connection(get_request())
And when I run the code for some reason it display the results twice. I did not include the all of the api results as it is super long.
{'#odata.context': 'https://apis.burning-glass.com/v3.5/ontology/$metadata#occupationgroups', 'value': [{'id': 2, 'ontologyIRI':
<class 'dict'>
{'#odata.context': 'https://apis.burning-glass.com/v3.5/ontology/$metadata#occupationgroups', 'value': [{'id': 2, 'ontologyIRI':
<class 'dict'>
CONNECTED
Process finished with exit code 0

Insert into table values from a dictionary in a Postgresql database

I have a dictionary in python that I created from a JSON file. Now, I need to pass its values to insert into a postgresql database.
dictionary
if(i['trailers']):
a = [
{'url': i['images'][0]['url'], 'type': i['images'][0]['type']},
{'url': i['images'][1]['url'], 'type': i['images'][1]['type']},
{'url': i['trailers'][0]['url'], 'type': 'Trailer'},
{'url': i['trailers'][1]['url'], 'type': 'Trailer'},
]
else:
a = [
{'url': i['images'][0]['url'], 'type': i['images'][0]['type']},
{'url': i['images'][1]['url'], 'type': i['images'][1]['type']},
]
length = len(a)
Here, I created the dictionary. If there is anything inside the trailer it goes A, else it goes B. In the B case, trailers doesn't exists. Then I get the length of the dictionary.
Now, I will try to insert these elements into the table media, that depends on movies. Their relation is movie(1):media(n).
INSERT INTO media
for x in range(length):
query = ("""INSERT INTO media VALUES (%s, %s, %(url)s, %(type)s);""")
data = (media_id, media_movie_id)
cur.execute(query, data)
conn.commit()
media_id += 1
Here is what I'm trying to do. Since movie can have many media, I'll create a for to move through all the elements and inserting them in the table. With their id being incremented.
The problem is, I don't know how to do this quiet right in Python, since I always create a query and a data and then cur.execute it and the example that I got, was a entire dictionary being used, without any other kind of value.

So, if anyone have this kind of problem, the solution is simple, actually.
I remade my dict in something like this:
i['trailers'] = i.get('trailers') or []
dictionary = [{'url': x['url'], 'type': x['type']} for x in i['images'] + i['trailers']]
This solution was made by #minboost here
Then, for the insertion, is something like that:
for i, dic in enumerate(dictionary):
query = ("""
INSERT INTO media (id, movie_id, url, type)
VALUES (%s, %s, %s, %s);
"""
)
data = (media_id, media_movie_id, dictionary[i]['url'], dictionary[i]['type'])
cur.execute (query, data)
conn.commit()
All working perfectly. :)

Iterating rows with Pyodbc

I am using Pyodbc to return a number of rows which are dumped into a JSON and sent to a server. I would like to iterate my SQL table and return all records. I am using cursor.fetchall() now, and the program returns one record. As shown below. When I use fetchone an error is returned AttributeError: 'unicode' object has no attribute 'SRNUMBER' and fetchmany returns one record as well. How do I successfully return all records? I am using Python 2.6.7
Code:
import pyodbc
import json
import collections
import requests
connstr = 'DRIVER={SQL Server};SERVER=server;DATABASE=ServiceRequest; UID=SA;PWD=pwd'
conn = pyodbc.connect(connstr)
cursor = conn.cursor()
cursor.execute("""
SELECT SRNUMBER, FirstName, LastName, ParentNumber
FROM MYLA311 """)
rows = cursor.fetchone()
objects_list = []
for row in rows:
d = collections.OrderedDict()
d['SRNUMBER']= row.SRNUMBER
d['FirstName']= row.FirstName
d['LastName']= row.LastName
d['ParentNumber']= row.ParentNumber
objects_list.append(d)
output = {"MetaData": {},
"SRData": d}
print output
j = json.dumps(output)
print json.dumps(output, sort_keys=True, indent=4)`
Output for fetchall and fetchmany:
{
"MetaData": {},
"SRData": {
"FirstName": "MyLAG",
"LastName": "ThreeEleven",
"ParentNumber": "021720151654176723",
"SRNUMBER": "1-3580171"
}
}

Use code from my answer here to build a list of dictionaries for the value of output['SRData'], then JSON encode the output dict as normal.
import pyodbc
import json
connstr = 'DRIVER={SQL Server};SERVER=server;DATABASE=ServiceRequest; UID=SA;PWD=pwd'
conn = pyodbc.connect(connstr)
cursor = conn.cursor()
cursor.execute("""SELECT SRNUMBER, FirstName, LastName, ParentNumber FROM MYLA311""")
# build list of column names to use as dictionary keys from sql results
columns = [column[0] for column in cursor.description]
results = []
for row in cursor.fetchall():
results.append(dict(zip(columns, row)))
output = {"MetaData": {}, "SRData": results}
print(json.dumps(output, sort_keys=True, indent=4))

For starters, the line
objects_list.append(d)
needs to be inside the for loop, not outside.