I have a dictionary in python that I created from a JSON file. Now, I need to pass its values to insert into a postgresql database.
dictionary
if(i['trailers']):
a = [
{'url': i['images'][0]['url'], 'type': i['images'][0]['type']},
{'url': i['images'][1]['url'], 'type': i['images'][1]['type']},
{'url': i['trailers'][0]['url'], 'type': 'Trailer'},
{'url': i['trailers'][1]['url'], 'type': 'Trailer'},
]
else:
a = [
{'url': i['images'][0]['url'], 'type': i['images'][0]['type']},
{'url': i['images'][1]['url'], 'type': i['images'][1]['type']},
]
length = len(a)
Here, I created the dictionary. If there is anything inside the trailer it goes A, else it goes B. In the B case, trailers doesn't exists. Then I get the length of the dictionary.
Now, I will try to insert these elements into the table media, that depends on movies. Their relation is movie(1):media(n).
INSERT INTO media
for x in range(length):
query = ("""INSERT INTO media VALUES (%s, %s, %(url)s, %(type)s);""")
data = (media_id, media_movie_id)
cur.execute(query, data)
conn.commit()
media_id += 1
Here is what I'm trying to do. Since movie can have many media, I'll create a for to move through all the elements and inserting them in the table. With their id being incremented.
The problem is, I don't know how to do this quiet right in Python, since I always create a query and a data and then cur.execute it and the example that I got, was a entire dictionary being used, without any other kind of value.
So, if anyone have this kind of problem, the solution is simple, actually.
I remade my dict in something like this:
i['trailers'] = i.get('trailers') or []
dictionary = [{'url': x['url'], 'type': x['type']} for x in i['images'] + i['trailers']]
This solution was made by #minboost here
Then, for the insertion, is something like that:
for i, dic in enumerate(dictionary):
query = ("""
INSERT INTO media (id, movie_id, url, type)
VALUES (%s, %s, %s, %s);
"""
)
data = (media_id, media_movie_id, dictionary[i]['url'], dictionary[i]['type'])
cur.execute (query, data)
conn.commit()
All working perfectly. :)
Related
im a beginner in python , Trying to connect access database to python with a json file loaded in my program so I can read it and eventually analyze it for certain things. But I can't connect to it and tried different approaches still getting the same error.
import mysql.connector
import json
# create the key
from mysql.connector import cursor
mydb = mysql.connector.connect(host='localhost', port='3306', user='root', password='nihad147', database='tweets')
mycursor = mydb.cursor()
sql_tweet = """INSERT INTO tweet ( tweet_id,
id_user,
text,
tweet_location,
created_at,
name_screen,
categorie_id,
)
VALUES (%s,%s,%s,%s,%s,%s,%s)"""
sql_user = """INSERT INTO tweetuser (
id_user,
name_screen,
location_user,
count_followers,
friends_count,
statuse_count)
VALUES (%s,%s,%s,%s,%s,%s)"""
sql_location = """"insert into tweet_location (
location_id,
latitude,
longitude
tweet_id
VALUES(%s,%s,%s,%s)"""
myJsonFile = open('tweets.json', encoding="utf-8")
mycursor.execute("DELETE FROM tweet")
mycursor.execute("DELETE FROM tweetuser")
mycursor.execute("DELETE FROM tweet_location")
c = 0
for line in myJsonFile:
c = c + 1
print("tweet number ", c, " is uploading to the server")
data = json.loads(line)
# insert into tweet
val_tweet = (
data['tweet_id'], data['user_id_str'], data['raw_text'],data['location']['address']['city'],data['date'], data['user_screen_name'])
mycursor.execute(sql_tweet,sql_location, val_tweet)
mydb.commit()
# testing ifthe user already exist
user = "SELECT * FROM tweetuser WHERE id_user = '" + str(data['user_id_str']) + "'"
mycursor.execute(user)
myresult = mycursor.fetchall()
row_count = mycursor.rowcount
if row_count == 0:
val_user = (data['user_id_str'], data['user_screen_name'], data['location']['address']['city'],data['user_followers_count'],
data['user_friends_count'], data['user_statuses_count'])
mycursor.execute(sql_user, val_user)
mydb.commit()
print('done')
here's an example of json file data :
{
"tweet_id":"1261276320878788609",
"date":"Fri May 15 12:44:42 +0000 2020",
"raw_text":"برنامج وطني لدعم المبدعين في مواجهة #كورون",
"geo_source":"user_location",
"location":{
"address":{
"country":"Tunisia",
"country_code":"tn",
"state_district":"غزالة",
"county":"العرب",
"state":"Bizerte"
},
"response":"{'place_id': 235309103, 'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. https://osm.org/copyright', 'osm_type': 'relation', 'osm_id': 7124228, 'boundingbox': ['37.105957', '37.2033466', '9.4739053', '9.6124953'], 'lat': '37.1551868', 'lon': '9.54834183807249', 'display_name': 'العرب, غزالة, Bizerte, Tunisia', 'class': 'boundary', 'type': 'administrative', 'importance': 0.45, 'icon': '/data/nominatimimages/mapicons/poi_boundary_administrative.p.20.png','address':{'county': 'العرب', 'state_district': 'غزالة', 'state': 'Bizerte', 'country': 'Tunisia', 'country_code': 'tn'}}",
"geohash":"snwg37buskzd",
"query_term":"arab",
"lon":9.54834183807249,
"lat":37.1551868
},
"user_friends_count":61,
"user_description":"I love UAE and his great leadership",
"user_created_at":"Wed Oct 09 11:41:41 +0000 2013",
"user_screen_name":"SikandarMirani",
"user_id_str":"706377881",
"user_verified":false,
"user_statuses_count":50804,
"user_followers_count":946,
"user_location":"Dubai United Arab Emirates"
}
thanks to you guys , i was able to solve the previous error since i didn't check tha data type of the id user it has to be bigint not int since it's a large data .
i had no problem connecting my jsonfile to my database but it got inserted only in tweetuser table but not in tweet table .
the tweet table is empty.
i would appreciate any kind of help thank you
The error
mysql.connector.errors.DataError: 1264 (22003): Out of range value for column 'id_user' at row 1
suggests that the value you are trying to use as the id_user is numerically too large.
Since you haven't posted the table definitions, my guess is you are using MEDIUMINT or SMALLINT or TINYINT for id_user and the actual user ID that you are trying to write into the database is too large for that data type.
In your example user_id_str is 706377881, however, the maximum value for MEDIUMINT is 8388607 and 16777215 (unsigned), respectively.
Check the data types in the table definitions.
You are connecting to your DB, that is not the problem.
The problem is that the user id that you are trying to insert has a length that surpasses the maximum allowed by MySQL for the datatype of that field. See here and here for more info related to your error.
My code is pulling JSON data from an api as such, however I'm having trouble getting my code to read from the JSON objects.
url = 'https://api.test.net/Vacancy'
payload = {
"APIKey": "0000",
"Action": "GetAllVacancies",
"Content-Type" : "json",
}
headers = {}
r = requests.post(url, data=json.dumps(payload), headers=headers)
print(r.content)
cursor = mydb.cursor()
json_obj = r.json()
for index in json_obj:
cursor.execute("INSERT INTO apidata (VacancyName, Department, Location) VALUES (%s, %s, %s)", (json_obj[index]["VacancyName"], (json_obj[index]["Department"], (json_obj[index]["Location"]))
cursor.close()
My JSON response looks like this
{
"isError":false,
"Status":0,
"Message":"",
"Result":[
{
"VacancyName":"Test Vacancy",
"VacancyDescription":"test data 123",
"Location":"location 1",
"Department":"Finance",
However I keep getting the error
TypeError 'bool' object is not subscriptable
Im trying to pull this JSON data and send it to a database, thanks!
look at
for index in json_obj:
cursor.execute("INSERT INTO apidata (VacancyName, Department, Location) VALUES (%s, %s, %s)", (json_obj[index]["VacancyName"], (json_obj[index]["Department"], (json_obj[index]["Location"]))
you loop over keys in json_obj and first key is isError. so json_obj[index] is False, thus not subscriptuble and you get error when try to access key ["VacancyName"].
You need to loop over elements in json_obj["Result"]
for result in json_obj["Result"]:
cursor.execute("INSERT INTO apidata (VacancyName, Department, Location) VALUES (%s, %s, %s)", (result["VacancyName"], result["Department"], result["Location"]))
Your JSON object is a dict. Iterating over a dict iterates over the keys.
The first key in your object is "isError" so when you try to access json_obj[index]["Department"] that is equivalent to (json_obj["isError"]["Department"] which is False["Department"] which gives the error you are seeing.
Please in future include the full error message including the traceback. It makes it a lot easier to answer these questions if you do.
I am trying to to insert record in mongodb but I dont want duplication so I am using update command with upsert=true
import pymongo
client = pymongo.MongoClient(settings.MONGO_DB_URI)
db = self.client[settings.MONGO_DB_NAME]
filter = {
'my_id': '1234',
'name': 'alok'
}
record = {
'my_id': '1234',
'name': 'alok',
'marks': 26
}
status = db['mycollection'].update(filter, {'$setOnInsert': record}, upsert=True)
print('id is ', status['my_id']) # this will not work but I want such behaviour
This code will insert record only if there is no existing record with matching filter values. So there are two case:
It will insert record
It will not insert record if already exist
In both the case I want to get my_id. How can I get my_id when update command execute?
You can search for the document and then print out its ID
print('id is ', db['mycollection'].find_one(filter)['my_id'])
I am trying to search for all users in an sql database whose first names are "blah" and return that data to my html through an ajax call. I have this functioning with a single user like this:
user = db.execute(
'SELECT * FROM user WHERE genres LIKE ?', (str,)
).fetchone()
user_details = {
'first': user['first'],
'last': user['last'],
'email': user['email']
}
y = json.dumps(user_details)
return jsonify(y)
Now for multiple users I want the struct to look something like this:
users{
user1_details = {
'first': user['first'],
'last': user['last'],
'email': user['email']
}
user2_details = {
'first': user2['first'],
'last': user2['last'],
'email': user2['email']
}
user3_details = {
'first': user3['first'],
'last': user3['last'],
'email': user3['email']
}
}
generating each user_details in a loop. I know I can use fetchall() to find all the users, but how do I concatenate the details?
Fetch all the rows after the query, then structure the results as you'd like.
Example:
db = mysql.connection.cursor()
# query
db.execute('SELECT * FROM user')
# returned columns
header = [x[0] for x in db.description]
# returned rows
results = db.fetchall()
#data to be returned
users_object = {}
#structure results
for result in results:
users_object[result["user_id"]] = dict(zip(header,result))
return jsonify(users_object)
As you can see in under "#structure results", you just loop through the results and insert the data for each row into the users_object with key equal to "user_id" for example.
If you want the results in an array just convert users_object into an array e.g. users_array and append the dict to the array within the loop instead
The keys in the desired users dictionary do not seem particularly useful so you could instead build a list of user dicts. It's easy to go directly from fetchall() to such a list:
result = db.execute('SELECT * FROM user WHERE genres LIKE ?', (str,))
users = [{'first': first, 'last': last, 'email': email} for first, last, email in result.fetchall()]
return jsonify(users)
To return a dict containing the user list:
return jsonify({'users': users})
I have a set of data that a user needs to query using their own query string. The current solution creates a temporary in-memory sqlite database that the query is run against.
The dataset is a list of "flat" dictionaries, i.e. there is no nested data. The query string does not need to be SQL, but it should be simple to define using an existing query framework.
It needs to support ordering (ascending, descending, custom) and filtering.
The purpose of this question is to get a range of different solutions that might work for this use case.
import sqlite3
items = [
{'id': 1},
{'id': 2, 'description': 'This is a description'},
{'id': 3, 'comment': 'This is a comment'},
{'id': 4, 'height': 1.78}
]
# Assemble temporary sqlite database
conn = sqlite3.connect(':memory:')
cur = conn.cursor()
knownTypes = { "id": "real", "height": "real", "comment": "text" }
allKeys = list(set().union(*(d.keys() for d in items)))
allTypes = list(knownTypes.get(k, "text") for k in allKeys)
createTable_query = "CREATE TABLE data ({});".format(", ".join(["{} {}".format(x[0], x[1]) for x in zip(allKeys, allTypes)]))
cur.execute(createTable_query)
conn.commit()
qs = ["?" for i in range(len(allKeys))]
insertRow_query = "INSERT INTO data VALUES ({});".format(", ".join(qs))
for p in items:
vals = list([p.get(k, None) for k in allKeys])
cur.execute(insertRow_query, vals)
conn.commit()
# modify user query here
theUserQuery = "SELECT * FROM data"
# Get data from query
data = [row for row in cur.execute(theUserQuery)]
YAQL is what I'm looking for.
It doesn't do SQL, but it does execute a query string - which is a simple way to do complex user-defined sorting and filtering.
There's a library called litebox that does what you want. It is backed by SQLite.
from litebox import LiteBox
items = [
{'id': 1},
{'id': 2, 'description': 'This is a description'},
{'id': 3, 'comment': 'This is a comment'},
{'id': 4, 'height': 1.78}
]
types = {"id": int, "height": float, "comment": str}
lb = LiteBox(items, types)
lb.find("height > 1.5")
Result: [{'id': 4, 'height': 1.78}]