I am trying to find a way to parse a JSON string and save them into mysql.
This is my json!
{"title": My title, "desc": mydesc, "url": http//example.com}
From now i don't have problem to save all json into one column usining json.dumps() so actually I'm trying to parse each joson data string to send him to mysql table. Title | Desc | Url.
This is my python code for desc example (pyspider-resultdb.py)
def _parse(self, data):
for key, value in list(six.iteritems(data)):
if isinstance(value, (bytearray, six.binary_type)):
data[key] = utils.text(value)
if 'result' in data:
decoded = json.loads(data['result'])
data['result'] = json.dumps(decoded['desc'])
return data
def _stringify(self, data):
if 'result' in data:
decoded = json.loads(data['result'])
data['result'] = json.dumps(decoded['desc'])
return data
It's unclear from your question what you trying to achieve, but If your question is how to convert JSON to python dict and then load to the table, then that's how you can do it:
my_dict = json.loads('{"title": "foo", "dest": "bar"}')
curs.execute('INSERT INTO test (title, dest) values(%(title)s, %(dest)s)', my_dict)
Related
I have the following model:
class Car(models.Model):
data = models.TextField(default="[]")
Also, I have the following serializer:
class CarSerializer(serializers.ModelSerializer):
data = serializers.ListField(child=serializers.CharField())
The REST API gets data and saves it as text field. In my to_dict method of Car, I want to convert self.data into JSON and return the dict:
def to_dict(self):
result = dict()
result['data']= json.loads(self.data)
return result
But it fails with the error:
json.decoder.JSONDecodeError: Expecting value: line 1 column 2 (char 1)
As I understand, the reason is that self.data is:
"['a', 'b', 'c']"
And not:
'["a", "b", "c"]'
I'm familiar with JsonField, but since I'm using SQLite without JSON1 externation, I can't use it. How can I convert self.data to JSON?
You can use python json.dumps() method to convert string into json format and then use json.loads() to convert json into python object.
import json
def to_dict(self):
result = dict()
data = json.dumps(self.data)
result['data'] = json.loads(data)
return result
The simplest way to solve this problem is json.loads(self.data.replace('\'','\"')).
Replace ' to ".
Or you can try eval(self.data)
you can watch a sample here about the usage of eval
In my DynamoDB table I have a primary key of 'repo' and a list of repos in my table.
I run this code to return a scan of the table and this is my response
Code:
response = table.scan()
data = response['Items']
print(data)
Output:
[{'repo': 'MPriv32/TackleTakeHome'}, {'repo': 'MPriv32/fleetwood-k8s-project'}, {'repo': 'MPriv32/terransible'}]
I need to convert that into a list of just the item values like this
repo_list = ["MPriv32/TackleTakeHome", "MPriv32/fleetwood-k8s-project", "MPriv32/terransible"]
I looked online and tried to do this from another post on here
response = table.scan()
data = response['Items']
data = json.loads(data)
repo_lists = dict(item.values() for item in data)
print(data)
That returned this error when I tried running it
data = json.loads(data)
File "C:\Users\Mitchell Privett\AppData\Local\Programs\Python\Python39\lib\json\__init__.py", line 339, in loads
raise TypeError(f'the JSON object must be str, bytes or bytearray, '
TypeError: the JSON object must be str, bytes or bytearray, not list
I really just need a list of the values from my DynamoDB table and I'm not sure how to get exactly that, without the key in the list as well
response = table.scan()
repo_lists = []
for item in response['Items']:
if 'repo' in item:
repo_lists.append(item.repo)
else:
print('Not a valid item')
print(item)
print(repo_lists)
Try this, to handle cases where the item in you list doesn’t contains repo key in it.
I have the following model:
class BaseTransaction(models.Model):
"""Model to store JSON data"""
name = models.CharField(max_length=255)
json_data = JSONField(null=True)
If I create an instance with the following data:
base_transaction = models.BaseTransaction.objects.create(
name="Test Transaction",
json_data={{"sales": 3.24, "date": "2020-06-05"},
{"sales": 5.50, "date": "2020-06-04"},
{"sales": 256.53, "date": "2020-06-02"}}
)
How would I access the second row of data without a key? Or is this the wrong format for JSON? I am using this format because the original data is from a CSV and this is how it converts to JSON.
No, the above structure is not inJSON format. You can always validate if it's JSON or not using JSON Formatter & Validator
You would want to restructure is according to the rules of JSON, and manually if that can be done so. Once it's in JSON format, you can access the second row without keys using a for loop and a counter, e.g.
counter = 0
for (key in obj) {
counter+=1
if (counter == 2):
# Do anything
else:
print("Key: " + key)
print("Value: " + obj[key])
}
Hi i'm trying to save data that i get from this api into my Json column in my postgresql using sqlalchemy and python requests.
r = requests.get(api)
content = r.content
data = json.loads(content)
crawl_item = {}
crawl_item = session.query(CrawlItem).filter_by(site_id=3, href=list_id).first()
crawl_item.description = data['ad']['body']
crawl_item.meta_data = {}
crawl_item.meta_data["ward"] = data['ad_params']['ward']['value']
try:
session.commit()
except:
session.rollback()
raise
finally:
ret_id = crawl_item.id
session.close()
my model:
class CrawlItem(Base):
...
description = Column(Text)
meta_data = Column(postgresql.JSON)
i want to get the value of ward :
"ward": {
"id": "ward",
"value": "Thị trấn Trạm Trôi",
"label": " Phường, thị xã, thị trấn"
}
I already encoding my postgresql to utf-8 so other fields that are not json column (description = Column(Text)) save utf-8 characters normally only my json column data are not decode:
{
"ward":"Th\u1ecb tr\u1ea5n Tr\u1ea1m Tr\u00f4i"
}
description column:
meta_data column:
i had tried using :
crawl_item.meta_data["ward"] = data['ad_params']['ward']['value'].decode('utf-8')
but the ward data don't get save
I have no idea what is wrong, hope someone can help me
EDIT:
i checked the data with psql and got these:
description column:
meta_data column:
It seems like only meta_data json column have trouble with the characters
Sqlalchemy serializes JSON field before save to db (see url and url and url).
json_serializer = dialect._json_serializer or json.dumps
By default, the PostgreSQL' dialect uses json.dumps and json.loads.
When you work with Text column, the data is converted in the following flow:
str -> bytes in utf-8 encoding
When you work with JSON column for PostgreSQL dialect, the data is converted in the following flow:
dict -> str with escaped non-ascii symbols -> bytes in utf-8 encoding
You can override the serializer in your engine configuration using json_serializer field:
json_serializer=partial(json.dumps, ensure_ascii=False)
use "jsonb" data type for your json column or cast "meta_data" field to "jsonb" like this:
select meta_data::jsonb from your_table;
I have the below list from which I have to retrieve the port number I want the value 50051 but what I get is port=50051 I know I can retrieve this by iterating the list and using string operations but wanted to see if there is some direct way to access this.
r = requests.get(url_service)
data = {}
data = r.json()
#Below is the json after printing
[{'ServerTag': [ 'abc-service=true',
'port=50051',
'protocol=http']
}]
print(data[0]["ServiceTags"][1]) // prints port=50051
You can do something like this perhaps:
received_dic = {
'ServerTag': [ 'abc-service=true',
'port=50051',
'protocol=http']
}
ServerTag = received_dic.get("ServerTag", None)
if ServerTag:
port = list(filter(lambda x: "port" in x, ServerTag))[0].split("=")[1]
print(port)
# 50051
Considering you have the following JSON:
[
{
"ServerTag": ["abc-service=true", "port=50051", "protocol=http"]
}
]
You can extract your value like this:
from functools import partial
# ...
def extract_value_from_tag(tags, name, default=None):
tags = map(partial(str.split, sep='='), tags)
try:
return next(value for key, value in tags if key == name)
except StopIteration:
# Tag was not found
return default
And then you just:
# Providing data is the deserialized JSON as a Python list
# Also assuming that data is not empty and ServerTag is present on the first object
tags = data[0].get('ServerTag', [])
port_number = extract_value_from_tag(tags, 'port', default='8080')