I am new to Django and am having some issues writing to a MySQL dB. What I am trying to do is loop through nested JSON data, and dynamically create a string to be used with a save() method.
I've looped through my nested JSON data and successfully created a string that contains the data I want to save in a single row to the MySQL table "mysqltable":
q = "station_id='thisid',stall_id='thisstaull',source='source',target='test'"
I then try to save this to the table in MySQL:
b = mysqltable(q)
b.save()
But I am getting the error:
TypeError: int() argument must be a string or a number, not 'mysqltable'
What I think is happening is that it doesn't like the fact I have created a string to use in b = mysqltable(q). When I just write out the statement like the below it works fine:
q = mysqltable(station_id='thisid',stall_id='thisstaull',source='source',target='test')
q.save()
But I am not sure how to take that string and make it available to use with b.save(). Any help would be greatly appreciated!
Instead string, create a dictionary, and then pass it directly to mysqltable:
mysqltable(**dictWithData)
Of course you can re-parse string onto dictionary, but this is useless work...
Related
I am using postgresql to extract some data from my database.One of the query i hit returns a nested dictionary to me for a specific date i have specified.
For eg:
query: select user_position_details from user_detail where last_login >= '2018-03-01';
This query return the follwing result:
"{\"user_position\": {\"LOGIN_s\": \"something\", \"X_SLS_AREA_s\": \"data\"}}"
"{\"user_position\": {\"LOGIN_so\": \"login_data\", \"X_SLS_AREA_s\": \"data\"}}"
I want only the field LOGIN_s out of this nested dictionay.
Is there any way i can do this using a specific query?
I have searched about this but could not find anything that would help.
Any help is appreciated.Thank you.
You are getting a json string output. Use the json module to convert it to JSON and access the required value using key.
Ex:
import json
s = ["{\"user_position\": {\"LOGIN_so\": \"something\", \"X_SLS_AREA_s\": \"data\"}}",
"{\"user_position\": {\"LOGIN_so\": \"login_data\", \"X_SLS_AREA_s\": \"data\"}}"]
for i in s:
v = json.loads(i)
print(v["user_position"].get(u'LOGIN_so'))
I'm trying to insert a list of element into a MySQL database (into a Blob column). This is an example of my code is:
myList = [1345,22,3,4,5]
myListString = str(myList)
myQuery = 'INSERT INTO table (blobData) VALUES (%s)'
cursor.execute(query, myListString)
Everything works fine and I have my list stored in my database. But, when I want to retrieve my list, because it's now a string I have no idea how to get a real integer list instead of a string.
For example, if now i do :
myQuery = 'SELECT blobData FROM db.table'
cursor.execute(myQuery)
myRetrievedList = cursor.fetch_all()
print myRetrievedList[0]
I ll get :
[
instead of :
1345
Is there any way to transform my string [1345,22,3,4,5] into a list ?
You have to pick a data format for your list, common solutions in order of my preference are:
json -- fast, readable, allows nested data, very useful if your table is ever used by any other system. checks if blob is valid format. use json.dumps() and json.loads() to convert to and from string/blob representation
repr() -- fast, readable, works across Python versions. unsafe if someone gets into your db. user repr() and eval() to get data to and from string/blob format
pickle -- fast, unreadable, does not work across multiple architectures (afaik). does not check if blob is truncated. use cPickle.dumps(..., protocol=(cPickle.HIGHEST_PROTOCOL)) and cPickle.loads(...) to convert your data.
As per the comments in this answer, the OP has a list of lists being entered as the blob field. In that case, the JSON seems a better way to go.
import json
...
...
myRetrievedList = cursor.fetch_all()
jsonOfBlob = json.loads(myRetrievedList)
integerListOfLists = []
for oneList in jsonOfBlob:
listOfInts = [int(x) for x in oneList]
integerListOfLists.append(listOfInts)
return integerListOfLists #or print, or whatever
I have what is likely an easy question. I'm trying to pull a JSON from an online source, and store it in a SQLite table. In addition to storing the data in a rich table, corresponding to the many fields in the JSON, I would like to also just dump the entire JSON into a table every time it is pulled.
The table looks like:
CREATE TABLE Raw_JSONs (ID INTEGER PRIMARY KEY ASC, T DATE DEFAULT (datetime('now','localtime')), JSON text);
I've pulled a JSON from some URL using the following python code:
from pyquery import PyQuery
from lxml import etree
import urllib
x = PyQuery(url='json')
y = x('p').text()
Now, I'd like to execute the following INSERT command:
import sqlite3
db = sqlite3.connect('a.db')
c = db.cursor()
c.execute("insert into Raw_JSONs values(NULL,DATETIME('now'),?)", y)
But I'm told that I've supplied the incorrect number bindings (i.e. thousands, instead of just 1). I gather it's reading the y variable as all the different elements of the JSON.
Can someone help me store just the JSON, in it's entirety?
Also, as I'm obviously new to this JSON game, any online resources to recommend would be amazing.
Thanks!
.execute() expects a sequence, better give it a one-element tuple:
c.execute("insert into Raw_JSONs values(NULL,DATETIME('now'),?)", (y,))
A Python string is a sequence too, one of individual characters. So the .execute() call tried to treat each separate character as a parameter for your query, and unless your string is one character short that means it'll not provide the right number of parameters.
Don't forget to commit your inserts:
db.commit()
or use the database connection as a context manager:
with db:
# inserts executed here will automatically commit if no exceptions are raised.
You may also be interested to know about the built in sqlite modules adapters. These can convert any python object to an sqlite column both ways. See the standard documentation and the adapters section.
Need help in understanding what is happening here and a suggestion to avoid this!
Here is my snippet:
result = [list of dictionary objects(dictionary objects have 2 keys and 2 String values)]
copyResults = list(results);
## Here I try to insert each Dict into MongoDB (Using PyMongo)
for item in copyResults:
dbcollection.save(item) # This is all saving fine in MongoDB.
But when I loop thru that original result list again it shows dictionary objects with a new field added
automatically which is ObjectId from MongoDB!
Later in code I need to transform that original result list to json but this ObjectId is causing issues.No clue why this is getting added to original list.
I have already tried copy or creating new list etc. It still adds up ObjectId in the original list after saving.
Please suggest!
every document saved in mongodb requires '_id' field - which has to be unique among documents in the collection. if you don't provide one, mongodb will automatically create one with ObjectId (bson.objectid.ObjectId for pymongo)
If you need to export documents to json, you have to pop '_id' field before jsonifying it.
Or you could use:
rows['_id'] = str(rows['_id'])
Remember to set it back if you then need to update
I apologize if this is a basic fix and the length of the post, but I am new to Python. I've also included a large chunk of the script for context purposes.
I am using a script to pull scanning data from JSON into a MySQL DB. The script was working fine until an update was released.
Now when I run the script I receive the error:
for result in resultc['response']['results']:
TypeError: string indices must be integers
Before this update I knew the data types for each value, but this has changed and I cannot pinpoint where. Is there a way to convert each value to be recognized as a string?
# Send the cumulative JSON and then populate the table
cumresponse, content = SendRequest(url, headers, cumdata)
resultc = json.loads(content)
off = 0
print "\nFilling cumvulndata table with vulnerabilities from the cumulative database. Please wait..."
for result in resultc['response']['results']:
off += 1
print off, result
cursor.execute ("""INSERT INTO cumvulndata(
offset,pluginName,repositoryID,
severity,pluginID,hasBeenMitigated,
dnsName,macAddress,familyID,recastRisk,
firstSeen,ip,acceptRisk,lastSeen,netbiosName,
port,pluginText,protocol) VALUES
(%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,(FROM_UNIXTIME(%s)),%s,%s,(FROM_UNIXTIME(%s)),%s,%s, %s,%s)""" , (off,result["pluginName"],result["repositoryID"]),result["severity"]),
result["pluginID"]), result["hasBeenMitigated"]),result["dnsName"],
result["macAddress"],result["familyID"]),result["recastRisk"]),
result["firstSeen"],result["ip"],result["acceptRisk"],result["lastSeen"],
result["netbiosName"],result["port"],result["pluginText"],result["protocol"]))
Put this before the for loop to work out which object is the string (I guess it's probably the second one)
print type(resultc)
print type(resultc['response'])