I'm trying to insert a list of element into a MySQL database (into a Blob column). This is an example of my code is:
myList = [1345,22,3,4,5]
myListString = str(myList)
myQuery = 'INSERT INTO table (blobData) VALUES (%s)'
cursor.execute(query, myListString)
Everything works fine and I have my list stored in my database. But, when I want to retrieve my list, because it's now a string I have no idea how to get a real integer list instead of a string.
For example, if now i do :
myQuery = 'SELECT blobData FROM db.table'
cursor.execute(myQuery)
myRetrievedList = cursor.fetch_all()
print myRetrievedList[0]
I ll get :
[
instead of :
1345
Is there any way to transform my string [1345,22,3,4,5] into a list ?
You have to pick a data format for your list, common solutions in order of my preference are:
json -- fast, readable, allows nested data, very useful if your table is ever used by any other system. checks if blob is valid format. use json.dumps() and json.loads() to convert to and from string/blob representation
repr() -- fast, readable, works across Python versions. unsafe if someone gets into your db. user repr() and eval() to get data to and from string/blob format
pickle -- fast, unreadable, does not work across multiple architectures (afaik). does not check if blob is truncated. use cPickle.dumps(..., protocol=(cPickle.HIGHEST_PROTOCOL)) and cPickle.loads(...) to convert your data.
As per the comments in this answer, the OP has a list of lists being entered as the blob field. In that case, the JSON seems a better way to go.
import json
...
...
myRetrievedList = cursor.fetch_all()
jsonOfBlob = json.loads(myRetrievedList)
integerListOfLists = []
for oneList in jsonOfBlob:
listOfInts = [int(x) for x in oneList]
integerListOfLists.append(listOfInts)
return integerListOfLists #or print, or whatever
Related
I use Scrapy to write a spider to get something from a website.And I want to put the item into database.In my code,there are five items ,two of items are unicode type,so I can put it into database directily,but two of items are list type,How can I put it into database?Here is my code about the items whose type are list:
descr = sel.xpath(
'//*[#id="root"]/div/main/div/div[2]/div[1]/div[2]/div/div/div/div[2]/div/div/span/p[1]/text()').extract()
print 'type is:', type(descr)
answer_time = sel.xpath(
'//*[#id="root"]/div/main/div/div[2]/div[1]/div[2]/div/div/div/div[2]/div/div/div/a/span/#data-tooltip').extract()
print 'type is:', type(answer_time)
It should depend strongly on how the data will be used once in the db and what kind of db you're using.
If you're using something like MongoDB, it fully supports just adding the list as part of the record. Some relational dbs such as postgresql have support for JSON column types.
Beyond that there are two main options. 1) Cast the list to a string using something like JSON and save it in a text column along with your other unicode columns. 2) Take full advantage of a relational db and use a second table to store a one:many relationship.
Casting it a string is much easier/faster to dev. Great for rapid prototyping. However, using multiple tables has huge efficiency perks if the data is going to be read for analysis on any data set that is not small.
From a mobile client, we're sending a list of lists as part of a json to the server(python 2.7). This needs to be split into individual lists for processing.
Example json = {"table":"[[1,2,3],[2,3,4],....]"}
What I need is to extract individual lists [1,2,3],[2,4,5] etc, add an extra object at the start of the list and insert them into a postgres DB.
For extracting the json I use :
post_data = json.loads(request.data)
table = post_data['table']
print table prints u'[[1,2,3],[2,3,4]]'.
I checked a couple of questions and use the following snippet for processing and inserting into db.
tbl = post_data['table']
table = ast.literal_eval(tbl)
print table
tup_table = []
for lst in table :
lst.insert(0,userid)
tup_table += [tuple(lst)]
connection = engine.raw_connection()
cur = connection.cursor()
records_list_template = ','.join(['%s'] * len(tup_table))
insert_query = 'insert into stats(userid,rid,val,fid) values {0}'.format(records_list_template)
cur.execute(insert_query, tup_table)
connection.commit()
connection.close()
The problem however is ast.literal_eval() works fine in this example but does not work when the list in json contains values of other types. For e.g when the list is [[1,abc,3],[2,bcd,4]] it returns an error ValueError('malformed string')
I've tried using table.encode('utf-8') instead. That returns a list type but the for loop loop does not extract a sub-list and it extracts each character in the list separately.
Is there a better way of extracting a list from the unicode and fetching the sublist for creating the tuple ?
Like your error message indicates, the given json file is malformed. You forgot the quotes.
Change [[1,abc,3],[2,bcd,4]] to [[1,"abc",3],[2,"bcd",4]].
I am using python 2.7 and psycopg2 for connecting to postgresql
I read a bunch of data from a source which has strings like 'Aéropostale'. I then store it in the database. However, in postgresql it is ending up as 'A\u00e9ropostale'. But I want it to get stored as 'Aéropostale'.
The encoding of postgresql database is utf-8.
Please tell me how can I store the actual string 'Aéropostale' instead.
I suspect that the problem is happening in python. Please advice.
EDIT:
Here is my data source
response_json = json.loads(response.json())
response is obtained via service call and looks like:
print(type(response.json())
>> <type'str'>
print(response.json())
>> {"NameRecommendation": ["ValueRecommendation": [{"Value": "\"Handmade\""}, { "Value": "Abercrombie & Fitch"}, {"Value": "A\u00e9ropostale"}, {"Value": "Ann Taylor"}}]
From the above data, my goal is to construct a list of all ValueRecommendation.Value and store in a postgresql json datatype column. So the python equivalent list that I want to store is
py_list = ["Handmade", "Abercrombie & Fitch", "A\u00e9ropostale", "Ann Taylor"]
Then I convert py_list in to json representation using json.dumps()
json_py_list = json.dumps(py_list)
And finally, to insert, I use psycopg2.cursor() and mogrify()
conn = psycopg2.connect("connectionString")
cursor = conn.cursor()
cursor.execute(cursor.mogrify("INSERT INTO table (columnName) VALUES (%s), (json_py_list,)))
As I mentioned earlier, using the above logic, string with special charaters like è are getting stored as utf8 character code.
Please spot my mistake.
json.dumps escapes non-ASCII characters by default so its output can work in non-Unicode-safe environments. You can turn this off with:
json_py_list = json.dumps(py_list, ensure_ascii=False)
Now you will get UTF-8-encoded bytes (unless you change that too with encoding=) so you'll need to make sure your database connection is using that encoding.
In general it shouldn't make any difference as both forms are valid JSON and even with ensure_ascii off there are still characters that get \u-encoded.
I have what is likely an easy question. I'm trying to pull a JSON from an online source, and store it in a SQLite table. In addition to storing the data in a rich table, corresponding to the many fields in the JSON, I would like to also just dump the entire JSON into a table every time it is pulled.
The table looks like:
CREATE TABLE Raw_JSONs (ID INTEGER PRIMARY KEY ASC, T DATE DEFAULT (datetime('now','localtime')), JSON text);
I've pulled a JSON from some URL using the following python code:
from pyquery import PyQuery
from lxml import etree
import urllib
x = PyQuery(url='json')
y = x('p').text()
Now, I'd like to execute the following INSERT command:
import sqlite3
db = sqlite3.connect('a.db')
c = db.cursor()
c.execute("insert into Raw_JSONs values(NULL,DATETIME('now'),?)", y)
But I'm told that I've supplied the incorrect number bindings (i.e. thousands, instead of just 1). I gather it's reading the y variable as all the different elements of the JSON.
Can someone help me store just the JSON, in it's entirety?
Also, as I'm obviously new to this JSON game, any online resources to recommend would be amazing.
Thanks!
.execute() expects a sequence, better give it a one-element tuple:
c.execute("insert into Raw_JSONs values(NULL,DATETIME('now'),?)", (y,))
A Python string is a sequence too, one of individual characters. So the .execute() call tried to treat each separate character as a parameter for your query, and unless your string is one character short that means it'll not provide the right number of parameters.
Don't forget to commit your inserts:
db.commit()
or use the database connection as a context manager:
with db:
# inserts executed here will automatically commit if no exceptions are raised.
You may also be interested to know about the built in sqlite modules adapters. These can convert any python object to an sqlite column both ways. See the standard documentation and the adapters section.
... but it's not escaping it the same way twice.
I'm trying to upload ASCII output from gpg to a website. So, the bit I've got, so far, just queries the table, shows me the data it got, and then shows it to me after it encodes it for a HTTP POST request:
cnx = connect()
sql = ("SELECT Data FROM SomeTable")
cursor = cnx.cursor()
cursor.execute(sql)
for (data) in cursor:
print "encoding : %s" % data
postdata = urllib.urlencode( { "payload" : data } )
print "encoded as %s" % postdata
... but what I get is:
encoding : -----BEGIN PGP MESSAGE-----
Version: GnuPG v1.4.12 (GNU/Linux)
.... etc...
encoded as payload=%28u%27-----BEGIN+PGP+MESSAGE-----%5CnVersion%3A+GnuPG+v1.4.12+... etc ...
The part to notice is that the newlines aren't getting turned into %0A, like I'd expect. Instead, they're somehow getting escaped into "\n", and then the backslashes are escaped to %5C, so a newline becomes "%5Cn". Even stranger, the data gets prepended with %28u%27, which comes out to "(u'".
Oddly, if I just do a basic test with:
data = "1\n2"
print data
print urllib.urlencode( { "payload" : data } )
I get what I expect, newlines turn into %0A...
1
2
payload=1%0A2
So, my hunch is that the data element returned from the mysql query isn't the same kind of string as my literal "1\n2" (maybe a 1-element dict... dunno), but I don't have the Python kung-fu to know how to inspect it.
Anybody know what's going on, here, and how I can fix it? If not, any suggestions for how to POST this via HTTP with everything getting escaped properly?
Assuming connect() is a function from some DB-API 2.0 compatible database interface (like the built-in sqlite3, or the most popular mysql interface), for (data) in cursor: is iterating Row objects, not strings.
When you print it out, you're effectively printing str(data) (by passing it to a %s format). If you want to encode the same thing, you have to encode str(data).
However, a better way to do it is to handle the rows as rows (of one column) in the first place, instead of relying on str to do what you want.
PS, if you were trying to rely on tuple unpacking to make data the first element of each row, you're doing it wrong:
for (data) in cursor:
… is identical to:
for data in cursor:
If you want a one-element tuple, you need a comma:
for data, in cursor:
(You can also add the parens if you want, but they still don't make a difference either way.)
Specifically, iterating the cursor will call the optional __iter__ method, which returns the cursor itself, then loop calling the next method on it, which does the same thing as calling fetchone() until the result set is exhausted, and fetchone is documented to return "a single sequence", whose type isn't defined. In most implementations, that's a special row type, like sqlite3.Row, which can be accessed as if it were a tuple but has special semantics for things like printing in tabular format, allowing by-name access, etc.