Develop Reference

saving python object in postgres table with pickle

I have a python script which creates some objects.
I would like to be able to save these objects into my postgres database for use later.
My thinking was I could pickle an object, then store that in a field in the db.
But I'm going round in circles about how to store and retrieve and use the data.
I've tried storing the pickle binary string as text but I can't work out how to encode / escape it. Then how to load the string as a binary string to unpickle.
I've tried storing the data as bytea both with psycopg2.Binary(data) and without.
Then reading into buffer and encoding with base64.b64encode(result) but it's not coming out the same and cannot be unpickled.
Is there a simple way to store and retrieve python objects in a SQL (postgres) database?

Following the comment from #SergioPulgarin I tried the following which worked!
N.B Edit2 following comment by #Tomalak
Storing:
Pickle the object to a binary string
pickle_string = pickle.dumps(object)
Store the pickle string in a bytea (binary) field in postgres. Use simple INSERT query in Psycopg2
Retrieval:
Select the field in Psycopg2. (simple SELECT query)
Unpickle the decoded result
retrieved_pickle_string = pickle.loads(decoded_result)
Hope that helps anybody trying to do something similar!

Dynamically Create Query String for Django > MySQL from JSON

I am new to Django and am having some issues writing to a MySQL dB. What I am trying to do is loop through nested JSON data, and dynamically create a string to be used with a save() method.
I've looped through my nested JSON data and successfully created a string that contains the data I want to save in a single row to the MySQL table "mysqltable":
q = "station_id='thisid',stall_id='thisstaull',source='source',target='test'"
I then try to save this to the table in MySQL:
b = mysqltable(q)
b.save()
But I am getting the error:
TypeError: int() argument must be a string or a number, not 'mysqltable'
What I think is happening is that it doesn't like the fact I have created a string to use in b = mysqltable(q). When I just write out the statement like the below it works fine:
q = mysqltable(station_id='thisid',stall_id='thisstaull',source='source',target='test')
q.save()
But I am not sure how to take that string and make it available to use with b.save(). Any help would be greatly appreciated!

Instead string, create a dictionary, and then pass it directly to mysqltable:
mysqltable(**dictWithData)
Of course you can re-parse string onto dictionary, but this is useless work...

Entire JSON into One SQLite Field with Python

I have what is likely an easy question. I'm trying to pull a JSON from an online source, and store it in a SQLite table. In addition to storing the data in a rich table, corresponding to the many fields in the JSON, I would like to also just dump the entire JSON into a table every time it is pulled.
The table looks like:
CREATE TABLE Raw_JSONs (ID INTEGER PRIMARY KEY ASC, T DATE DEFAULT (datetime('now','localtime')), JSON text);
I've pulled a JSON from some URL using the following python code:
from pyquery import PyQuery
from lxml import etree
import urllib
x = PyQuery(url='json')
y = x('p').text()
Now, I'd like to execute the following INSERT command:
import sqlite3
db = sqlite3.connect('a.db')
c = db.cursor()
c.execute("insert into Raw_JSONs values(NULL,DATETIME('now'),?)", y)
But I'm told that I've supplied the incorrect number bindings (i.e. thousands, instead of just 1). I gather it's reading the y variable as all the different elements of the JSON.
Can someone help me store just the JSON, in it's entirety?
Also, as I'm obviously new to this JSON game, any online resources to recommend would be amazing.
Thanks!

.execute() expects a sequence, better give it a one-element tuple:
c.execute("insert into Raw_JSONs values(NULL,DATETIME('now'),?)", (y,))
A Python string is a sequence too, one of individual characters. So the .execute() call tried to treat each separate character as a parameter for your query, and unless your string is one character short that means it'll not provide the right number of parameters.
Don't forget to commit your inserts:
db.commit()
or use the database connection as a context manager:
with db:
# inserts executed here will automatically commit if no exceptions are raised.

You may also be interested to know about the built in sqlite modules adapters. These can convert any python object to an sqlite column both ways. See the standard documentation and the adapters section.

Python Database update error

Usually i use Django orm for making database related query in python but now i am using the python itself
I am trying to update a row of my mysql database
query ='UPDATE callerdetail SET upload="{0}" WHERE agent="{1}" AND custid="{2}"AND screenname="{3}" AND status="1"'.format(get.uploaded,get.agent,get.custid,get.screenname)
But i am getting the error
query ='UPDATE callerdetail SET upload="{0}" WHERE agent="{1}" AND custid="{2}"AND screenname="{3}" AND status="1"'.format(get.uploaded,get.agent,get.custid,get.screenname)
AttributeError: 'C' object has no attribute 'uploaded'
Please help me what is wrong with my query ?

Get is probably mapping to a c object. Try renaming your "get" object to something else.
Here is a list of reserved words. I don't see get in there, but it sound like it could be part of a c library that's being included. If you're including something with from x import *, you could be importing it without knowing.

In short - get probably isn't what you think it is.
However, before you go much further building SQL queries with string formatting, I strongly advise you not to! Search for "SQL injection" and you'll see why. Python DB API compliant libraries utilise "placeholders" which the library can use to insert the variables into a query for you providing any necessary escaping/quoting.
So instead of:
query ='UPDATE callerdetail SET upload="{0}" WHERE agent="{1}" AND custid="{2}"AND screenname="{3}" AND status="1"'.format(get.uploaded,get.agent,get.custid,get.screenname)
An example using SQLite3 (using ? as a placeholder - others use %s or :1 or %(name)s - or any/all of the above - but that'll be detailed in the docs of your library):
query = "update callerdetail set upload=? where agent=? and custid=? and screename=? and status=?"
Then when it comes to execute the query, you provide the values to be substituted as a separate argument:
cursor.execute(query, (get.uploaded, get.agent, get.custid, get.screenname))
If you really wanted, you could have a convenience function, and reduce this to:
from operator import attrgetter
get_fields = attrgetter('uploaded', 'agent', 'custid', 'screenname')
cursor.execute(query, get_fields(get))

python SPARQL query RESULTS BINDINGS: IF statement on BINDINGs value?

Using Python with SPARQLWrapper, JSON, urlib2 & cgi. Had trouble passing a working SPARQL query with some NULL values to python so I populated the blanks with a literal and will try to filter at the output. I have this results section example:
for result in results["results"]["bindings"]:
project = result["project"]["value"].encode('utf-8')
filename = result["filename"]["value"].encode('utf-8')
url = result["url"]["value"].encode('utf-8')
...and I print the %s. Is there a way to filter a value, i.e., IF VALUE NE "string" then PRINT? Or is there another workaround? I'm at the tail-end of a small project, I know I need a better wrapper, I just need to get these results filtered before I can move on. T very much IA...

I'm one of the developers of the SPARQLWrapper library, and the question had been already answered at the mailing list.
Regarding optionals values on the original query, the result set will come with no values for those variables. The problems is that we'd need to parse the query to populate such missing entries, and we want to avoid such parsing; therefore you'd need to check it for avoiding runtime problems with KeyError.
Usually I use a code like:
for result in results["results"]["bindings"]:
party = result["party"]["value"] if ("party" in result) else None