I am trying to do a batch insert for a single value (row), I am trying to use the executemany function to do so, but it will not work it returns TypeError: not all arguments converted during string formatting. However, when I add an extra value it does work
So...
It will return an error here:
entries_list=[("/helloworld"),
("/dfadfadsfdas")]
cursor.executemany('''INSERT INTO fb_pages(fb_page)
VALUES (%s)''', entries_list)
But not here:
entries_list=[("/helloworld", 'test'),
("/dfadfadsfdas", 'rers')]
cursor.executemany('''INSERT INTO fb_pages(fb_page, page_name)
VALUES (%s, %s)''', entries_list)
Writing ("/helloworld") is the same as writing "/helloworld". To create a tuple you need ("/helloworld", ).
What you're doing is actually running this:
cursor.executemany('''INSERT INTO fb_pages(fb_page)
VALUES (%s)''', ["/helloworld","/dfadfadsfdas"])
And now the error you're receiving makes perfect sense - you're supplying two arguments with only one place holder. Defining entries_list as follows would solve the problem:
entries_list=[("/helloworld",),
("/dfadfadsfdas",)]
In addition to making tuple with an additional , at the end as pointed by Karem
entries_list=[
("/helloworld",),
("/dfadfadsfdas",)
]
You could also just pass in lists and that will sort it out for you just as good when you are using parameterized query.
entries_list=[
["/helloworld"],
["/dfadfadsfdas"]
]
Related
I am using execute_values to insert a list of lists of values into a postgres database using psycopg2. Sometimes I get "not all arguments converted during string formatting", indicating that one of the values in one of the lists is not the expected data type (and also not NoneType). When it is a long list, it can be a pain to figure out which value in which list was causing the problem.
Is there a way to get postgres/psycopg2 to tell me the specific 'argument which could not be converted'?
If not, what is the most efficient way to look through the list of lists and find any incongruent data types per place in the list, excluding NoneTypes (which obviously are not equal to a value but also are not the cause of the error)?
Please note that I am not asking for help with the specific set of values I am executing it, but trying to find a general method to more quickly inspect the problem query so I can debug it.
I'm pretty new to SQLite and Python and have run into a bit of confusion. I'm trying to return all elements in a column that contain a substring which is passed to a function as a variable in Python. My code is running, but it's returning an empty result instead of the correct result.
Here's the code with the names generalized:
def myFunc(cursor,myString):
return cursor.execute("""select myID from Column where name like '%'+?'%' """,(myString,))
Like I said, the code does run without error but returns an empty result instead of the result that I know it should be. I'm assuming it has something to do with my use of the wildcard and/or question mark, but I can't be sure. Anyone have any ideas? Thanks in advance for your time/help! Also, this is my first post, so I apologize in advance if I missed any of the recommended protocols for asking questions.
Well, '%'+?'%' definitely isn't going to work—you're trying to concatenate with + on the left, but with no operator…
You can compute LIKE-search fields if you do it right—'%'+?+'%', in this case. That will cause problems with some databases (from not working, to doing a less efficient search), but, at least according to CL.'s comment, sqlite3 will be fine.
But the easy thing to do is to just substitute a complete parameter, rather than part of one. You can put % into the parameters, and it'll be interpreted just fine. So:
return cursor.execute("""select myID from Column where name like ?""",
('%'+myString+'%',))
And this also has the advantage that if you want to do a search for initial substrings ('foo%'), it'll be the same SQL statement but with a different parameter.
Try this:
def myFunc(cursor,myString):
return cursor.execute('select myID from Column where name like "{0}"'.format(myString))
I have the following Boolean statement in Python:
db_connection.query(
'select storage_time from traces where id=' + trace_id
).dictresult()[0]['storage_time'] == None
It basically checks if there is a value in storage_time and I would like to do the same thing in Matlab, but I can't find anything equivalent to None.
Could you please help me out?
Thanks
Equivalent to None values from Python are NULLs. And since you connect to your database
via Matlab Database Toolbox, you need to specify how NULL values retrieved from database
are to be presented in Matlab. This may be done by setting of 'NullNumberRead'
via calling of setdbprefs function from Matlab Database Toolbox. For instance, you can do this so
setdbprefs('NullNumberRead','NaN')
or so
setdbprefs('NullNumberRead','0')
Unfortunately, there is no guarantee that the NULL representation value specified by this way won't be confused
with real non-NULL values obtained as a result of your query (it is your own responsibility in this case to
guarantee the query always do not contain NaNs or zeros, respectively, among non-NULL values in its results).
But if you have to connect to PostgreSQL, as far as I know, there exists at least one Matlab and PostgreSQL connector that
respects NULLs in a fully consistent manner. This is a high-performance PostgreSQL client library PgMex.
In PostgreSQL both a value itself and the value elements can be NULL (for array types). This makes a representation of NULLs in Matlab less trivial than expected.
To illustrate the way PgMex uses to represent NULLs in Matlab, let us consider the following example. Suppose you have to retrieve results for a query returning one field myfield of float8[] type with two tuples. And suppose the value of myfield for the first tuple equals to NULL as itself, while for the second tuple the corresponding value equals {0,NULL,NaN}. Results are obtained as follows
(we assume that the argument of the very first command connect below is corrected properly and that the table mytable containing
myfield of float8[] type among its fields already exists within the respective database):
% Create the database connection
dbConn=com.allied.pgmex.pgmexec('connect',[...
'host=<yourhost> dbname=<yourdb> port=<yourport> '...
'user=<your_postgres_username> password=<your_postgres_password>']);
pgResult=com.allied.pgmex.pgmexec('exec',dbConn,...
'select myfield from mytable'); % perform the query
SMyField=com.allied.pgmex.pgmexec('getf',pgResult,...
'%float8[]',0); % retrieve the results
Here SMyField is a structure with three fields: valueVec, isNullVec and isValueNullVec.
isValueNullVec equals a column logical array [true;false], that is the entire value for the first tuple equals NULL,
while the value for the second tuple does not equal NULL as itself. isNullVec equals the following column cell array:
{[];[false,true,false]}. By this way it is possible to indicate that only the second element of the array being the value
of myfield for the second tuple is NULL. At last, valueVec equals to the column cell array {[];[0 0 NaN]}. Only
the first and the third elements of the second cell are relevant, in contrast to the second element of this array.
This is because isNullVec clearly indicates that this second element is NULL, so this zero value does not matter
(some default value is chosen for each particular data type).
What concerns your example, the respective code in Matlab may be the following (we assume that dbConn obtained as above already exists
as well as that the query below is correct with storage_time field of timestamp type and the variable trace_id being already defined):
pgResult=com.allied.pgmex.pgmexec('exec',dbConn,...
['select storage_time from traces where id=' + trace_id]); % perform the query
SStorageTime=com.allied.pgmex.pgmexec('getf',pgResult,...
'%timestamp',0); % retrieve the results
% check that the value for the first tuple is not NULL
isStorageTime=~SStorageTime.isValueNullVec(1);
Hence it is sufficient to check only isValueNullVec.
EDIT: There are free academic licenses for PgMex.
MATLAB's Database Toolbox has preferences how to handle NULL values. Depending on those settings you can get different values. See SETDBPREFS for details. You can change preferences in GUI as well.
By default you will get NaN if you read the data as numeric, and 'NULL' strings if you read as strings. In the first case for numbers check for NaN with ISNAN function.
null_idx = isnan(fetcheddata);
For strings use STRCMP:
null_idx= strcmp(upper(fetcheddata), 'NULL');
In addition, if you fetch the data as cell array, you may need to deal with them with CELLFUN or convert to matrix with CELL2MAT.
Matlab idiom is usually to use the isempty() function.
isempty(somefunction(someargument))
returns true if somefunction(someargument) returns any empty result, and false otherwise.
I have not worked with the Matlab DB toolbox much, so I'm not sure what the full translation of your Python statement is.
If you use this query you can check for True or False in instead of None:
trace_id_exists = db_connection.query("""\
select exists (
select 1
from traces
where id = %s
) "exists"
""" % trace_id
).dictresult()[0]['exists']
if trace_id_exists:
...
You could also return something else like 1 or 0.
I am fetching data from database which is stored in arrays, I have to match the output of this array with a string. But array outputs the result into Unicode format (u'aviesta',) thus it does not match the string.
My code.
// fblike is an arry in which the output of query stores.
for i in fblike:
if i=="Aviesta":
like=1
return render_to_response('showroom.html')
i have also try to encode this as variable.encode('utf8') but it only encode the specific element of an array such as i[0].encode('utf8') but i do not know which element of array have aviesta as a value.Thus i need to encode whole array but5 i don't know how to do that.
Updated::
In views.py is use
cursor = connection.cursor()
cursor.execute("SELECT name FROM django_facebook_facebooklike WHERE user_id = %s", request.user.id)
rowfb = cursor.fetchall() return render_to_response('showroom.html',{'rowfb':rowfbthis}
and print the {{rowfb}} variable in my template..and result array is
((u'Mukesh Chapagain',), (u'Ghrix Technologies Private Limited',), (u'FirstLALimo',), (u'Aviesta',), (u'Awkward Group',), (u'FB.Canvas.setDoneLoading',), (u'99recharge',), (u'AllThingsCustomized.com',), (u'celebrity aviesta',), (u'FTC',))
So please suggest me some way so that i can match the elements of array with the given string.
Thanks
Firstly, you should have posted the code as an update to your question, rather than a comment.
Secondly, I have no idea why you are accessing the data via a manual SQL query, rather than using Django's ORM. If you had done it the normal way, you would not be having this problem.
Finally, your problem has nothing to do with encodings. Your data is as follows (reposted for clarity):
((u'Mukesh Chapagain',), (u'Ghrix Technologies Private Limited',), (u'FirstLALimo',), (u'Aviesta',), (u'Awkward Group',), (u'FB.Canvas.setDoneLoading',), (u'99recharge',), (u'AllThingsCustomized.com',), (u'celebrity aviesta',), (u'FTC',))
This is a tuple of tuples. Each row of data is represented by a tuple, and in turn each column within that row is a tuple. In your case, since you're only selecting one column, you have a tuple of single-element tuples. That means, in each iteration of your loop, you have a tuple, not a string.
This would work:
for i in fblike:
if i[0] == "Aviesta":
like = 1
but to be honest, you would be better off going and doing a simple Python tutorial, and then going back to the Django tutorial and learning how to do queries via the ORM.
I don't know if your question has anything to do with arrays.
If you only need to find if the given string is in your array, you could simply do
if "Aviesta" in fblike:
like+=1
I'm having an issue when trying to pass a sqlite query to another function.
The issue is that the sqlite query MAY contains a list and therefore I cannot use *args as it unpacks the tuple but then ignores the list, example query I'm attempting to pass to the function:
'SELECT postname FROM history WHERE postname = ? COLLATE NOCASE', [u'Test']
So in this case I could use args as opposed to *args in the destination function, however I may have a sqlite query that doesn't contain a list and therefore I can't always do this e.g.
'SELECT * FROM history'
so I guess my question in a nutshell is how can I successfully pass a sqlite query to another function whether it contains a list or not, using args?
Can you just try,except it?
try:
func(*args)
except TypeError:
func(args)
Of course, this will catch TypeErrors inside your function as well. As such, you may want to create another function which actually deals with the unpacking and makes sure to give you an unpackable object in return. This also doesn't work for strings since they'll unpack too (see comments).
Here's a function which will make sure an object can be unpacked.
def unpackable(obj):
if hasattr(obj,'__iter__'):
return obj
else:
return (obj,)
func(*unpackable(args))
I would argue the best answer here is to try and ensure you are always putting in an iterable, rather than trying to handle the odd case of having a single item.
Where you have ('SELECT postname FROM history WHERE postname = ? COLLATE NOCASE', [u'Test']) in one place, it makes more sense to pass in a tuple of length one - ('SELECT * FROM history', ) as opposed to the string.
You haven't said where the strings are coming from, so it's possible you simply can't change the way the data is, but if you can, the tuple is the much better option to remove the edge case from your code.
If you truly can't do that, then what you want is to unpack any non-string iterable, checking for that can be done as shown in this question.