I'm using the dataset library to attempt to back up a postgres database into an sqlite file. The code I'm running goes as follows:
local_db = "sqlite:///backup_file.db"
with dataset.connect(local_db) as save_to:
with dataset.connect(postgres_db) as download_from:
for row in download_from['outlook']:
save_to['outlook'].insert(row)
If I print one row of the table, it looks like this:
OrderedDict([
('id', 4400),
('first_sighting', '2014-08-31'),
('route', None),
('sighted_by', None),
('date', None)
])
However, when I get to the line save_to['outlook'].insert(row) I get an error with the following stack trace:
Traceback (most recent call last):
File "/home/anton/Development/Python/TTC/backup_db.py", line 25, in <module>
save_to['outlook'].insert(dict(row))
File "/home/anton/.virtualenvs/flexity/lib/python3.6/site-packages/dataset/table.py", line 79, in insert
row = self._sync_columns(row, ensure, types=types)
File "/home/anton/.virtualenvs/flexity/lib/python3.6/site-packages/dataset/table.py", line 278, in _sync_columns
self._sync_table(sync_columns)
File "/home/anton/.virtualenvs/flexity/lib/python3.6/site-packages/dataset/table.py", line 245, in _sync_table
self._table.append_column(column)
File "/home/anton/.virtualenvs/flexity/lib/python3.6/site-packages/sqlalchemy/sql/schema.py", line 681, in append_column
column._set_parent_with_dispatch(self)
File "/home/anton/.virtualenvs/flexity/lib/python3.6/site-packages/sqlalchemy/sql/base.py", line 431, in _set_parent_with_dispatch
self._set_parent(parent)
File "/home/anton/.virtualenvs/flexity/lib/python3.6/site-packages/sqlalchemy/sql/schema.py", line 1344, in _set_parent
self.key, table.fullname))
sqlalchemy.exc.ArgumentError: Trying to redefine primary-key column 'id' as a non-primary-key column on table 'outlook'
Any ideas as to what I'm doing wrong? I've tried this in python 2.7.14 and 3.6.3
Assuming you have a schema and table made for "outlook", did you make a PK field? Did you let sqlite decide which field to make a PK field?
It is highly that you are trying to insert id twice. Once, sqlite is inserting itself, and other comes from the other table records.
I figured it out! So, the trick is that by default the database library makes tables with an auto-incrementing integer primary-key. But, my data already has an 'id' column. In order to avoid this problem, I should define my table before I try to add lines to it, and define it with no primary key as follows:
with dataset.connect(local_db) as save_to:
with dataset.connect(postgres_db) as download_from:
table_to_save_to = save_to.create_table('outlook', primary_id=False)
for row in download_from['outlook']:
table_to_save_to.insert(row)
By doing .create_table(table_name, primary_key=False) I can make sure that i can insert my own id values into the table.
I found this solution by reading the docs.
Related
I'm missing a table called 'timetable'.
Traceback (most recent call last):
File "C:\Users\samsshow\Desktop\microsoft-teams-class-attender-main\bot.py", line 265, in <module>
add_timetable()
File "C:\Users\samsshow\Desktop\microsoft-teams-class-attender-main\bot.py", line 108, in add_timetable
c.execute("INSERT INTO timetable VALUES ('%s','%s','%s','%s')"%(name,start_time,end_time,day))
sqlite3.OperationalError: no such table: timetable
Did you create the table? If not:
CREATE TABLE timetable (id INTEGER PRIMARY KEY AUTOINCREMENT, name VARCHAR(255), start_time DATETIME, end_time DATETIME, day DATE);
If you already did, maybe you misspelled your table name.
You don't have the table timetable in your database, but you probably already figured this out.
It's impossible to say anything else based on the information you provided. What are you trying to do? What have you tried? Error log is not enough.
I've below two tables in tables.py -
db.define_table('spr_details',
Field('SPR_name', 'string', unique=True, notnull=True),
Field('Object_location',notnull=True),
Field('NSK_System',notnull=True,),
primarykey = ['SPR_name'],migrate=True)
db.define_table('my_master_table',
Field('Test_id',notnull=True),
Field('Test_suite',notnull=True),
Field('Test_category',notnull=True),
Field('Test_unit',notnull=True),
Field('Test_case',notnull=True),
Field ('Applicability', db.spr_details,'string'),migrate=True)
I inserted one row in spr_details table. Now when I am inserting record in my_master table, I am selecting previously inserted value from Applicability drop-down column. But on submitting it, I am getting
FOREIGN KEY constraint failed error.
below is the stack trace -
Stack trace
Traceback
Traceback (most recent call last):
File "C:\Users\pandeyar\Downloads\web2py_src\web2py\gluon\restricted.py", line 219, in restricted
exec(ccode, environment)
File "C:/Users/pandeyar/Downloads/web2py_src/web2py/applications/JDBC_E2E/controllers/default.py", line 183, in <module>
File "C:\Users\pandeyar\Downloads\web2py_src\web2py\gluon\globals.py", line 419, in <lambda>
self._caller = lambda f: f()
File "C:/Users/pandeyar/Downloads/web2py_src/web2py/applications/JDBC_E2E/controllers/default.py", line 99, in admin
user_signature=False,
File "C:\Users\pandeyar\Downloads\web2py_src\web2py\gluon\sqlhtml.py", line 3338, in smartgrid
user_signature=user_signature, **kwargs)
File "C:\Users\pandeyar\Downloads\web2py_src\web2py\gluon\sqlhtml.py", line 2534, in grid
onsuccess=oncreate)
File "C:\Users\pandeyar\Downloads\web2py_src\web2py\gluon\html.py", line 2300, in process
self.validate(**kwargs)
File "C:\Users\pandeyar\Downloads\web2py_src\web2py\gluon\html.py", line 2238, in validate
if self.accepts(**kwargs):
File "C:\Users\pandeyar\Downloads\web2py_src\web2py\gluon\sqlhtml.py", line 1965, in accepts
self.vars.id = self.table.insert(**fields)
File "C:\Users\pandeyar\Downloads\web2py_src\web2py\gluon\packages\dal\pydal\objects.py", line 753, in insert
ret = self._db._adapter.insert(self, row.op_values())
File "C:\Users\pandeyar\Downloads\web2py_src\web2py\gluon\packages\dal\pydal\adapters\base.py", line 486, in insert
raise e
File "C:\Users\pandeyar\Downloads\web2py_src\web2py\gluon\packages\dal\pydal\adapters\base.py", line 481, in insert
self.execute(query)
File "C:\Users\pandeyar\Downloads\web2py_src\web2py\gluon\packages\dal\pydal\adapters\__init__.py", line 67, in wrap
return f(*args, **kwargs)
File "C:\Users\pandeyar\Downloads\web2py_src\web2py\gluon\packages\dal\pydal\adapters\base.py", line 412, in execute
rv = self.cursor.execute(command, *args[1:], **kwargs)
sqlite3.IntegrityError: FOREIGN KEY constraint failed
Any help would be extremely helpful.
Please refer to the documentation on keyed tables (i.e., tables whose primary key is not the default auto-incrementing integer ID field), as spr_details is such a table.
Note, there are two problems with your reference field definition:
Field('Applicability', db.spr_details, 'string')
First, as noted in the documentation, when referencing a keyed table, the format for the type argument to Field() must be 'reference tablename.fieldname', not db.tablename. Second, you do not separately specify the type of a reference field (i.e., you should not pass 'string' as the third argument to Field(), as the third argument is the length argument, not the type -- the type is inferred based on the type of the referenced field). So, the field definition should be:
Field('Applicability', 'reference spr_details.SPR_name')
Note, with your current incorrect definition of the Applicability field, the field becomes an integer field (as it expects to store a reference to an integer ID field in the foreign table). If you attempt to insert an integer, you will get a foreign key constraint failure error, and if you attempt to insert a string, you will get an error about inserting a string where an int is expected. You should not get any errors if you use the correct field definition as shown above and insert string values that match values that exist in the db.spr_details.SPR_name field.
Also note that web2py assumes keyed tables will be used together (i.e., both the referenced and the referencing table will be keyed). So, for example, if you use SQLFORM with my_master_table, by default it will assume that because my_master_table is not keyed, neither is the spr_details table that it references. It will therefore attempt to convert the value of SPR_details to an integer ID (which results in a conversion to 0, which leads to a foreign key constraint error). To work around this, you should handle the database inserts and updates manually -- for example:
form = SQLFORM(db.my_master_table).process(dbio=False)
if form.accepted:
db.my_master_table.insert(**form.vars)
Above, setting dbio=False prevents SQLFORM from doing the insert itself (which will result in an error). Instead, the insert is done explicitly using the values in form.vars.
For some reason memcache does not seem to like
result
in this following code
db.query("select * from TABLE order by ID desc limit 70")
result = db.store_result()
m.set('1',result,60)
This is the error in apache error_log:
m.set('1',result,60)
File "/usr/lib/python2.6/site-packages/memcache.py", line 466, in set
return self._set("set", key, val, time, min_compress_len)
File "/usr/lib/python2.6/site-packages/memcache.py", line 639, in _set
store_info = self._val_to_store_info(val, min_compress_len)
File "/usr/lib/python2.6/site-packages/memcache.py", line 615, in _val_to_store_info
pickler.dump(val)
UnpickleableError: Cannot pickle objects
Something is probably going on with the "result".
otherwise instead of "result".. something else such as..
m.set('1','test',60)
works just fine.
store_result instructs MySQL to store the result of your query locally, and returns a reference to that "result object". It does not actually return a list of rows.
To actually get the rows:
rows = result.fetch_row(maxrows=0) # Actually fetches all the rows
m.set('1', rows, 60)
Now, it'd be best to instantiate a cursor than to use _mysql directly.
I am working with Python 3.3, pypyodbc 1.2.1, and a Quickbooks Enterprise 12 company file being access over Flexquarters QODBC version 14. I'm new to programming and python, so still learning :) I can run a query using the pypyodbc examples just fine, and produces expected results.
Notice the hardcoded email address in the execute. This works as expected:
def get_customer_id(search_col,search_str):
'''(str,str) --> str
>>>get_customer_id(email, foo#foo.com)
80000001-1385782702
'''
cur.execute("SELECT listid FROM CUSTOMER WHERE email='foo#foo.com'")
for row in cur.fetchall():
for field in row:
return field
If I try to do the same thing using the parameters that I am reading from the pypyodbc documentation, I throw an error. I'm having problems with the quotes, and parameter markers I think.
def get_customer_id(search_col,search_str):
'''(str,str) --> str
>>>get_customer_id(email, foo#foo.com)
80000001-1385782702
'''
cur.execute("SELECT listid FROM CUSTOMER WHERE email=?",(search_str,))
for row in cur.fetchall():
for field in row:
return field
Trying to be more pythonistic? I really want to reuse the function to search different columns. Something like:
cur.execute("SELECT listid FROM CUSTOMER WHERE search_str=search_col")
I have looked at a few other threads, and most of them seem to just be dealing with the parameter, and not the column to search. Can anyone help me learn this?
PS forgot to include the traceback:
Traceback (most recent call last):
File "C:\Users\Mike\Documents\Projects\qb_sync\quickbooks.py", line 32, in <module>
print(get_customer_id('email','foo#foo.com'))
File "C:\Users\Mike\Documents\Projects\qb_sync\quickbooks.py", line 27, in get_customer_id
cur.execute("SELECT listid FROM CUSTOMER WHERE email=?",[search_str,])
File "C:\Python\lib\site-packages\pypyodbc.py", line 1457, in execute
self._BindParams(param_types)
File "C:\Python\lib\site-packages\pypyodbc.py", line 1420, in _BindParams
check_success(self, ret)
File "C:\Python\lib\site-packages\pypyodbc.py", line 982, in check_success
ctrl_err(SQL_HANDLE_STMT, ODBC_obj.stmt_h, ret, ODBC_obj.ansi)
File "C:\Python\lib\site-packages\pypyodbc.py", line 960, in ctrl_err
raise Error(state,err_text)
pypyodbc.Error: ('HY004', '[HY004] [Microsoft][ODBC Driver Manager] SQL data type out of range')
[Finished in 1.7s]
I think the use of
cur.execute("""SELECT listid FROM CUSTOMER WHERE ?=?""",[column, email])
can not be accepted by database engine rather than pypyodbc or any other odbc interface. It's the database engine refuse to accept the query for the use of parameter on column names.
Probably you would have to try this instead to reuse the function:
# First construct your dynamic query for the targeted column
sql = """SELECT listid FROM CUSTOMER WHERE %s=?""" %(column)
# Then provide the dynamic value for the dynamic query string
cur.execute(sql, (value,))
Python 3 also has the str.format() method which will do string replacement on {index} items within your string. This is useful if you have many values to inject into your strings like:
myStr = "I like {0} and {1}, but I don't like {2}.".format("apples","bananas","spinach")
myStr
"I like apples and bananas, but I don't like spinach."
# First construct your dynamic query for the targeted column
sql = """SELECT listid FROM CUSTOMER WHERE {0}=?""".format(column)
# Then provide the dynamic value for the dynamic query string
cur.execute(sql, (value,))
It's worth noting that this method of replacing values in a string query can be subject to sql injection.
The safer way to do this would be with parameterized stored procs.
I got 1/2 the answer so far. This works for one parameter, IF I format the string before calling the function;
print(get_custid_email(b'foo#foo.org'))
cur.execute("""SELECT listid FROM CUSTOMER WHERE email=?""",[email])
I still can't get it to do the same thing with column name though.
print(get_custid_email(b'email',b'foo#foo.org'))
cur.execute("""SELECT listid FROM CUSTOMER WHERE ?=?""",[column, email])
That throws a differnt error:
Traceback (most recent call last):
File "C:\Users\Mike\Documents\Projects\qb_sync\quickbooks.py", line 34, in <module>
print(get_custid_email(b'wendy.lindsay#gmail.com'))
File "C:\Users\Mike\Documents\Projects\qb_sync\quickbooks.py", line 29, in get_custid_email
cur.execute("""SELECT listid FROM CUSTOMER WHERE ?=?""",['email',email])
pyodbc.ProgrammingError: ('42S00', '[42S00] [QODBC] Data type of parameter cannot be determined (11023) (SQLPrepare)')
I'm having a problem using SQLAlchemy with PySide(PyQt). I'm trying to pop-up a QtGui.QDialog, but when I do this SQLAlchemy throws an exception:
Traceback (most recent call last):
File "C:\Python27\lib\site-packages\preo\preodb\dbviewandmodel.py", line 32, in rowCount
return len(self.rows())
File "C:\Python27\lib\site-packages\preo\preodb\dbviewandmodel.py", line 30, in rows
return self.tableobj.query.all()
File "C:\Python27\lib\site-packages\sqlalchemy-0.6.6-py2.7.egg\sqlalchemy\orm\query.py", line 1579, in all
return list(self)
File "C:\Python27\lib\site-packages\sqlalchemy-0.6.6-py2.7.egg\sqlalchemy\orm\query.py", line 1688, in __iter__
self.session._autoflush()
File "C:\Python27\lib\site-packages\sqlalchemy-0.6.6-py2.7.egg\sqlalchemy\orm\session.py", line 862, in _autoflush
self.flush()
File "C:\Python27\lib\site-packages\sqlalchemy-0.6.6-py2.7.egg\sqlalchemy\orm\session.py", line 1388, in flush
self._flush(objects)
File "C:\Python27\lib\site-packages\sqlalchemy-0.6.6-py2.7.egg\sqlalchemy\orm\session.py", line 1469, in _flush
flush_context.execute()
File "C:\Python27\lib\site-packages\sqlalchemy-0.6.6-py2.7.egg\sqlalchemy\orm\unitofwork.py", line 302, in execute
rec.execute(self)
File "C:\Python27\lib\site-packages\sqlalchemy-0.6.6-py2.7.egg\sqlalchemy\orm\unitofwork.py", line 446, in execute
uow
File "C:\Python27\lib\site-packages\sqlalchemy-0.6.6-py2.7.egg\sqlalchemy\orm\mapper.py", line 1878, in _save_obj
execute(statement, params)
File "C:\Python27\lib\site-packages\sqlalchemy-0.6.6-py2.7.egg\sqlalchemy\engine\base.py", line 1191, in execute
params)
File "C:\Python27\lib\site-packages\sqlalchemy-0.6.6-py2.7.egg\sqlalchemy\engine\base.py", line 1271, in _execute_clauseelement
return self.__execute_context(context)
File "C:\Python27\lib\site-packages\sqlalchemy-0.6.6-py2.7.egg\sqlalchemy\engine\base.py", line 1302, in __execute_context
context.parameters[0], context=context)
File "C:\Python27\lib\site-packages\sqlalchemy-0.6.6-py2.7.egg\sqlalchemy\engine\base.py", line 1401, in _cursor_execute
context)
File "C:\Python27\lib\site-packages\sqlalchemy-0.6.6-py2.7.egg\sqlalchemy\engine\base.py", line 1394, in _cursor_execute
context)
File "C:\Python27\lib\site-packages\sqlalchemy-0.6.6-py2.7.egg\sqlalchemy\engine\default.py", line 299, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.IntegrityError: (IntegrityError) ('23000', "[23000] [Microsoft][ODBC
SQL Server Driver][SQL Server]Violation of UNIQUE KEY
constraint 'UQ__users__F3DBC5720DAF0CB0'. Cannot insert duplicate key in
object 'dbo.users'. (2627) (SQLExecDirectW); [01000] [Microsoft][ODBC SQL Server
Driver][SQL Server]The statement has been terminated. (3621)") u'INSERT INTO users
(username, fullname, email, passwordmd5) OUTPUT inserted.id VALUES (?, ?, ?, ?)'
(None, None, None, None)
This is particularly troubling because I have no code, anywhere, that even attempts to insert records into SQL; I am only ever attempting to query data from the database. In fact, my DB model is read-only with respect to what PySide/PyQt are doing (i.e., I'm using a QtGui.QTableView model/view and there is no insertRows function in that model).
I have no idea what's going on or how to solve it - again, I have no code to modify SQL records at all, but still SQLAlchemy attempts to be inserting blank records into one of my SQL tables. All I can see, in the background, is the QTableView data model is querying the database A LOT. It just seems that when I popup this QDialog (which does have some code in it to query some table column) this error is thrown. Oddly, this isn't consistent, sometime the popup appears first before the exception, sometimes the popup appears after the exception. Under normal circumstances, the QTableView data model works great, just not when I popup this dialog (and ironically, the popup isn't using any QTableView at all, just standard widgets like QLineEdit, QTextEdit, etc.)
If it helps, I'm using Python 2.7 with SQLAlchemy 0.6.6 (also with Elixir 0.7.1), and PySide 1.0.0 (and PyQt4 4.8.3). I'm on Windows 7 using SQL 2008 R2 (Express). And yes, I've tried rebooting the PC, but the problem still occurs after a reboot. I'm reluctant to post more code because I have a lot of it in this particular project and I can't nail down the problem anything specific.
I'm hoping someone might know of oddities in SQLAlchemy and/or PyQt that might be related to this. I'm also hoping I can continue using SQLAlchemy as I have a large data model built; I'm reluctant, at this point, to abandon this and use PyQt's SQL features.
I've managed to make this problem go away, but it's still not really clear to me why SQLAlchemy was trying to insert rows in my database - that really bothers me, but it's not happening anymore.
At any rate, what was, I think, happening, was related to my SQLAlchemy data model and the way I was accessing it, here is a snippet of that model:
from elixir import *
metadata.bind = 'mssql+pyodbc://username:password/dbname'
metadata.bind.echo = False
class Users(Entity):
using_options(tablename = 'users')
username = Field(String(50), unique=True)
fullname = Field(String(255))
email = Field(String(255))
passwordmd5 = Field(String(32))
def __repr__(self):
return "<Users ({})({})({})>".format(self.username, self.fullname, self.email)
def prettyname(self):
return {'username':'User Name', 'fullname':'Full Name', 'email':'Email Address', 'passwordmd5':'$hidden$'}
In my code I needed a way of getting 'pretty' label names for a GUI without having to hard code this in a GUI (I've been trying to create a dynamic way of building GUI forms). So, I added the 'prettyname' method to my data model to give me some application specific metadata in that data model. All I'm doing is returning a dictionary of items.
I had a secondary problem in that sometimes I needed to get this data from the class instance for Users and sometimes for a query result for Users (for example, Users.get_by(id=1)). As it turned out, retrieving this data had to be done in two ways. In the class instances I had to get the value this way:
prettyname = Users().prettyname()['username']
But when I was using query results it was:
prettyname = queryresult.prettyname()['username']
SQLAlchemy seems to have a real problem when I was using the former method (the class instance method) - as this was being used everytime I was seeing the crash. When I was using the latter instance I was never seeing a crash. Still, I needed access to that metadata in the class instance.
The fix, or should I say what turned out to fix this came from another Stackoverflow article (thank you everyone at Stackoverflow - I'd be nothing without you). I changed the structure of the dbmodel:
class Users(Entity):
using_options(tablename = 'users')
username = Field(String(50), unique=True, info={'prettyname':'User Name'})
fullname = Field(String(255), info={'prettyname':'Full Name'})
email = Field(String(255), info={'prettyname':'Email Address'})
passwordmd5 = Field(String(32), info={'hidden':True})
def __repr__(self):
return "<Users ({})({})({})>".format(self.username, self.fullname, self.email)
This allows me to use a common method of introspection to get the dictionary data in the info argument, regardless if I'm looking at a class instance, or a query result. In this case I use the '.table' method of either the class or query result, then get the column that I need (.c), then use the .info method of that column to return the dictionary.
Whatever the case, now SQLAlchemy no longer tries to arbitrarily insert rows in the database anymore.