I'd like to use the ON DUPLICATE KEY UPDATE optionality provided by SQLAlchemy to upsert a bunch of records.
These records have been sucessfully inserted with python using the following (where connection is engine.connect() object and table is a Table object)
record_list = [{'col1': 'name1', 'col2': '2015-01-31', 'col3': 27.2},
{'col1': 'name1', 'col2': '2016-01-31', 'col3': 25.2}]
query = insert(table)
results = connection.execute(query, record_list)
Looking at the docs at https://docs.sqlalchemy.org/en/13/dialects/mysql.html#insert-on-duplicate-key-update-upsert as well as a number of SO questions (including the suggestion it's possible under the comments on SQLAlchemy ON DUPLICATE KEY UPDATE ) I've tried a number of different examples, but there were none that I could see that address multiple records with the upsert statement using this method.
I'm trying along the lines of
query = insert(table).values(record_list)
upsert_query = query.on_duplicate_key_update()
results = connection.execute(upsert_query)
but either get the issue that the .on_duplicate_key_update() requires cant be empty or that the SQL syntax is wrong.
If anyone has sucessfully managed and could help me with the code structure here I'd really appreciate it.
I just ran into a similar problem and creating a dictionary out of query.inserted solved it for me.
query = insert(table).values(record_list)
update_dict = {x.name: x for x in query.inserted}
upsert_query = query.on_duplicate_key_update(update_dict)
#user12730260’s answer is great! but has a little bug, the correct code is:
query = insert(table).values(record_list) # each record is a dict
update_dict = {x.name: x for x in query.inserted} # specify columns for update, u can filter some of it
upsert_query = query.on_duplicate_key_update(**update_dict) # here's the modification: u should expand the columns dict
Your on_duplicate_key_update function requires arguments that define the data to be inserted in the update. Please have a look at the example in the documentation that you have already found.
insert().on_duplicate_key_update({"key": "value"})
Related
so I have looked around for days and tried various solutions to other people issues that seem the same as my own but unfortunately I have not made any progress. I am still very new with python, tkinter, and mysqli databases. What I trying to do is have an inventory program, it's just that simple. You can login and add, delete or simply view what is in the database. I can add and delete items just fine but I am having an issue pulling the data in the correct columns.
So far I have tried to pull each as a variable and assign it accordingly, tried to pull one row at a time, even switch to a new database but have gotten no correct results. If I use sqlite 3 it works fine so is it because mysql is on a server and not local? Any way, I'd like some advice to point me in the right direction so any help is much appreciated.
Edit:
tree.insert("", 1, "dirIso", text="ProductID")
for n, dirIso in enumerate(results,1):
list_of_column_values = [list(_dict.values())[0] for _dict in dirIso]
tree.insert('dirIso', n, text=list_of_column_values[0],
values=list_of_column_values[1:])
cursor.close()
conn.close()
This is what I have done, I am now getting a 'str' object has no attribute 'values'. Do I need to change n? Or is it looking for the name inside of my database as values and not the columns?
Outcome I'm trying to get is for the table to display its respective data for each item.
Question: Treeview is placing mysql data into one column
Edit your Question, showing three rows of results data, if your results does not look like the following.
Assuming the following results == list of dict!
results = [{'produkt_id': '1234', 'Name': 'John', 'Address': 'NewYork'},
{'produkt_id': '5678', 'Name': 'Peter', 'Address': 'Boston'}
]
You didn't need Tree Heading, as you want a Tabelview
# Dict's are unordered,
# therfore you need a list of fieldnames in your desired order
fieldnames = ['produkt_id', 'Name', 'Address']
# Loop results
for n, _dict in enumerate(results,1):
# Create a list of values from this _dict
_list = []
for key in fieldnames:
_list.append(_dict[key])
# The first value goes to 'text'
# All others goes to 'values'
tree.insert('', 'end', n, text=_list[0], values=_list[1:])
Output:
1234 John NewYork
5678 Peter Boston
Assuming we have a table consisting of column_1, column_2, ... , column_n and all of them are string fields. The conditions we are going to do case-insensitive query are stored in a dictionary d like d[column_1] = "Hello", which may or may not contains all columns. How can we do the query?
I checked the question Case Insensitive Flask-SQLAlchemy Query. It contains a lot of awesome answers, but none of them works if we do not know what conditions we have until runtime.
You would need to build the query looping through each key of the dictionary.
As you didn't give any code sample, I'm going to call the table model class TableModel and each column will be column_1, column_2, etc.
Something like this should work:
d = {'column_1': 'some_string', 'column_3': 'another_string'}
# skipping 'column_2' just to exemplify how every column is optional in the dictionary
my_query = TableModel.query
for k in d:
my_query = my_query.filter(getattr(TableModel, k).ilike(d[k]))
And that's about it. Afterwards you can use my_query as any other query, e.g., my_query.count() or my_query.all()
I'm trying to set the order of columns when building a table with SQLAlchemy, as of right now the columns are appearing in alphabetical order, I currently have:
def data_frame(query, columns):
def make_row(x):
return dict([(c, getattr(x, c)) for c in columns])
return pd.DataFrame([make_row(x) for x in query])
PackL = create_engine('mssql+pyodbc://u:pass#Server/db1?driver=SQL Server', echo=False)
Fabr = create_engine('mssql+pyodbc://u:pass#Server/db2?driver=SQL Server', echo=False)
Session = sessionmaker(bind=PackL)
session = Session()
Base = declarative_base()
metadata = MetaData()
class Tranv(Base):
__tablename__= "Transactions"
__table_args__= {'autoload': True, 'autoload_with':PackL}
newvarv = session.query(Tranv).filter_by(status='SCRAP').filter(Tranv.time_stamp.
between('2015-10-01', '2015-10-09'))
session.close()
dfx = data_frame(newvarv, ['action', 'employee_number', 'time_stamp', 'qty',
'part_number', 'card_number'])
Current dfx has the columns in alphabetical order, but I want it to order them by the order in which I define the columns when I create the data frame dfx. Therefore the order would be action, employee_number, time_stamp, qty, part_number, card_number. I can easily do this with Pandas, but that seems like extra (and unnecessary) steps.
I've searched the documentation, google, & stackoverflow but nothing really seems to fit my needs. As I'm still new with SQLAlchemy I appreciate any help. Am I right in thinking that because I'm autoloading the table, I can not easily define the order of my columns (I'm sure there is a workaround, but don't have a clue where in the documentation that might be found)?
The reason your columns are not in the order you specify, has nothing to do with the sql query or sqlalchemy. This is caused by the fact that you convert the query output to a dictionary, which you then feed to DataFrame.
As a dictionary has no order in python, pandas will sort it alphabetically to have a predictable output.
Using the current approach of the dict, you can always change the order of the columns afterwards by doing dfx.reindex(columns=['action', ..., 'card_number'])
Apart from the explanation why it is not ordered in your case, there are maybe better approaches to tackle this:
Use the builtin pd.read_sql_query. When working with sessions and Query objects, you can pass the selectable attribute to read_sql_query to convert it to a DataFrame:
query = session.query(Table)...
df = pd.read_sql_query(query.selectable, engine)
Do not convert to a dictionary, but keep the output as tuples which you feed to DataFrame: this will keep the order of the query output.
I have a database table with multiple fields which I am querying and pulling out all data which meets certain parameters. I am using psycopg2 for python with the following syntax:
cur.execute("SELECT * FROM failed_inserts where insertid='%s' AND site_failure=True"%import_id)
failed_sites= cur.fetchall()
This returns the correct values as a list with the data's integrity and order maintained. However I want to query the list returned somewhere else in my application and I only have this list of values, i.e. it is not a dictionary with the fields as the keys for these values. Rather than having to do
desiredValue = failed_sites[13] //where 13 is an arbitrary number of the index for desiredValue
I want to be able to query by the field name like:
desiredValue = failed_sites[fieldName] //where fieldName is the name of the field I am looking for
Is there a simple way and efficient way to do this?
Thank you!
cursor.description will give your the column information (http://www.python.org/dev/peps/pep-0249/#cursor-objects). You can get the column names from it and use them to create a dictionary.
cursor.execute('SELECT ...')
columns = []
for column in cursor.description:
columns.append(column[0].lower())
failed_sites = {}
for row in cursor:
for i in range(len(row)):
failed_sites[columns[i]] = row[i]
if isinstance(row[i], basestring):
failed_sites[columns[i]] = row[i].strip()
The "Dictionary-like cursor", part of psycopg2.extras, seems what you're looking for.
Is it possible to do SELECT * in SQLAlchemy?
Specifically, SELECT * WHERE foo=1?
Is no one feeling the ORM love of SQLAlchemy today? The presented answers correctly describe the lower-level interface that SQLAlchemy provides. Just for completeness, this is the more-likely (for me) real-world situation where you have a session instance and a User class that is ORM mapped to the users table.
for user in session.query(User).filter_by(name='jack'):
print(user)
# ...
And this does an explicit select on all columns.
The following selection works for me in the core expression language (returning a RowProxy object):
foo_col = sqlalchemy.sql.column('foo')
s = sqlalchemy.sql.select(['*']).where(foo_col == 1)
If you don't list any columns, you get all of them.
query = users.select()
query = query.where(users.c.name=='jack')
result = conn.execute(query)
for row in result:
print row
Should work.
You can always use a raw SQL too:
str_sql = sql.text("YOUR STRING SQL")
#if you have some args:
args = {
'myarg1': yourarg1
'myarg2': yourarg2}
#then call the execute method from your connection
results = conn.execute(str_sql,args).fetchall()
Where Bar is the class mapped to your table and session is your sa session:
bars = session.query(Bar).filter(Bar.foo == 1)
Turns out you can do:
sa.select('*', ...)
I had the same issue, I was trying to get all columns from a table as a list instead of getting ORM objects back. So that I can convert that list to pandas dataframe and display.
What works is to use .c on a subquery or cte as follows:
U = select(User).cte('U')
stmt = select(*U.c)
rows = session.execute(stmt)
Then you get a list of tuples with each column.
Another option is to use __table__.columns in the same way:
stmt = select(*User.__table__.columns)
rows = session.execute(stmt)
In case you want to convert the results to dataframe here is the one liner:
pd.DataFrame.from_records(rows, columns=rows.keys())
For joins if columns are not defined manually, only columns of target table are returned. To get all columns for joins(User table joined with Group Table:
sql = User.select(from_obj(Group, User.c.group_id == Group.c.id))
# Add all coumns of Group table to select
sql = sql.column(Group)
session.connection().execute(sql)
I had the same issue, I was trying to get all columns from a table as a list instead of getting ORM objects back. So that I can convert that list to pandas dataframe and display.
What works is to use .c on a subquery or cte as follows:
U = select(User).cte('U')
stmt = select(*U.c)
rows = session.execute(stmt)
Then you get a list of tuples with each column.
Another option is to use __table__.columns in the same way:
stmt = select(*User.__table__.columns)
rows = session.execute(stmt)
In case you want to convert the results to dataframe here is the one liner:
pd.DataFrame.from_records(dict(zip(r.keys(), r)) for r in rows)
If you're using the ORM, you can build a query using the normal ORM constructs and then execute it directly to get raw column values:
query = session.query(User).filter_by(name='jack')
for cols in session.connection().execute(query):
print cols
every_column = User.__table__.columns
records = session.query(*every_column).filter(User.foo==1).all()
When a ORM class is passed to the query function, e.g. query(User), the result will be composed of ORM instances. In the majority of cases, this is what the dev wants and will be easiest to deal with--demonstrated by the popularity of the answer above that corresponds to this approach.
In some cases, devs may instead want an iterable sequence of values. In these cases, one can pass the list of desired column objects to query(). This answer shows how to pass the entire list of columns without hardcoding them, while still working with SQLAlchemy at the ORM layer.