I'm new to python and I was trying to insert data from one table to another and the code is executing but it is not reflecting any changes in new table in which I want to insert data.
import mysql.connector
conn=mysql.connector.connect(user='root',password='',host='localhost',database='xyz')
mycursor=conn.cursor()
mycursor.execute("insert into newTable select * from oldTable group by mac,date,time order by mac")
I suppose it is because you are not committing the changes. Use the below code and see if it works.
conn.commit()
You need to use conn because that is the variable name for your connection to the database. A commit must be sent to the server, which in turn will commit your changes.
You need to add commit() to make the changes.
conn.commit()
Related
Below is the last part of my selenium web scraper that loops through the different tabs of this website page, selects the "export data" button, downloads the data, adds a "yearid" column, then loads the data into a MySQL table.
df = pd.read_csv(desired_filepath)
df["yearid"] = datetime.today().year
df[df.columns[df.columns.str.contains('%')]] = \
(df.filter(regex='%')
.apply(lambda x: pd.to_numeric(x.str.replace(r'[\s%]', ''),
errors='coerce')))
df.to_csv(desired_filepath)
engine = create_engine("mysql+pymysql://{user}:{pw}#localhost/{db}"
.format(user="walker",
pw="password",
db="data"))
df.to_sql(con=engine, name='fg_test_hitting_{}'.format(button_text), if_exists='replace')
time.sleep(10)
driver.quit()
Everything works great, but I would like to import the data into the MySQL table and replace only if the yearid=2018. Does anyone know if it is possible to load data and replace given a specific condition? Thanks in advance!
I think rather than deleting from your table it may be better to just let MySQL handle the replacing. You can do this by creating a temporary table with the new data, replace into the permanent table, then delete the temp table. The big caveat here is that you will need to set the keys in your table (Ideally only once). I don't know what your key fields are so its tough to help in this regard.
Replace the commented line with this:
# df.to_sql(con=engine, name='fg_test_hitting_{}'.format(button_text), if_exists='replace')
conn = engine.connect()
# should fail if temporary table already exists (we want it to fail in this case)
df.to_sql('fg_test_hitting_{}_tmp'.format(button_text), conn)
# Will create the permanent table if it does not already exist (will only matter in the first run)
# note that you may have to create keys here so that mysql knows what constitutes a replacement
conn.execute('CREATE TABLE IF NOT EXISTS fg_test_hitting_{} LIKE fg_test_hitting_{}_tmp;'.format(button_text, button_text))
# updating the permanent table and dropping the temporary table
conn.execute('REPLACE INTO fg_test_hitting_{} (SELECT * FROM fg_test_hitting_{}_tmp);'.format(button_text, button_text))
conn.execute('DROP TABLE IF EXISTS fg_test_hitting_{}_tmp;'.format(button_text))
As described by #Leo in comments first delete that part of data (from MySQL table) that you were going to update and then save it to MySQL table:
conn = engine.connect()
cur = conn.cursor()
...
cur.execute('delete from fg_test_hitting_{} where yearid=?'.format(button_text),
(pd.datetime.today().year,))
df.to_sql(con=engine, name='fg_test_hitting_{}'.format(button_text), if_exists='replace')
I'm querying a json on a website for data, then saving that data into a variable so I can put it into a sqlite table. I'm 2 out of 3 for what I'm trying to do, but the sqlite side is just mystifying. I'm able to request the data, from there I can verify that the variable has data when I test it with a print, but all of my sqlite stuff is failing. It's not even creating a table, much less updating the table (but it is printing all the results to the buffer for some reason) Any idea what I'm doing wrong here? Disclaimer: Bit of a python noob. I've successfully created test tables just copying the stuff off of the python sqlite doc
# this is requesting the data and seems to work
for ticket in zenpy.search("bananas"):
id = ticket.id
subj = ticket.subject
created = ticket.created_at
for comment in zenpy.tickets.comments(ticket.id):
body = comment.body
# connecting to sqlite db that exists. things seem to go awry here
conn = sqlite3.connect('example.db')
c = conn.cursor()
# Creating the table table (for some reason table is not being created at all)
c.execute('''CREATE TABLE tickets_test
(ticket id, ticket subject, creation date, body text)''')
# Inserting the variables into the sqlite table
c.execute("INSERT INTO ticketstest VALUES (id, subj, created, body)")
# committing changes the changes and closing
c.commit()
c.close()
I'm on Windows 64bit and using pycharm to do this.
Your table likely isn't created because you haven't committed yet, and your sql fails before it commits. It should work when you fix your 2nd sql statement.
You're not inserting the variables you've created into the table. You need to use parameters. There are two ways of parameterizing your sql statement. I'll show the named placeholders one:
c.execute("INSERT INTO ticketstest VALUES (:id, :subj, :created, :body)",
{'id':id, 'subj':subj, 'created':created, 'body':body}
)
So, I have the following code that inserts the data of an old database to a new one:
...
cur_old.execute("""SELECT DISTINCT module FROM all_students_users_log_course266""")
module_rows = cur_old.fetchall()
for row in module_rows:
cur_new.execute("""INSERT INTO modules(label) SELECT %s WHERE NOT EXISTS (SELECT 1 FROM modules WHERE label=%s)""", (row[0], row[0]))
...
The last line executes a query where labels are inserted into the new database table. I tested this query on pgAdmin and it works as I want.
However, when execute the script, nothing is inserted on the modules table. (Actually the sequences are updated, but none data is stored on the table).
Do I need to do anything else after I call the execute method from the cursor?
(Ps. The script is running till the end without any errors)
You forgot to do connection.commit(). Any alteration in the database has to be followed by a commit on the connection. For example, the sqlite3 documentation states it clearly in the first example:
# Save (commit) the changes.
conn.commit()
And the first example in the psycopg2 documentation does the same:
# Make the changes to the database persistent
>>> conn.commit()
As Evert said, the commit() was missing. An alternative to always specifying it in your code is using the autocommit feature.
http://initd.org/psycopg/docs/connection.html#connection.autocommit
For example like this:
with psycopg2.connect("...") as dbconn:
dbconn.autocommit=True
I'm connecting mysql on my Kivy application.
import mysql.connector
con = mysql.connector.Connect(host='XXX', port=XXX, user='XXX', password='XXX', database='XXX')
cur = con.cursor()
db = cur.execute("""select SELECT SQL_NO_CACHE * from abc""")
data = cur.fetchall()
print (data)
After inserting or deleting on table abc from another connection; i call the same query on python; but data is not updating.
I add the query "SET SESSION query_cache_type = OFF;" before select query, but it didn't work. Someone said "select NOW() ..." query is not cachable but it didn't work again. What should I do?
I solved this by adding the code after fetchall()
con.commit()
Calling the same select query without doing a commit, won't update the results.
The solution is to use:
Once:
con.autocommit(True)
Or, after each select query:
con.commit()
With this option, there will be a commit after each select query.
Otherwise, subsequent selects will render the same result.
This error seems to be Bug #42197 related to Query cache and auto-commit in MySQL. The status is won't fix!
In a few months, this should be irrelevant because MySQL 8.0 is dropping Query Cache.
I encounterd the same problem that has been solved and used the above method.
conn.commit()
and I found that different DBMS has different behavior,not all DBMS exist in the connection cache
try this,
conn.autocommit(True);
this will auto commit after each of you select query.
The MySQL query cache is flushed when tables are modified, so it wouldn't have that effect. It's impossible to say without seeing the rest of your code, but it's most likely that your INSERT / DELETE query is failing to run.
I'm writing a python script that would reset the database to an initial state (some hardcoded entries in every table). The db consists of multiple tables with primary and foreign keys.
Every time I would run the script, it should remove all the old entries in all of the tables, reset the primary key counter and insert the sample entries.
Currently I am trying to achieve this like this:
# Delete all the entries from the tables
cursor.execute("DELETE FROM table1")
cursor.execute("DELETE FROM table2")
cursor.execute("DELETE FROM table3")
# Reset the primary key counter and insert sample entries
cursor.execute("ALTER TABLE table1 AUTO_INCREMENT = 1")
cursor.execute("INSERT INTO table1(username, password) VALUES('user01', '123')")
cursor.execute("ALTER TABLE table2 AUTO_INCREMENT = 1")
cursor.execute("INSERT INTO table2(column1, column2) VALUES('column1_data', 'column2_data')")
This isn't working due to the presence of foreign keys in some tables (it won't let me delete them).
I generate the tables using a models.py script (I also use Django), so I thought I could solve this the following way:
remove the database programatically and create a new one with the same name
call the models.py script to generate empty tables in the db
insert sample data using the script I wrote
Is this a good solution or am I overlooking something?
I use scripts monthly to purge a transaction table, after archiving the contents.
Try using the 'truncate' command, ie.
truncate table [tablename];
It resets the counter (auto-increment) for primary key, automatically.
Then use your insert statements to populate base info.
Also, this preserves all of the table base settings (keys,indexes,.).