How can a function be run "in a transaction" (see http://code.google.com/appengine/docs/python/datastore/functions.html#run_in_transaction with MySQL and SQLite? (and other RDBMSs if anyone knows how)?
EDIT: I want to do so in Python, but a way to do it in other programming languages would also be okay.
That depends on the RDBMS you use, what programming language you're writing in, and what library you use to connect.
With MySQL, you issue a START TRANSACTION query before the queries you want to be run in a transaction, and a COMMIT query to run that transaction.
http://dev.mysql.com/doc/refman/5.0/en/commit.html
Related
I have a question and hope someone can direct me in the right direction; Basically every week I have to run a query (SSMS) to get a table containing some information (date, clientnumber, clientID, orderid etc) and then I copy all the information and that table and past it in a folder as a CSV file. it takes me about 15 min to do all this but I am just thinking can I automate this, if yes how can I do that and also can I schedule it so it can run by itself every week. I believe we live in a technological era and this should be done without human input; so I hope I can find someone here willing to show me how to do it using Python.
Many thanks for considering my request.
This should be pretty simple to automate:
Use some database adapter which can work with your database, for MSSQL the one delivered by pyodbc will be fine,
Within the script, connect to the database, perform the query, parse an output,
Save parsed output to a .csv file (you can use csv Python module),
Run the script as the periodic task using cron/schtask if you work on Linux/Windows respectively.
Please note that your question is too broad, and shows no research effort.
You will find that Python can do the tasks you desire.
There are many different ways to interact with SQL servers, depending on your implementation. I suggest you learn Python+SQL using the built-in sqlite3 library. You will want to save your query as a string, and pass it into an SQL connection manager of your choice; this depends on your server setup, there are many different SQL packages for Python.
You can use pandas for parsing the data, and saving it to a ~.csv file (literally called to_csv).
Python does have many libraries for scheduling tasks, but I suggest you hold off for a while. Develop your code in a way that it can be run manually, which will still be much faster/easier than without Python. Once you know your code works, you can easily implement a scheduler. The downside is that your program will always need to be running, and you will need to keep checking to see if it is running. Personally, I would keep it restricted to manually running the script; you could compile to an ~.exe and bind to a hotkey if you need the accessibility.
Having learned SQL before learning any Python, I have a fairly lengthy program/query that I wrote in SQL server which heavily transforms and calculates the data (Basically taking forecast, inventory, and Bill of Materials, and efficiency data and then automatically generating a production plan. While I am sure there are things I could optimize, the query/program itself is aorund 3,000 lines).
While I have figured out how to update the data in SQL Server using a combination of pandas, pyodbc, and fast_to_sql, I have not been able to find a simple method for running a SQL Server script through Python.
I am sure that I could achieve the same thing by just having the data manipulation occur in python rather than SQL Server, it would be fairly time intensive to translate everything.
If there is anything I can do to clarify please let me know. For reference I am using the 2017 version of Microsoft SQL Server python version 3.8.3.
Try to combine all of your MSSQL scripts into Stored Procedures and then call it from Python.
I have three programs running, one of which iterates over a table in my database non-stop (over and over again in a loop), just reading from it, using a SELECT statement.
The other programs have a line where they insert a row into the table and a line where they delete it. The problem is, that I often get an error sqlite3.OperationalError: database is locked.
I'm trying to find a solution but I don't understand the exact source of the problem (is reading and writing in the same time what make this error occur? or the writing and deleting? maybe both aren't supposed to work).
Either way, I'm looking for a solution. If it were a single program, I could match the database I/O with mutexes and other multithreading tools, but it's not. How can I wait until the database is unlocked for reading/writing/deleting without using too much CPU?
you need to switch databases..
I would use the following:
postgresql as my database
psycopg2 as the driver
the syntax is fairly similar to SQLite and the migration shouldn't be too hard for you
I'm connecting to MySQL with the MySQLdb module. I don't want to use Python's time functions: I want to know how long the query ran within MySQL, i.e. the number I see after I've run a query within MySQL directly.
I do see a thread where this is addressed as something one could eventually dig down to, but I was hoping that since MySQL reports that number, the Python connection would have picked it up somewhere.
May this help?
SET profiling = 1;
Run your query;
SHOW PROFILES;
See here:http://dev.mysql.com/doc/refman/5.7/en/show-profile.html
Ｂｅｃａｕｓｅ ｏｆ ｔｈｅ ａｂｏｖｅ ｃｏｍｍａｎｄｓ ｗｉｌｌ ｂｅ ｒｅｍｏｖｅｄ ｉｎ ｔｈｅ ｆｕｔｕｒｅ ｖｅｒｓｉｏｎ， Performance Schema ｃａｎ ｂｅ ｕｓｅｄ http://dev.mysql.com/doc/refman/5.7/en/performance-schema.html ａｎｄ http://dev.mysql.com/doc/refman/5.7/en/performance-schema-query-profiling.html．
Ｏｎ ｔｈｅ ａｂｏｖｅ ｌｉｎｋｓ， ｔｈｅｒｅ ａｒｅ ｍｏｒｅ ｄｅｔａｉｌｓ ｏｎ Query Profiling Using Performance Schema．
I used to be able to run and execute python using simply execute statement. This will insert value 1,2 into a,b accordingly. But started last week, I got no error , but nothing happened in my database. No flag - nothing... 1,2 didn't get insert or replace into my table.
connect.execute("REPLACE INTO TABLE(A,B) VALUES(1,2)")
I finally found the article that I need commit() if I have lost the connection to the server. So I have add
connect.execute("REPLACE INTO TABLE(A,B) VALUES(1,2)")
connect.commit()
now it works , but I just want to understand it a little bit , why do I need this , if I know I my connection did not get lost ?
New to python - Thanks.
This isn't a Python or ODBC issue, it's a relational database issue.
Relational databases generally work in terms of transactions: any time you change something, a transaction is started and is not ended until you either commit or rollback. This allows you to make several changes serially that appear in the database simultaneously (when the commit is issued). It also allows you to abort the entire transaction as a unit if something goes awry (via rollback), rather than having to explicitly undo each of the changes you've made.
You can make this functionality transparent by turning auto-commit on, in which case a commit will be issued after each statement, but this is generally considered a poor practice.
Not commiting puts all your queries into one transaction which is safer (and possibly better performance wise) when queries are related to each other. What if the power goes between two queries that doesn't make sense independently - for instance transfering money from one account to another using two update queries.
You can set autocommit to true if you don't want it, but there's not many reasons to do that.