im very new to Python but want to preform some mathmatic functions using Python's libraries getting interger values from a mysql table i have running,
ive sucessfully established a connection using mysql.connector however im at a loss,
I can select and print Rows and columbs but im unsure of the Syntax to physically define my query as an "x" or "y" in order to preform mathmatic operations with the varible.
Any help would be greatly appreciated.
EDIT
sql_select_Query = "select * from ATABLE"
cursor = mySQLconnection .cursor()
cursor.execute(sql_select_Query)
records = cursor.fetchall()`
and
for row in records:
print("Name = ", row[1], )
print("X_num = ", row[2])
print("Y_num = ", row[3])
print("Signal_Strength = ", row[4], "\n")
cursor.close()
gives me as an example
Name = X,
X_num = Y,
Y_num = Z,
SS = Q
what i would prefer in my selection operation is to define the X , Y, Z, Q to a Global name that i could then use for atleast my application math operations using Numpy libraries for example being able to perform an operator
X*Y-Z+Q
I hope that is a bit clearer
From the gate, I would recommend following the advice of this thread highlighting the use of select *. Turning a field into an integer is possible with your SQL selection statement int the way of CAST or CONVERT. Sort of like this (my daily language is SQL Server; check out the mysql documentation for exacts):
sql_select_Query = "select Name, CAST(X as INT),CAST(Y as BIGINT) from ATABLE"
In my personal experience, SQL tends to age better than Python (tongue in cheek). Aside, if your SQL instance is on a server; I code to the workhorse as error catching is better.
But coming from it in the other direction, if you want these elements to be re-callable later, I'm suggest fetching your feedback into a dictionary.
Information about Python dictionaries can be found here. At least that way, you're pretty much working from a global but fairly structured set of captured data.
It is a bad idea to play with locals() and globals() if you don't exactly know what you're doing. Create a dictionary.
sql_select_Query = "select * from ATABLE"
cursor = mySQLconnection.cursor()
cursor.execute(sql_select_Query)
records = cursor.fetchall()
columns = [item[0] for item in cursor.description] # Grab the table column names
for record in records:
# Create a dictionary {column_name: value, ...} for each row
variable_dict = dict(zip(columns, record))
print("X variable is: ", variable_dict['X'])
# <Calculation here>
You can also configure MySQL to return values as a dictionary but this is probably an easier starting point.
This way, your "variable X" value would just be variable_dict['X'] and there's no need to make any global values other than the dictionary.
Related
I am new to python (as I am sure you can tell from my code) and have basically written a sample script to test some core commands so I can better understand how things work. I have gotten everything running as designed except the last "insert" command -- after hours of googling and experimenting I cannot figure out what's wrong, so I would appreciate it if anyone can show me what needs to be changed and helping me understand why (I'm sure it's basic, but am stumped!).
Below is the line that is giving me trouble:
c.execute("INSERT OR IGNORE INTO {tn} ({cn1}, {cn2}, {cn3}, {cn4}) VALUES ({VID}, {VSnu}, {VIN}, {VName})".\
format(tn=table_name, cn1=column1, cn2=column2, cn3=column3, cn4=column4, VID=ID, VSnu=Snu, VIN=IN, VName=Name))
Here's the entire script for context, if that helps:
import sqlite3
sqlite_file = '/test_database.sqlite' # name of the sqlite database file
table_name = 'test_table'
column1 = 'my_1st_column'
column2 = 'my_2nd_column'
column3 = 'my_3rd_column'
column4 = 'my_4th_column'
ID = int(123456)
Base = 'Arnold'
Snu = 'test'
conn = sqlite3.connect(sqlite_file)
c = conn.cursor()
c.execute("UPDATE {tn} SET {cn2}=('Snu'), {cn3}=('Muh'), {cn4}=('Arnold_A') WHERE {cn1}=({NID})".\
format(tn=table_name, cn1=column1, cn2=column2, cn3=column3, cn4=column4, NID=ID))
i = 1
while(i<15):
if i == 1: IN = 'B'
if i == 2: IN = 'C'
if i == 3: IN = 'D'
if i == 4: IN = 'E'
if i == 5: IN = 'F'
if i == 6: IN = 'G'
if i == 7: IN = 'H'
if i == 8: IN = 'I'
if i == 9: IN = 'J'
ID = ID+1
i = i+1
Name = Base + '_' + IN
params = (Snu, IN, Name)
c.execute("INSERT OR IGNORE INTO {tn} ({cn1}, {cn2}, {cn3}, {cn4}) VALUES ({VID}, {VSnu}, {VIN}, {VName})".\
format(tn=table_name, cn1=column1, cn2=column2, cn3=column3, cn4=column4, VID=ID, VSnu=Snu, VIN=IN, VName=Name))
if(i == 10): break
conn.commit()
conn.close()
It will write the "VID" item just fine if isolated (which is an integer column and the primary key of the table), but everything after that it interprets as a column and I get the "no such column: [value]" error.
I get this error regardless of what comes after "VID" -- be it any variable, as demonstrated with the command above, or if I try to insert a direct string value. These other columns are simply text columns, if that helps.
I know you are just learning, so don't be scared by what I'm going to say next :-)
You are improperly forming SQL commands. There's a special feature known as parameter substitution that is present in many database APIs, including that of SQLite in Python.
You don't want to concatenate values like you are doing, because it opens up room for something called SQL injection. In simple scenarios like yours it may not a problem, but when people do it in services exposed on the internet, it can cause great damage (data theft, data loss, data corruption etc). This comic illustrates the problem: https://xkcd.com/327/
In your case, writing the SQL command with parameter substitution is not only going to make your code safer and easier to read, it will also solve the problem you are seeing.
See the example below:
import sqlite3
conn = sqlite3.connect(":memory:")
c = conn.cursor()
# Here we don't need parameter substitution
c.execute("CREATE TABLE mytable (name text, quantity int)")
# We set the values somewhere else. For example, this could come from a web
# form
entry = ("myname", 2)
# Now we execute the query, with the proper parameter substitution.
# Note the `?` characters. Python will automatically replace those with the
# values in `entry`, making sure that things are right when forming the
# command before passing to SQLite.
c.execute("INSERT OR IGNORE INTO mytable VALUES (?, ?)", entry)
print(c.execute("SELECT * from mytable").fetchall())
The example above assumes you don't need to parametrize the column names. In your case, you are apparently reading all columns, so you don't really need to pass all the names, just do like I did in the example above and it will read all columns. If you really need column names to be parameters to read a subset of data, then you'll have to resort to a concatenation mechanism similar to what you were using. But in that case, as always, be really careful with user input to make sure it's not forming bad queries and commands.
More details here: https://docs.python.org/3/library/sqlite3.html
Below is the code. In iterating the dictionary, the code is querying multiple times. Is it the best practice to execute the query or Pinging the DB multiple times ?
import cx_Oracle
connDev = 'username/password#hostname:port/service'
connDev = cx_Oracle.connect(connDev)
cursor = connDev.cursor()
d = {'2006': '20170019201',
'2006172': '2017000002',
'200617123': '200003'
}
for key,value in d.items():
cursDev.execute('SELECT columnName from tableName where columnName={}'.format(key))
if len(cursDev.fetchall())!=0:
# cursDev.execute('UPDATE tableName SET columnName= {0} WHERE columnName= {1} '.format(value, key))
else:
continue
connDev.commit()
cursDev.close()
connDev.close()
You could run a single query and get everything:
cursDev.execute(
'SELECT columnName FROM tableName WHERE columnName IN ({})'.format(
','.join(':p{}'.format(n) for n in range(len(d))),
{'p{}'.format(n): k for n, k in enumerate(d)}
)
or run the updates directly - they do nothing if the row is not found:
for k, v in d.items():
cursDev.execute(
'UPDATE tableName SET columnName = {} WHERE columnName = :value',
{'value': v}
)
Note that both examples are using parameterized queries - the data is being passed separated from the query and it is the job of the database to do the parameter interpolation - thus freeing you from quote hell and preventing injection automatically, besides performing better.
The code uses :value named-style parameter placeholders because that is what cx_Oracle uses - see documentation on cx_Oracle.paramstyle.
For a 'batch' update like this, the executemany() call is going to be most efficient way.
As #nosklo noted, the SELECT calls aren't necessary - they just take time. And with executemany() you don't need to do repeated execute() calls, which is another saving.
From samples/ArrayDMLRowCounts.py:
# delete the following parent IDs only
parentIdsToDelete = [20, 30, 50]
print("Deleting Parent IDs:", parentIdsToDelete)
print()
# enable array DML row counts for each iteration executed in executemany()
cursor.executemany("""
delete from ChildTable
where ParentId = :1""",
[(i,) for i in parentIdsToDelete],
arraydmlrowcounts = True)
# display the number of rows deleted for each parent ID
rowCounts = cursor.getarraydmlrowcounts()
for parentId, count in zip(parentIdsToDelete, rowCounts):
print("Parent ID:", parentId, "deleted", count, "rows.")
See Efficient and Scalable Batch Statement Execution in Python cx_Oracle for more examples, including those with multiple binds like you will need.
[Never ever (except is specialized cases) build up a SQL statement by concatenating strings. It is a security hole and can also give poor performance. Always use bind variables]
I apologize if my question seems novice, but I have hit a roadblock when it comes to assigning variables based on a string for tuples in python (2.7).
In the past I have had no issues assigning a variable and using a string to give it a name (eg: rowId = '%sDays' %workoutMode). But in a tuple environment, I am having some issues.
I have three different tables I want to pull from a database and apply the same code to them. In this instance I want to pull the data and print it based on the string from the defined list. But I am having a problem assigning variables based on strings. Here is my code:
def workoutCycle():
catagories = 'Legs', 'Arms', 'Back'
for catagory in catagories:
conn = sqlite3.connect('workoutData.db')
c = conn.cursor()
c.execute ('SELECT round(%sDays, 1), round(%sDaysTotal, 1) FROM Profile_%s' %(catagory, catagory, catagory))
originalData = c.fetchall()
('%sDays' %catagory, '%sDaysTotal' %catagory) = tuple(originalData[0])
print originalData
print '%sDays' %catagory
print '%sDaysTotal' %catagory
This code returns:
SyntaxError: invalid syntax
I've tried different modifications and I'm not having luck. Is there a specific format that I am missing for strings/tuples?
**********EDIT**************
It seems like I wasn't very clear with what I was trying to do. Essentially I'm trying to create a loop to replace having to write a code out for each catagory. So for example I want to print the data pertaining to "Arms", this code works:
def armCycle():
conn = sqlite3.connect('workoutData.db')
c = conn.cursor()
c.execute ('SELECT round(ArmsDays, 1), round(ArmsDaysTotal, 1) FROM Profile_Arms')
#originalData = c.fetchall()
originalData = c.fetchall()
(ArmsDays, ArmsDaysTotal) = tuple(originalData[0])
print originalData
print ArmsDays
print ArmsDaysTotal
I'm trying to create a code that is a little more dynamic than just creating a function for each catagory. I'm sure I'm going about this the wrong way. I apologize, I'm new to programming.
It looks like you want to create new variables (ArmsDays, ArmsDaysTotal) that are named on the basis of the variable category that contains the value "Arms". The simplest and usual way to do this is not to create individual variables dynamically, but instead use a dict.
c.execute ('SELECT round(%sDays, 1), round(%sDaysTotal, 1) FROM Profile_%s' %(category, category, category))
originalData = c.fetchall()
my_dynamic_data = {}
my_dynamic_data['%sDays' %category] = originalData[0][0]
my_dynamic_data['%sDaysTotal' %category] = originalData[0][1]
The keys of the dict are formed on exactly the same principle as the way you construct the column names in the select statement. Then, instead of
print '%sDays' %category
print '%sDaysTotal' %category
do
print my_dynamic_data['%sDays' %category]
print my_dynamic_data['%sDaysTotal' %category]
You can create new variables dynamically in Python, but if you need to ask how, you shouldn't be doing that. You need a very good reason not to use a dict for this.
I am trying to streamLine queries to SQLITE3. I use it for financial price modelling and so am re-using the same basic query alot, but have to keep changing the hard coding to get out different column queries each time. So I want a generic query where I just write in what I want once, then it spits out the columns as lists. This is a basic version of what I want but basically still hard coded so you can see what I am trying to create.
dbName = 'NASDAQ_Equities'
ticker = 'AAPL'
def pullDataTest(dbPathName, ticker, *args):
datep = []
openp = []
highp = []
db = sqlite3.connect(dbPathName + '.mydb', detect_types=sqlite3.PARSE_DECLTYPES | sqlite3.PARSE_COLNAMES, timeout=3)
cursor = db.cursor()
cursor.execute('''SELECT ''' + str(args) + ''' FROM ''' + ticker)
for row in cursor:
datep.append(row[0])
openp.append(row[1])
highp.append(row[2])
pullData(dbName, ticker, 'datep', 'openp', 'highp')
At the moment I am lost on how to enter an *args into the select statement as it rejects it because of the () brackets. Also what will be an issue is creating empty lists and appending to those lists from from an *args. Would it be better to create a order dict to append to, then brake that into lists at the end somehow? On returning values for use later down the track I was thinking of making them globals? Any suggestions? Thanks
I am trying to get the numbers of rows returned from an sqlite3 database in python but it seems the feature isn't available:
Think of php mysqli_num_rows() in mysql
Although I devised a means but it is a awkward: assuming a class execute sql and give me the results:
# Query Execution returning a result
data = sql.sqlExec("select * from user")
# run another query for number of row checking, not very good workaround
dataCopy = sql.sqlExec("select * from user")
# Try to cast dataCopy to list and get the length, I did this because i notice as soon
# as I perform any action of the data, data becomes null
# This is not too good as someone else can perform another transaction on the database
# In the nick of time
if len(list(dataCopy)) :
for m in data :
print("Name = {}, Password = {}".format(m["username"], m["password"]));
else :
print("Query return nothing")
Is there a function or property that can do this without stress.
Normally, cursor.rowcount would give you the number of results of a query.
However, for SQLite, that property is often set to -1 due to the nature of how SQLite produces results. Short of a COUNT() query first you often won't know the number of results returned.
This is because SQLite produces rows as it finds them in the database, and won't itself know how many rows are produced until the end of the database is reached.
From the documentation of cursor.rowcount:
Although the Cursor class of the sqlite3 module implements this attribute, the database engine’s own support for the determination of “rows affected”/”rows selected” is quirky.
For executemany() statements, the number of modifications are summed up into rowcount.
As required by the Python DB API Spec, the rowcount attribute “is -1 in case no executeXX() has been performed on the cursor or the rowcount of the last operation is not determinable by the interface”. This includes SELECT statements because we cannot determine the number of rows a query produced until all rows were fetched.
Emphasis mine.
For your specific query, you can add a sub-select to add a column:
data = sql.sqlExec("select (select count() from user) as count, * from user")
This is not all that efficient for large tables, however.
If all you need is one row, use cursor.fetchone() instead:
cursor.execute('SELECT * FROM user WHERE userid=?', (userid,))
row = cursor.fetchone()
if row is None:
raise ValueError('No such user found')
result = "Name = {}, Password = {}".format(row["username"], row["password"])
import sqlite3
conn = sqlite3.connect(path/to/db)
cursor = conn.cursor()
cursor.execute("select * from user")
results = cursor.fetchall()
print len(results)
len(results) is just what you want
Use following:
dataCopy = sql.sqlExec("select count(*) from user")
values = dataCopy.fetchone()
print values[0]
When you just want an estimate beforehand, then simple use COUNT():
n_estimate = cursor.execute("SELECT COUNT() FROM user").fetchone()[0]
To get the exact number before fetching, use a locked "Read transaction", during which the table won't be changed from outside, like this:
cursor.execute("BEGIN") # start transaction
n = cursor.execute("SELECT COUNT() FROM user").fetchone()[0]
# if n > big: be_prepared()
allrows=cursor.execute("SELECT * FROM user").fetchall()
cursor.connection.commit() # end transaction
assert n == len(allrows)
Note: A normal SELECT also locks - but just until it itself is completely fetched or the cursor closes or commit() / END or other actions implicitely end the transaction ...
I've found the select statement with count() to be slow on a very large DB. Moreover, using fetch all() can be very memory-intensive.
Unless you explicitly design your database so that it does not have a rowid, you can always try a quick solution
cur.execute("SELECT max(rowid) from Table")
n = cur.fetchone()[0]
This will tell you how many rows your database has.
I did it like
cursor.execute("select count(*) from my_table")
results = cursor.fetchone()
print(results[0])
this code worked for me:
import sqlite3
con = sqlite3.connect(your_db_file)
cursor = con.cursor()
result = cursor.execute("select count(*) from your_table").fetchall() #returns array of tupples
num_of_rows = result[0][0]
A simple alternative approach here is to use fetchall to pull a column into a python list, then count the length of the list. I don't know if this is pythonic or especially efficient but it seems to work:
rowlist = []
c.execute("SELECT {rowid} from {whichTable}".\
format (rowid = "rowid", whichTable = whichTable))
rowlist = c.fetchall ()
rowlistcount = len(rowlist)
print (rowlistcount)
The following script works:
def say():
global s #make s global decleration
vt = sqlite3.connect('kur_kel.db') #connecting db.file
bilgi = vt.cursor()
bilgi.execute(' select count (*) from kuke ') #execute sql command
say_01=bilgi.fetchone() #catch one query from executed sql
print (say_01[0]) #catch a tuple first item
s=say_01[0] # assign variable to sql query result
bilgi.close() #close query
vt.close() #close db file