I am new to python (as I am sure you can tell from my code) and have basically written a sample script to test some core commands so I can better understand how things work. I have gotten everything running as designed except the last "insert" command -- after hours of googling and experimenting I cannot figure out what's wrong, so I would appreciate it if anyone can show me what needs to be changed and helping me understand why (I'm sure it's basic, but am stumped!).
Below is the line that is giving me trouble:
c.execute("INSERT OR IGNORE INTO {tn} ({cn1}, {cn2}, {cn3}, {cn4}) VALUES ({VID}, {VSnu}, {VIN}, {VName})".\
format(tn=table_name, cn1=column1, cn2=column2, cn3=column3, cn4=column4, VID=ID, VSnu=Snu, VIN=IN, VName=Name))
Here's the entire script for context, if that helps:
import sqlite3
sqlite_file = '/test_database.sqlite' # name of the sqlite database file
table_name = 'test_table'
column1 = 'my_1st_column'
column2 = 'my_2nd_column'
column3 = 'my_3rd_column'
column4 = 'my_4th_column'
ID = int(123456)
Base = 'Arnold'
Snu = 'test'
conn = sqlite3.connect(sqlite_file)
c = conn.cursor()
c.execute("UPDATE {tn} SET {cn2}=('Snu'), {cn3}=('Muh'), {cn4}=('Arnold_A') WHERE {cn1}=({NID})".\
format(tn=table_name, cn1=column1, cn2=column2, cn3=column3, cn4=column4, NID=ID))
i = 1
while(i<15):
if i == 1: IN = 'B'
if i == 2: IN = 'C'
if i == 3: IN = 'D'
if i == 4: IN = 'E'
if i == 5: IN = 'F'
if i == 6: IN = 'G'
if i == 7: IN = 'H'
if i == 8: IN = 'I'
if i == 9: IN = 'J'
ID = ID+1
i = i+1
Name = Base + '_' + IN
params = (Snu, IN, Name)
c.execute("INSERT OR IGNORE INTO {tn} ({cn1}, {cn2}, {cn3}, {cn4}) VALUES ({VID}, {VSnu}, {VIN}, {VName})".\
format(tn=table_name, cn1=column1, cn2=column2, cn3=column3, cn4=column4, VID=ID, VSnu=Snu, VIN=IN, VName=Name))
if(i == 10): break
conn.commit()
conn.close()
It will write the "VID" item just fine if isolated (which is an integer column and the primary key of the table), but everything after that it interprets as a column and I get the "no such column: [value]" error.
I get this error regardless of what comes after "VID" -- be it any variable, as demonstrated with the command above, or if I try to insert a direct string value. These other columns are simply text columns, if that helps.
I know you are just learning, so don't be scared by what I'm going to say next :-)
You are improperly forming SQL commands. There's a special feature known as parameter substitution that is present in many database APIs, including that of SQLite in Python.
You don't want to concatenate values like you are doing, because it opens up room for something called SQL injection. In simple scenarios like yours it may not a problem, but when people do it in services exposed on the internet, it can cause great damage (data theft, data loss, data corruption etc). This comic illustrates the problem: https://xkcd.com/327/
In your case, writing the SQL command with parameter substitution is not only going to make your code safer and easier to read, it will also solve the problem you are seeing.
See the example below:
import sqlite3
conn = sqlite3.connect(":memory:")
c = conn.cursor()
# Here we don't need parameter substitution
c.execute("CREATE TABLE mytable (name text, quantity int)")
# We set the values somewhere else. For example, this could come from a web
# form
entry = ("myname", 2)
# Now we execute the query, with the proper parameter substitution.
# Note the `?` characters. Python will automatically replace those with the
# values in `entry`, making sure that things are right when forming the
# command before passing to SQLite.
c.execute("INSERT OR IGNORE INTO mytable VALUES (?, ?)", entry)
print(c.execute("SELECT * from mytable").fetchall())
The example above assumes you don't need to parametrize the column names. In your case, you are apparently reading all columns, so you don't really need to pass all the names, just do like I did in the example above and it will read all columns. If you really need column names to be parameters to read a subset of data, then you'll have to resort to a concatenation mechanism similar to what you were using. But in that case, as always, be really careful with user input to make sure it's not forming bad queries and commands.
More details here: https://docs.python.org/3/library/sqlite3.html
Related
im very new to Python but want to preform some mathmatic functions using Python's libraries getting interger values from a mysql table i have running,
ive sucessfully established a connection using mysql.connector however im at a loss,
I can select and print Rows and columbs but im unsure of the Syntax to physically define my query as an "x" or "y" in order to preform mathmatic operations with the varible.
Any help would be greatly appreciated.
EDIT
sql_select_Query = "select * from ATABLE"
cursor = mySQLconnection .cursor()
cursor.execute(sql_select_Query)
records = cursor.fetchall()`
and
for row in records:
print("Name = ", row[1], )
print("X_num = ", row[2])
print("Y_num = ", row[3])
print("Signal_Strength = ", row[4], "\n")
cursor.close()
gives me as an example
Name = X,
X_num = Y,
Y_num = Z,
SS = Q
what i would prefer in my selection operation is to define the X , Y, Z, Q to a Global name that i could then use for atleast my application math operations using Numpy libraries for example being able to perform an operator
X*Y-Z+Q
I hope that is a bit clearer
From the gate, I would recommend following the advice of this thread highlighting the use of select *. Turning a field into an integer is possible with your SQL selection statement int the way of CAST or CONVERT. Sort of like this (my daily language is SQL Server; check out the mysql documentation for exacts):
sql_select_Query = "select Name, CAST(X as INT),CAST(Y as BIGINT) from ATABLE"
In my personal experience, SQL tends to age better than Python (tongue in cheek). Aside, if your SQL instance is on a server; I code to the workhorse as error catching is better.
But coming from it in the other direction, if you want these elements to be re-callable later, I'm suggest fetching your feedback into a dictionary.
Information about Python dictionaries can be found here. At least that way, you're pretty much working from a global but fairly structured set of captured data.
It is a bad idea to play with locals() and globals() if you don't exactly know what you're doing. Create a dictionary.
sql_select_Query = "select * from ATABLE"
cursor = mySQLconnection.cursor()
cursor.execute(sql_select_Query)
records = cursor.fetchall()
columns = [item[0] for item in cursor.description] # Grab the table column names
for record in records:
# Create a dictionary {column_name: value, ...} for each row
variable_dict = dict(zip(columns, record))
print("X variable is: ", variable_dict['X'])
# <Calculation here>
You can also configure MySQL to return values as a dictionary but this is probably an easier starting point.
This way, your "variable X" value would just be variable_dict['X'] and there's no need to make any global values other than the dictionary.
Whenever I try the code below, I get near "?": syntax error
I tried multiple things including prepping it into a variable
Is this possible in python? Or am I thinking in the wrong direction?
import sqlite3
sqlite_file = 'DATABASE.db'
conn = sqlite3.connect(sqlite_file)
c = conn.cursor()
word = ()
question = int(input("What would you like to see?"))
if question == 1:
word = ("Batchnumbers")
if question == 2:
word = ("Worker IDs")
c.execute('SELECT * FROM ?', (word))
data = c.fetchall()
for x in range(len(data)):
print(data[x])
Query parameters can only be used to supply column values, not table or column names. If you need to supply a table name you will have to use dynamic SQL, e.g.,
c.execute('SELECT * FROM "{}"'.format(word))
Note that this approach is vulnerable to SQL injection issues, so you really should consider mitigating those, e.g., ensuring that word does not contain double-quote characters that would cause errors (or worse).
Indeed use this line of code
word =
c.execute('SELECT * FROM "{}"'.format(word))
I apologize if my question seems novice, but I have hit a roadblock when it comes to assigning variables based on a string for tuples in python (2.7).
In the past I have had no issues assigning a variable and using a string to give it a name (eg: rowId = '%sDays' %workoutMode). But in a tuple environment, I am having some issues.
I have three different tables I want to pull from a database and apply the same code to them. In this instance I want to pull the data and print it based on the string from the defined list. But I am having a problem assigning variables based on strings. Here is my code:
def workoutCycle():
catagories = 'Legs', 'Arms', 'Back'
for catagory in catagories:
conn = sqlite3.connect('workoutData.db')
c = conn.cursor()
c.execute ('SELECT round(%sDays, 1), round(%sDaysTotal, 1) FROM Profile_%s' %(catagory, catagory, catagory))
originalData = c.fetchall()
('%sDays' %catagory, '%sDaysTotal' %catagory) = tuple(originalData[0])
print originalData
print '%sDays' %catagory
print '%sDaysTotal' %catagory
This code returns:
SyntaxError: invalid syntax
I've tried different modifications and I'm not having luck. Is there a specific format that I am missing for strings/tuples?
**********EDIT**************
It seems like I wasn't very clear with what I was trying to do. Essentially I'm trying to create a loop to replace having to write a code out for each catagory. So for example I want to print the data pertaining to "Arms", this code works:
def armCycle():
conn = sqlite3.connect('workoutData.db')
c = conn.cursor()
c.execute ('SELECT round(ArmsDays, 1), round(ArmsDaysTotal, 1) FROM Profile_Arms')
#originalData = c.fetchall()
originalData = c.fetchall()
(ArmsDays, ArmsDaysTotal) = tuple(originalData[0])
print originalData
print ArmsDays
print ArmsDaysTotal
I'm trying to create a code that is a little more dynamic than just creating a function for each catagory. I'm sure I'm going about this the wrong way. I apologize, I'm new to programming.
It looks like you want to create new variables (ArmsDays, ArmsDaysTotal) that are named on the basis of the variable category that contains the value "Arms". The simplest and usual way to do this is not to create individual variables dynamically, but instead use a dict.
c.execute ('SELECT round(%sDays, 1), round(%sDaysTotal, 1) FROM Profile_%s' %(category, category, category))
originalData = c.fetchall()
my_dynamic_data = {}
my_dynamic_data['%sDays' %category] = originalData[0][0]
my_dynamic_data['%sDaysTotal' %category] = originalData[0][1]
The keys of the dict are formed on exactly the same principle as the way you construct the column names in the select statement. Then, instead of
print '%sDays' %category
print '%sDaysTotal' %category
do
print my_dynamic_data['%sDays' %category]
print my_dynamic_data['%sDaysTotal' %category]
You can create new variables dynamically in Python, but if you need to ask how, you shouldn't be doing that. You need a very good reason not to use a dict for this.
I am trying to read 1000 tweets from a file.
http://rasinsrv07.cstcis.cti.depaul.edu/CSC455/Twitter_2013_11_12.txt
The tweets are stored on line-by-line basis.
I have to create a SQL table for the 'geo' entry. 'Geo' is a dictionary inside the tweets dictionary. In some cases dictionary 'geo' is completely blank and in come cases it has values. I need to keep track of how many Geo dictionaries are blank, and how many have vlaues. I need to generate a unique ID for that table. In addition to the ID column, the geo table should have “type”, “longitude” and “latitude” columns. If Geo dictionary has values, it looks like this:
{u'type': u'Point', u'coordinates': [44.49241705, 11.33374359]}
Since I am new to Python and SQLITE, my code is basic (as I want to be able to understamd my code), and it is not working as expected. I am trying to do the insert in the Geo table if the length of geo dictionary is greater than 1, but it is not working. Any input will be greatly appreciated.
import urllib2, time, json, sqlite3
conn = sqlite3.connect('Tweets_Database_A6.db')
c = conn.cursor()
wFD = urllib2.urlopen('http://rasinsrv07.cstcis.cti.depaul.edu/CSC455/Twitter_2013_11_12.txt')
numLines = 1000
tweets = []
while numLines > 0:
line = wFD.readline()
numLines = numLines - 1
try:
tweets.append(json.loads(line))
except:
print line
wFD.close()
#create geo table using sqlite3
TblGeo = """create table Geo(Id number, Type text, Longitude number, latitude number);"""
c.execute(TblGeo)
HasGeo=0
NoGeo=0
for tweet in tweets:
tweet_geo = tweet['geo']
if len(tweet_geo) > 1:
HasGeo = HasGeo+1
try:
c.execute("insert into Geo(id, Type, Longitude, Latitude) values ('%s', '%s', '%s', '%s')" %(HasGeo, tweet_geo['type'], tweet_geo['coordinates'][0], tweet_geo['coordinates'][1]))
except:
print "no entry for " , i
else:
NoGeo = NoGeo+1
print HasGeo, " ", NoGeo
Your code is failing for a few reasons. Since this appears to be an assignment, I will not post the working code here, but I will attempt to point you in the right direction. Here are some of the things I've noticed while testing your code:
You made the assumption that tweet['geo'] would be an empty string. It actually is not. Essentially, the data sets this value to a json "null" when no geo information is available; this gets translated to the Nonetype in python, and not an empty string. Therefore, you should not be checking for the length of that value, but rather whether that value is True (hint: python considers '', "", [], ,{}, None as False)
I don't think your indentation on lines 28-31 is correct. Shouldn't that logic execute in the if block? right now, you are always executing that code, which I think is a logical error.
In your exception trapping at line 31, where do you define the variable "i"?
I hope this is helpful; feel free to ask additional clarifications if you are stumped.
I've got a SQLite query, which depends on 2 variables, gender and hand. Each of these can have 3 values, 2 which actually mean something (so male/female and left/right) and the third is 'all'. If a variable has a value of 'all' then I don't care what the particular value of that column is.
Is it possible to achieve this functionality with a single query, and just changing the variable? I've had a look for a wildcard or don't care operator but haven't been able to find any except for % which doesn't work in this situation.
Obviously I could make a bunch of if statements and have different queries to use for each case but that's not very elegant.
Code:
select_sql = """ SELECT * FROM table
WHERE (gender = ? AND hand = ?)
"""
cursor.execute(select_sql, (gender_var, hand_var))
I.e. this query works if gender_val = 'male' and hand_var = 'left', but not if gender_val or hand_var = 'all'
You can indeed do this with a single query. Simply compare each variable to 'all' in your query.
select_sql = """ SELECT * FROM table
WHERE ((? = 'all' OR gender = ?) AND (? = 'all' OR hand = ?))
"""
cursor.execute(select_sql, (gender_var, gender_var, hand_var, hand_var))
Basically, when gender_var or hand_var is 'all', the first part of each OR expression is always true, so that branch of the AND is always true and matches all records, i.e., it is a no-op in the query.
It might be better to build a query dynamically in Python, however, that has just the fields you actually need to test. It might be noticeably faster, but you'd have to benchmark that to be sure.