I currently have a python script that can connect to a mySQL db and execute queries. I wish to modify it so that I can connect to a different SQL db to run a separate query. I am having trouble doing this, running osx 10.11. This is my first question and I'm a newbie programmer so please take it easy on me...
Here is the program i used to for mySQL
sf_username = "user"
sf_password = "pass"
sf_api_token = "token"
sf_update = beatbox.PythonClient()
password = str("%s%s" % (sf_password, sf_api_token))
sf_update.login(sf_username, password)
t = Terminal()
hub = [stuff]
def FINAL_RUN():
cnx = alternate_modify(hub)
cur = cnx.cursor()
queryinput = """
SQL QUERY I WANT
"""
cur.execute(queryinput)
rez = cur.fetchone()
while rez is not None:
write_to_sf(rez)
rez = cur.fetchone()
FINAL_RUN()
You can use a Python library called SQLAlchemy. It abstracts away the "low-level" operations you would do with a database (e.g. specifying queries manually).
A tutorial for using SQLAlchemy can be found here.
I was able to get connected using SQL Alchemy--thank you. If anyone else tries, I think you'll need a ODBC driver per this page:
http://docs.sqlalchemy.org/en/latest/dialects/mssql.html
Alternatively, pymssql is a nice tool. If you run into trouble installing like I did, there is a neat workaround here:
mac - pip install pymssql error
Thanks again
Related
I'm using Python (and Peewee) to connect to a SQLite database. My data access layer (DAL) is a mix of peewee ORM and SQL-based functions. I would like to enable EXPLAIN PLAN for all queries upon connecting to the database and toggle it via configuration or CLI parameter ... how can I do that using the Python API?
from playhouse.db_url import connect
self._logger.info("opening db connection to database, creating cursor and initializing orm model ...")
self.__db = connect(url)
# add support for a REGEXP and POW implementation
# TODO: this should be added only for the SQLite case and doesn't apply to other vendors.
self.__db.connection().create_function("REGEXP", 2, regexp)
self.__db.connection().create_function("POW", 2, pow)
self.__cursor = self.__db.cursor()
self.__cursor.arraysize = 100
# what shall I do here to enable EXPLAIN PLANs?
That is a feature of the sqlite interactive shell. To get the query plans, you will need to explicitly request it. This is not quite straightforward with Peewee because it uses parameterized queries. You can get the SQL executed by peewee in a couple of ways.
# Print all queries to stderr.
import logging
logger = logging.getLogger('peewee')
logger.addHandler(logging.StreamHandler())
logger.setLevel(logging.DEBUG)
Or for an individual query:
query = SomeModel.select()
sql, params = query.sql()
# To get the query plan:
curs = db.execute_sql('EXPLAIN ' + sql, params)
print(curs.fetchall()) # prints query plan
I have the following code in flask
sql = text('select * from person')
results = self.db.engine.execute(sql)
for row in results:
print(".............", row) # prints nothing
people = Person.query.all() # shows all person data
Now given this situation, it's obvious, the self.db is not using the same connection somehow that Person.query is using. However, given this situation, can I get the connection somehow from Person.query object?
PS. This is for testing and I'm using SQLite3 database. I tried this in postgres, but outcome is the same.
Just figured out. Try Person.query.session.execute(sql). Voila!
I don't know what suddenly caused this (I recently reinstalled Anaconda and all my python libraries, but I am back to the same versions as before), but when sqlalchemy tries to connect to the SQL server, it fails because it looks for the server version and tries to run a string operation on it.
The following had no problems prior to my reinstall of packages. I'd connect like so:
sqlalchemy_conn_string = 'mssql+pyodbc://myDSN'
sqlalchemy.create_engine(sqlalchemy_conn_string, module=pypyodbc)
Then it gets all the way to a file called pyodbc.py and fails at this function:
def _get_server_version_info(self, connection):
try:
raw = connection.scalar("SELECT SERVERPROPERTY('ProductVersion')")
except exc.DBAPIError:
#...
else:
version= []
r = re.compile(r'[.\-]')
for n in r.split(raw): # culprit here
try:
version.append(int(n))
except ValueError:
version.append(n)
return tuple(version)
Out[1]: TypeError: cannot use a string pattern on a bytes-like object
That's because at this step, raw is not a string that can be split:
# from PyCharm's debugger window
raw = {bytes}b'13.0.5026.0'
At this point, I don't know if I should submit a bug report for sqlalchemy and/or pypyodbc, or if there's something I can do to fix this myself. But I'd like a solution that doesn't involve editing the code for sqlalchemy on my own machine (like handling the bytes-like object specifically), because we have other team members who will also be downloading vanilla sqlalchemy & pypyodbc and won't have the confidence in editing that source code.
I have confirmed the pypyodbc behaviour under Python 3.6.4.
print(pypyodbc.version) # 1.3.5
sql = """\
SELECT SERVERPROPERTY('ProductVersion')
"""
crsr.execute(sql)
x = crsr.fetchone()[0]
print(repr(x)) # b'12.0.5207.0'
Note that SQLAlchemy's mssql+pyodbc dialect is coded for pyodbc, not pypyodbc, and the two are not guaranteed to be 100% compatible.
The obvious solution would be to use pyodbc instead.
UPDATE:
Check your version of SQLAlchemy. I just looked at the current source code for the mssql+pyodbc dialect and it does
def _get_server_version_info(self, connection):
try:
# "Version of the instance of SQL Server, in the form
# of 'major.minor.build.revision'"
raw = connection.scalar(
"SELECT CAST(SERVERPROPERTY('ProductVersion') AS VARCHAR)")
which should avoid the issue, even when using pypyodbc.
If you are using the latest production release of SQLAlchemy (currently version 1.2.15), then you might have better luck with version 1.3.0b1.
I would like to use data from SQL server in Pycharm using python. I have my database connection set up in Pycharm, but not sure how to access this data within my python code. I would like to query the data within the python code (similar to what I would do in R using the RODBC package).
Any suggestions on what to do or where to look would be much appreciated.
I have been having issues with this over learning this the last few days. (database / python) For me I am working in flask but it doesn't really seem to matter.
I did get this to work though not exactly what you ask but might get you a start
import MySQLdb
def database():
db = MySQLdb.connect(host="localhost", port=3306, user="root", passwd="admin", db="echo")
cursor = db.cursor()
cursor.execute( "INSERT INTO `post` (`hello`) VALUES (null), ('hello_world')" )
db.commit()
db.close()
I had to just set up my database from the command line. Its not pretty or intuitive but should get you started.
If you want to work with Python objects rather than SQL, I'd use SqlAlchemy and reflection.
from sqlalchemy import MetaData, create_engine
from sqlalchemy.orm import Session
from sqlalchemy.ext.automap import automap_base
engine = create_engine('mysql+mysqldb://...', pool_recycle=3600)
metadata = MetaData()
metadata.reflect(bind=engine)
session = Session(engine)
Base = automap_base(metadata=metadata)
Base.prepare()
# assuming I have a table named 'users'
Users = Base.classes.users
someUsers = Users.query.filter(Users.name.in_(['Jack', 'Bob', 'YakMan')).all()
import mysql.connector
connection=mysql.connector.connect(user='root', password='daniela', host='localhost', database='girrafe')
mycursor=connection.cursor()
There is a concept called OR(Object Relational) Mapping in python, which can be used for database connections. One of the modules that you need to import for the purpose is SQLAlchemy.
First, you will need to install sqlalchemy by:
pip install sqlalchemy
Now, for database connection, we have an Engine class in the sqlalchemy, which is responsible for the database connectivity. We create an object of the Engine class for establishing connection.
from sqlalchemy import create_engine,MetaData,select
engine=create_engine("mysql://user:pwd#localhost/dbname", echo=True)
connection=engine.connect()
The process of reading the database and creating metadata is called Reflection.
metadata=MetaData()
query=select([Student]) #Assuming that my database already has a table named Student
result=connection.execute(query)
row=result.fetchall() #This works similar to the select* query
In this way, you can manipulate data through other queries too, using sqlalchemy!
I just found out that the way I am using _mysql is causing a major SQL Injection problem.
My current code looks like:
db = _mysql.connect('', 'user', 'pass', 'db')
query = """SELECT * FROM stuff WHERE thing="{0}" """.format(user_input)
cur.query(query)
What am I doing wrong and how can I fix it so that it is safe?
I have tried using _mysql.escape_string() but that still returns an SQL syntax error.
You can use MySQLdb on its own:
conn = MySQLdb.connect();
curs = conn.cursor();
curs.execute("SELECT * FROM stuff WHERE thing = %s", (user_input));
If you want to stick with _mysql, use db.escape_string(user_input).
Documentation: http://mysql-python.sourceforge.net/MySQLdb.html
A nice handy reference is available via the bobby tables website.
You may also find value in In this powerpoint reference which shows some examples of sql injection as well as possible ways to mitigate the issue.