I would like to use data from SQL server in Pycharm using python. I have my database connection set up in Pycharm, but not sure how to access this data within my python code. I would like to query the data within the python code (similar to what I would do in R using the RODBC package).
Any suggestions on what to do or where to look would be much appreciated.
I have been having issues with this over learning this the last few days. (database / python) For me I am working in flask but it doesn't really seem to matter.
I did get this to work though not exactly what you ask but might get you a start
import MySQLdb
def database():
db = MySQLdb.connect(host="localhost", port=3306, user="root", passwd="admin", db="echo")
cursor = db.cursor()
cursor.execute( "INSERT INTO `post` (`hello`) VALUES (null), ('hello_world')" )
db.commit()
db.close()
I had to just set up my database from the command line. Its not pretty or intuitive but should get you started.
If you want to work with Python objects rather than SQL, I'd use SqlAlchemy and reflection.
from sqlalchemy import MetaData, create_engine
from sqlalchemy.orm import Session
from sqlalchemy.ext.automap import automap_base
engine = create_engine('mysql+mysqldb://...', pool_recycle=3600)
metadata = MetaData()
metadata.reflect(bind=engine)
session = Session(engine)
Base = automap_base(metadata=metadata)
Base.prepare()
# assuming I have a table named 'users'
Users = Base.classes.users
someUsers = Users.query.filter(Users.name.in_(['Jack', 'Bob', 'YakMan')).all()
import mysql.connector
connection=mysql.connector.connect(user='root', password='daniela', host='localhost', database='girrafe')
mycursor=connection.cursor()
There is a concept called OR(Object Relational) Mapping in python, which can be used for database connections. One of the modules that you need to import for the purpose is SQLAlchemy.
First, you will need to install sqlalchemy by:
pip install sqlalchemy
Now, for database connection, we have an Engine class in the sqlalchemy, which is responsible for the database connectivity. We create an object of the Engine class for establishing connection.
from sqlalchemy import create_engine,MetaData,select
engine=create_engine("mysql://user:pwd#localhost/dbname", echo=True)
connection=engine.connect()
The process of reading the database and creating metadata is called Reflection.
metadata=MetaData()
query=select([Student]) #Assuming that my database already has a table named Student
result=connection.execute(query)
row=result.fetchall() #This works similar to the select* query
In this way, you can manipulate data through other queries too, using sqlalchemy!
Related
I have created a MySQL db using the code:
my_engine = create_engine(
"mysql+pymysql://{user}:{pw}#{host}/{db}".format(
host='localhost:8000',
user="user_name",
pw="user_pass",
db="my_db"),
echo=True,
)
the code
my_engine.execute( 'DROP TABLE IF EXISTS {}'.format(table) )
works intermittently, causing python to stall (and the db to freeze) for certain tables. Do you know what this may depend upon?
I'm trying to make my update db quicker that's why I want to use bulk operations. I've a TEST table witch contains around 200k rows. Every day I've to clean the table and load fresh data. When I do this one by one it takes my 2 hours.
I'd like put all the data into dictionary and insert in one operations.
I use a code like below but something is wrong. Do You know what should I change ?
sqlalchemy.exc.UnboundExecutionError: Could not locate a bind
configured on mapper Mapper|TEST|TEST or this Session
my db table:
class TEST(db.Model):
ID = db.Column(db.Integer, primary_key=True)
PN = db.Column(db.String(45))
AMOUNT = db.Column(db.String(6))
and insert code:
from sqlalchemy.orm import mapper, Session
s=Session()
s.bulk_insert_mappings(TEST,
[dict(PN='TEST2', AMOUNT=200), dict(PN='TEST3', AMOUNT=300), dict(PN='TEST5', AMOUNT=500)]
)
I suggest to import Session from flask_sqlalchemy.
from flask_sqlalchemy import Session
And if you configure flask-sqlalchemy correctly flask_sqlalchemy internally configured below mess in behind the scene.
When do I make a sessionmaker?
You are importing the Session wrong way. You have to bind it to the create_engine
from sqlalchemy.orm import mapper, Session
engine = create_engine('sqlite3://...')
s = Session(bind=engine)
I have a code that executes queries to redshift like this:
def send_sql_query(source, sql_query, lst=None):
connection = psycopg2.connect(
host=os.environ["REDSHIFT_HOST"],
port="5439",
dbname="dbname",
user=os.environ["REDSHIFT_USERNAME"],
password=os.environ["REDSHIFT_PASSWORD"],
cursor = connection.cursor()
cursor.execute(sql_query, lst)
sql_results = cursor.fetchall()
return sql_results
finally:
if connection:
connection.close()
I would like to mock the method in a way that it will retrieve and sql_query, and the method will hold a fake db data (preferable in json), but will execute the SQL on the fake data with the sql_query and return the result.
Using mock.return_value and mock.side_effect will not help, because I want to verify that the SQL query is correct. Writing a code to return results doesn't really check the SQL query
Is there a framework in python for it?
Testing the SQL requires a SQL engine. As different databases use different dialects and as you use PostgreSQL as you main database, you should install a PostgreSQL instance on you dev environment with fake data and redirect your queries there while testing.
As you use the environment to store the reference of the database, you have just to setup a test environment pointing to the test database.
I am using sqlalchemy/pyodbc to connect to a MS SQL 2012 server. I chose sqlalchemy because of the direct integration with pandas dataframes using .read_sql and .to_sql.
At a high level, my code is:
df = dataframe.read_sql("EXEC sp_getsomedata")
<do some stuff here>
finaldf.to_sql("loader_table", engine,...)
This part works great, very easy to read, etc. The problem is that I have to run a final stored proc to insert the data from the loader table into the live table. Normally, sqlalchemy knows to commit after INSERT/UPDATE/DELETE, but doesn't want to do the commit for me when I run this final stored proc.
After having tried multiple approaches, I see the transaction in the db sitting uncommitted. I know sqlalchemy is very flexible and I am using about 3% of its functionality, what is the simplest way to get this working? I think I need to be using sqlalchemy core and not ORM. I saw examples using sessionmaker, but I think that monopolizes the engine object and doesn't let pandas access it.
connection = engine.connect()
transaction = connection.begin()
connection.execute("EXEC sp_doLoaderStuff")
transaction.commit()
connection.close()
I have tried calling .execute from the connection level, from a cursor level, and even using the .raw_connection() method without success.
connection = engine.raw_connection()
connection.autocommit = True
cursor = connection.cursor()
cursor.execute("EXEC sp_doLoaderStuff")
connection.commit()
connection.close()
Any ideas what I am missing here?
Completely self-inflicted. The correct working code using the raw_connection() method that is working fine is:
connection = engine.raw_connection()
cursor = connection.cursor()
cursor.execute("EXEC sp_doLoaderStuff")
connection.commit()
connection.close()
In sqlalchemy, I make the connection:
conn = engine.connect()
I found this will set autocommit = 0 in my mysqld log.
Now I want to set autocommit = 1 because I do not want to query in a transaction.
Is there a way to do this?
From The SQLAlchemy documentation: Understanding autocommit
conn = engine.connect()
conn.execute("INSERT INTO users VALUES (1, 'john')") # autocommits
The “autocommit” feature is only in effect when no Transaction has otherwise been declared. This means the feature is not generally used with the ORM, as the Session object by default always maintains an ongoing Transaction.
Full control of the “autocommit” behavior is available using the generative Connection.execution_options() method provided on Connection, Engine, Executable, using the “autocommit” flag which will turn on or off the autocommit for the selected scope. For example, a text() construct representing a stored procedure that commits might use it so that a SELECT statement will issue a COMMIT:
engine.execute(text("SELECT my_mutating_procedure()").execution_options(autocommit=True))
What is your dialect for mysql connection?
You can set the autocommit to True to solve the problem, like this mysql+mysqldb://user:password#host:port/db?charset=foo&autocommit=true
You can use this:
from sqlalchemy.sql import text
engine = create_engine(host, user, password, dbname)
engine.execute(text(sql).execution_options(autocommit=True))
In case you're configuring sqlalchemy for a python application using flask / django, you can create the engine like this:
# Configure the SqlAlchemy part of the app instance
app.config['SQLALCHEMY_DATABASE_URI'] = conn_url
session_options = {
'autocommit': True
}
# Create the SqlAlchemy db instance
db = SQLAlchemy(app, session_options=session_options)
I might be little late here, but for fox who is using sqlalchemy >= 2.0.*, above solution might not work as it did not work for me.
So, I went through the official documentation, and below solution worked for me.
from sqlalchemy import create_engine
db_engine = create_engine(database_uri, isolation_level="AUTOCOMMIT")
Above code works if you want to set autocommit engine wide.
But if you want use autocommit for a particular query then you can use below -
with engine.connect().execution_options(isolation_level="AUTOCOMMIT") as connection:
with connection.begin():
connection.execute("<statement>")
Official Documentation
This can be done using the autocommit option in the execution_option() method:
engine.execute("UPDATE table SET field1 = 'test'").execution_options(autocommit=True))
This information is available within the documentation on Autocommit