py.test - How to inherit other tests - python

So let's say that I have two files (test_file1.py, test_file2.py) for integration testing using py.test.
The test_file1.py is something like this:
import datetime
import pytest
Datetime = datetime.datetime.now()
def test_connect():
#1st Query to a mysql database
#2nd Query to a mysql database
..
#N Query to a mysql database
Now I'm writing the test_file2.py which is an extention of test_file1.py but I don't want to write the same mysql queries that I wrote in the above test.
How can I make py.test to inherit the above test and run both after executing py.test test_file2.py?
Something like this (test_file2.py Contents):
import datetime
import pytest
from testDirectory import test_file1
Datetime = datetime.datetime.now()
def test_connect():
#Here should run all the tests from 'test_file1' somehow...
#1st new additional Query to a mysql database
#2nd new additional Query to a mysql database
..
#N new additional Query to a mysql database
Thanks!!

When you import a module, it will execute all of the code inside it. So just write the code you want executed in your original file. For example add the call to the function in your file like this:
test_file1.py:
import datetime
import pytest
Datetime = datetime.datetime.now()
def test_connect():
#1st Query to a mysql database
#2nd Query to a mysql database
..
#N Query to a mysql database
test_connect() # This will run your function when you import
So then in your py.test when you call import test_file1, it will execute the test_connect() and any other code you would like without doing anything else.
In other words, here is a really simple example with 3 files:
File 1: hello_world.py:
def hello_world():
print('hello world!')
hello_world()
File 2: print_text.py:
def print_text():
print('foo bar baz')
print_text()
File 3: run_everything.py:
import hello_world
import print_text
Result when you run run_everything.py:
>>>hello world!
>>>foo bar baz
If you want the function to be executed when the file is executed directly, but not imported as a module, you can do this:
test_file1.py:
import datetime
import pytest
Datetime = datetime.datetime.now()
def test_connect():
#1st Query to a mysql database
#2nd Query to a mysql database
..
#N Query to a mysql database
def main():
# This will _not_ run your function when you import. You would
# have to use test_file1.test_connect() in your py.test.
test_connect()
if __name__ == '__main__':
main()
So in this example, your py.test would be:
import test_file1
test_file1.test_connect()

First one create a fixture in conftest.py:
import pytest
import MySQLdb
def db_cursor(request):
db = MySQLdb.connect(host="localhost", user="root")
cursor = db.cursor()
cursor.execute("SELECT USER()")
data = cursor.fetchone()
assert 'root#localhost' in data
yield cursor
db.close()
Then use it in your test modules:
# test_file1.py
def test_a(db_cursor)
pass
# test_file2.py
def test_b(db_cursor)
res = db_cursor.execute("SELECT VERSION()")
assert '5.5' in res.fetchone()
P.S.
It possible to use any other modules, just inject they are into your tests with pytest_plugins directive:
# conftest.py
pytest_plugins = '_mysql.cursor'
# _mysql/__init__.py
# _mysql/cursor.py
import pytest
import MySQLdb
def db_cursor(request):
db = MySQLdb.connect(host="localhost", user="root")
cursor = db.cursor()
cursor.execute("SELECT USER()")
data = cursor.fetchone()
assert 'root#localhost' in data
yield cursor
db.close()

Related

Python 3 not completing mysql delete query

I'm working on a legacy script upgrade to Python 3 however the script is hanging during a database delete command (DELETE FROM). The script is showing no error and the logger contains only the is_connected result which is true. Here's my test script based on the main.py file but with only the call to delete the contents of a table and reset it's auto increment prior to repopulating the table.
Here's my test.py file.
#!/usr/bin/env python3
import json
import hashlib
from pprint import pprint
import mysql.connector
import configparser
import re
import random
import requests
import sys
import logging
# Load config for database variables
config = configparser.ConfigParser()
config.read("config.ini")
logFile = "logger.log"
# Set up logging
logging.basicConfig(format='%(asctime)s %(message)s', filename=logFile,level=logging.DEBUG)
# Connect to the MySQL database
cnx = mysql.connector.connect(
host=config["MySQL"]["host"],
user=config["MySQL"]["user"],
port=config["MySQL"]["port"],
passwd=config["MySQL"]["password"],
database=config["MySQL"]["database"]
)
cur = cnx.cursor()
logging.debug(cnx.is_connected())
# Clear the database ready to re-import
# Clear lookup tables first
cur.execute("DELETE FROM member_additional_skills")
logging.debug("Delete Done")
cur.execute("ALTER TABLE member_additional_skills AUTO_INCREMENT = 1")
cnx.commit()
logging.debug("Finished!")
print("Done")
I've left this running for 20 minutes and still nothing else is logged after it declares Teue to being connectedand the process is still running. Is there anything I've missed here?
*** EDIT ***
The process is still in htop but is not using cpu so seems crashed right? And as i write this I have the following as the output to my python3 test.py command line:
client_loop: send disconnect: Broken pipe and the process is no longer in htop
I should point out that this table has no more than 30 rows in it so would expect it to complete in milliseconds
thanks
You can use TrUNCAtE member_additional_skills OR DROP member_additional_skills and CREATE member_additional_skills(....)
that is much faster
You might be having a lock issue:
Sometime with the MySQL connector, the table lock's aren't released properly. Making the table unchangeable.
Try resetting the database and trying again!

Use python function in a MySQL query

my workflow is
Extract a CSV file from MySQL database--> open the CSV on Python---> filter the necessary information based on a Python function.
However, I am started to deal with datasets that don't fit on memory. It is also inconvenient to have to import and filter over and over again.
My question is: Is there a way to apply a Python function in a MySQL database? I mean in a way that I only download from MySQL the values that attend my filter based on a Python function.
Note: I use Datagrip.
which python function to apply in MySQL database? Try to apply tests of some libraries! Which one do you like? Tests with a query like "SELECT 1"
import time
def query_1k(cur):
t = time.time()
for _ in range(1000):
cur.execute("SELECT 1,2,3,4,5")
try:
res = cur.fetchall()
assert len(res) == 1
assert res[0] == (1,2,3,4,5)
except:
pass
return time.time() - t
# pip search mysql-connector | grep --color mysql-connector-python
# pip install mysql-connector-python-rf
def mysql_connector_python():
import mysql.connector
conn = mysql.connector.connect(user='root', host='localhost')
print("MySQL Connector/Python:", query_1k(conn.cursor()), "[sec]")
# pip install mysqlclient
# pip3 install mysqlclient
# sudo yum install gcc
def mysqlclient():
pass
try:
import pymysql
pymysql.install_as_MySQLdb()
except ImportError:
pass
import MySQLdb
conn = MySQLdb.connect(user='root', host='localhost')
print("MySQLdb mysqlclient:", query_1k(conn.cursor()), "[sec]")
def pymysql():
import pymysql
conn = pymysql.connect(user='root', host='localhost')
print("PyMySQL:", query_1k(conn.cursor()), "[sec]")
def msqlchemy():
from sqlalchemy import create_engine
conn = create_engine('sqlite:///:memory:')
print("sqlalchemy mysqlclient:", query_1k(conn), "[sec]")
def pewee():
# from pewee import *
user = 'root'
password = '1234'
db_name = 'information_schema'
conn = MySQLDatabase(
db_name, user=user,
password=password,
host='localhost'
)
print("pewee:", query_1k(conn.cursor()), "[sec]")
'''
Warning: (3090, u"Changing sql mode 'NO_AUTO_CREATE_USER' is deprecated.
SET SESSION sql_mode="NO_ENGINE_SUBSTITUTION,NO_AUTO_CREATE_USER";
'''
for _ in range(10): # for PyPy warmup
print('-------------')
mysql_connector_python()
mysqlclient()
msqlchemy()
pymysql()
# pewee()

How to mock mongodb when it is called from another function?

I need help while mocking mongodb. I am using mongomock to mock mongodb.
My project structure is:
-- my_mongo.py
-- code.py
-- my_test.py
my_mongo.py has :
from pymongo import MongoClient
def get_db():
client = MongoClient(os.environ['MONGODB_URI'])
db = client['my_db']
return db
def insert(id, data):
return get_db().results.insert_one(
{
"id": id,
"data":df.to_json(),
}).id
and code.py has
import my_mongo
def action():
#do somethings
my_mongo.insert(id, data)
and my_test.py has
import mongomock
import my_mongo
from unittest import mock
with patch.object(my_mongo.get_db().client, "client", mongomock.MongoClient()):
import code
def test_action_1():
my_mongo.insert = mock.Mock(return_value=1)
code.action()
def test_action_2():
with patch.object(my_mongo.get_db(), "get_db", mongomock.MongoClient().db):
code.action()
It throws pymongo.errors.ServerSelectionTimeoutError for both tests. So, It still goes into the insert_one() method in my_mongo.py.
I expect in test_action_1 my_mongo.insert returns 1, but it doesn't.
What am I missing?
I'm not entirely sure what mongomock is for, but it looks like it's for mocking an entire mongo database and not actually using python mocking. I'm going to answer without including mongomock since I don't think you really need it, so you can take that for what it's worth.
There were a few issues:
Calling patch.object will patch the given method on whatever object you give it. If you call get_db in the test, then code.action calls get_db, those are 2 different objects. Maybe this works? But I'm skeptical, so I just changed it.
Don't use code as your module name. That's already a module included with python.
code.action was missing args and a return statement.
You'll also notice that I changed how and what was being mocked to illustrate different ways to accomplish the mocking. Test 1 mocks the insert call with a function decorator. Test 2 mocks the get_db call with a contextmanager. Either is correct, just showing that you have options.
Here's the finished product:
my_mongo.py:
from pymongo import MongoClient
def get_db():
client = MongoClient(os.environ['MONGODB_URI'])
db = client['my_db']
return db
def insert(id, data):
return get_db().results.insert_one({"id": id, "data":data.to_json()}).id # df was undefined, updated to data
my_code.py:
import my_mongo
# I added id and data args. They were undefined
def action(id, data):
return my_mongo.insert(id, data) # I added a return here
my_test.py
from unittest import mock
import my_code
# I removed the contextmanager import. Nothing is being evaluated here that would
# need to be patched, so I'm pretty certain it has no effect
#mock.patch('my_mongo.insert')
def test_action_1(mock_insert):
expected_id = 1
mock_insert.return_value = expected_id
ret = my_code.action(expected_id, mock.Mock())
assert ret == expected_id
def test_action_2():
with mock.patch('my_mongo.get_db') as mock_get_db:
expected_id = 'some id'
mock_db = mock.Mock()
mock_db.results.insert_one.return_value.id = expected_id
mock_get_db.return_value = mock_db
ret = my_code.action(expected_id, mock.Mock())
assert ret == expected_id
That line of code for patching mongodb is wrong. Instead of using patch.object(my_mongo.get_db(), "get_db", mongomock.MongoClient().db), you should use patch.object("my_mongo.get_db", return_value=mongomock.MongoClient()['my_db']).
Following is the complete runable code for your example:
my_test.py
import mongomock
from unittest.mock import patch
import my_code
import my_mongo
def test_action_2():
mocked_mongo = mongomock.MongoClient()
with patch("my_mongo.get_db", return_value=mongomock.MongoClient()['my_db']):
my_code.action()
assert mocked_mongo.my_db.results.count_documents({'id': 'some_id'}) == 1
my_mongo.py
from pymongo import MongoClient
def get_db():
client = MongoClient(os.environ['MONGODB_URI'])
db = client['my_db']
return db
def insert(id, data):
return get_db().results.insert_one(
{
"id": id,
"data": data,
})
my_code.py
import my_mongo
def action():
#do somethings
return my_mongo.insert('some_id', '{"a": 3}')

Shebang line #!/usr/bin/python3 preventing server run

Here's the code I have. Basically I have the Shebang line in there because the psycopg2 wasn't working without it.
But now when I have this line in there it doesn't allow me to run the database, it just says "no module named 'flask'"
#!/usr/bin/python3.4
#
# Small script to show PostgreSQL and Pyscopg together
#
from flask import Flask, render_template
from flask import request
from flask import *
from datetime import datetime
from functools import wraps
import time
import csv
import psycopg2
app = Flask(__name__)
app.secret_key ='lukey'
def getConn():
connStr=("dbname='test' user='lukey' password='lukey'")
conn=psycopg2.connect(connStr)
return conn
#app.route('/')
def home():
return render_template(index.html)
#app.route('/displayStudent', methods =['GET'])
def displayStudent():
residence = request.args['residence']
try:
conn = None
conn = getConn()
cur = conn.cursor()
cur.execute('SET search_path to public')
cur.execute('SELECT stu_id,student.name,course.name,home_town FROM student,\
course WHERE course = course_id AND student.residence = %s',[residence])
rows = cur.fetchall()
if rows:
return render_template('stu.html', rows = rows, residence = residence)
else:
return render_template('index.html', msg1='no data found')
except Exception as e:
return render_template('index.html', msg1='No data found', error1 = e)
finally:
if conn:
conn.close()
##app.route('/addStudent, methods =['GET','POST']')
#def addStudent():
if __name__ == '__main__':
app.run(debug = True)
This is an environment problem, not a flask, postgres or shebang problem. A specific version of Python is being called, and it is not being given the correct path to its libraries.
Depending on what platform you are on, changing you shebang to #! /usr/bin/env python3 can fix the problem, but if not (very likely not, though using env is considered better/portable practice these days), then you may need to add your Python3 libs location manually in your code.
sys.path.append("/path/to/your/python/libs")
If you know where your Python libs are (or maybe flask is installed somewhere peculiar?) then you can add that to the path and imports following the line where you added to the path will include it in their search for modules.

python, accessing a psycopg2 form a def?

i'm trying to make a group of defs in one file so then i just can import them whenever i want to make a script in python
i have tried this:
def get_dblink( dbstring):
"""
Return a database cnx.
"""
global psycopg2
try
cnx = psycopg2.connect( dbstring)
except Exception, e:
print "Unable to connect to DB. Error [%s]" % ( e,)
exit( )
but i get this error: global name 'psycopg2' is not defined
in my main file script.py
i have:
import psycopg2, psycopg2.extras
from misc_defs import *
hostname = '192.168.10.36'
database = 'test'
username = 'test'
password = 'test'
dbstring = "host='%s' dbname='%s' user='%s' password='%s'" % ( hostname, database, username, password)
cnx = get_dblink( dbstring)
can anyone give me a hand?
You just need to import psycopg2 in your first snippet.
If you need to there's no problem to 'also' import it in the second snippet (Python makes sure the modules are only imported once). Trying to use globals for this is bad practice.
So: at the top of every module, import every module which is used within that particular module.
Also: note that from x import * (with wildcards) is generally frowned upon: it clutters your namespace and makes your code less explicit.

Categories

Resources