I am currently trying to write a little web-application using python, flask and sqlite and I'm not sure about how to handle the database-connections.
Basically I've been following the "official" Tutorial (and http://flask.pocoo.org/docs/patterns/sqlite3/#sqlite3 ), leaving me with code like this in my main Flask/App module (drastically shortened):
#vs_app.before_request
def before_request():
g.db = sqlite3.connect("somedb.db")
def execute_db(command):
return g.db.cursor().execute(command).fetchall()
#app.route("/givemeallusers")
def foo():
return execute_db("SELECT * FROM users")
So this creates a DB-Connection for each request that can be used by my application to execute sql-commands and it works fine for smaller applications.
But for a larger project I'd like to put my database-code (with some utility methods) and app-code in different modules, so I could (for example) write:
from my_database_module import user_auth #
#app.route("/login")
def foo():
if user_auth(request.form["name"], request.form["pw"]):
pass
But for that I would need access to g (or g.db) from my database-class and the only way I could get that to work was by "manually" handing over the db-connection to each method, leaving me with code like this:
#app.route("/login")
def foo():
if user_auth(g.db, request.form["name"], request.form["pw"]):
pass
and in my database-module
def user_auth(database, name, pw):
pass
I don't think thats the best approach, but other ideas I had (like importing the g object into my database-class) don't work. I also don't know whether my approach is safe with regards to concurrent db-access, so any help with this would be appreciated.
tl;dr How to split Flask-App and Database/Database-Utility the right way?
Just in case someone stumbles upon this question:
The correct answer is to not bother with the approaches described in the Question, and instead just start working with Flask-SQLAlchemy.
Related
An Example code (which does not work)
#pytest.fixture(scope="function")
def my_method_user(db):
user = User(name="method_user").save()
return user
#pytest.fixture(scope="class")
def my_class_user(db):
user = User(name="class_user").save()
return user
#pytest.mark.usefixtures("my_class_user")
class TestClass:
def test_user_count(self):
assert len(User.objects.all()) == 1
def test_user_count_2(self, my_method_user):
assert len(User.objects.all()) == 2
However, doing this (or anything similar where I tried to access DB directly through queryset etc) basically always gives me ScopeMistmatch.
Basically, what I'm trying to do is
Create a User that will be "globally" used within the class - fixture with class scope (or maybe rather, some rows that were "setup" in prior to performing a test for class)
In some of the test functions, I would like a user to be created temporarily (but with a certain condition, thus I hope to use fixture). When the function tested is done, the user is automatically removed.
Later on, the datas might be more nested than simple class-function, the entire 5 layers might be all used. So rows that were created by class-scoped fixtures should also be deleted after, but that were created by module/package/session should stay
I've researched about using django_db_blocker.unblock() trying to avoid scopemismatch and keep datas throughout multiple tests, however it gives me a problem where the datas are not being removed after the tests are done (unless I remove them manually, which may be quite inaccurate).
Am i fundamentally wrong about how I'm supposed to use pytest/fixture? or am I missing something? Is there a different way to achieve this?
I have a large Flask project spread across multiple modules. I want all of those modules to get access to my MongoDB (PyMongo) database connection. According to best practices, I want exactly one connection pool to persist throughout the application (i.e. throughout all modules).
Infeasible Solutions (from other related StackOverflow answers)
I cannot create a global variable and pass that around to each module I call because that is not thread-safe (especially when my db object is not read-only). Example follows.
I cannot create a new connection for every request because that is inefficient.
I cannot use the Flask.g "global" variable because it is wiped off with every request, making it essentially equivalent to point 2.
I cannot simply put the database connection code into a new module (like config.py) and call it when required from other modules because then I would be creating new connections every time I call it from the different modules.
Problem
How do I create a single and persistent MongoDB connection in a thread- and process- safe manner?
How do I give access to this across modules? In other words, how to make it global across modules?
Is there no best-practice about this? The literature is either outdated or non-existent. Please answer for MongoDB (specifically PyMongo, if possible).
Example
This is a minimum working example of how I would create a global variable and use it across modules (Point 1 in Infeasible Solutions).
main.py
from module1 import helper_function
app = Flask(__name__)
db = new_db_connection()
#app.route('/page1'):
def page1():
db_return = db.query1()
db.cache = new_value # this makes this entire code thread-unsafe
return db_return
#app.route('/page2'):
def page2():
db_return = db.query2()
db_return2 = helper_function(db) # doesn't seem right that I have to pass this object around all the time
return db_return
module1.py
def helper_function(db):
db_return = db.query3()
return db_return
I've been in much the same boat, and the answer I came up with was to pass a parameter to every function. If there's a better way of doing it I would love to know.
Rather than pass the db value, I actually pass a custom context object which has the mongoclient db as well as bunch of other helpful methods (such as recording result ids etc).
class Context:
def __init__(self, cs: str = None) -> None:
self.cs = cs
try:
self.db = pymongo.MongoClient(self.cs).get_database()
except Exception as e:
raise ValueError(f'Failed to connect to server using connection string {self.cs}')
I am using GAE with python, and I am using many forms. Usually, my code looks something like this:
class Handler(BaseHandler):
#...
def post(self):
name = self.request.get("name")
last_name = self.request.get("last_name")
# More variables...
n = self.request.get("n")
#Do something with the variables, validations, etc.
#Add them to a dictionary
data = dict(name=name, last_name=last_name, n=n)
info = testdb.Test(**data)
info.put()
I have noticed lately that it gets too long when there are many inputs in the form (variables), so I thought maybe I could send a stringified JSON object (which can be treated as a python dictionary using json.loads). Right now it looks like this:
class Handler(BaseHandler):
#...
def post(self):
data = validate_dict(json.loads(self.request.body))
#Use a variable like this: data['last_name']
test = testdb.Test(**data)
test.put()
Which is a lot shorter. I am inclined to do things this way (and stop using self.request.get("something")), but I am worried I may be missing some disadvantage of doing this apart from the client needing javascript for it to even work. Is it OK to do this or is there something I should consider before rearranging my code?
There is absolutely nothing wrong with your short JSON-focused code variant (few web apps today bother supporting clients w/o Javascript anyway:-).
You'll just need to adapt the client-side code preparing that POST, from being just a traditional HTML form, to a JS-richer approach, of course. But, I'm pretty sure you're aware of that -- just spelling it out!-)
BTW, there is nothing here that's App Engine - specific: the same considerations would apply no matter how you chose to deploy your server.
I'm working on a client library for a popular API. Currently, all of my unit tests of said client are making actual API calls against a test account.
Here's an example:
def test_get_foo_settings(self):
client = MyCustomClient(token, account)
results = client.get_foo_settings()
assert_is(type(results), list)
I'd like to stop making actual API calls against my test account.
How should I tackle this? Should I be using Mock to mock the calls to the client and response?
Also, I'm confused on the philosophy of what to test with this client library. I'm not interested in testing the actual API, but when there are different factors involved like the method being invoked, the permutations of possible return results, etc - I'm not sure what I should test and/or when it is safe to make assumptions (such as a mocked response).
Any direction and/or samples of how to use Mock in my type of scenario would be appreciated.
I would personally do it by first creating a single interface or function call which your library uses to actually contact the service, then write a custom mock for that during tests.
For example, if the service uses HTTP and you're using Requests to contact the service:
class MyClient(…):
def do_stuff(self):
result = requests.get(self.service_url + "/stuff")
return result.json()
I would first write a small wrapper around requests:
class MyClient(…):
def _do_get(self, suffix):
return requests.get(self.service_url + "/" + suffix).json()
def do_stuff(self):
return self._do_get("stuff")
Then, for tests, I would mock out the relevant functions:
class MyClientWithMocks(MyClient):
def _do_get(self, suffix):
self.request_log.append(suffix)
return self.send_result
And use it in tests like this:
def test_stuff(self):
client = MyClientWithMocks(send_result="bar")
assert_equal(client.do_stuff(), "bar")
assert_contains(client.request_log, "stuff")
Additionally, it would likely be advantageous to write your tests so that you can run them both against your mock and against the real service, so that if things start failing, you can quickly figure out who's fault it is.
I'm using HTTmock and I'm pretty happy with it : https://github.com/patrys/httmock
In general I want to disable as little code as possible, and I want it to be explicit: I don't want the code being tested to decide whether it's a test or not, I want the test to tell that code "hey, BTW, I'm running a unit test, can you please not make your call to solr, instead can you please stick what you would send to solr in this spot so I can check it". I have my ideas but I don't like any of them, I am hoping that there's a good pythonic way to do this.
You can use Mock objects to intercept the method calls that you do not want to execute.
E.g. You have some class A, where you don't want method no() to be called during a test.
class A:
def do(self):
print('do')
def no(self):
print('no')
A mock object could inherit from A and override no() to do nothing.
class MockA(A):
def no(self):
pass
You would then create MockA objects instead of As in your test code. Another way to do mocking would be to have A and MockA implement a common interface say InterfaceA.
There are tons of mocking frameworks available. See StackOverflow: Python mocking frameworks.
In particular see: Google's Python mocking framework.
Use Michael Foord's Mock
in your unit test do this:
from mock import Mock
class Person(object):
def __init__(self, name):
super(Person, self).__init__()
self.name = name
def say(self, str):
print "%s says \"%s\"" % (self.name, str)
...
#In your unit test....
#create the class as normal
person = Person("Bob")
#now mock all of person's methods/attributes
person = Mock(spec=person)
#talkto is some function you are testing
talkTo(person)
#make sure the Person class's say method was called
self.assertTrue(person.say.called, "Person wasn't asked to talk")
#make sure the person said "Hello"
args = ("Hello")
keywargs = {}
self.assertEquals(person.say.call_args, (args, keywargs), "Person did not say hello")
The big problem that I was having was with the mechanics of the dependency injection. I have now figured that part out.
I need to import the module in the exact same way in both places to successfully inject the new code. For example, if I have the following code that I want to disable:
from foo_service.foo import solr
solr.add(spam)
I can't seem to do this in the in my test runner:
from foo import solr
solr = mock_object
The python interpreter must be treating the modules foo_service.foo and foo as different entries. I changed from foo import solr to the more explicit from foo_service.foo import solr and my mock object was successfully injected.
Typically when something like this arises you use Monkey Patching (also called Duck Punching) to achieve the desired results. Check out this link to learn more about Monkey Patching.
In this case, for example, you would overwrite solr to just print the output you are looking for.
You have two ways to do this is no ,or minimal in the case of DI, modifications to your source code
Dependency injection
Monkey patching
The cleanest way is using dependency injection, but I don't really like extensive monkeypatching, and there are some things that are non-possible/difficult to do that dependency injection makes easy.
I know it's the typical use case for mock objects, but that's also an old argument... are Mock objects necessary at all or are they evil ?
I'm on the side of those who believe mocks are evil and would try to avoid changing tested code at all. I even believe such need to modify tested code is a code smell...
If you wish to change or intercept an internal function call for testing purpose you could also make this function an explicit external dependency set at instanciation time that would be provided both by your production code and test code. If you do that the problem disappear and you end up with a cleaner interface.
Note that doing that there is not need to change the tested code at all neither internally nor by the test being performed.