At a certain point in my project, I need to query a SQLAlchemy object for columns that are NOT NULL. In my code, I do:
session.query(MyModel).filter(MyModel.my_column != None).all()
...and it works great. Problem is, whenever I run pep8 or one of the other linters on the file, it raises an error E711: Comparison to None should be if cond is not None:. I agree with the spirit of this guideline, and as such I don't want to silence the warning just because of one little line.
Is there another way to write this? Preferably one where we don't have to dig into the func module?
PEP8 isn't meant to be followed to the letter.
You're recommended to use is None instead of == None because is cannot be overloaded (unlike ==):
>>> class Bad(object):
... def __eq__(self, other):
... return True
...
>>> instance = Bad()
>>> instance == None
True
>>> instance is None
False
In your case you want to use the overloaded == operator, so the warning is incorrect.
There are three ways of making the warning go away:
You can disable that particular warning for that single line:
For Flake8/PyFlakes:
session.query(MyModel).filter(MyModel.my_column != None).all() # noqa: E711
For Pylint:
session.query(MyModel).filter(MyModel.my_column != None).all() # pylint: disable=singleton-comparison
You can avoid using the equality operator by directly calling the class's __eq__ (for ==) and __ne__ (for !=) methods:
session.query(MyModel).filter(MyModel.my_column.__ne__(None)).all()
You can use .is_ and .isnot:
session.query(MyModel).filter(MyModel.my_column.isnot(None)).all()
Note that .is_ and .isnot are not the same as == and !=. MyModel.my_column != x can mean both IS NOT NULL and != x depending on the value of x and your database, so solutions 1 and 2 are probably better unless you really don't like using __-methods and know the type of the right-hand-side of your expression.
Related
Let's say I have a python module with the following function:
def is_plontaria(plon: str) -> bool:
if plon is None:
raise RuntimeError("None found")
return plon.find("plontaria") != -1
For that function, I have the unit test that follows:
def test_is_plontaria_null(self):
with self.assertRaises(RuntimeError) as cmgr:
is_plontaria(None)
self.assertEqual(str(cmgr.exception), "None found")
Given the type hints in the function, the input parameter should always be a defined string. But type hints are... hints. Nothing prevents the user from passing whatever it wants, and None in particular is a quite common option when previous operations fail to return the expected results and those results are not checked.
So I decided to test for None in the unit tests and to check the input is not None in the function.
The issue is: the type checker (pylance) warns me that I should not use None in that call:
Argument of type "None" cannot be assigned to parameter "plon" of type "str" in function "is_plontaria"
Type "None" cannot be assigned to type "str"
Well, I already know that, and that is the purpose of that test.
Which is the best way to get rid of that error? Telling pylance to ignore this kind of error in every test/file? Or assuming that the argument passed will be always of the proper type and remove that test and the None check in the function?
This is a good question. I think that silencing that type error in your test is not the right way to go.
Don't patronize the user
While I would not go so far as to say that this is universally the right way to do it, in this case I would definitely recommend getting rid of your None check from is_plontaria.
Think about what you accomplish with this check. Say a user calls is_plontaria(None) even though you annotated it with str. Without the check he causes an AttributeError: 'NoneType' object has no attribute 'find' with a traceback to the line return plon.find("plontaria") != -1. The user thinks to himself "oops, that function expects a str". With your check he causes a RuntimeError ideally telling him that plon is supposed to be a str.
What purpose did the check serve? I would argue none whatsoever. Either way, an error is raised because your function was misused.
What if the user passes a float accidentally? Or a bool? Or literally anything other than a str? Do you want to hold the user's hand for every parameter of every function you write?
And I don't buy the "None is a special case"-argument. Sure, it is a common type to be "lying around" in code, but that is still on the user, as you pointed out yourself.
If you are using properly type annotated code (as you should) and the user is too, such a situation should never happen. Say the user has another function foo that he wants to use like this:
def foo() -> str | None:
...
s = foo()
b = is_plontaria(s)
That last line should cause any static type checker worth its salt to raise an error, saying that is_plontaria only accepts str, but a union of str and None was provided. Even most IDEs mark that line as problematic.
The user should see that before he even runs his code. Then he is forced to rethink and either change foo or introduce his own type check before calling your function:
s = foo()
if isinstance(s, str):
b = is_plontaria(s)
else:
# do something else
Qualifier
To be fair, there are situations where error messages are very obscure and don't properly tell the caller what went wrong. In those cases it may be useful to introduce your own. But aside from those, I would always argue in the spirit of Python that the user should be considered mature enough to do his own homework. And if he doesn't, that is on him, not you. (So long as you did your homework.)
There may be other situations, where raising your own type-errors makes sense, but I would consider those to be the exception.
If you must, use Mock
As a little bonus, in case you absolutely do want to keep that check in place and need to cover that if-branch in your test, you can simply pass a Mock as an argument, provided your if-statement is adjusted to check for anything other than str:
from unittest import TestCase
from unittest.mock import Mock
def is_plontaria(plon: str) -> bool:
if not isinstance(plon, str):
raise RuntimeError("None found")
return plon.find("plontaria") != -1
class Test(TestCase):
def test_is_plontaria(self) -> None:
not_a_string = Mock()
with self.assertRaises(RuntimeError):
is_plontaria(not_a_string)
...
Most type checkers consider Mock to be a special case and don't complain about its type, assuming you are running tests. mypy for example is perfectly happy with such code.
This comes in handy in other situations as well. For example, when the function being tested expects an instance of some custom class of yours as its argument. You obviously want to isolate the function from that class, so you can just pass a mock to it that way. The type checker won't mind.
Hope this helps.
You can disable type checking for on a specific line with a comment.
def test_is_plontaria_null(self):
with self.assertRaises(RuntimeError) as cmgr:
is_plontaria(None) # type: ignore
self.assertEqual(str(cmgr.exception), "None found")
How can I validate that a function includes a return keyword? I frequently forget the return line, so I am worried that the users of my package will too when they provide a function-based input.
def appler():
a = "apple"
# `return` is missing
def bananer():
b = "banana"
return b
I could parse the actual code string of the function for a final line that includes "return" but that isn't very robust (it could be triggered by comments).
def validate_funk(funk):
if condition_to_check_that_it_contains_rtrn:
pass
else:
raise ValueError(f"Yikes - The function you provided not contain a `return` statement:\n\n{funk}")
>>> validate_funk(appler)
#triggers ValueError
>>> validate_funk(bananer)
# passes
EDIT: ideally without running the function.
What you actually care about is probably not the return statement itself, but that the function returns something of a certain type. This you can most easily accomplish by using type hints (PEP 484):
def appler() -> str:
a = "apple"
# `return` is missing
def bananer() -> str:
b = "banana"
return b
Now, running a static analysis tool like mypy or Pyre (or many others): will emit a warning about a wrong return type in function appler (expected str, got NoneType).
Look for sabik's answer for a more general answer. Writing (unit) tests is another good practice that catches many more issues and - if done well - is an invest in code maintainability.
A function without return statement returns None by default.
>>> def abc():
pass
>>> print(abc())
None
>>>
You can add a check using this:
def validate_func(function):
if function() == None:
raise ValueError("Yikes - Does not contain a `return` statement")
There are few cons though.
You have to execute the function
It wont work if you are returning None in a function
Not much practical but yea, that is one way. You can also get a list of local functions or a list of methods in a class and loop through them without having to check each function individually.
For the question as asked, the ast module will let you check that.
However, it doesn't seem very useful just by itself - as others have pointed out, a function without a return is valid (it returns None), and just because a function does have a return doesn't mean that it returns the correct value, or even any value (the return could be in an if statement).
There are a couple of standard ways of dealing with this:
Unit tests - separate code that calls your function with various combinations of inputs (possibly just one, possibly hundreds) and checks that the answers match the ones you calculated manually, or otherwise satisfy requirements.
A more general implementation of the idea of checking for a return statement is "lint", in the case of Python pylint; that looks through your code and checks for various patterns that look like they could be mistakes. A side benefit is that it already exists and it checks dozens of common patterns.
Another, different more general implementation is the mypy type checker; that not only checks that there's a return statement, but also that it returns the correct type, as annotated in the header of the function.
Typically, these would be used together with a "gated trunk" development process; manual changes to the main version are forbidden, and only changes which pass the tests, lint and/or mypy are accepted into the main version.
As others have mentioned, simply calling the function is not enough: a return statement might only be present in a conditional, and thus, specific input would need to be passed in order to execute the return statement. That, too, is not a good indicator of the presence of a return, since it could return None, causing greater ambiguity. Instead, the inspect and ast module can be used:
Test functions:
def appler():
a = "apple"
# `return` is missing
def bananer():
b = "banana"
return b
def deeper_test(val, val1):
if val and val1:
if val+val1 == 10:
return
def gen_func(v):
for i in v:
if isinstance(i, list):
yield from gen_func(i)
else:
yield i
inspect.getsource returns the entire source of the function as a string, which can then be passed to ast.parse. From there, the syntax tree can be recursively traversed, searching for the presence of a return statement:
import inspect, ast
fs = [appler, bananer, deeper_test, gen_func]
def has_return(f_obj):
return isinstance(f_obj, ast.Return) or \
any(has_return(i) for i in getattr(f_obj, 'body', []))
result = {i.__name__:has_return(ast.parse(inspect.getsource(i))) for i in fs}
Output:
{'appler': False, 'bananer': True, 'deeper_test': True, 'gen_func': False}
With a defined validate_funk:
def validate_funk(f):
if not has_return(ast.parse(inspect.getsource(f))):
raise ValueError(f"function '{f.__name__}' does not contain a `return` statement")
return True
Notes:
This solution does not require the test functions to be called.
The solution must be run in a file. If it is run in the shell, an OSError will be raised. For the file, see this Github Gist.
You can simplify return checking with a decorator:
def ensure_return(func):
def wrapper(*args, **kwargs):
res = func(*args, **kwargs)
if res is None:
raise ValueError(f'{func} did not return a value')
return res
return wrapper
#ensure_return
def appler():
a = "apple"
# `return` is missing
#ensure_return
def bananer():
b = "banana"
return b
then:
>>> appler()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 5, in wrapper
ValueError: <function appler at 0x7f99d1a01160> did not return a value
>>> bananer()
'banana'
I need to use UserString to create my own str class, but its implementation seems problematic.
For example, in the class definition, it reads:
def __eq__(self, string):
if isinstance(string, UserString):
return self.data == string.data
return self.data == string
But since an empty list ([]) is actually an instance of UserString:
isinstance([], UserString) == True
Now this code doesn't work:
s = UserString("")
if s in [None, [], {}, ()]:
# do whatever
because in operator will use UserString's __eq__ to check membership but [] does not have .data attribute. This issue doesn't exist in the built-in str class.
I know this is a trivial, non-realistic example, but anyone encountered this problem before using UserString and what is the best way to circumvent this (maybe method override in my own subclass)? Any other caveats?
Note: I am aware of this SO thread, but I don't think my question is a duplicate of it.
It seems like no one can reproduce isinstance([], UserString) == True. But this is a screenshot from my PyCharm IDE:
Is it possible in python to call dot-syntax-function based on condition. Simple example to turn:
if condition:
foo().bar().baz()
lots_of_code()
else:
foo().baz()
lots_of_code()
def lots_of_code():
# lots of code
into:
foo().(if condition: bar()).baz()
# lots of code only once
No, it is not possible.
The syntax for attribute reference is
attributeref ::= primary "." identifier
Quoting the documentation,
An attribute reference is a primary followed by a period and a name
name must be a regular Python identifier and identifiers can't contain special characters like (.
However, you can use a simple conditional expression to select primary:
(foo().bar() if condition else foo()).baz()
It's equivalent to
if condition:
primary = foo().bar()
else:
primary = foo()
primary.baz()
Note that in this case we have to use parentheses, because attribute reference has higher precedence than conditional expression.
Since foo() is called in either case, start by doing so unconditionally. Save that object to f, with the intention of calling f.baz(). Before that, though, check your condition to see if f should really be the result of foo().bar().
f = foo()
if condition:
f = f.bar()
f.baz()
How do I do this? Can I do this?
def aFunction(argument):
def testSomething():
if thisValue == 'whatItShouldBe':
return True
else:
return False
if argument == 'theRightValue': # this is actually a switch using elif's in my code
testSomething()
else:
return False
def aModuleEntryPoint():
if aFunction(theRightValue) == True:
doMoreStuff()
else:
complain()
aModuleEntryPoint()
aModuleEntryPoint() needs to first make sure that a condition is true before it starts doing things. Because of encapsulation, aModuleEntryPoint doesn't know how to check the condition, but aFunction() has a sub-function called testSomething() that does know how to check the condition. aModuleEntryPoint() calls aFunction(theRightValue).
Because theRightValue was passed to aFunction() as an argument, aFunction() calls testSomething(). testSomething() performs the logic test, and either returns True or False.
I need for aModuleEntryPoint() to know what testSomething() decided. I do not want aModuleEntryPoint() to know anything about how testSomething() came to its conclusion.
It would actually be an accomplishment to post my actual source while removing other functions and what-not, so I had to setup the general gist like this.
The only thing I see wrong right now is you need a return before testSomething() on line 9.
Perhaps a sub-function is not the right encapsulation tool for you here. You want to expose internal functionality to an external entity. Python classes provide a better mechanism for expressing this than sub-function. Having a class, you can expose whatever parts of internal functionality you want in a very controlled manner.
My first thought upon looking at your code is that it's a little too complicated. Why have aFunction at all? You could just write
def aModuleEntryPoint():
argument = ...
if argument in (theRightValue, theOtherRightValue, theOtherOtherRightValue)\
and testSomething():
doMoreStuff()
else:
complain()
This if clause will first check whether argument is one of the possible right values, and if it is, then it will proceed to call testSomething() and check the return value of that. Only if that return value is true will it call doMoreStuff(). If either of the tests fails (that's why I used and), it will complain().