Develop Reference

Python: Clean Code and Performance on For Loops

I have problems to apply clean code principles to this code section.
In clean code principles is written, that a function should do one thing and do it well.
But in this case I do not know how to refractor this function according to those principles without decreasing performance.
for netobject in netObjectList:
for key, value in netobject.getObjectParams().items():
if value.getValue() == '' and key is not 'comment':
Errorhandler().error_on_pos(value.getRow(), value.getCol(),
str(key) + ' is missing')
else:
#do syntax check on values in the dictionary
NValidator.checkSyntaxOfValue(key, value)
A netobject is dictionary containing all its parameters.
So I know that this function does more than one thing. And I think it is quite hard to unittest this function.
But if I proof syntax and missing parameters separatley, I have to iterate twice over all network objects bundled in that neObjectList. But otherwise it is no clean code to check syntax and missing parameters in one function.
I very often have these inner conflicts, when I am writing code.
Do you have some tips or suggestions to fix these problems?
If you need more description for that code section, please let me know.
Errorhandler is just a way to print out my errors, if an missing parameter is found.
The function checkSyntaxOfValue(key, value) calls suiteable check methods depending on the key of the bundled parameters. I think there would be also a better solution to handle syntax checks. But I do not know.

As far as I'm concerned, your code IS "doing one thing": validating your netobjects values. The fact that the validation is done in two steps (first checking if there's a value where expected, then checking the value's 'syntax') is an implementation detail.
Now what can be questionned is whether checking for empty values is distinct from checking the value's syntax - ie if an empty value is not a valid syntax for a given key, shouldn't the check be a responsability of NValidator.checkSyntaxOfValue ?
Also if for whatever reason you really do have to keep the tests (empty value / value syntax) distinct, you should probably encapsulate the emptyness test in a distinct function so you have the same level of abstraction for both tests, ie:
def checkForNotEmptyValue(key, value):
if key == 'comment':
# we don't care about empty values here
return True
if value.getValue() == '':
Errorhandler().error_on_pos(
value.getRow(),
value.getCol(),
'{} is missing'.format(key)
)
return False
return True
and then
for netobject in netObjectList:
for key, value in netobject.getObjectParams().items():
if checkForNotEmptyValue(key, value):
NValidator.checkSyntaxOfValue(key, value)
NB: this is just an example based on your code snippet and question - I don't have enough context to tell whether it's the "best" solution here.
As a last word: don't get over the top with "best practices" either. Writing clean code IS important, but "clean" is a relative notion, and trying to strictly follow all "golden rules" is downright impossible (specially since no one really agrees on those rules xD).
Actually, the point of "golden rules" etc is to make you aware of what problem you might have if you don't follow the rule - once you know, it's your responsability to decide when the rule applies and when it's doesn't. It's like raising children: at first you just forbid them to cross the road alone because they are too young to really understand the dangers. Once they are mature enough to get the point, the rule becomes useless.

Let me comment on your code first:
for netobject in netObjectList:
for key, value in netobject.getObjectParams().items():
"""
Below you seem to be assuming that first item must have 'comment' as key?
What if the key 'comment' actually exists? If you iterate like this you will be
Throwing errors until you iterate through all the items (and either hit the required key or not)
"""
if value.getValue() == '' and key is not 'comment':
Errorhandler().error_on_pos(value.getRow(), value.getCol(), str(key) + ' is missing')
else:
NValidator.checkSyntaxOfValue(key, value)
So, I'd choose a different approach. If you definitely want to work with item having key 'comment'
(and assuming here that your keys are unique - ie no multiple 'comment' keys in a single netobject object) you could do this instead:
for netobject in netObjectList:
try:
# try to get the value directly
# (of course you could use .keys() instead of .items() to evaluate existing keys first)
value = netobject.getObjectParams()['comment']
NValidator.checkSyntaxOfValue('comment', value)
except:
# Due to KeyError we now know for sure that such key definitely doesn't exist
# So we can handle the Exception, or throw your own such as:
Errorhandler().error_on_pos(...)
This example will either throw you just one True (ie not false positive) error, or it will succeed with syntax check
EDIT:
Decided to illustrate with better example:
Here I'm first emulating your netobject
class netobject_class(object):
def __init__(self, datadict):
self.data = datadict
def getObjectParams(self):
return self.data
def __str__(self):
return "I'm a netobject_class instance (hash=" + str(self.__hash__()) + ") containing a data Dict"
Now we create the netObjectList which contains 3 sample netobject_class instances. Note that second Dict has no 'comment' key.
netObjectList = [
netobject_class({'comment': 'comment1'}),
netobject_class({'somekey':'somevalue','somekey2':'somevalue2'}),
netobject_class({'comment': 'comment2'})
]
Now we run the code:
print("-" * 40)
for netobject in netObjectList:
# print("Working with netobject: %s" % netobject)
try:
# try to get the value directly
# (of course you could use .keys() instead of .items() to evaluate existing keys first)
value = netobject.getObjectParams()['comment']
print("****** value='%s' # aka comment" % value)
#NValidator.checkSyntaxOfValue('comment', value)
except Exception as e:
# Due to KeyError we now know for sure that such key definitely doesn't exist
# So we can handle the Exception, or throw your own such as:
print('Error: key', str(e), 'does not exist! Skipping....')
print("-" * 40)
And this produces:
----------------------------------------
****** value='comment1' # aka comment
----------------------------------------
Error: key 'comment' does not exist! Skipping....
----------------------------------------
****** value='comment2' # aka comment
----------------------------------------

using get function if key must exist in dictionary?

One question about using dictionary get function in python. I understand get function can provide default value for dict if key does not exist. What if during the program, we know key must be existed. like following code. should we still use get function or we can just use dict[key] to get value. does it mean get function can replace dict[key]
value = 'default'
dict_get = dict(key='value')
def test_get(dict_get):
return dict_get.get('key', 'default_value')
test_get.get('key')
test_get['key']

If the key must exist, you should use yourdict[key], i.e. the __getitem__ method.
If for some reason the key does not exist, you want your program to crash with a KeyError, because clearly there's something wrong with your program that needs to be fixed.
If the key should exist, but may not due to reasons other than faulty program logic, you can take a more defensive approach. For example, if a user is queried to input a valid key but fails to do so, you could fall back to a default value using dict.get or ask the user again.

dict.get will return None if the key doesn't exist. In some applications you just want to raise an exception there instead, which is what dict[key] does. For example:
scores = {"Larry": 5}
bob_score = scores.get("Bob") # None
if bob_score:
print("You're doing great Bob!")
else:
print("Too bad Bob, you're out.")
Our code assumes that the key Bob is in the dictionary, and maps to an integer. Even though Bob isn't in the dictionary because None is a falsy value we still get a meaningful answer from this code, even though that conclusion is based on a mistake. It would be much better for an exception to be raised here.

Do you have to check if an array element exists (not null string) in Python3?

If an associative array exists in Python3, is it a waste to check if an element of it exists rather than just using it?
Should you:
if 'name' in array and not array['name'].startswith('something'):
Or should you just:
if not array['name'].startswith('something'):
... And will Python3 handle it "for" you?

You can do -
if not array.get('name', 'something').startswith('something'):
get() function returns the second value by default if the key ( name ) is not found.
So in above case , this would return something , if key is not found in the dictionary, and because of the .startwith() and the not , the complete conditional expression would evaluate to False , as it would be doing in OP's first example .

The answer really depends on where does your array come from. Also on what can you do as a result of the check.
For example if it comes from the user, then yes, you definitely should check if name exists, so that you can respond with a better error message than some big stacktrace.... KeyError: "name".
If it's an internal structure you created and it's not exposed for user modifications, or if it's exposed via API but the expectation is that nobody should touch it, then don't bother checking. If it's missing, then it's an internal exception the developers should see and fix.
Of course there may be a situation where you don't really care if it was provided or not, because you have a reasonable fallback. Then a default value like in Anand's answer is a good solution too. array.get('name', some_default)

Python won't handle it for you. You'll have to do some work in both cases. Try to write code that makes sense to you later, because you'll write it once and read it (and maintain it) many times over.
You can do it two ways, pick the one that makes the most sense to you:
obj = array.get('name')
if obj and not obj.startswith('something'):
pass
In your second option, you'll have to catch the exception:
try:
if not array['name'].startswith('something'):
pass
except KeyError:
print('name does not exist in array')

Just do ...
if not array.get('name',"").startswith('something')
If name exists then it returns array['name'] else it will return empty string .

Python idiomatic unpacking assignment or False

If function returns a two value list or tuple on success or False on failure, how can I best unpack the return list into two variables while also checking for False?
def get_key_value():
if (cond != True):
return False
return [val1, val2]
# Call it
# How can I also check for False while unpacking?
key, value = get_key_value()

Coverting #Felix Kling's great comment into an answer.
If not being able to find a (key, value) pair indicates some kind of system failure, it would be better to throw an exception. If your failure doesn't really fall into any of the standard exceptions, you should build a new exception type of your own.
The cond != True is better written as not cond. Also it's better to not create a list if it's not necessary.
class DataNotFound(Exception): pass
def get_key_value():
if not cond:
raise DataNotFound("Couldn't find it!")
return val1, val2
try:
key,value = get_key_value()
except DataNotFound:
#handle the failure somehow
key, value = 'ERROR', 'ERROR'

This falls under the "Easier to Ask for Forgiveness than Permission" policy of Python. I avoid catching TypeError in your function, in case there's some other unforeseen problem.
data = get_key_value()
try:
key, value = data
except TypeError:
#handle the failure somehow
key, value = 'ERROR', 'ERROR'

I don't think there is an idiomatic way to do this -- not least because a function that behaves that way is itself unidiomatic. If you have to do it, I suggest you simply make use of the fact that your 2-element list or tuple is a "truthy" rather than a "falsy" value (this isn't Python terminology but it's useful):
pair_or_false = get_key_value()
if pair:
key,value = val
else:
# handle failure in whatever way
The obvious alternative is to treat the not-found case as an exception:
try:
key,value = get_key_value()
except TypeError:
# deal with not-found case
but if there's any possibility at all that something other than the unsuccessful unpacking could raise a TypeError then you run the risk of masking a genuine error that way.

You're running into problems because you're mixing return types. Just because you can doesn't mean you should.
Although I agree with the others here that an exception is one appropriate way to go, it may depend on whether you expect to find a valid key & value most of the time. If so, use an exception (something like KeyError) to indicate that the function failed. But if you expect it to fail at a high rate, you may not want the exception overhead. In that case, return something like [None, None] from get_key_value and then your calling code would look like:
key, value = get_key_value()
if key:
# take action
else:
# handle the error appropriately

Is this common practice to avoid key not found in a dictionary

I was wondering, whether the following style is a common practice to avoid key not found in a dictionary?
# default is 0
value = my_dic[100] if 100 in my_dic else 0

value = my_dic.get(100, 0)

If you need the "default value" everywhere, consider defaultdict as a possible alternative. (A factory/callback approach offers a good bit of flexibility with the "default value".)
Happy coding.

If you try to access a key that is not in the dictionary, python will raise an exception that will crash your program. Instead of this, you should "catch" the exception and do something more graceful.
it's better practice to catch the exception:
try:
value = my_dic[100]
except KeyError:
print("key not found in dictionary") #or handle the error more elegantly
I haven't read the article, but you can learn more here:
http://en.wikipedia.org/wiki/Exception_handling

It is valid python syntax, however I would say that according to Python Coding Style Guidelines You should structure your if else statements like so,
if <condition>:
<statements>
else:
<statements>

first_name = params.get('first_name', None)
if first_name:
profile.first_name = first_name