If I'm dealing with dictionaries everywhere, should I make them "classes"? - python

Right now, I have many python files. Each has many functions.
I'm passing hash maps around everywhere. Basic dictionaries, with a word as a key and a score as the value.
{ 'dog': 33, 'cat': 294 }
I'm also doing some more complicated strcutures, like:
{ 'dog': [ 33, 66, 11, 88 ], 'cat': [11, 66, 22] }
Do I need to turn these into my own "objects"? if so, what would they be?
I don't do OOP very much so I'm asking these noob questions.

Having maintained a large codebase for many years that favored raw dicts over objects, my opinion is that if this codebase is going to be maintained by either A) another person or B) several people, you should start migrating toward real classes and OOP. Primarily, the fact that you can easily see what attributes a class instance is supposed to have by looking at the source or even interactively using dir() and other introspection techniques makes your code a lot easier to learn, modify, and maintain. Comprehension and debugging are just that much easier with classes that have intuitive names, well-defined properties, documentation, and readable source code as opposed to a giant mess of functions that take dicts as arguments, and transform them at runtime in myriad ways that can only be understood by reading the entire codebase. Tracebacks are also generally easier to read and decipher when classe are involved since you don't get generic low-level error messages.
As further evidence, go look at mature and successful python projects. You'll see lots of OOP, classes, and documentation. Raw dicts, lists, sets, and tuples are great when appropriate (your data is truly simple) and for small projects and scripts, but the maintainability doesn't last past a certain project size. The notion of "X is just a dict" only survives while your concept of X is extremely simple.

Depending on what exactly you may be doing a namedTuple may be the best solution for your problem. Not enough information to say for sure.

My rule of thumb is that if you have functions which act on the dictionary, then it should probably be a class, and those functions should be methods.
As a (mostly ex)-perl programmer the passing the above style is very familiar. However the Python OO tools are much lighter weight and worth deploying earlier rather than later.
For example
{ 'dog': 33, 'cat': 294 }
Becomes something like
class AnimalCount(object):
"""Some useful documentation you didn't have a place for before"""
def __init__(self, dog=0, cat=0):
self.dog = dog
self.cat = cat
def add_cat(self):
self.cat += 1
def add_dog(self):
self.dog += 1
def dog_eats_cat(self):
self.cat -= 1

OOP (or any other design philosophy) is a strategy to simplify your code, not a religion you follow because it has lots of hits on google.
The goal of OOP is to factor out repeated patterns in your code: If you're often writing the same code to handle these dicts, you should factor these patterns out and write functions. Sometimes you'll have many of these functions, acting on the same data structure, then you group them together on a class.
So, should you use a class? Only if you think it can simplify your code.

dicts are fine for many uses. when you find you need to add some attributes, simply subclass a dict.

Using a dictionary is object oriented, since the dictionary is a class and your instance is an object. The reason to extend it with your own class shouldn't just be because you want to program OO, it should be because the dictionary class needs to have additional methods or attributes added to it.
We can't answer that question without more detail.
You can also nest dictionaries, so the key is a string and the value is another dictionary. That would at least allow you to pass the wrapper dictionary around instead of a whole bunch of separate dictionaries.

Related

How to determine main program vs class [duplicate]

I have been programming in python for about two years; mostly data stuff (pandas, mpl, numpy), but also automation scripts and small web apps. I'm trying to become a better programmer and increase my python knowledge and one of the things that bothers me is that I have never used a class (outside of copying random flask code for small web apps). I generally understand what they are, but I can't seem to wrap my head around why I would need them over a simple function.
To add specificity to my question: I write tons of automated reports which always involve pulling data from multiple data sources (mongo, sql, postgres, apis), performing a lot or a little data munging and formatting, writing the data to csv/excel/html, send it out in an email. The scripts range from ~250 lines to ~600 lines. Would there be any reason for me to use classes to do this and why?
Classes are the pillar of Object Oriented Programming. OOP is highly concerned with code organization, reusability, and encapsulation.
First, a disclaimer: OOP is partially in contrast to Functional Programming, which is a different paradigm used a lot in Python. Not everyone who programs in Python (or surely most languages) uses OOP. You can do a lot in Java 8 that isn't very Object Oriented. If you don't want to use OOP, then don't. If you're just writing one-off scripts to process data that you'll never use again, then keep writing the way you are.
However, there are a lot of reasons to use OOP.
Some reasons:
Organization:
OOP defines well known and standard ways of describing and defining both data and procedure in code. Both data and procedure can be stored at varying levels of definition (in different classes), and there are standard ways about talking about these definitions. That is, if you use OOP in a standard way, it will help your later self and others understand, edit, and use your code. Also, instead of using a complex, arbitrary data storage mechanism (dicts of dicts or lists or dicts or lists of dicts of sets, or whatever), you can name pieces of data structures and conveniently refer to them.
State: OOP helps you define and keep track of state. For instance, in a classic example, if you're creating a program that processes students (for instance, a grade program), you can keep all the info you need about them in one spot (name, age, gender, grade level, courses, grades, teachers, peers, diet, special needs, etc.), and this data is persisted as long as the object is alive, and is easily accessible. In contrast, in pure functional programming, state is never mutated in place.
Encapsulation:
With encapsulation, procedure and data are stored together. Methods (an OOP term for functions) are defined right alongside the data that they operate on and produce. In a language like Java that allows for access control, or in Python, depending upon how you describe your public API, this means that methods and data can be hidden from the user. What this means is that if you need or want to change code, you can do whatever you want to the implementation of the code, but keep the public APIs the same.
Inheritance:
Inheritance allows you to define data and procedure in one place (in one class), and then override or extend that functionality later. For instance, in Python, I often see people creating subclasses of the dict class in order to add additional functionality. A common change is overriding the method that throws an exception when a key is requested from a dictionary that doesn't exist to give a default value based on an unknown key. This allows you to extend your own code now or later, allow others to extend your code, and allows you to extend other people's code.
Reusability: All of these reasons and others allow for greater reusability of code. Object oriented code allows you to write solid (tested) code once, and then reuse over and over. If you need to tweak something for your specific use case, you can inherit from an existing class and overwrite the existing behavior. If you need to change something, you can change it all while maintaining the existing public method signatures, and no one is the wiser (hopefully).
Again, there are several reasons not to use OOP, and you don't need to. But luckily with a language like Python, you can use just a little bit or a lot, it's up to you.
An example of the student use case (no guarantee on code quality, just an example):
Object Oriented
class Student(object):
def __init__(self, name, age, gender, level, grades=None):
self.name = name
self.age = age
self.gender = gender
self.level = level
self.grades = grades or {}
def setGrade(self, course, grade):
self.grades[course] = grade
def getGrade(self, course):
return self.grades[course]
def getGPA(self):
return sum(self.grades.values())/len(self.grades)
# Define some students
john = Student("John", 12, "male", 6, {"math":3.3})
jane = Student("Jane", 12, "female", 6, {"math":3.5})
# Now we can get to the grades easily
print(john.getGPA())
print(jane.getGPA())
Standard Dict
def calculateGPA(gradeDict):
return sum(gradeDict.values())/len(gradeDict)
students = {}
# We can set the keys to variables so we might minimize typos
name, age, gender, level, grades = "name", "age", "gender", "level", "grades"
john, jane = "john", "jane"
math = "math"
students[john] = {}
students[john][age] = 12
students[john][gender] = "male"
students[john][level] = 6
students[john][grades] = {math:3.3}
students[jane] = {}
students[jane][age] = 12
students[jane][gender] = "female"
students[jane][level] = 6
students[jane][grades] = {math:3.5}
# At this point, we need to remember who the students are and where the grades are stored. Not a huge deal, but avoided by OOP.
print(calculateGPA(students[john][grades]))
print(calculateGPA(students[jane][grades]))
Whenever you need to maintain a state of your functions and it cannot be accomplished with generators (functions which yield rather than return). Generators maintain their own state.
If you want to override any of the standard operators, you need a class.
Whenever you have a use for a Visitor pattern, you'll need classes. Every other design pattern can be accomplished more effectively and cleanly with generators, context managers (which are also better implemented as generators than as classes) and POD types (dictionaries, lists and tuples, etc.).
If you want to write "pythonic" code, you should prefer context managers and generators over classes. It will be cleaner.
If you want to extend functionality, you will almost always be able to accomplish it with containment rather than inheritance.
As every rule, this has an exception. If you want to encapsulate functionality quickly (ie, write test code rather than library-level reusable code), you can encapsulate the state in a class. It will be simple and won't need to be reusable.
If you need a C++ style destructor (RIIA), you definitely do NOT want to use classes. You want context managers.
I think you do it right. Classes are reasonable when you need to simulate some business logic or difficult real-life processes with difficult relations.
As example:
Several functions with share state
More than one copy of the same state variables
To extend the behavior of an existing functionality
I also suggest you to watch this classic video
dantiston gives a great answer on why OOP can be useful. However, it is worth noting that OOP is not necessary a better choice most cases it is used. OOP has the advantage of combining data and methods together. In terms of application, I would say that use OOP only if all the functions/methods are dealing and only dealing with a particular set of data and nothing else.
Consider a functional programming refactoring of dentiston's example:
def dictMean( nums ):
return sum(nums.values())/len(nums)
# It's good to include automatic tests for production code, to ensure that updates don't break old codes
assert( dictMean({'math':3.3,'science':3.5})==3.4 )
john = {'name':'John', 'age':12, 'gender':'male', 'level':6, 'grades':{'math':3.3}}
# setGrade
john['grades']['science']=3.5
# getGrade
print(john['grades']['math'])
# getGPA
print(dictMean(john['grades']))
At a first look, it seems like all the 3 methods exclusively deal with GPA, until you realize that Student.getGPA() can be generalized as a function to compute mean of a dict, and re-used on other problems, and the other 2 methods reinvent what dict can already do.
The functional implementation gains:
Simplicity. No boilerplate class or selfs.
Easily add automatic test code right after each
function for easy maintenance.
Easily split into several programs as your code scales.
Reusability for purposes other than computing GPA.
The functional implementation loses:
Typing in 'name', 'age', 'gender' in dict key each time is not very DRY (don't repeat yourself). It's possible to avoid that by changing dict to a list. Sure, a list is less clear than a dict, but this is a none issue if you include an automatic test code below anyway.
Issues this example doesn't cover:
OOP inheritance can be supplanted by function callback.
Calling an OOP class has to create an instance of it first. This can be boring when you don't have data in __init__(self).
A class defines a real world entity. If you are working on something that exists individually and has its own logic that is separate from others, you should create a class for it. For example, a class that encapsulates database connectivity.
If this not the case, no need to create class
It depends on your idea and design. If you are a good designer, then OOPs will come out naturally in the form of various design patterns.
For simple script-level processing, OOPs can be overhead.
Simply consider the basic benefits of OOPs like reusability and extendability and make sure if they are needed or not.
OOPs make complex things simpler and simpler things complex.
Simply keep the things simple in either way using OOPs or not using OOPs. Whichever is simpler, use that.

Is it Pythonic to use objects wherever possible?

Newbie Python question here - I am writing a little utility in Python to do disk space calculations when given the attributes of 2 different files.
Should I create a 'file' class with methods appropriate to the conversion and then create each file as an instance of that class? I'm pretty new to Python, but ok with Perl, and I believe that in Perl (I may be wrong, being self-taught), from the examples that I have seen, that most Perl is not OO.
Background info - These are IBM z/OS (mainframe) data sets, and when given the allocation attributes for a file on a specific disk type and file organisation (it's block size) and then given the allocation parameters for a different disk type & organisation, the space requirements can vary enormously.
Definition nitpicking preface: Everything in Python is technically an object, even functions and numbers. I'm going to assume you mean classes vs. functions in your question.
Actually I think one of the great things about Python is that it doesn't embrace classes for absolutely everything as some other languages (e.g., Java and C#).
It's perfectly acceptable in Python (and the built-in modules do this a lot) to define module level functions rather than encapsulating all logic in objects.
That said, classes do have their place, for example when you perform multiple actions on a single piece of data, and especially when these actions change the data and you want to keep its state encapsulated.
For Your Question and you requirements ..a short answer is "No"
The use of objects is not in itself "object oriented". Functional programming uses objects, too, just not in the same way. Python can be used in a very FP way, even though Python uses objects heavily behind the scenes.
Overuse of primitives can be a problem, but it's impossible to say whether that applies to your case without more data.
I think of OO as an interface design approach: If you are creating tools that are straightforward to interact with (and substitutable) as objects with predictable methods, then by all means, create objects. But if the interactions are straightforward to describe with module-level functions, then don't try too hard to engineer your code into classes.
First and foremost - Pythonic is a term that needs to disappear, preferably with everyone who uses it. It doesn't mean anything and it's used by people who can't use reason to justify anything, so they need a mandatory term to justify their nonsense.
But to the point you never HAVE to use object oriented concepts in your software development, as everything OOP can as easily be written with functions and solid spaghetti stringers. But the question is - do use of objects makes sense in my solution?
To understand when and how to use it, you have to ask what exactly is object oriented programming. And this was already very well explained in very old, but also free, book called Thinking in java which I consider to be the 101 bible of thinking on OO terms. I strongly urge you to grab a free copy and read couple of first chapters.
Because if you don't understand the object oriented approach, how can you apply it properly? When you do - then when to use it, or not use it, becomes clear, because you can clearly translate real life items and interactions into abstract objects. And this is the guideline - when the translation of given action, item or data to OOP model is straightforward and logical - then you should do it.

Declaring types for complex data structures in python

I am quite new to python programming (C/C++ background).
I'm writing code where I need to use complex data structures like dictionaries of dictionaries of lists.
The issue is that when I must use these objects I barely remember their structure and so how to access them.
This makes it difficult to resume working on code that was untouched for days.
A very poor solution is to use comments for each variable, but that's very inflexible.
So, given that python variables are just pointers to memory and they cannot be statically type-declared, is there any convention or rule that I could follow to ease complex data structures usage?
If you use docstrings in your classes then you can use help(vargoeshere) to see how to use it.
Whatever you do, do NOT, I repeat, do NOT use Hungarian Notation! It causes severe brain & bit rot.
So, what can you do? Python and C/C++ are quite different. In C++ you typically handle polymorphic calls like so:
void doWithFooThing(FooThing *foo) {
foo->bar();
}
Dynamic polymorphism in C++ depends on inheritance: the pointer passed to doWithFooThing may point only to instances of FooThing or one of its subclasses. Not so in Python:
def do_with_fooish(fooish):
fooish.bar()
Here, any sufficiently fooish thing (i.e. everything that has a callable bar attribute) can be used, no matter how it is releated to any other fooish thing through inheritance.
The point here is, in C++ you know what (base-)type every object has, whereas in Python you don't, and you don't care. What you try to achieve in Python is code that is reusable in as many situations as possible without having to force everthing under the rigid rule of class inheritance. Your naming should also reflect that. You dont write:
def some_action(a_list):
...
but:
def some_action(seq):
...
where seq might be not only a list, but any iterable sequence, be it list, tuple, dict, set, iterator, whatever.
In general, you put emphasis on the intent of your code, instead of its the type structure. Instead of writing:
dict_of_strings_to_dates = {}
you write:
users_birthdays = {}
It also helps to keep functions short, even more so than in C/C++. Then you'll be easily able to see what's going on.
Another thing: you shouldn't think of Python variables as pointers to memory. They're in fact dicionary entries:
assert foo.bar == getattr(foo, 'bar') == foo.__dict__['bar']
Not always exactly so, I concur, but the details can be looked up at docs.python.org.
And, BTW, in Python you don't declare stuff like you do in C/C++. You just define stuff.
I believe you should take a good look some of your complex structures, what you are doing with them, and ask... Is This Pythonic? Ask here on SO. I think you will find some cases where the complexity is an artifact of C/C++.
Include an example somewhere in your code, or in your tests.

What to consider before subclassing list?

I was recently going over a coding problem I was having and someone looking at the code said that subclassing list was bad (my problem was unrelated to that class). He said that you shouldn't do it and that it came with a bunch of bad side effects. Is this true?
I'm asking if list is generally bad to subclass and if so, what are the reasons. Alternately, what should I consider before subclassing list in Python?
The abstract base classes provided in the collections module, particularly MutableSequence, can be useful when implementing list-like classes. These are available in Python 2.6 and later.
With ABCs you can implement the "core" functionality of your class and it will provide the methods which logically depend on what you've defined.
For example, implementing __getitem__ in a collections.Sequence-derived class will be enough to provide your class with __contains__, __iter__, and other methods.
You may still want to use a contained list object to do the heavy lifting.
There are no benefits to subclassing list. None of the methods will use any methods you override, so you can have unexpected bugs. Further, it's very often confusing doing things like self.append instead of self.foos.append or especially self[4] rather than self.foos[4] to access your data. You can make something that works exactly like a list or (better) howevermuch like a list you really want while just subclassing object.
I think the first question I'd ask myself is, "Is my new object really a list?". Does it walk like a list, talk like a list? Or is is something else?
If it is a list, then all the standard list methods should all make sense.
If the standard list methods don't make sense, then your object should contain a list, not be a list.
In old python (2.2?) sub-classing list was a bad idea for various technical reasons, but in a modern python it is fine.
Nick is correct.
Also, while I can't speak to Python, in other OO languages (Java, Smalltalk) subclassing a list is a bad idea. Inheritance in general should be avoided and delegation-composition used instead.
Rather, you make a container class and delegate calls to the list. The container class has a reference to the list and you can even expose the calls and returns of the list in your own methods.
This adds flexibility and allows you to change the implementation (a different list type or data structure) later w/o breaking any code. If you want your list to do different listy-type things then your container can do this and use the plain list as a simple data structure.
Imagine if you had 47 different uses of lists. Do you really want to maintain 47 different subclasses?
Instead you could do this via the container and interfaces. One class to maintain and allow people to call your new and improved methods via the interface(s) with the implementation remaining hidden.

Python design question

I'm a C programmer and I'm getting quite good with Python. But I still have some problems getting my mind around the OO awesomeness of Python.
Here is my current design problem:
The end "product" is a JSON data structure created in Python (and passed to Javascript code) containing different types of data like:
{ type:url, {urlpayloaddict) }
{ type:text, {textpayloaddict}
...
My Javascript knows how to parse and display each type of JSON response.
I'm happy with this design. My question comes from handling this data in the Python code.
I obtain my data from a variety of sources: MySQL, a table lookup, an API call to a web service...
Basically, should I make a super class responseElement and specialise it for each type of response, then pass around a list of these objects in the Python code OR should I simply pass around a list of dictionaries that contain the response data in key value pairs. The answer seems to result in significantly different implementations.
I'm a bit unsure if I'm getting too object happy ??
In my mind, it basically goes like this: you should try to keep things the same where they are the same, and separate them where they're different.
If you're performing the exact same operations on and with the data, and it can all be represented in a common format, then there's no reason to have separate objects for it - translate it into a common format ASAP and Don't Repeat Yourself when it comes to implementing things that don't distinguish.
If each type/source of data requires specialized operations specific to it, and there isn't much in the way of overlap between such at the layer your Python code is dealing with, then keep things in separate objects so that you maintain a tight association between the specialized code and the specific data on which it is able to operate.
Do the different response sources represent fundamentally different categories or classes of objects? They don't appear to, the way you've described it.
Thus, various encode/decode functions and passing around only one type seems the best solution for you.
That type can be a dict or your own class, if you have special methods to use on the data (but those methods would then not care what input and output encodings were), or you could put the encode/decode pairs into the class. (Decode would be a classmethod, returning a new instance.)
Your receiver objects (which can perfectly well be instances of different classes, perhaps generated by a Factory pattern depending on the source of incoming data) should all have a common method that returns the appropriate dict (or other directly-JSON'able structure, such as a list that will turn into a JSON array).
Differently from what one answer states, this approach clearly doesn't require higher level code to know what exact kind of receiver it's dealing with (polymorphism will handle that for you in any OO language!) -- nor does the higher level code need to know "names of keys" (as, again, that other answer peculiarly states), as it can perfectly well treat the "JSON'able data" as a pretty opaque data token (as long as it's suitable to be the argument for a json.dumps later call!).
Building up and passing around a container of "plain old data" objects (produced and added to the container in various ways) for eventual serialization (or other such uniform treatment, but you can see JSON translation as a specific form of serialization) is a common OO pattern. No need to carry around anything richer or heavier than such POD data, after all, and in Python using dicts as the PODs is often a perfectly natural implementation choice.
I've had success with the OOP approach. Consider a base class with a "ToJson" method and have each subclass implement it appropriately. Then your higher level code doesn't need to know any detail about how the data was obtained...it just knows it has to call "ToJson" on every object in the list you mentioned.
A dictionary would work too, but it requires your calling code to know names of keys, etc and won't scale as well.
OOP I say!
Personally, I opt for the latter (passing around a list of data) wherever and whenever possible. I think OO is often misused/abused for certain things. I specifically avoid things like wrapping data in an object just for the sake of wrapping it in an object. So this, {'type':'url', 'data':{some_other_dict}} is better to me than:
class DataObject:
def __init__(self):
self.type = 'url'
self.data = {some_other_dict}
But, if you need to add specific functionality to this data, like the ability for it to sort its data.keys() and return them as a set, then creating an object makes more sense.

Categories

Resources