Related
Newbie Python question here - I am writing a little utility in Python to do disk space calculations when given the attributes of 2 different files.
Should I create a 'file' class with methods appropriate to the conversion and then create each file as an instance of that class? I'm pretty new to Python, but ok with Perl, and I believe that in Perl (I may be wrong, being self-taught), from the examples that I have seen, that most Perl is not OO.
Background info - These are IBM z/OS (mainframe) data sets, and when given the allocation attributes for a file on a specific disk type and file organisation (it's block size) and then given the allocation parameters for a different disk type & organisation, the space requirements can vary enormously.
Definition nitpicking preface: Everything in Python is technically an object, even functions and numbers. I'm going to assume you mean classes vs. functions in your question.
Actually I think one of the great things about Python is that it doesn't embrace classes for absolutely everything as some other languages (e.g., Java and C#).
It's perfectly acceptable in Python (and the built-in modules do this a lot) to define module level functions rather than encapsulating all logic in objects.
That said, classes do have their place, for example when you perform multiple actions on a single piece of data, and especially when these actions change the data and you want to keep its state encapsulated.
For Your Question and you requirements ..a short answer is "No"
The use of objects is not in itself "object oriented". Functional programming uses objects, too, just not in the same way. Python can be used in a very FP way, even though Python uses objects heavily behind the scenes.
Overuse of primitives can be a problem, but it's impossible to say whether that applies to your case without more data.
I think of OO as an interface design approach: If you are creating tools that are straightforward to interact with (and substitutable) as objects with predictable methods, then by all means, create objects. But if the interactions are straightforward to describe with module-level functions, then don't try too hard to engineer your code into classes.
First and foremost - Pythonic is a term that needs to disappear, preferably with everyone who uses it. It doesn't mean anything and it's used by people who can't use reason to justify anything, so they need a mandatory term to justify their nonsense.
But to the point you never HAVE to use object oriented concepts in your software development, as everything OOP can as easily be written with functions and solid spaghetti stringers. But the question is - do use of objects makes sense in my solution?
To understand when and how to use it, you have to ask what exactly is object oriented programming. And this was already very well explained in very old, but also free, book called Thinking in java which I consider to be the 101 bible of thinking on OO terms. I strongly urge you to grab a free copy and read couple of first chapters.
Because if you don't understand the object oriented approach, how can you apply it properly? When you do - then when to use it, or not use it, becomes clear, because you can clearly translate real life items and interactions into abstract objects. And this is the guideline - when the translation of given action, item or data to OOP model is straightforward and logical - then you should do it.
Let me say first that I'm NOT searching for automagical solutions here. I want to translate code from Python to Smalltalk because I've noticed some very simple sentences can be automatically translated, examples:
Assigning a variable to a value
Python
i = 1
Smalltalk
i := 1.
Creating a new instance of a class
Python
instance = module.ClassName()
Smalltalk
instance := ClassName new.
A for loop
Python
for a in [0,1,2]:
print (str(a)+str(a))
Smalltalk
#(0 1 2) do: [: a | Transcript show: a + a; cr ]
and so on (while loops, conditionals, etc). My idea is to have a tool which translates all this extremely "simple" cases, and then I may complete or teach a rule system by hand.
Do you know any programming translation tool or library which can help me?
In case you haven't heard of any tool, what technique/pattern you will use to implement such translation? Can you provide a link to an example?
Thanks
You need to parse the Python code, walk the abstract syntax tree that is generated by the parser and output your Smalltalk. There's a nice article about Python ASTs by Eli Bendersky and a slightly older one here. Python makes this relatively straight forward as the Python standard library exposes a lot of the internal tooling of the interpreter and the documentation is reasonably comprehensive.
I am not aware of any such tool, and in general case it might be complicated and/or inefficient. So your route would depend on your more precise need: porting an existing python library, just using it from smalltalk, or making nice clean smalltalk code that does the same thing as python one.
Routes I would consider:
leaving python library as is, and calling it from smalltalk through c interface
implementing python parser in pettit parser an then:
implement smalltalk generator maybe assisted by human through user interface
python interpreter in smalltalk
Note that generator variant might face some difficult issues in general cases, for instance smalltalk has fixed number of instance variables, while in python you can attach then as you go. You could work around that, but resulting smalltalk code might not be pretty.
As for implementing python inside smalltalk take a look at the helvetia presentation from Lukas Renggli, it is on the subject of including other languages inside smalltalk IDE.
Take a look at ply, which is a Lex-Yacc Python implementation. I've used it mostly for translating some other language into Python byte code by building a Python AST with it, but the opposite should be also possible.
I'm a very experienced developer - done a lot of heavy duty work with Delphi, C# and C++ for years. I have always adhered very closely to the guidelines for structured programming, OOP, loosely coupled modular designs etc - and since all the languages I've used have built-in ways of enforcing these concepts - access control, static types, interface and abstract class support etc - I rely on these to structure my code.
Now, I have been doodling with Python for a few months. I am impressed by its many wonderful features - but I sorely miss the built-in constraints that make it easy to keep code modularized and organized. And, unfortunately, I see an awful lot of 'spaghetti code' out there written in Python, even from very respectable sources. I won't single anyone out but I have a few books written by major league pythonistas with examples replete with designs (better put - 'anti-designs') that make me shudder. It seems to me that because Python is so easy to use, it's also very easy to abuse.
I do try to discipline myself when I code in Python, but I find it takes a lot of extra work to implement and often I have to set up and adhere to constraints simply based on my own memory of the design with no help from the language at all. And since there is no 'compile time' checking, it's doubly difficult - often you don't discover a design flaw until you actually RUN that segment of code.
So, I'm looking for very specific information: some examples or better still a book of WELL STRUCTURED Python designs and design techniques - how to best implement encapsulation, indirection, very loosely coupled designs, etc.
Bad design IMO from a prominent python book author - (with obfuscation)
def populateList(self, selecteddisk=None):
selected = None ***#Bundling - coupling:***
self.listWidget.clear()
for disk in self.disks.inOrder():
item = QListWidgetItem(QString("%1 of %2/%3 (%L4)") \
.arg(disk.name).arg(disk.owner).arg(disk.country) \
.arg(disk.teu))
self.listWidget.addItem(item)
***#Bundling - coupling:***
if selecteddisk is not None and selecteddisk == id(disk):
selected = item
if selected is not None:
selected.setSelected(True)
self.listWidget.setCurrentItem(selected)
I find it takes a lot of extra work to implement and compile code that adheres to constraints simply based on my own memory of the design with no help from the language when writing the code itself before compiling. Some IDE's offer help, but the language itself offers me no help at all.
And since 'compile time' checking never seems to help me find ordinary logic bugs it's doubly difficult - often you don't discover a design flaw until you actually RUN that segment of code.
Python designs and design techniques ...
how to best implement encapsulation,
By encapsulating. In languages like Java and C++, "encapsulation" has grown to mean "uses private stuff all over the place." In Python that's simply not supported.
We're All Adults Here.
You still do encapsulation just like you did in every other language. Without the word private, however.
Python offers properties, decorators and overrides to __getattribute__ to implement various kinds of encapsulation techniques.
indirection,
By referencing other objects. I'm not clear on what specific problems you have here, but perhaps you've passed some wrong-type argument to a function. The way to avoid this is to read the docstrings you wrote for yourself.
very loosely coupled designs, etc.
By doing dependency injection. Again. Python works just like every other language with respect to loose coupling.
You should investigate -- and use -- docstrings heavily.
You might want to use https://www.sphinx-doc.org to generate nice documentation from your docstrings.
You can also use Python's built-in help() function to read the docstrings you wrote when you wrote your code.
On design pattern support:
There is a proposition that also puts that Design Patterns as a crude metric of gaps in a programming language. It would be an interesting read and counterpoint to your suggestion that design patterns are constraints supported by language.
http://www.oscon.com/oscon2008/public/schedule/detail/2810
The section below goes into more detail, but basically someone stated that the Ruby-written DSL RSpec couldn't be rewritten in Python. Is that true? If so, why?
I'm wanting to better understand the technical differences between Ruby and Python.
Update: Why am I asking this question?
The Running away from RSpec discussion has some statements about it being "impossible" to recreate RSpec in Python. I was trying to make the question a little broader in hopes of learning more of the technical differences between Ruby and Python. In hindsight, maybe I should have tightened the question's scope to just asking if it truly is impossible to recreate RSpec in Python, and if so why.
Below are just a few quotes from the Running away from RSpec discussion.
Initial Question
For the past few weeks I have been thinking a lot about RSpec and why there is no clear, definite answer when someone asks:
"I'm looking for a Python equivalent of RSpec. Where can I find such a
thing?"
Probably the most common (and understandable) answer is that Python syntax
wouldn't allow such a thing whereas in Ruby it is possible.
First Response to Initial Question
Not syntax exactly. Rspec monkeypatches every object inside of its
scope, inserting the methods "should" and "should_not". You can do
something in python, but you can't monkeypatch the built-in types.
Another Response
As you suggest, it's impossible. Mote and PySpec are just fancy ways
to name your tests: weak implementations of one tiny corner of RSpec.
Mote uses horrible settrace magic; PySpec adds a bunch of
domain-irrelevant noise. Neither even supports arbitrary context
strings. RSpec is more terse, more expressive, removes the noise, and
is an entirely reasonable thing to build in Ruby.
That last point is important: it's not just that RSpec is possible in
Ruby; it's actually idiomatic.
If I had to point out one great difficulty for creating a Python RSpec, it would be the lack of a good syntax in Python for creating anonymous functions (as in JavaScript) or blocks (as in Ruby). The only option for a Python programmer is to use lambdas, which is not an option at all because lambdas just accept one expression. The do ... end blocks used in RSpec would have to be written as a function before calling describe and it, as in the example below:
def should_do_stuff():
# ...
it("should do stuff", should_do_stuff)
Not so sexy, right?
There are some difficulties in creating the should methods, but I bet it would be a smaller problem. Actually, one does not even need to use such an unusual syntax—you could get similar results (maybe even better, depending on your taste) using the Jasmine syntax, which can be trivially implemented.
That said, I feel that Python syntax is more focused on efficiently representing the usual program components such as classes, functions, variables, etc. It is not well suited to be extended. I, for one, think that a good Python program is one where I can see objects, and functions, and variables, and I understand what each one of these elements do. Ruby programmers, OTOH, seem to seek for a more prose-like style, where a new language is defined for a new problem. It is a good way of doing things, too, but not a Pythonic way. Python is good to represent algorithms, not prose.
Sometimes it is a draconian limit. How could one use BDD for example? Well, the usual way of pushing these limits in Python is to effectively write your own DSL, but it should REALLY be another language. That is what Pyccuracy is, for example: another language for BDD. A more mainstream example is doctest. (Actually, if I would write some BDD Python library, I would write it based on doctest.) Another example of Python DSL is Twill. And yet another example is reStructuredText, used in Sphinx.
Summarizing: IMHO the hardest barrier to DSLs in Python is the lack of a flexible syntax for creating anonymous functions. And it is not a fault*: Python is not fond of having its syntax heavily explored anyway—it is considered to make code less clear in the Python universe. If you want a new syntax in Python you are well advised to write your own language, or at least it is the way I feel.
* Or maybe it is - I have to confess that I miss anonymous functions. However, I recognize that they would be hard to implement elegantly given the Python semantic indentation.
I set out on an attempt to implement something like rspec in Python.
I got this:
with It('should pass') as test:
test.should_be_equal(1, 1)
source: https://gist.github.com/2029866
(thoughts?)
EDIT: My answer to your question is that the lack of anonymous blocks prevents a Ruby DSL like RSpec from being rewritten in Python but you can get a close approximation using with statements.
One of Ruby's strengths is in the creation of DSLs. However the reasons given for it being difficult in python can be sidestepped. For example you can easily subclass the builtin types, e.g:
>>> class myint(int): pass
>>> i = myint(5)
>>> i
5
If I were going to create a DSL in python I'd use pyparsing or Parsley and something like the above behind the scenes, optimizing the syntax for the problem, not the implementation language.
By mixing Mamba and Expects, I think you can get very close to what RSpec is for Rails...
https://github.com/nestorsalceda/mamba
https://github.com/jaimegildesagredo/expects
Also, I think Specter should match your expectations with testing:
https://github.com/jmvrbanac/Specter
http://specter.readthedocs.io/en/latest/writing_tests/index.html
I think this is what you are looking for. Yes, we made the "impossible" in python
"sure" is an utility belt for expressive python tests, created by Gabriel Falcão
I would be interested to learn about large scale development in Python and especially in how do you maintain a large code base?
When you make incompatibility changes to the signature of a method, how do you find all the places where that method is being called. In C++/Java the compiler will find it for you, how do you do it in Python?
When you make changes deep inside the code, how do you find out what operations an instance provides, since you don't have a static type to lookup?
How do you handle/prevent typing errors (typos)?
Are UnitTest's used as a substitute for static type checking?
As you can guess I almost only worked with statically typed languages (C++/Java), but I would like to try my hands on Python for larger programs. But I had a very bad experience, a long time ago, with the clipper (dBase) language, which was also dynamically typed.
Don't use a screw driver as a hammer
Python is not a statically typed language, so don't try to use it that way.
When you use a specific tool, you use it for what it has been built. For Python, it means:
Duck typing : no type checking. Only behavior matters. Therefore your code must be designed to use this feature. A good design means generic signatures, no dependences between components, high abstraction levels.. So if you change anything, you won't have to change the rest of the code. Python will not complain either, that what it has been built for. Types are not an issue.
Huge standard library. You do not need to change all your calls in the program if you use standard features you haven't coded yourself. And Python come with batteries included. I keep discovering them everyday. I had no idea of the number of modules I could use when I started and tried to rewrite existing stuff like everybody. It's OK, you can't get it all right from the beginning.
You don't write Java, C++, Python, PHP, Erlang, whatever, the same way. They are good reasons why there is room for each of so many different languages, they do not do the same things.
Unit tests are not a substitute
Unit tests must be performed with any language. The most famous unit test library (JUnit) is from the Java world!
This has nothing to do with types. You check behaviors, again. You avoid trouble with regression. You ensure your customer you are on tracks.
Python for large scale projects
Languages, libraries and frameworks
don't scale. Architectures do.
If you design a solid architecture, if you are able to make it evolves quickly, then it will scale. Unit tests help, automatic code check as well. But they are just safety nets. And small ones.
Python is especially suitable for large projects because it enforces some good practices and has a lot of usual design patterns built-in. But again, do not use it for what it is not designed. E.g : Python is not a technology for CPU intensive tasks.
In a huge project, you will most likely use several different technologies anyway. As a SGBD (French for DBMS) and a templating language, or else. Python is no exception.
You will probably want to use C/C++ for the part of your code you need to be fast. Or Java to fit in a Tomcat environment. Don't know, don't care. Python can play well with these.
As a conclusion
My answer may feel a bit rude, but don't get me wrong: this is a very good question.
A lot of people come to Python with old habits. I screwed myself trying to code Java like Python. You can, but will never get the best of it.
If you have played / want to play with Python, it's great! It's a wonderful tool. But just a tool, really.
I had some experience with modifying "Frets On Fire", an open source python "Guitar Hero" clone.
as I see it, python is not really suitable for a really large scale project.
I found myself spending a large part of the development time debugging issues related to assignment of incompatible types, things that static typed laguages will reveal effortlessly at compile-time.
also, since types are determined on run-time, trying to understand existing code becomes harder, because you have no idea what's the type of that parameter you are currently looking at.
in addition to that, calling functions using their name string with the __getattr__ built in function is generally more common in Python than in other programming languages, thus getting the call graph to a certain function somewhat hard (although you can call functions with their name in some statically typed languages as well).
I think that Python really shines in small scale software, rapid prototype development, and gluing existing programs together, but I would not use it for large scale software projects, since in those types of programs maintainability becomes the real issue, and in my opinion python is relatively weak there.
Since nobody pointed out pychecker, pylint and similar tools, I will: pychecker and pylint are tools that can help you find incorrect assumptions (about function signatures, object attributes, etc.) They won't find everything that a compiler might find in a statically typed language -- but they can find problems that such compilers for such languages can't find, too.
Python (and any dynamically typed language) is fundamentally different in terms of the errors you're likely to cause and how you would detect and fix them. It has definite downsides as well as upsides, but many (including me) would argue that in Python's case, the ease of writing code (and the ease of making it structurally sound) and of modifying code without breaking API compatibility (adding new optional arguments, providing different objects that have the same set of methods and attributes) make it suitable just fine for large codebases.
my 0.10 EUR:
i have several python application in 'production'-state. our company use java, c++ and python. we develop with the eclipse ide (pydev for python)
unittests are the key-solution for the problem. (also for c++ and java)
the less secure world of "dynamic-typing" will make you less careless about your code quality
BY THE WAY:
large scale development doesn't mean, that you use one single language!
large scale development often uses a handful of languages specific to the problem.
so i agree to the-hammer-problem :-)
PS: static-typing & python
Here are some items that have helped me maintain a fairly large system in python.
Structure your code in layers. i.e separate biz logic, presentation logic and your persistence layers. Invest a bit of time in defining these layers and make sure everyone on the project is brought in. For large systems creating a framework that forces you into a certain way of development can be key as well.
Tests are key, without unit tests you will likely end up with an unmanagable code base several times quicker than with other languages. Keep in mind that unit tests are often not sufficient, make sure to have several integration/acceptance tests you can run quickly after any major change.
Use Fail Fast principle. Add assertions for cases you feel your code maybe vulnerable.
Have standard logging/error handling that will help you quickly navigate to the issue
Use an IDE( pyDev works for me) that provides type ahead, pyLint/Checker integration to help you detect common typos right away and promote some coding standards
Carefull about your imports, never do from x import * or do relative imports without use of .
Do refactor, a search/replace tool with regular expressions is often all you need to do move methods/class type refactoring.
Incompatible changes to the signature of a method. This doesn't happen as much in Python as it does in Java and C++.
Python has optional arguments, default values, and far more flexibility in defining method signatures. Also, duck typing means that -- for example -- you don't have to switch from some class to an interface as part of a significant software change. Things just aren't as complex.
How do you find all the places where that method is being called? grep works for dynamic languages. If you need to know every place a method is used, grep (or equivalent IDE-supported search) works great.
How do you find out what operations an instance provides, since you don't have a static type to lookup?
a. Look at the source. You don't have the Java/C++ problem of object libraries and jar files to contend with. You don't need all the elaborate aids and tools that those languages require.
b. An IDE can provide signature information under many common circumstances. You can, easily, defeat your IDE's reasoning powers. When that happens, you should probably review what you're doing to be sure it makes sense. If your IDE can't reason out your type information, perhaps it's too dynamic.
c. In Python, you often work through the interactive interpreter. Unlike Java and C++, you can explore your instances directly and interactively. You don't need a sophisticated IDE.
Example:
>>> x= SomeClass()
>>> dir(x)
How do you handle/prevent typing errors? Same as static languages: you don't prevent them. You find and correct them. Java can only find a certain class of typos. If you have two similar class or variable names, you can wind up in deep trouble, even with static type checking.
Example:
class MyClass { }
class MyClassx extends MyClass { }
A typo with these two class names can cause havoc. ["But I wouldn't put myself in that position with Java," folks say. Agreed. I wouldn't put myself in that position with Python, either; you make classes that are profoundly different, and will fail early if they're misused.]
Are UnitTest's used as a substitute for static type checking? Here's the other Point of view: static type checking is a substitute for clear, simple design.
I've worked with programmers who weren't sure why an application worked. They couldn't figure out why things didn't compile; the didn't know the difference between abstract superclass and interface, and the couldn't figure out why a change in place makes a bunch of other modules in a separate JAR file crash. The static type checking gave them false confidence in a flawed design.
Dynamic languages allow programs to be simple. Simplicity is a substitute for static type checking. Clarity is a substitute for static type checking.
My general rule of thumb is to use dynamic languages for small non-mission-critical projects and statically-typed languages for big projects. I find that code written in a dynamic language such as python gets "tangled" more quickly. Partly that is because it is much quicker to write code in a dynamic language and that leads to shortcuts and worse design, at least in my case. Partly it's because I have IntelliJ for quick and easy refactoring when I use Java, which I don't have for python.
The usual answer to that is testing testing testing. You're supposed to have an extensive unit test suite and run it often, particularly before a new version goes online.
Proponents of dynamically typed languages make the case that you have to test anyway because even in a statically typed language conformance to the crude rules of the type system covers only a small part of what can potentially go wrong.