Related
I was recently reading over PEP 572 on assignment expressions and stumbled upon an interesting use case:
# Compute partial sums in a list comprehension
total = 0
partial_sums = [total := total + v for v in values]
print("Total:", total)
I began exploring the snippet on my own and soon discovered that :+= wasn't valid Python syntax.
# Compute partial sums in a list comprehension
total = 0
partial_sums = [total :+= v for v in values]
print("Total:", total)
I suspect there may be some underlying reason in how := is implemented that wisely precludes :+=, but I'm not sure what it could be. If someone wiser in the ways of Python knows why :+= is unfeasible or impractical or otherwise unimplemented, please share your understanding.
The short version: The addition of the walrus operator was incredibly controversial, and they wanted to discourage overuse, so they limited it to only those cases for which a strong motivating use case was put forward, leaving = the convenient tool for all other cases.
There's a lot of things the walrus operator won't do that it could do (assigning to things looked up on a sequence or mapping, assigning to attributes, etc.), but it would encourage using it all the time, to the detriment of the readability of typical code. Sure, you could choose to write more readable code, ignoring the opportunity to use weird and terrible punctuation-filled nonsense, but if my (and many people's) experience with Perl is any guide, people will use shortcuts to get it done faster now even if the resulting code is unreadable, even by them, a month later.
There are other minor hurdles that get in the way (supporting all of the augmented assignment approaches with the walrus would add a ton of new byte codes to the interpreter, expanding the eval loop's switch significantly, and potentially inhibiting optimizations/spilling from CPU cache), but fundamentally, your motivating case of using a list comprehension for side-effects is a misuse of list comprehensions (a functional construct that, like all functional programming tools, is not intended to have side-effects), as are most cases that would rely on augmented assignment expressions. The strong motivations for introducing this feature were things you could reasonably want to do and couldn't without the walrus, e.g.
Regexes, replacing this terrible, verbose arrow pattern:
m = re.match(r'pattern', string)
if m:
do_thing(m)
else:
m = re.match(r'anotherpattern', string)
if m:
do_another_thing(m)
else:
m = re.match(r'athirdpattern', string)
if m:
do_a_third_thing(m)
with this clean chain of tests:
if m := re.match(r'pattern', string):
do_thing(m)
elif m := re.match(r'anotherpattern', string):
do_another_thing(m)
elif m := re.match(r'athirdpattern', string):
do_a_third_thing(m)
Reading a file by block, replacing:
while True:
block = file.read(4096)
if not block:
break
with the clean:
while block := file.read(4096):
Those are useful things people really need to do with some frequency, and even the "canonical" versions I posted are often misimplemented in other ways (e.g. applying the regex test twice to avoid the arrow pattern, duplicating the block = file.read(4096) once before loop and once at the end so you can run while block:, but in exchange now continue doesn't work properly, and you risk the size of the block changing in one place but not another); the walrus operator allowed for better code.
The listcomp accumulator isn't (much) better code. itertools.accumulate exists, and even if it didn't, the problem can be worked around in other ways with simple generator functions or hand-rolled loops. The PEP does describe it as a benefit (it's why they allowed walrus assignments to "escape" comprehensions), but the discussion relating to this scoping special case was even more divided than the discussion over adding the walrus itself; you can do it, and it's arguably useful, but it's not something where you look at the two options and immediately say "Man, if not for the walrus, this would be awful".
And just to satisfy the folks who want evidence of this, note that they explicitly blocked some use cases that would have worked for free (it took extra work to make the grammar prohibit these usages), specifically to prevent walrus overuse. For example, you can't do:
x := 1
on a line by itself. There's no technical reason you can't, at all, but they intentionally made it a syntax error for the walrus to be used without being wrapped in something. (x := 1) works at top level, but it's annoying enough no one will ever choose it over x = 1, and that was the goal.
Until someone comes up with a common code pattern for which the lack of :+= makes it infeasibly/unnecessarily ugly (and it would have to be really common and really ugly to justify the punctuation filled monstrosity that is :+=), they won't consider it.
The itertools module provides a lot of the mechanisms to achieve the same result without bloating the language:
partial_sums = list(accumulate(values))
total = partial_sums[-1]
My understanding is that a major part of comprehension efficiency is that the calculations for each element in the original iterable can be performed in parallel. If you're calculating a running total, that has to be performed in series.
We already have sum(values), so what does this add in this use case?
If you want something for more general use cases, functools has reduce.
I know Ruby very well. I believe that I may need to learn Python presently. For those who know both, what concepts are similar between the two, and what are different?
I'm looking for a list similar to a primer I wrote for Learning Lua for JavaScripters: simple things like whitespace significance and looping constructs; the name of nil in Python, and what values are considered "truthy"; is it idiomatic to use the equivalent of map and each, or are mumble somethingaboutlistcomprehensions mumble the norm?
If I get a good variety of answers I'm happy to aggregate them into a community wiki. Or else you all can fight and crib from each other to try to create the one true comprehensive list.
Edit: To be clear, my goal is "proper" and idiomatic Python. If there is a Python equivalent of inject, but nobody uses it because there is a better/different way to achieve the common functionality of iterating a list and accumulating a result along the way, I want to know how you do things. Perhaps I'll update this question with a list of common goals, how you achieve them in Ruby, and ask what the equivalent is in Python.
Here are some key differences to me:
Ruby has blocks; Python does not.
Python has functions; Ruby does not. In Python, you can take any function or method and pass it to another function. In Ruby, everything is a method, and methods can't be directly passed. Instead, you have to wrap them in Proc's to pass them.
Ruby and Python both support closures, but in different ways. In Python, you can define a function inside another function. The inner function has read access to variables from the outer function, but not write access. In Ruby, you define closures using blocks. The closures have full read and write access to variables from the outer scope.
Python has list comprehensions, which are pretty expressive. For example, if you have a list of numbers, you can write
[x*x for x in values if x > 15]
to get a new list of the squares of all values greater than 15. In Ruby, you'd have to write the following:
values.select {|v| v > 15}.map {|v| v * v}
The Ruby code doesn't feel as compact. It's also not as efficient since it first converts the values array into a shorter intermediate array containing the values greater than 15. Then, it takes the intermediate array and generates a final array containing the squares of the intermediates. The intermediate array is then thrown out. So, Ruby ends up with 3 arrays in memory during the computation; Python only needs the input list and the resulting list.
Python also supplies similar map comprehensions.
Python supports tuples; Ruby doesn't. In Ruby, you have to use arrays to simulate tuples.
Ruby supports switch/case statements; Python does not.
Ruby supports the standard expr ? val1 : val2 ternary operator; Python does not.
Ruby supports only single inheritance. If you need to mimic multiple inheritance, you can define modules and use mix-ins to pull the module methods into classes. Python supports multiple inheritance rather than module mix-ins.
Python supports only single-line lambda functions. Ruby blocks, which are kind of/sort of lambda functions, can be arbitrarily big. Because of this, Ruby code is typically written in a more functional style than Python code. For example, to loop over a list in Ruby, you typically do
collection.each do |value|
...
end
The block works very much like a function being passed to collection.each. If you were to do the same thing in Python, you'd have to define a named inner function and then pass that to the collection each method (if list supported this method):
def some_operation(value):
...
collection.each(some_operation)
That doesn't flow very nicely. So, typically the following non-functional approach would be used in Python:
for value in collection:
...
Using resources in a safe way is quite different between the two languages. Here, the problem is that you want to allocate some resource (open a file, obtain a database cursor, etc), perform some arbitrary operation on it, and then close it in a safe manner even if an exception occurs.
In Ruby, because blocks are so easy to use (see #9), you would typically code this pattern as a method that takes a block for the arbitrary operation to perform on the resource.
In Python, passing in a function for the arbitrary action is a little clunkier since you have to write a named, inner function (see #9). Instead, Python uses a with statement for safe resource handling. See How do I correctly clean up a Python object? for more details.
I, like you, looked for inject and other functional methods when learning Python. I was disappointed to find that they weren't all there, or that Python favored an imperative approach. That said, most of the constructs are there if you look. In some cases, a library will make things nicer.
A couple of highlights for me:
The functional programming patterns you know from Ruby are available in Python. They just look a little different. For example, there's a map function:
def f(x):
return x + 1
map(f, [1, 2, 3]) # => [2, 3, 4]
Similarly, there is a reduce function to fold over lists, etc.
That said, Python lacks blocks and doesn't have a streamlined syntax for chaining or composing functions. (For a nice way of doing this without blocks, check out Haskell's rich syntax.)
For one reason or another, the Python community seems to prefer imperative iteration for things that would, in Ruby, be done without mutation. For example, folds (i.e., inject), are often done with an imperative for loop instead of reduce:
running_total = 0
for n in [1, 2, 3]:
running_total = running_total + n
This isn't just a convention, it's also reinforced by the Python maintainers. For example, the Python 3 release notes explicitly favor for loops over reduce:
Use functools.reduce() if you really need it; however, 99 percent of the time an explicit for loop is more readable.
List comprehensions are a terse way to express complex functional operations (similar to Haskell's list monad). These aren't available in Ruby and may help in some scenarios. For example, a brute-force one-liner to find all the palindromes in a string (assuming you have a function p() that returns true for palindromes) looks like this:
s = 'string-with-palindromes-like-abbalabba'
l = len(s)
[s[x:y] for x in range(l) for y in range(x,l+1) if p(s[x:y])]
Methods in Python can be treated as context-free functions in many cases, which is something you'll have to get used to from Ruby but can be quite powerful.
In case this helps, I wrote up more thoughts here in 2011: The 'ugliness' of Python. They may need updating in light of today's focus on ML.
My suggestion: Don't try to learn the differences. Learn how to approach the problem in Python. Just like there's a Ruby approach to each problem (that works very well givin the limitations and strengths of the language), there's a Python approach to the problem. they are both different. To get the best out of each language, you really should learn the language itself, and not just the "translation" from one to the other.
Now, with that said, the difference will help you adapt faster and make 1 off modifications to a Python program. And that's fine for a start to get writing. But try to learn from other projects the why behind the architecture and design decisions rather than the how behind the semantics of the language...
I know little Ruby, but here are a few bullet points about the things you mentioned:
nil, the value indicating lack of a value, would be None (note that you check for it like x is None or x is not None, not with == - or by coercion to boolean, see next point).
None, zero-esque numbers (0, 0.0, 0j (complex number)) and empty collections ([], {}, set(), the empty string "", etc.) are considered falsy, everything else is considered truthy.
For side effects, (for-)loop explicitly. For generating a new bunch of stuff without side-effects, use list comprehensions (or their relatives - generator expressions for lazy one-time iterators, dict/set comprehensions for the said collections).
Concerning looping: You have for, which operates on an iterable(! no counting), and while, which does what you would expect. The fromer is far more powerful, thanks to the extensive support for iterators. Not only nearly everything that can be an iterator instead of a list is an iterator (at least in Python 3 - in Python 2, you have both and the default is a list, sadly). The are numerous tools for working with iterators - zip iterates any number of iterables in parallel, enumerate gives you (index, item) (on any iterable, not just on lists), even slicing abritary (possibly large or infinite) iterables! I found that these make many many looping tasks much simpler. Needless to say, they integrate just fine with list comprehensions, generator expressions, etc.
In Ruby, instance variables and methods are completely unrelated, except when you explicitly relate them with attr_accessor or something like that.
In Python, methods are just a special class of attribute: one that is executable.
So for example:
>>> class foo:
... x = 5
... def y(): pass
...
>>> f = foo()
>>> type(f.x)
<type 'int'>
>>> type(f.y)
<type 'instancemethod'>
That difference has a lot of implications, like for example that referring to f.x refers to the method object, rather than calling it. Also, as you can see, f.x is public by default, whereas in Ruby, instance variables are private by default.
I've seen some elegant python snippets using list comprehension and map reduce. Can you share some of these code or a web site.
Thanks.
Python is not lisp. Please don't try to make it look that way. It only reduces one of python's biggest strengths, which is its readability and understandability later on.
If you like functional programming, learn Haskell, ML, or F#. You will be amazed at what those languages offer (pure functions to start with).
There are some nice functional style snippets in here: Functional Programming HOWTO
Be careful when programming python in functional style. The only reason to ever do so is for readability. If the algorithm is more elegantly expressed functionally than imperatively, and it doesn't cause performance problems (it usually doesn't), then go right ahead.
However, python does not optimize tail recursion, and has a fixed recursion limit of 1000, so you generally can't do O(n) recursion, only O(log(n)).
Also, reduce() is removed in python 3, for good reason ( http://www.artima.com/weblogs/viewpost.jsp?thread=98196 ). Most non-trivial uses of reduce are more readable as a normal loop instead of a reduction, and sum() is already built in.
Here is quick sort:
def qsort (list):
if (len(list) > 1):
list = qsort(filter (lambda x: x <= list[0], list[1:])) + [list[0]] + qsort(filter (lambda x: x > list[0], list[1:]))
return list
This one is a solution to a programming puzzle of finding a missing number among the integers from 1 to 100:
from random import randint
nos = range(1,101)
to_remove = randint(1,100)
nos.remove(to_remove)
print "Removed %d from list" % to_remove
found = 5050 - reduce (lambda x,y: x+y, nos)
print "You removed %d " % found
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 10 years ago.
I've decided to learn Python 3. For those that have gone before, what did you find most useful along the way and wish you'd known about sooner?
I learned Python back before the 1.5.2 release, so the things that were key for me back then may not be the key things today.
That being said, a crucial thing that took me a little bit to realize, but I now consider crucial: much functionality that other languages would make intrinsic is actually made available by the standard library and the built-ins.
The language itself is small and simple, but until you're familiar with the built-ins and the "core parts" of the standard library (e.g., nowadays, sys, itertools, collections, copy, ...), you'll be reinventing the wheel over and over. So, the more time you invest in getting familiar with those parts, the smoother your progress will be. Every time you have a task you want to do, that doesn't seem to be directly supported by the language, first ask yourself: what built-ins or modules in the standard library will make the task much simpler, or even do it all for me? Sometimes there won't be any, but more often than not you'll find excellent solutions by proceeding with this mindset.
I wished I didn't know Java.
More functional programming. (see itertools module, list comprehension, map(), reduce() or filter())
List comprehension (makes a list cleanly):
[x for x in y if x > z]
Generator expansion (same as list comprehension but doesn't evaluate until it is used):
(x for x in y if x > z)
Two brain-cramping things. One of which doesn't apply to Python 3.
a = 095
Doesn't work. Why? The leading zero is an octal literal. The 9 is not valid in an octal literal.
def foo( bar=[] ):
bar.append( 1 )
return bar
Doesn't work. Why? The mutable default object gets reused.
What enumerate is for.
That seq = seq.append(item) and seq = seq.sort() both set seq to None.
Using set to remove duplicates.
Pretty much everything in the itertools and collections modules.
How the * and ** prefixes for function arguments work.
How default arguments to functions work internally (i.e. what f.func_defaults is).
How (why, really) to design functions so that they are useful in conjunction with map and zip.
The role of __dict__ in classes.
What import actually does.
Learn how to use iPython
It's got Tab completion.
View all the elements in your namespace with 'whos'.
After you import a module, it's easy to view the code:
>>> import os
>>> os?? # this display the actual source of the method
>>> help() # Python's interactive help. Fantastic!
Most Python modules are well documented; in theory, you could learn iPython and the rest of what you'd need to know could be learned through the same tool.
iPython also has a debug mode, pdb().
Finally, you can even use iPython as a python enabled command line. The basic UNIX commands work as %magic methods. Any commands that aren't magic command can be executed:
>>> os.system('cp file1 file2')
Don't have variable names that are types. For example, don't name a variable "file" or "dict"
Decorators. Writing your own is not something you might want to do right away, but knowing that #staticmethod and #classmethod are available from the beginning (and the difference between what they do) is a real plus.
using help() in the shell on any object, class or path
you can run import code;
code.interact(local=locals()) anywhere in your code and it will start a python shell at that exact point
you can run python -i yourscript.py to start a shell at the end of yourscript.py
Most helpful: Dive Into Python. As a commenter points out, if you're learning Python 3, Dive Into Python 3 is more applicable.
Known about sooner: virtualenv.
That a tuple of a single item must end with a comma, or it won't be interpreted as a tuple.
pprint() is very handy (yes, 2 p's)
reload() is useful when you're re-testing a module while making lots of rapid changes to a dependent module.
And learn as many common "idioms" as you can, otherwise you'll bang your head looking for a better way to do something, when the idiom really is regarded as the best way (e.g. ugly expressions like ' '.join(), or the answer to why there is no isInt(string) function.... the answer is you can just wrap the usage of a "possible" integer with a try: and then catch the exception if it's not a valid int. The solution works well, but it sounds like a terrible answer when you first encounter it, so you can waste a lot of time convincing yourself it really is a good approach.
Those are some things that wasted several hours of my time to determine that my first draft of some code which felt wrong, really was acceptable.
Readings from python.org:
http://wiki.python.org/moin/BeginnerErrorsWithPythonProgramming
http://wiki.python.org/moin/PythonWarts
List comprehensions, if you're coming to Python fresh (not from an earlier version).
Closures. Clean and concise, without having to resort to using a Strategy Pattern unlike languages such as Java
If you learn from a good book, it will not only teach you the language, it will teach you the common idioms. The idioms are valuable.
For example, here is the standard idiom for initializing a class instance with a list:
class Foo(object):
def __init__(self, lst=None):
if lst is None:
self.lst = []
else:
self.lst = lst
If you learn this as an idiom from a book, you don't have to learn the hard way why this is the standard idiom. #S.Lott already explained this one: if you try to make the default initializer be an empty list, the empty list gets evaluated just once (at compile time) and every default-initialized instance of your class gets the same list instance, which was not what was intended here.
Some idioms protect you from non-intended effects; some help you get best performance out of the language; and some are just small points of style, which help other Python fans understand your code better.
I learned out of the book Learning Python and it introduced me to some of the idioms.
Here's a web page devoted to idioms: http://python.net/~goodger/projects/pycon/2007/idiomatic/handout.html
P.S. Python code that follows the best-practice Python idioms often is called "Pythonic" code.
I implemented plenty of recursive directory walks by hand before I learned about os.walk()
Lambda functions
http://www.diveintopython.org/power_of_introspection/lambda_functions.html
One of the coolest things I learned about recently was the commands module:
>>> import commands
>>> commands.getoutput('uptime')
'18:24 up 10:22, 7 users, load averages: 0.37 0.45 0.41'
It's like os.popen or os.system but without all of the DeprecationWarnings.
And let's not forget PDB (Python Debugger):
% python -m pdb poop.py
Dropping into interactive mode in IPython
from IPython.Shell import IPShellEmbed
ipshell = IPShellEmbed()
ipshell()
When I started with python, started out with main methods from the examples. This was because I didn't know better, after that I found this on how to create a better main method.
Sequential imports overwrite:
If you import two files like this:
from foo import *
from bar import *
If both foo.py and bar.py have a function named fubar(), having imported the files this way, when you call fubar, fubar as defined in bar.py will be executed. The best way to avoid this is to do this:
import foo
import bar
and then call foo.fubar or bar.fubar. This way, you ALWAYS know which file's definition of fubar will be executed.
Maybe a touch more advanced, but I wish I'd known that you don't use threads to take advantage of multiple cores in (C)python. You use the multiprocessing library.
Tab completion and general readline support, including histories, even in the regular python shell.
$ cat ~/.pythonrc.py
#!/usr/bin/env python
try:
import readline
except ImportError:
print("Module readline not available.")
else:
import rlcompleter
readline.parse_and_bind("tab: complete")
import os
histfile = os.path.join(os.environ["HOME"], ".pyhist")
try:
readline.read_history_file(histfile)
except IOError:
pass
import atexit
atexit.register(readline.write_history_file, histfile)
del os, histfile
and then add a line to your .bashrc
export PYTHONSTARTUP=~/.pythonrc.py
These two things lead to an exploratory programming style of "it looks like this library might do what I want", so then I fire up the python shell and then poke around using tab-completion and the help() command until I find what I need.
Generators and list comprehensions are more useful than you might think. Don't just ignore them.
I wish I knew well a functional language. After playing a bit with Clojure, I realized that lots of Python's functional ideas are borrowed from Lisp or other functional langs
I wish I'd known right off the bat how to code idiomatically in Python. You can pick up any language you like and start coding in it like it's C, Java, etc. but ideally you'll learn to code in "the spirit" of the language. Python is particularly relevant, as I think it has a definite style of its own.
While I found it a little later in my Python career than I would have liked, this excellent article wraps up many Python idioms and the little tricks that make it special. Several of the things people have mentioned in their answers so far are contained within:
Code Like a Pythonista: Idiomatic Python.
Enjoy!
Pretty printing:
>>> print "%s world" %('hello')
hello world
%s for string
%d for integer
%f for float
%.xf for exactly x many decimal places of a float. If the float has lesser decimals that indicated, then 0s are added
I really like list comprehension and all other semifunctional constructs. I wish I had known those when I was in my first Python project.
What I really liked: List comprehensions, closures (and high-order functions), tuples, lambda functions, painless bignumbers.
What I wish I had known about sooner: The fact that using Python idioms in code (e.g. list comprehensions instead of loops over lists) was faster.
That multi-core was the future. Still love Python. It's writes a fair bit of my code for me.
Functional programming tools, like all and any
There seems to be a lot of heated discussion on the net about the changes to the reduce() function in python 3.0 and how it should be removed. I am having a little difficulty understanding why this is the case; I find it quite reasonable to use it in a variety of cases. If the contempt was simply subjective, I cannot imagine that such a large number of people would care about it.
What am I missing? What is the problem with reduce()?
As Guido says in his The fate of reduce() in Python 3000 post:
So now reduce(). This is actually the one I've always hated most, because, apart from a few examples involving + or *, almost every time I see a reduce() call with a non-trivial function argument, I need to grab pen and paper to diagram what's actually being fed into that function before I understand what the reduce() is supposed to do. So in my mind, the applicability of reduce() is pretty much limited to associative operators, and in all other cases it's better to write out the accumulation loop explicitly.
There is an excellent example of a confusing reduce in the Functional Programming HOWTO article:
Quick, what's the following code doing?
total = reduce(lambda a, b: (0, a[1] + b[1]), items)[1]
You can figure it out, but it takes time to disentangle the expression to figure out
what's going on. Using a short nested def statements makes things a little bit better:
def combine (a, b):
return 0, a[1] + b[1]
total = reduce(combine, items)[1]
But it would be best of all if I had simply used a for loop:
total = 0
for a, b in items:
total += b
Or the sum() built-in and a generator expression:
total = sum(b for a,b in items)
Many uses of reduce() are clearer when written as for loops.
reduce() is not being removed -- it's simply being moved into the functools module. Guido's reasoning is that except for trivial cases like summation, code written using reduce() is usually clearer when written as an accumulation loop.
People worry it encourages an obfuscated style of programming, doing something that can be achieved with clearer methods.
I'm not against reduce myself, I also find it a useful tool sometimes.
The primary reason of reduce's existence is to avoid writing explicit for loops with accumulators. Even though python has some facilities to support the functional style, it is not encouraged. If you like the 'real' and not 'pythonic' functional style - use a modern Lisp (Clojure?) or Haskell instead.
Using reduce to compute the value of a polynomial with Horner's method is both compact and expressive.
Compute polynomial value at x.
a is an array of coefficients for the polynomial
def poynomialValue(a,x):
return reduce(lambda value, coef: value*x + coef, a)