Develop Reference

Why does bytearray_obj.extend(bytes) differ from bytearray_obj += bytes?

Just saw (and enjoyed) the video of Brandon Rhodes talking on PyCon 2015 about bytearrays.
He said that .extend method is slow, but += operation is implemented differently and is much more efficient. Yes, indeed:
>>> timeit.timeit(setup='ba=bytearray()', stmt='ba.extend(b"xyz123")', number=1000000)
0.19515220914036036
>>> timeit.timeit(setup='ba=bytearray()', stmt='ba += b"xyz123"', number=1000000)
0.09053478296846151
What is the reason of having two ways of extending a bytearray? Are they performing exactly the same task? If not, what is the difference? Which one should be used when?

What is the reason of having two ways of extending a bytearray?
An operator is not chainable like function calls, whereas a method is.
The += operator cannot be used with nonlocal variables.
The += is slightly faster
.extend() may/might/could sometimes possibly be more readable
Are they performing exactly the same task?
Depends on how you look at it. The implementation is not the same, but the result usually is. For a bunch of examples and explanations, maybe try the SO search, and for example this question: Concatenating two lists - difference between '+=' and extend()
Which one should be used when?
If you care about the small performance difference, use the operator when you can. Other than that, just use whichever you like to look at, of course given the restrictions I mentioned above.
But the main question is why += and .extend do not share the same internal function to do the actual work of extending a bytearray.
Because one is faster but has limitations, so we need the other for the cases where we do have the limitations.
Bonus:
The increment operator might cause some funny business with tuples:
if you put a list in a tuple and use the += operator to extend the list, the increment succeeds and you get a TypeError
Source: https://emptysqua.re/blog/python-increment-is-weird-part-ii/

Side-effects in Python map (Python "do" block) [duplicate]

This question already has answers here:
Is it Pythonic to use list comprehensions for just side effects?
(7 answers)
Closed 4 months ago.
What is the preferred way to tell someone "I want to apply func to each element in iterable for its side-effects"?
Option 1... clear, but two lines.
for element in iterable:
func(element)
Option 2... even more lines, but could be clearer.
def walk_for_side_effects(iterable):
for element in iterable:
pass
walk_for_side_effects(map(func, iterable)) # Assuming Python3's map.
Option 3... builds up a list, but this how I see everyone doing it.
[func(element) for element in iterable]
I'm liking Option 2; is there a function in the standard library that is already the equivalent?

Avoid the temptation to be clever. Use option 1, it's intent is clear and unambiguous; you are applying the function func() to each and every element in the iterable.
Option 2 just confuses everyone, looking for what walk_for_side_effects is supposed to do (it certainly puzzled me until I realized you needed to iterate over map() in Python 3).
Option 3 should be used when you actually get results from func(), never for the side effects. Smack anyone doing that just for the side-effects. List comprehensions should be used to generate a list, not to do something else. You are instead making it harder to comprehend and maintain your code (and building a list for all the return values is slower to boot).

This has been asked many times, e.g., here and here. But it's an interesting question, though. List comprehensions are meant to be used for something else.
Other options include
use map() - basically the same as your sample
use filter() - if your function returns None, you will get an empty list
Just a plain for-loop
while the plain loop is the preferable way to do it. It is semantically correct in this case, all other ways, including list comprehension, abuse concepts for their side-effect.
In Python 3.x, map() and filter() are generators and thus do nothing until you iterate over them. So we'd need, e.g., a list(map(...)), which makes it even worse.

List comprehensions for side effects - idiomatically correct or an abomination? [duplicate]

This question already has answers here:
Is it Pythonic to use list comprehensions for just side effects?
(7 answers)
Closed 4 years ago.
I have a list of cheese objects. Cheese has a method that does a bunch of stuff with the db and whatnot, called out_of_stock().
So:
[cheese.out_of_stock() for cheese in cheeses]
It feels sloppy to me. Like I'm doing something (creating a list) for a bunch of side effects (zeroing inventory).
Am I being foolish? Is this subjective to the method?

List comprehensions are for creating a list. That's not what you're doing here, so I'd avoid it.

Personally, I would do
for cheese in cheeses:
cheese.out_of_stock()
This feels a bit more explicit to me. It's another line, but I suspect that another programmer reading that code would understand it well, whereas the list comprehension version might be slightly confusing. In addition, I could imagine that someone else might come along later (not knowing that out_of_stock doesn't return a value) and try to use that list later.
I guess there's nothing wrong with it, though. Ultimately, it comes down to who's going to be reading that code later (yourself or others) and whether or not they'll know what's going on.
EDIT: Also, you're using a lovely functional programming construct taken from Haskell to do nasty stateful things...

Semantically, if cheese.out_of_stock returned a boolean, then it would be fine. However this is not very recommended if cheese.out_of_stock does not return a value, then it would be very confusing and unmaintainable.
import this
for more information

List comprehensions are usually used for filtering (or something similar where the result of the comprehsion is being used later) but not for replacing a loop calling a method on the iteration data. So this works but it locks ugly.

If you are doing something with the return values of cheese.out_of_stock() this is perfectly Pythonic. If you are discarding the result of the list comprehension, it is bad style.
Creating the list is a waste of time and memory, just use the two line for loop from Peter's answer

The answer hinges entirely on whether you want the list. A list comprehension is often a good replacement for a loop like this:
stock = []
for cheese in cheeses:
stock.append(cheese.out_of_stock())
But rarely if ever for a loop like this:
for cheese in cheeses:
cheese.out_of_stock()
When people read a list comprehension, they don't typically expect it to have side effects - aside from something like cacheing that doesn't affect the meaning of the program, and, of course, building a list. Certainly, any side effects it has shouldn't be necessary for your program's correctness - this also means they can be worth avoiding for the first type of loop if you're using an API without command-query separation - that is, even if you do want the list, if .out_of_stock() also has side-effects that you depend on, you should consider using the loop rather than the comprehension.

It's ugly, don't do it.
There is a quite similar way of doing it that is not ugly. It requires a pre-defined function like this:
def consume(iter):
for i in iter:
pass
Now you can write code like this:
consume(cheese.out_of_stock() for cheese in cheeses)
So really, this is just Peters suggestion wrapped in a function which is called with a generator expression. And though much better than the list comprehension, it's still not exactly beautiful. It would really be best (most explicit) to just use the for loop directly.

for cheese in cheeses: cheese.out_of_stock()
Still a one liner and avoids the side effect of creating an unnecessary list.

Learning Python from Ruby; Differences and Similarities

I know Ruby very well. I believe that I may need to learn Python presently. For those who know both, what concepts are similar between the two, and what are different?
I'm looking for a list similar to a primer I wrote for Learning Lua for JavaScripters: simple things like whitespace significance and looping constructs; the name of nil in Python, and what values are considered "truthy"; is it idiomatic to use the equivalent of map and each, or are mumble somethingaboutlistcomprehensions mumble the norm?
If I get a good variety of answers I'm happy to aggregate them into a community wiki. Or else you all can fight and crib from each other to try to create the one true comprehensive list.
Edit: To be clear, my goal is "proper" and idiomatic Python. If there is a Python equivalent of inject, but nobody uses it because there is a better/different way to achieve the common functionality of iterating a list and accumulating a result along the way, I want to know how you do things. Perhaps I'll update this question with a list of common goals, how you achieve them in Ruby, and ask what the equivalent is in Python.

Here are some key differences to me:
Ruby has blocks; Python does not.
Python has functions; Ruby does not. In Python, you can take any function or method and pass it to another function. In Ruby, everything is a method, and methods can't be directly passed. Instead, you have to wrap them in Proc's to pass them.
Ruby and Python both support closures, but in different ways. In Python, you can define a function inside another function. The inner function has read access to variables from the outer function, but not write access. In Ruby, you define closures using blocks. The closures have full read and write access to variables from the outer scope.
Python has list comprehensions, which are pretty expressive. For example, if you have a list of numbers, you can write
[x*x for x in values if x > 15]
to get a new list of the squares of all values greater than 15. In Ruby, you'd have to write the following:
values.select {|v| v > 15}.map {|v| v * v}
The Ruby code doesn't feel as compact. It's also not as efficient since it first converts the values array into a shorter intermediate array containing the values greater than 15. Then, it takes the intermediate array and generates a final array containing the squares of the intermediates. The intermediate array is then thrown out. So, Ruby ends up with 3 arrays in memory during the computation; Python only needs the input list and the resulting list.
Python also supplies similar map comprehensions.
Python supports tuples; Ruby doesn't. In Ruby, you have to use arrays to simulate tuples.
Ruby supports switch/case statements; Python does not.
Ruby supports the standard expr ? val1 : val2 ternary operator; Python does not.
Ruby supports only single inheritance. If you need to mimic multiple inheritance, you can define modules and use mix-ins to pull the module methods into classes. Python supports multiple inheritance rather than module mix-ins.
Python supports only single-line lambda functions. Ruby blocks, which are kind of/sort of lambda functions, can be arbitrarily big. Because of this, Ruby code is typically written in a more functional style than Python code. For example, to loop over a list in Ruby, you typically do
collection.each do |value|
...
end
The block works very much like a function being passed to collection.each. If you were to do the same thing in Python, you'd have to define a named inner function and then pass that to the collection each method (if list supported this method):
def some_operation(value):
...
collection.each(some_operation)
That doesn't flow very nicely. So, typically the following non-functional approach would be used in Python:
for value in collection:
...
Using resources in a safe way is quite different between the two languages. Here, the problem is that you want to allocate some resource (open a file, obtain a database cursor, etc), perform some arbitrary operation on it, and then close it in a safe manner even if an exception occurs.
In Ruby, because blocks are so easy to use (see #9), you would typically code this pattern as a method that takes a block for the arbitrary operation to perform on the resource.
In Python, passing in a function for the arbitrary action is a little clunkier since you have to write a named, inner function (see #9). Instead, Python uses a with statement for safe resource handling. See How do I correctly clean up a Python object? for more details.

I, like you, looked for inject and other functional methods when learning Python. I was disappointed to find that they weren't all there, or that Python favored an imperative approach. That said, most of the constructs are there if you look. In some cases, a library will make things nicer.
A couple of highlights for me:
The functional programming patterns you know from Ruby are available in Python. They just look a little different. For example, there's a map function:
def f(x):
return x + 1
map(f, [1, 2, 3]) # => [2, 3, 4]
Similarly, there is a reduce function to fold over lists, etc.
That said, Python lacks blocks and doesn't have a streamlined syntax for chaining or composing functions. (For a nice way of doing this without blocks, check out Haskell's rich syntax.)
For one reason or another, the Python community seems to prefer imperative iteration for things that would, in Ruby, be done without mutation. For example, folds (i.e., inject), are often done with an imperative for loop instead of reduce:
running_total = 0
for n in [1, 2, 3]:
running_total = running_total + n
This isn't just a convention, it's also reinforced by the Python maintainers. For example, the Python 3 release notes explicitly favor for loops over reduce:
Use functools.reduce() if you really need it; however, 99 percent of the time an explicit for loop is more readable.
List comprehensions are a terse way to express complex functional operations (similar to Haskell's list monad). These aren't available in Ruby and may help in some scenarios. For example, a brute-force one-liner to find all the palindromes in a string (assuming you have a function p() that returns true for palindromes) looks like this:
s = 'string-with-palindromes-like-abbalabba'
l = len(s)
[s[x:y] for x in range(l) for y in range(x,l+1) if p(s[x:y])]
Methods in Python can be treated as context-free functions in many cases, which is something you'll have to get used to from Ruby but can be quite powerful.
In case this helps, I wrote up more thoughts here in 2011: The 'ugliness' of Python. They may need updating in light of today's focus on ML.

My suggestion: Don't try to learn the differences. Learn how to approach the problem in Python. Just like there's a Ruby approach to each problem (that works very well givin the limitations and strengths of the language), there's a Python approach to the problem. they are both different. To get the best out of each language, you really should learn the language itself, and not just the "translation" from one to the other.
Now, with that said, the difference will help you adapt faster and make 1 off modifications to a Python program. And that's fine for a start to get writing. But try to learn from other projects the why behind the architecture and design decisions rather than the how behind the semantics of the language...

I know little Ruby, but here are a few bullet points about the things you mentioned:
nil, the value indicating lack of a value, would be None (note that you check for it like x is None or x is not None, not with == - or by coercion to boolean, see next point).
None, zero-esque numbers (0, 0.0, 0j (complex number)) and empty collections ([], {}, set(), the empty string "", etc.) are considered falsy, everything else is considered truthy.
For side effects, (for-)loop explicitly. For generating a new bunch of stuff without side-effects, use list comprehensions (or their relatives - generator expressions for lazy one-time iterators, dict/set comprehensions for the said collections).
Concerning looping: You have for, which operates on an iterable(! no counting), and while, which does what you would expect. The fromer is far more powerful, thanks to the extensive support for iterators. Not only nearly everything that can be an iterator instead of a list is an iterator (at least in Python 3 - in Python 2, you have both and the default is a list, sadly). The are numerous tools for working with iterators - zip iterates any number of iterables in parallel, enumerate gives you (index, item) (on any iterable, not just on lists), even slicing abritary (possibly large or infinite) iterables! I found that these make many many looping tasks much simpler. Needless to say, they integrate just fine with list comprehensions, generator expressions, etc.

In Ruby, instance variables and methods are completely unrelated, except when you explicitly relate them with attr_accessor or something like that.
In Python, methods are just a special class of attribute: one that is executable.
So for example:
>>> class foo:
... x = 5
... def y(): pass
...
>>> f = foo()
>>> type(f.x)
<type 'int'>
>>> type(f.y)
<type 'instancemethod'>
That difference has a lot of implications, like for example that referring to f.x refers to the method object, rather than calling it. Also, as you can see, f.x is public by default, whereas in Ruby, instance variables are private by default.

Idiomatic Python: 'times' loop [duplicate]

This question already has answers here:
Is it possible to implement a Python for range loop without an iterator variable?
(15 answers)
Closed 7 years ago.
Say I have a function foo that I want to call n times. In Ruby, I would write:
n.times { foo }
In Python, I could write:
for _ in xrange(n): foo()
But that seems like a hacky way of doing things.
My question: Is there an idiomatic way of doing this in Python?

You've already shown the idiomatic way:
for _ in range(n): # or xrange if you are on 2.X
foo()
Not sure what is "hackish" about this. If you have a more specific use case in mind, please provide more details, and there might be something better suited to what you are doing.

If you want the times method, and you need to use it on your own functions, try this:
def times(self, n, *args, **kwargs):
for _ in range(n):
self.__call__(*args, **kwargs)
import new
def repeatable(func):
func.times = new.instancemethod(times, func, func.__class__)
return func
now add a #repeatable decorator to any method you need a times method on:
#repeatable
def foo(bar):
print bar
foo.times(4, "baz") #outputs 4 lines of "baz"

Fastest, cleanest is itertools.repeat:
import itertools
for _ in itertools.repeat(None, n):
foo()

The question pre-supposes that calling foo() n times is an a priori necessary thing. Where did n come from? Is it the length of something iterable? Then iterate over the iterable. As I am picking up Python, I find that I'm using few to no arbitrary values; there is some more salient meaning behind your n that got lost when it became an integer.
Earlier today I happened upon Nicklaus Wirth's provocative paper for IEEE Computer entitled Good Ideas - Through the Looking Glass (archived version for future readers). In section 4 he brings a different slant on programming constructs that everyone (including himself) has taken for granted but that hold expressive flaws:
"The generality of Algol’s for
statement should have been a warning
signal to all future designers to
always keep the primary purpose of a
construct in mind, and to be weary of
exaggerated generality and complexity,
which may easily become
counter-productive."
The algol for is equivalent to the C/Java for, it just does too much. That paper is a useful read if only because it makes one not take for granted so much that we so readily do. So perhaps a better question is "Why would you need a loop that executes an arbitrary number of times?"