I have a filter in Python3 that I am applying with a lambda function. Here is my function:
affy_reader = csv.DictReader(filter(lambda row:
not row[0].startswith('#') and
str(row[0]).isdigit(),
file_pointer),
delimiter='\t',
fieldnames=affy_column_headers)
Is there a way that I can print the value of row from within this lambda function? I think I need to do it like this because row is only scoped within the lambda. For example, were this a LISP Lambda procedure, I believe I could do something like this:
affy_reader = csv.DictReader(filter(lambda row: print(row)
not row[0].startswith('#') and
str(row[0]).isdigit(),
file_pointer),
delimiter='\t',
fieldnames=affy_column_headers)
Because the print() is read and executed in-line. Is there some way in Python to do this? Or if not, what is a good way for me to see this value? Thanks!
*I realize my "LISP" example is Python not LISP. I was just trying to illustrate further what it is I am trying to do.
I don't think you can do it with a lambda. Just define a helper function. That way you can also display whether a certain row is or isn't going to be filtered:
def filter_plus_print(row):
result = not row[0].startswith('#') and str(row[0]).isdigit()
print("Keeping:" if result else "Removing:", row)
return result
and then do
affy_reader = csv.DictReader(filter(filter_plus_print, file_pointer),
delimiter='\t',
fieldnames=affy_column_headers)
General Python 3.x trick to print debug info in a lambda without changing its semantics:
Original:
lambda: 4
Instrumented:
lambda: (print (3), 4) [1]
Explanation:
Both arguments of the tuple will be evaluated.
The left one, print (3) in the example, is relying purely on a sideeffect, i.e. in this case it will print something. You can call any function here, since any Python function will return a value. This value will be the Python predefined value None if a return statement is missing. This doesn't matter since the return value isn't used anywhere.
The second argument, 4 in the example, can be any expression including a function call or a call to a functor (object with overloaded round brackets). This argument is returned by the lambda function by selecting [1], i.e. the second element of the tuple (indexing in Python starts at 0].
The reason why this works specifically in Python 3.x is that print is a "perfectly ordinary" function there, whereas in Python 2.x it was a statement.
You can view the values within file_pointer by printing it directly.
print(file_pointer)
affy_reader = csv.DictReader(filter(lambda row:
not row[0].startswith('#') and
str(row[0]).isdigit(),
file_pointer),
delimiter='\t',
fieldnames=affy_column_headers)
There is no pythonic way to execute multiple statements in an anonymous function. I would suggest breaking the lambda out to a named function, like so:
def relevant(row):
print(row)
return not row[0].startswith('#') and str(row[0]).isdigit()
affy_reader = csv.DictReader(filter(relevant, file_pointer),
delimiter='\t',
fieldnames=affy_column_headers)
The problems is that print(row) returns None and you have to do something with it. Rather unpleasant in most cases.
Related
I have sample dataset df like;
tel_no
tel:+1-860-752-8792
tel:+1-949-722-8838
Th goal is to get the output as;
tel_no
18607528792
19497228838
Here is my attempt;
df['tel_no'].apply(lambda x: x.replace(i, '') for i in ['+','-','tel:'])
But this gives an error message;
TypeError: 'generator' object is not callable
I am aware that it can be done in 3 separate lines, on for each character. But I was wondering can we do it one line as above. Help is appreciated.
An easier way to go would be to use pandas str methods, namely findall (to find all digits using the regex \d+) and join (to join the resulting list of digit substrings together):
>>> df.tel_no.str.findall("\d+").str.join("")
0 18607528792
1 19497228838
Name: tel_no, dtype: object
I agree that using regex matching is a good solution to your problem, but I can at least address the problem with your code.
You current code is:
df['tel_no'].apply(lambda x: x.replace(i, '') for i in ['+','-','tel:'])
Python parses this (perhaps surprisingly) as:
df['tel_no'].apply(
(
(lambda x: x.replace(i, ''))
for i in ['+','-','tel:'])
)
)
That is, you have written a generator comprehension, creating a new anonymous function at each iteration of the loop. You have not created a single anonymous function with a generator comprehension inside it!
Obviously, generators are not callable, which is what caused the error.
Your attempt reflects two additional misunderstandings:
Comprehension syntax cannot be used outside of an actual comprehension. Perhaps you meant to write lambda x: (x.replace(i, '')) for i in ['+','-','tel:']), which would at least be one function that contains a generator comprehension.
String functions like str.replace do not modify the string. They return a new string. See the example below.
s1 = 'hello'
s2 = s1.replace('e', 'f')
# s1 will be unchanged
assert s1 == 'hello'
# s2 will be changed
assert s2 == 'hfllo'
To write this as a function, you would need to use def, not `lambda:
def clean_tel(x):
for bad_string in ['+', '-', 'tel:']:
x = x.replace(bad_string, '')
return x
df['tel_no'].apply(clean_tel)
Or you can omit the loop and write it like this:
df['tel_no'].apply(
lambda x: x.replace('+', '').replace('-', '').replace('tel:', '')
)
Seeking guidance to understand a lambda-map function. In the below, I see that the file "feedback" is read line by line and stored in a list "feedback". I'm unable to get my head around the variable x. I don't see the variable "x" declared anywhere. Can someone help me understand the statement?Thanks in advance
f = open('feedback.txt','r')
feedback = list(map(lambda x:x[:-1],f.readlines())
f.close()
The map function will execute the given function for every element in the list.
In your code the map function will get lambda x:x[:-1].
You can read that like: for every x in f.readlines() return everything except the last element of x.
So x will be every line the file. lambda x: you could see as def thing(x):.
I replaced lambda with a standard func:
def read_last(x): #x means a line
return x[:-1]
f = open('feedback.txt','r')
feedback = list(map(read_last, f.readlines())
f.close()
Maybe it will help.
lambda function is a simple anonymous function that takes any number of arguments, but has only one expression.
lambda arguments : expression
It is anonymous because we have not assigned it to an object, and thus it has no name.
example f and g are somewhat same:
def f(x):
# take a string and return all but last value
return x[:-1]
g = lambda x: x[:-1]
so:
f('hello') == g('hello') #True ->'hell'
But g is not how we would use lambda. The whole aim is to avoid assigning ;)
Now map takes in a function and applies it to an iteratable:it returns a generator in Python 3+ and thus a list is used to case that generator to a list
data = ['we are 101','you are 102','they are 103']
print(list(map(lambda x:x[:-1],data)))
#->['we are 10','you are 10','they are 10']
In principle, same as passing a function:
data = ['we are 101','you are 102','they are 103']
print(list(map(f,data)))
but often faster and awesome. I love lambdas
Keep in mind, while explaining lambda is solved here, it is not the implementation of choice for your particular example. Suggestion:
f = open('feedback.txt', 'r')
feedback = f.read().splitlines()
f.close()
See also 'Reading a file without newlines'.
I am a c++ guy, learning the lambda function in python and wanna know it inside out. did some seraches before posting here. anyway, this piece of code came up to me.
<1> i dont quite understand the purpose of lambda function here. r we trying to get a function template? If so, why dont we just set up 2 parameters in the function input?
<2> also, make_incrementor(42), at this moment is equivalent to return x+42, and x is the 0,1 in f(0) and f(1)?
<3> for f(0), does it not have the same effect as >>>f = make_incrementor(42)? for f(0), what are the values for x and n respectively?
any commments are welcome! thanks.
>>> def make_incrementor(n):
... return lambda x: x + n
...
>>> f = make_incrementor(42)
>>> f(0)
42
>>> f(1)
43
Yes, this is similar to a C++ int template. However, instead of at compile time (yes, Python (at least for CPython) is "compiled"), the function is created at run time. Why the lambda is used in this specific case is unclear, probably only for demonstration that functions can be returned from other functions rather than practical use. Sometimes, however, statements like this may be necessary if you need a function taking a specified number of arguments (e.g. for map, the function must take the same number of arguments as the number of iterables given to map) but the behaviour of the function should depend on other arguments.
make_incrementor returns a function that adds n (here, 42) to any x passed to that function. In your case the x values you tried are 0 and `1``
f = make_incrementor(42) sets f to a function that returns x + 42. f(0), however, returns 0 + 42, which is 42 - the returned types and values are both different, so the different expressions don't have the same effect.
The purpose is to show a toy lambda return. It lets you create a function with data baked in. I have used this less trivial example of a similar use.
def startsWithFunc(testString):
return lambda x: x.find(testString) == 0
Then when I am parsing, I create some functions:
startsDesctription = startsWithFunc("!Sample_description")
startMatrix = startsWithFunc("!series_matrix_table_begin")
Then in code I use:
while line:
#.... other stuff
if startsDesctription(line):
#do description work
if startMatrix(line):
#do matrix start work
#other stuff ... increment line ... etc
Still perhaps trival, but it shows creating general funcitons with data baked it.
I've got this block of code in a real Django function. If certain conditions are met, items are added to the list.
ret = []
if self.taken():
ret.append('taken')
if self.suggested():
ret.append('suggested')
#.... many more conditions and appends...
return ret
It's very functional. You know what it does, and that's great...
But I've learned to appreciate the beauty of list and dict comprehensions.
Is there a more Pythonic way of phrasing this construct, perhaps that initialises and populates the array in one blow?
Create a mapping dictionary:
self.map_dict = {'taken': self.taken,
'suggested': self.suggested,
'foo' : self.bar}
[x for x in ['taken', 'suggested', 'foo'] if self.map_dict.get(x, lambda:False)()]
Related: Most efficient way of making an if-elif-elif-else statement when the else is done the most?
Not a big improvement, but I'll mention it:
def populate():
if self.taken():
yield 'taken'
if self.suggested():
yield 'suggested'
ret = list(populate())
Can we do better? I'm skeptical. Clearly there's a need of using another syntax than a list literal, because we no longer have the "1 expression = 1 element in result" invariant.
Edit:
There's a pattern to our data, and it's a list of (condition, value) pairs. We might try to exploit it using:
[value
for condition, value
in [(self.taken(), 'taken'),
(self.suggested(), 'suggested')]
if condition]
but this still is a restriction for how you describe your logic, still has the nasty side effect of evaluating all values no matter the condition (unless you throw in a ton of lambdas), and I can't really see it as an improvement over what we've started with.
For this very specific example, I could do:
return [x for x in ['taken', 'suggested', ...] if getattr(self, x)()]
But again, this only works where the item and method it calls to check have the same name, ie for my exact code. It could be adapted but it's a bit crusty. I'm very open to other solutions!
I don't know why we are appending strings that match the function names, but if this is a general pattern, we can use that. Functions have a __name__ attribute and I think it always contains what you want in the list.
So how about:
return [fn.__name__ for fn in (self.taken, self.suggested, foo, bar, baz) if fn()]
If I understand the problem correctly, this works just as well for non-member functions as for member functions.
EDIT:
Okay, let's add a mapping dictionary. And split out the function names into a tuple or list.
fns_to_check = (self.taken, self.suggested, foo, bar, baz)
# This holds only the exceptions; if a function isn't in here,
# we will use the .__name__ attribute.
fn_name_map = {foo:'alternate', bar:'other'}
def fn_name(fn):
"""Return name from exceptions map, or .__name__ if not in map"""
return fn_name_map.get(fn, fn.__name__)
return [fn_name(fn) for fn in fns_to_check if fn()]
You could also just use #hcwhsa's mapping dictionary answer. The main difference here is I'm suggesting just mapping the exceptions.
In another instance (where a value will be defined but might be None - a Django model's fields in my case), I've found that just adding them and filtering works:
return filter(None, [self.user, self.partner])
If either of those is None, They'll be removed from the list. It's a little more intensive than just checking but still fairly easy way of cleaning the output without writing a book.
One option is to have a "sentinel"-style object to take the place of list entries that fail the corresponding condition. Then a function can be defined to filter out the missing items:
# "sentinel indicating a list element that should be skipped
Skip = object()
def drop_missing(itr):
"""returns an iterator yielding all but Skip objects from the given itr"""
return filter(lambda v: v is not Skip, itr)
With this simple machinery, we come reasonably close to list-comprehension style syntax:
return drop_skips([
'taken' if self.taken else Skip,
'suggested' if self.suggested else Skip,
100 if self.full else Skip,
// many other values and conditions
])
ret = [
*('taken' for _i in range(1) if self.taken()),
*('suggested' for _i in range(1) if self.suggested()),
]
The idea is to use the list comprehension syntax to construct either a single element list with item 'taken', if self.taken() is True, or an empty list, if self.taken() is False, and then unpack it.
Assume you have a function, that sometimes returns a value, and sometimes doesn't, because there really is nothing you could return in this case, not even a default value or something. Now you want to do something with the result, but of course only when there is one.
Example:
result = function_call(params)
if result:
print result
Is there a way to write this in a more pythonic way, maybe even in one line?
Like that:
print function_call(params) or #nothing
(Note that I mean it shouldn't print "nothing" or "None". It should actually just not print at all, if the result is None)
No; in Python, name binding is a statement and so cannot be used as an expression within a statement. Since print is also a statement you're going to require 3 lines; in Python 3 you could write:
result = function_call(params)
print(result) if result else None
This isn't quite true for name binding within a comprehension or generator, where name binding is a syntax item that has statement-like semantics:
[print(result) for result in generator_call(params) if result]
As Kos says, you can abuse this to create a one-element comprehension:
[print(result) for result in (function_call(params), ) if result]
Another syntax item that performs name binding and can similarly be abused is the lambda expression:
(lambda result: print(result) if result else None)(function_call(params))
Note that in both these cases the operation on the return value must be an expression and not a statement.
I think the more Pythonic version is actually closer to your original:
result = function_call(params)
if result is not None:
do_something(result)
Checking for is (not) None seems very idiomatic to me - I've used it several times myself and I've also seen it used elsewhere[citation-needed].
From the answers up to now I would do that:
>>> from __future__ import print_function #if Python2.7
>>> def filtered_print(txt):
... txt and print(txt)
...
>>> filtered_print('hello world')
hello world
>>> filtered_print('None')
None
>>> filtered_print(None)
>>>
If someone else has a better solution in mind, I am still open for alternatives, though!