difference between "function()" and "function" - python

I see
df["col2"] = df["col1"].apply(len)
len(df["col1"])
My question is,
Why use "len" function without parenthesis in 1, but use it with parenthesis in 2?
What is the difference between the two?
I see this kind of occasion a lot, where using a function with and without parenthesis.
Can someone explain to me what exactly is going on?
Thanks.

The first example that you mentioned(the above code) maps the function len to the target variable df["col1"]
df["col2"] = df["col1"].apply(len)
Whenever we have to map a function to any iterable object, the syntax needs the function to be without parenthesis.
In your case, df["col1"] must be having elements whose length can be calculated. And it will return a Pandas Series will lengths of all the elements.
Take the following example.
a = ["1", "2","3","4"]
z = list( map( int, a ) ) >> [1, 2, 3, 4]
Here, we mapped the builtin int function(which does typecasting), to the entire list.
The second example that you mentioned would give out the length of the df["col1"] series.
len(df["col1"])
It won't do any operations on the elements within that Series.
Take the following example.
a = ["1", "2","3","4"]
z = len(a) >> 4
Since, on both the occasions, the function len was fed an iterable object, it didn't give any error. But, the outputs are completely different as I explained!

In 1, the function len is being passed to a method called apply. That method presumably will apply the function len along the first axis (probably returning something like a list of lengths). In 2, the function len is being called directly, with an argument df["col2"], presumably to get the length of the data frame.
The use in 1 is sometimes called a "higher order function", but in principle it's just passing a function to another function for it to use.

In the second case you are directly calling the len method and will get the result, i.e. how many rows are in col1 in the df.
In the first you are giving the reference to the len function to the apply function.
This is a shortcut for df["col2"] = df["col1"].apply(lambda x: len(x))
This version you use if you want to make the behavior of a method flexible by letting the user of the method hand in the function to influence some part of an algorithm. Like here in the case with the apply method. Depending of the conents in the column you want to fill the new column with something, and here it was decided to fill this with the lengths of the content of other column.

len(s) will return the lenght of the s variable
len will return the function itslelf. So if I do a=len, then I can do a(s). Of course, it is not recommended to do such thing as a=len.

Let's have a look at the documentation of DataFrame.apply:
its first parameter is func: function which is a function that we'll apply to each column or row of the DataFrame. In your case this function is len().
Now let's see what happens when you pass len as a parameter with parenthesis:
df.apply(len())
-----------------------------------------------------------------------------------
TypeError Traceback (most recent call last)
/tmp/ipykernel_11920/3211016940.py in <module>
----> 1 df.apply(len())
TypeError: len() takes exactly one argument (0 given)
While this perfectly works when we use df.apply(len).
This is because your parameter must be a function and the way Python uses to distinguish between functions and the return value of the call to a function is the use of parenthesis in the second case.

Related

How to pass arguments when using apply in DataFrame

I want to apply a function along a column, eg coco.convert, and pass to it, along with the value in the row, the argument to = 'ISO3'.
df['alpha-3'] = df['Country'].apply(coco.convert('ISO3'))
misses 1 positional argument:
TypeError: convert() missing 1 required positional argument: 'names'
but
df['alpha-3'] = df['Country'].apply(coco.convert)
works fine (I assume it uses default values).
How do I pass the positional argument here?
Also, what exactly is happening here - can someone explain a little how the function is passed to apply?
Probably the most normal way to solve this would be to use the **kwds argument for .apply which are covered in the apply documentation. Basically you can just pass any other named arguments to .apply() and it uses them with the passed function.
df['alpha-3'] = df['Country'].apply(coco.convert, to='ISO3')
An alternative way to do it would be to define your own new function with the arguments pre-passed such as below.
def my_fun(x):
return coco.convert(names = x, to = 'ISO3')
df['alpha-3'] = df['Country'].apply(my_fun)
To your other about how .apply works...
Take the column of data you give it, and loop through each element
For each element, feed its value into the supplied function as the *first
For each return of that function, convert it into another Pandas series.

dict.get(x,y) always evaluates the right hand argument

try this:
d = {"One":1,"Two":2,"Three":3}
d.get("Two", Print("Fail"))
Expected Output:
2
Actual Output:
Fail
2
Anyone know why the right hand argument of this expression is eagerly evaluated? What's the point of processing the right hand argument if there's a chance it's not returned?
Is there a logic to this I'm missing?
It's always evaluated because any functions has to be resolved before the return value can be sent to get.
Instead, you supply a default value to use if the key is not present, or if you need to check multiple keys to get a value, the get call will return None:
val = d.get("Two")
if val is None:
print("Fail..")
This is similar to:
if 'Two' not in d:
print("Fail..")
If you don't want to provide a default value, the first example can still be useful if you have multiple possible values to check - chaining .get calls or having multiple if .. is None: .. checks all the way down.
If you want lazy evaluation, you should use the or operator
d = {"One":1,"Two":2,"Three":3}
d.get("Two") or (print("Fail"))
Here's the logic.
x = d.get("b", expensiveComputation())
This is computed like so:
Compute "b", its value is the first argument to .get.
Call expensiveComputation(), its value is the second argument to .get.
Call .get with these two arguments
Assign the returned value to x.
If you want short-circuiting, Python is very ergonomic.
x = d.get("a") or expensiveComputation()
This is computed like so:
Compute "a", its value is the first and only argument to .get.
Call .get with the one argument.
Since it's an or operator which requires Booleans, call bool() on the returned value.
If the result is True, use the value returned by .get, else call expensiveComputation(), and use the value returned by it.
Assign that to x.
Of course, not only None is falsy, so are empty lists, empty strings, etc; if such falsy objects may occur among your values, this is not what you want.
Python 3.8 introduces assignment expressions so that you could still write one-liners like so:
x if ((x := d.get("a")) is not None) else expensiveComputation()
Here expensiveComputation() will only run if d does not have a value for "a". But this arguably balances on the verge of unreadable.

'generator' object is not callable when if inside a for loop in list compression

I am getting the following error when i ran this:
df['initial_referrer'].apply(lambda x: value.split("utm_campaign=",1)[1] if 'utm_campaign' in value else np.nan for value in x.split('&'))
TypeError: 'generator' object is not callable
I am not sure the meaning of the error and how to modify my this to get rid of this. I have read couple of similar questions here but could not figure our what could be the issue.
So I have values in df['initial_referrer'] be like:
df['initial_referrer'].head()
0 /login/index.php
1 /login/index.php?utm_source=INTERNAL&utm_medium=EMAIL&utm_campaign=login-day1
2 /login/index.php
3 /login/index.php?utm_source=INTERNAL&utm_medium=EMAIL&utm_campaign=login-day1
4 /login/index.php
And in this, I wanted to extract the value of utm_campaign which is login-day1 thats why I was using the for loop and then if statement it was taking a lot of time/days to process 20mil rows. Therefore I wanted to use generator expression or list compression to process it faster.
It's instructive to first use apply with a regular function:
def func(x):
return [value.split("utm_campaign=",1)[1] if 'utm_campaign' in value else np.nan \
for value in x.split('&')]
df['initial_referrer'].apply(func)
Notice the square brackets representing the list comprehension. You need to translate this to your lambda function:
df['initial_referrer'].apply(lambda x: [value.split("utm_campaign=",1)[1] if 'utm_campaign' in value else np.nan for value in x.split('&')])
But the latter is unreadable. You are better off writing a regular function.
Note pd.Series.apply is a Python-level loop. You can use map instead and will likely see a performance improvement:
df['initial_referrer'] = list(map(func, df['initial_referrer'].values))
Or even a list comprehension:
df['initial_referrer'] = [func(x) for x in df['initial_referrer'].values]

Swap method adaptable to any data structure

I meet this problem when implementing my own swap() method with Python
def swap(a,b):
temp=a
a=b
b=temp
list_=[5,4,6,3,7]
swap(list_[4],list_[2])
I expexted list_ to be updated with swap() call, since list_[4] and list_[2] are to be assigned new values during function call. However, list_ remains unchanged:
list_
[5, 4, 6, 3, 7]
I misunderstand why swap function call is dealing with a copy. I don't want to add a list argument to my swap function nor return the list during swap() call since I want the method to be adaptable to other datastructures, like in
swap(mat[0][1],mat[2,3])
You are off on how Python works:
list_=[5,4,6,3,7]
swap(list_[4],list_[2]) # this is absolutely the same as
swap(7,6)
Python fundamental concept is the use of names and values. When you write down a name in code it stands for the value attached to it during runtime. In this case list_[4] is a name that stands for the value 7.
When you want to change something you must use one of it's names. Here you want to change the list_, so you have to do this:
def swap(data, i1, i2):
data[i1], data[i2] = data[i2], data[i1]
swap(list_, 4,2) # swaps list index 4 and 2
python passes by value so swap does not affect the original list.
just use
list_[4], list_[2] = list_[2], list_[4]

Declaring Unknown Type Variable in Python?

I have a situation in Python(cough, homework) where I need to multiply EACH ELEMENT in a given list of objects a specified number of times and return the output of the elements. The problem is that the sample inputs given are of different types. For example, one case may input a list of strings whose elements I need to multiply while the others may be ints. So my return type needs to vary. I would like to do this without having to test what every type of object is. Is there a way to do this? I know in C# i could just use "var" but I don't know if such a thing exists in Python?
I realize that variables don't have to be declared, but in this case I can't see any way around it. Here's the function I made:
def multiplyItemsByFour(argsList):
output = ????
for arg in argsList:
output += arg * 4
return output
See how I need to add to the output variable. If I just try to take away the output assignment on the first line, I get an error that the variable was not defined. But if I assign it a 0 or a "" for an empty string, an exception could be thrown since you can't add 3 to a string or "a" to an integer, etc...
Here are some sample inputs and outputs:
Input: ('a','b') Output: 'aaaabbbb'
Input: (2,3,4) Output: 36
Thanks!
def fivetimes(anylist):
return anylist * 5
As you see, if you're given a list argument, there's no need for any assignment whatsoever in order to "multiply it a given number of times and return the output". You talk about a given list; how is it given to you, if not (the most natural way) as an argument to your function? Not that it matters much -- if it's a global variable, a property of the object that's your argument, and so forth, this still doesn't necessitate any assignment.
If you were "homeworkically" forbidden from using the * operator of lists, and just required to implement it yourself, this would require assignment, but no declaration:
def multiply_the_hard_way(inputlist, multiplier):
outputlist = []
for i in range(multiplier):
outputlist.extend(inputlist)
return outputlist
You can simply make the empty list "magicaly appear": there's no need to "declare" it as being anything whatsoever, it's an empty list and the Python compiler knows it as well as you or any reader of your code does. Binding it to the name outputlist doesn't require you to perform any special ritual either, just the binding (aka assignment) itself: names don't have types, only objects have types... that's Python!-)
Edit: OP now says output must not be a list, but rather int, float, or maybe string, and he is given no indication of what. I've asked for clarification -- multiplying a list ALWAYS returns a list, so clearly he must mean something different from what he originally said, that he had to multiply a list. Meanwhile, here's another attempt at mind-reading. Perhaps he must return a list where EACH ITEM of the input list is multiplied by the same factor (whether that item is an int, float, string, list, ...). Well then:
define multiply_each_item(somelist, multiplier):
return [item * multiplier for item in somelist]
Look ma, no hands^H^H^H^H^H assignment. (This is known as a "list comprehension", btw).
Or maybe (unlikely, but my mind-reading hat may be suffering interference from my tinfoil hat, will need to go to the mad hatter's shop to have them tuned) he needs to (say) multiply each list item as if they were the same type as the first item, but return them as their original type, so that for example
>>> mystic(['zap', 1, 23, 'goo'], 2)
['zapzap', 11, 2323, 'googoo']
>>> mystic([23, '12', 15, 2.5], 2)
[46, '24', 30, 4.0]
Even this highly-mystical spec COULD be accomodated...:
>>> def mystic(alist, mul):
... multyp = type(alist[0])
... return [type(x)(mul*multyp(x)) for x in alist]
...
...though I very much doubt it's the spec actually encoded in the mysterious runes of that homework assignment. Just about ANY precise spec can be either implemented or proven to be likely impossible as stated (by requiring you to solve the Halting Problem or demanding that P==NP, say;-). That may take some work ("prove the 4-color theorem", for example;-)... but still less than it takes to magically divine what the actual spec IS, from a collection of mutually contradictory observations, no examples, etc. Though in our daily work as software developer (ah for the good old times when all we had to face was homework!-) we DO meet a lot of such cases of course (and have to solve them to earn our daily bread;-).
EditEdit: finally seeing a precise spec I point out I already implemented that one, anyway, here it goes again:
def multiplyItemsByFour(argsList):
return [item * 4 for item in argsList]
EditEditEdit: finally/finally seeing a MORE precise spec, with (luxury!-) examples:
Input: ('a','b') Output: 'aaaabbbb' Input: (2,3,4) Output: 36
So then what's wanted it the summation (and you can't use sum as it wouldn't work on strings) of the items in the input list, each multiplied by four. My preferred solution:
def theFinalAndTrulyRealProblemAsPosed(argsList):
items = iter(argsList)
output = next(items, []) * 4
for item in items:
output += item * 4
return output
If you're forbidden from using some of these constructs, such as built-ins items and iter, there are many other possibilities (slightly inferior ones) such as:
def theFinalAndTrulyRealProblemAsPosed(argsList):
if not argsList: return None
output = argsList[0] * 4
for item in argsList[1:]:
output += item * 4
return output
For an empty argsList, the first version returns [], the second one returns None -- not sure what you're supposed to do in that corner case anyway.
Very easy in Python. You need to get the type of the data in your list - use the type() function on the first item - type(argsList[0]). Then to initialize output (where you now have ????) you need the 'zero' or nul value for that type. So just as int() or float() or str() returns the zero or nul for their type so to will type(argsList[0])() return the zero or nul value for whatever type you have in your list.
So, here is your function with one minor modification:
def multiplyItemsByFour(argsList):
output = type(argsList[0])()
for arg in argsList:
output += arg * 4
return output
Works with::
argsList = [1, 2, 3, 4] or [1.0, 2.0, 3.0, 4.0] or "abcdef" ... etc,
Are you sure this is for Python beginners? To me, the cleanest way to do this is with reduce() and lambda, both of which are not typical beginner tools, and sometimes discouraged even for experienced Python programmers:
def multiplyItemsByFour(argsList):
if not argsList:
return None
newItems = [item * 4 for item in argsList]
return reduce(lambda x, y: x + y, newItems)
Like Alex Martelli, I've thrown in a quick test for an empty list at the beginning which returns None. Note that if you are using Python 3, you must import functools to use reduce().
Essentially, the reduce(lambda...) solution is very similar to the other suggestions to set up an accumulator using the first input item, and then processing the rest of the input items; but is simply more concise.
My guess is that the purpose of your homework is to expose you to "duck typing". The basic idea is that you don't worry about the types too much, you just worry about whether the behaviors work correctly. A classic example:
def add_two(a, b):
return a + b
print add_two(1, 2) # prints 3
print add_two("foo", "bar") # prints "foobar"
print add_two([0, 1, 2], [3, 4, 5]) # prints [0, 1, 2, 3, 4, 5]
Notice that when you def a function in Python, you don't declare a return type anywhere. It is perfectly okay for the same function to return different types based on its arguments. It's considered a virtue, even; consider that in Python we only need one definition of add_two() and we can add integers, add floats, concatenate strings, and join lists with it. Statically typed languages would require multiple implementations, unless they had an escape such as variant, but Python is dynamically typed. (Python is strongly typed, but dynamically typed. Some will tell you Python is weakly typed, but it isn't. In a weakly typed language such as JavaScript, the expression 1 + "1" will give you a result of 2; in Python this expression just raises a TypeError exception.)
It is considered very poor style to try to test the arguments to figure out their types, and then do things based on the types. If you need to make your code robust, you can always use a try block:
def safe_add_two(a, b):
try:
return a + b
except TypeError:
return None
See also the Wikipedia page on duck typing.
Python is dynamically typed, you don't need to declare the type of a variable, because a variable doesn't have a type, only values do. (Any variable can store any value, a value never changes its type during its lifetime.)
def do_something(x):
return x * 5
This will work for any x you pass to it, the actual result depending on what type the value in x has. If x contains a number it will just do regular multiplication, if it contains a string the string will be repeated five times in a row, for lists and such it will repeat the list five times, and so on. For custom types (classes) it depends on whether the class has an operation defined for the multiplication operator.
You don't need to declare variable types in python; a variable has the type of whatever's assigned to it.
EDIT:
To solve the re-stated problem, try this:
def multiplyItemsByFour(argsList):
output = argsList.pop(0) * 4
for arg in argsList:
output += arg * 4
return output
(This is probably not the most pythonic way of doing this, but it should at least start off your output variable as the right type, assuming the whole list is of the same type)
You gave these sample inputs and outputs:
Input: ('a','b') Output: 'aaaabbbb' Input: (2,3,4) Output: 36
I don't want to write the solution to your homework for you, but I do want to steer you in the correct direction. But I'm still not sure I understand what your problem is, because the problem as I understand it seems a bit difficult for an intro to Python class.
The most straightforward way to solve this requires that the arguments be passed in a list. Then, you can look at the first item in the list, and work from that. Here is a function that requires the caller to pass in a list of two items:
def handle_list_of_len_2(lst):
return lst[0] * 4 + lst[1] * 4
Now, how can we make this extend past two items? Well, in your sample code you weren't sure what to assign to your variable output. How about assigning lst[0]? Then it always has the correct type. Then you could loop over all the other elements in lst and accumulate to your output variable using += as you wrote. If you don't know how to loop over a list of items but skip the first thing in the list, Google search for "python list slice".
Now, how can we make this not require the user to pack up everything into a list, but just call the function? What we really want is some way to accept whatever arguments the user wants to pass to the function, and make a list out of them. Perhaps there is special syntax for declaring a function where you tell Python you just want the arguments bundled up into a list. You might check a good tutorial and see what it says about how to define a function.
Now that we have covered (very generally) how to accumulate an answer using +=, let's consider other ways to accumulate an answer. If you know how to use a list comprehension, you could use one of those to return a new list based on the argument list, with the multiply performed on each argument; you could then somehow reduce the list down to a single item and return it. Python 2.3 and newer have a built-in function called sum() and you might want to read up on that. [EDIT: Oh drat, sum() only works on numbers. See note added at end.]
I hope this helps. If you are still very confused, I suggest you contact your teacher and ask for clarification. Good luck.
P.S. Python 2.x have a built-in function called reduce() and it is possible to implement sum() using reduce(). However, the creator of Python thinks it is better to just use sum() and in fact he removed reduce() from Python 3.0 (well, he moved it into a module called functools).
P.P.S. If you get the list comprehension working, here's one more thing to think about. If you use a list comprehension and then pass the result to sum(), you build a list to be used once and then discarded. Wouldn't it be neat if we could get the result, but instead of building the whole list and then discarding it we could just have the sum() function consume the list items as fast as they are generated? You might want to read this: Generator Expressions vs. List Comprehension
EDIT: Oh drat, I assumed that Python's sum() builtin would use duck typing. Actually it is documented to work on numbers, only. I'm disappointed! I'll have to search and see if there were any discussions about that, and see why they did it the way they did; they probably had good reasons. Meanwhile, you might as well use your += solution. Sorry about that.
EDIT: Okay, reading through other answers, I now notice two ways suggested for peeling off the first element in the list.
For simplicity, because you seem like a Python beginner, I suggested simply using output = lst[0] and then using list slicing to skip past the first item in the list. However, Wooble in his answer suggested using output = lst.pop(0) which is a very clean solution: it gets the zeroth thing on the list, and then you can just loop over the list and you automatically skip the zeroth thing. However, this "mutates" the list! It's better if a function like this does not have "side effects" such as modifying the list passed to it. (Unless the list is a special list made just for that function call, such as a *args list.) Another way would be to use the "list slice" trick to make a copy of the list that has the first item removed. Alex Martelli provided an example of how to make an "iterator" using a Python feature called iter(), and then using iterator to get the "next" thing. Since the iterator hasn't been used yet, the next thing is the zeroth thing in the list. That's not really a beginner solution but it is the most elegant way to do this in Python; you could pass a really huge list to the function, and Alex Martelli's solution will neither mutate the list nor waste memory by making a copy of the list.
No need to test the objects, just multiply away!
'this is a string' * 6
14 * 6
[1,2,3] * 6
all just work
Try this:
def timesfourlist(list):
nextstep = map(times_four, list)
sum(nextstep)
map performs the function passed in on each element of the list(returning a new list) and then sum does the += on the list.
If you just want to fill in the blank in your code, you could try setting object=arglist[0].__class__() to give it the zero equivalent value of that class.
>>> def multiplyItemsByFour(argsList):
output = argsList[0].__class__()
for arg in argsList:
output += arg * 4
return output
>>> multiplyItemsByFour('ab')
'aaaabbbb'
>>> multiplyItemsByFour((2,3,4))
36
>>> multiplyItemsByFour((2.0,3.3))
21.199999999999999
This will crash if the list is empty, but you can check for that case at the beginning of the function and return whatever you feel appropriate.
Thanks to Alex Martelli, you have the best possible solution:
def theFinalAndTrulyRealProblemAsPosed(argsList):
items = iter(argsList)
output = next(items, []) * 4
for item in items:
output += item * 4
return output
This is beautiful and elegant. First we create an iterator with iter(), then we use next() to get the first object in the list. Then we accumulate as we iterate through the rest of the list, and we are done. We never need to know the type of the objects in argsList, and indeed they can be of different types as long as all the types can have operator + applied with them. This is duck typing.
For a moment there last night I was confused and thought that you wanted a function that, instead of taking an explicit list, just took one or more arguments.
def four_x_args(*args):
return theFinalAndTrulyRealProblemAsPosed(args)
The *args argument to the function tells Python to gather up all arguments to this function and make a tuple out of them; then the tuple is bound to the name args. You can easily make a list out of it, and then you could use the .pop(0) method to get the first item from the list. This costs the memory and time to build the list, which is why the iter() solution is so elegant.
def four_x_args(*args):
argsList = list(args) # convert from tuple to list
output = argsList.pop(0) * 4
for arg in argsList:
output += arg * 4
return output
This is just Wooble's solution, rewritten to use *args.
Examples of calling it:
print four_x_args(1) # prints 4
print four_x_args(1, 2) # prints 12
print four_x_args('a') # prints 'aaaa'
print four_x_args('ab', 'c') # prints 'ababababcccc'
Finally, I'm going to be malicious and complain about the solution you accepted. That solution depends on the object's base class having a sensible null or zero, but not all classes have this. int() returns 0, and str() returns '' (null string), so they work. But how about this:
class NaturalNumber(int):
"""
Exactly like an int, but only values >= 1 are possible.
"""
def __new__(cls, initial_value=1):
try:
n = int(initial_value)
if n < 1:
raise ValueError
except ValueError:
raise ValueError, "NaturalNumber() initial value must be an int() >= 1"
return super(NaturalNumber, cls).__new__ (cls, n)
argList = [NaturalNumber(n) for n in xrange(1, 4)]
print theFinalAndTrulyRealProblemAsPosed(argList) # prints correct answer: 24
print NaturalNumber() # prints 1
print type(argList[0])() # prints 1, same as previous line
print multiplyItemsByFour(argList) # prints 25!
Good luck in your studies, and I hope you enjoy Python as much as I do.

Categories

Resources