Related
I'm working on getting a better grasp of Python 3 fundamentals, specifically objects and modifying them in the context of a list (for now).
I created a simple class called MyThing() that just has a number, letter, and instance method for incrementing the number. My goal with this program was to create a list of 3 "MyThings", and manipulate the list in various ways. To start, I iterated through the list (obj_list_1) and incremented each number using each object's instance method. Easy enough.
What I'm trying to figure out how to do is perform the same operation in one line using the map function and lambda expressions (obj_list_2).
#!/usr/bin/env py
import copy
class MyThing:
def __init__(self, letter='A', number=0):
self.number = number
self.letter = letter
def __repr__(self) -> str:
return("(letter={}, number={})".format(self.letter, self.number))
def incr_number(self, incr=0):
self.number += incr
# Test program to try different ways of manipulating lists
def main():
obj1 = MyThing('A', 1)
obj2 = MyThing('B', 2)
obj3 = MyThing('C', 3)
obj_list_1 = [obj1, obj2, obj3]
obj_list_2 = copy.deepcopy(obj_list_1)
# Show the original list
print("Original List: {}".format(obj_list_1))
# output: [(letter=A, number=1), (letter=B, number=2), (letter=C, number=3)]
# Standard iterating over a list and incrementing each object's number.
for obj in obj_list_1:
obj.incr_number(1)
print("For loop over List, adding one to each number:\n{}".format(obj_list_1))
# output: [(letter=A, number=2), (letter=B, number=3), (letter=C, number=4)]
# Try using map function with lambda
obj_list_2 = list(map(lambda x: x.incr_number(1), obj_list_2))
print("Using maps with incr_number instance method:\n{}".format(obj_list_2))
# actual output: [None, None, None] <--- If I don't re-assign obj_list_2...it shows the proper sequence
# expected output: [(letter=A, number=2), (letter=B, number=3), (letter=C, number=4)]
if __name__ == "__main__":
main()
What I can't figure out is how to get map() to return the correct type, a list of "MyThing"s.
I understand that between Python 2 and Python 3, map changed to return an iterable instead of a list, so I made sure to cast the output. What I get is a list of 'None' objects.
What I noticed, though, is that if I don't re-assign obj_list_2, and instead just call list(map(lambda x: x.incr_number(1), obj_list_2)), then print obj_list_2 in the next line, the numbers get updated as I expect.
However, if I don't cast the map iterable and just do map(lambda x: x.incr_number(1), obj_list_2), the following print statement shows the list as having not been updated. I read in some documentation that the map function is lazy and doesn't operate until it's use by something...so this makes sense.
Is there a way that I can get the output of list(map(lambda x: x.incr_number(1), obj_list_2)) to actually return my list of objects?
Are there any other cool one-liner solutions for updating a list of objects with their instance methods that I'm not thinking of?
TL;DR: Just use the for-loop. There's no advantage to using a map in this case.
Firstly:
You're getting a list of Nones because the mapped function returns None. That is, MyThing.incr_number() doesn't return anything, so it returns None implicitly.
Fewer lines is not necessarily better. Two simple lines are often easier to read than one complex line.
Notice that you're not creating a new list in the for-loop, you're only modifying the elements of the existing list.
list(map(lambda)) is longer and harder to read than a list comprehension:
[x.incr_number(1) for x in obj_list_2]
vs
list(map(lambda x: x.incr_number(1), obj_list_2))
Now, take a look at Is it Pythonic to use list comprehensions for just side effects? The top answer says no, it creates a list that never gets used. So there's your answer: just use the for-loop instead.
This is because, your incr_number doesn't return anything. Change it to:
def incr_number(self, incr=0):
self.number += incr
return self
The loop is clearly better, but here's another way anyway. Your incr_number doesn't return anything, or rather returns the default None. Which is a false value, so if you simply append or x, then you do get the modified value instead of the None
Change
list(map(lambda x: x.incr_number(1), obj_list_2))
to this:
list(map(lambda x: x.incr_number(1) or x, obj_list_2))
I have created a class and am attempting to create a function that will, in essence, act as a binary operator between two objects of that class. The function I was trying to create is called 'combine'.
I know I could create the function outside of the class, but I want it to be associated with the class.
Background (not really necessary for answering the question) - the class is modelling the S4 mathematical group and is some practice with objects and classes for me. My next function was to be to simplify an element into it's simplest expression in cycles, which I am confident that I could do, but I would rather get this one sorted first.
When I create a function with one argument, it runs fine - as demonstrated in the code below, with the function 'cycletype', which works as expected.
class s4:
# Define an element of s4, elementlist is a nested list, which is a list of the cycles composing the element.
def __init__(self, elementlist):
self.elementlist = elementlist
# One simple function is to ascertain the cycletype of the element.
def cycletype(self):
cycles = []
for i in self.elementlist:
cycles.append(len(i))
return cycles
# Combining two elements using the group operation is the first function to define.
def combine(first, second):
for i in second:
first.append(i)
return first
double = s4([[1,2],[3,4]])
triple = s4([[1,2,3]])
print(combine(double,triple))
I was expecting [[1,2],[3,4],[1,2,3]] to be printed, however, it showed a NameError, not recognising combine.
You should be creating a new instance from the lists wrapped by the two arguments:
class S4:
# Define an element of S4, elementlist is a nested list, which is a list of the cycles composing the element.
def __init__(self, elementlist):
self.elementlist = elementlist
# One simple function is to ascertain the cycletype of the element.
def cycletype(self):
cycles = []
for i in self.elementlist:
cycles.append(len(i))
return cycles
# Combining two elements using the group operation is the first function to define.
def combine(self, first, second):
return S4(first.element_list + second.element_list)
It also appears that you could simply define __add__ instead of combine,
# Simplified implementation, no error checking
def __add__(self, second):
return S4(self.element_list + second.element_list)
allowing you to write
print(double + triple)
instead of
print(combine(double, triple))
There's two problems with the code
putting the function inside the class means that it's a method. So, you have to access it using the object s4.combine(s1), not just combine(...). Otherwise, it would be a function, not a method.
After you change that: you can't write for i in second, because instances of your class are not iterable. You have to implement __iter__ to be able to use that syntax
This is because the combine function isn't present in the global scope ( I hope that's what its called, I mess up names real bad).
You can only call functions present in the global scope, in the case of classes, you need objects to call these functions as these functions are part of those specific classes and not the global scope in general.
Hope this helped
There's a scope problem, you defined combine within the s4 class, so you should call it from a instance of s4. Anyways, here's is how I would do it.
class s4:
# Define an element of s4, elementlist is a nested list, which is a list of the cycles composing the element.
def __init__(self, elementlist):
self.elementlist = elementlist
# One simple function is to ascertain the cycletype of the element.
def cycletype(self):
cycles = []
for i in self.elementlist:
cycles.append(len(i))
return cycles
# Combining two elements using the group operation is the first function to define.
def combine(self, second):
#Here I changed first to self, first would be the s4 class that calls the function, in this example double and second would be the one that you want to combine
first = self.elementlist
for i in second.elementlist:
first.append(i)
return first
double = s4([[1,2],[3,4]])
triple = s4([[1,2,3]])
##As double() is defined within the s4 class, you can only call it from an instance of s4 (in this case you could use doble.combine() or triple.combine())
print(double.combine(triple))
Hope it helps.
I have a list of patterns:
patterns_trees = [response.css("#Header").xpath("//a/img/#src"),
response.css("#HEADER").xpath("//a/img/#src"),
response.xpath("//header//a/img/#src"),
response.xpath("//a[#href='"+response.url+'/'+"']/img/#src"),
response.xpath("//a[#href='/']/img/#src")
]
After I traverse it and find the right pattern I have to send the pattern as an argument to a callback function
for pattern_tree in patterns_trees:
...
pattern_response = scrapy.Request(...,..., meta={"pattern_tree": pattern_tree.extract_first()})
By doing this I get the value of the regex not the pattern
THINGS I TRIED:
I tried isolating the patterns in a separate class but still I have the problem that I can not store them as pattern but as values.
I tried to save them as strings and maybe I can make it work but
What is the most efficient way of storing list of functions
UPDATE: Possible solution but too hardcoded and it's too problematic when I want to add more patterns:
def patter_0(response):
response.css("#Header").xpath("//a/img/#src")
def patter_1(response):
response.css("#HEADER").xpath("//a/img/#src")
.....
class patternTrees:
patterns = [patter_0,...,patter_n]
def length_patterns(self):
return len(patterns)
If you're willing to consider reformatting your list of operations, then this is a somewhat neat solution. I've changed the list of operations to a list of tuples. Each tuple contains (a ref to) the appropriate function, and another tuple consisting of arguments.
It's fairly easy to add new operations to the list: just specify what function to use, and the appropriate arguments.
If you want to use the result from one operation as an argument in the next: You will have to return the value from execute() and process it in the for loop.
I've replaced the calls to response with prints() so that you can test it easily.
def response_css_ARG_xpath_ARG(args):
return "response.css(\"%s\").xpath(\"%s\")" % (args[0],args[1])
#return response.css(args[0]).xpath(args[1])
def response_xpath_ARG(arg):
return "return respons.xpath(\"%s\")" % (arg)
#return response.xpath(arg)
def execute(function, args):
response = function(args)
# do whatever with response
return response
response_url = "https://whatever.com"
patterns_trees = [(response_css_ARG_xpath_ARG, ("#Header", "//a/img/#src")),
(response_css_ARG_xpath_ARG, ("#HEADER", "//a/img/#src")),
(response_xpath_ARG, ("//header//a/img/#src")),
(response_xpath_ARG, ("//a[#href='"+response_url+"/"+"']/img/#src")),
(response_xpath_ARG, ("//a[#href='/']/img/#src"))]
for pattern_tree in patterns_trees:
print(execute(pattern_tree[0], pattern_tree[1]))
Note that execute() can be omitted! Depending on if you need to process the result or not. Without the executioner, you may just call the function directly from the loop:
for pattern_tree in patterns_trees:
print(pattern_tree[0](pattern_tree[1]))
Not sure I understand what you're trying to do, but could you make your list a list of lambda functions like so:
patterns_trees = [
lambda response : response.css("#Header").xpath("//a/img/#src"),
...
]
And then, in your loop:
for pattern_tree in patterns_trees:
intermediate_response = scrapy.Request(...) # without meta kwarg
pattern_response = pattern_tree(intermediate_response)
Or does leaving the meta away have an impact on the response object?
I've got this block of code in a real Django function. If certain conditions are met, items are added to the list.
ret = []
if self.taken():
ret.append('taken')
if self.suggested():
ret.append('suggested')
#.... many more conditions and appends...
return ret
It's very functional. You know what it does, and that's great...
But I've learned to appreciate the beauty of list and dict comprehensions.
Is there a more Pythonic way of phrasing this construct, perhaps that initialises and populates the array in one blow?
Create a mapping dictionary:
self.map_dict = {'taken': self.taken,
'suggested': self.suggested,
'foo' : self.bar}
[x for x in ['taken', 'suggested', 'foo'] if self.map_dict.get(x, lambda:False)()]
Related: Most efficient way of making an if-elif-elif-else statement when the else is done the most?
Not a big improvement, but I'll mention it:
def populate():
if self.taken():
yield 'taken'
if self.suggested():
yield 'suggested'
ret = list(populate())
Can we do better? I'm skeptical. Clearly there's a need of using another syntax than a list literal, because we no longer have the "1 expression = 1 element in result" invariant.
Edit:
There's a pattern to our data, and it's a list of (condition, value) pairs. We might try to exploit it using:
[value
for condition, value
in [(self.taken(), 'taken'),
(self.suggested(), 'suggested')]
if condition]
but this still is a restriction for how you describe your logic, still has the nasty side effect of evaluating all values no matter the condition (unless you throw in a ton of lambdas), and I can't really see it as an improvement over what we've started with.
For this very specific example, I could do:
return [x for x in ['taken', 'suggested', ...] if getattr(self, x)()]
But again, this only works where the item and method it calls to check have the same name, ie for my exact code. It could be adapted but it's a bit crusty. I'm very open to other solutions!
I don't know why we are appending strings that match the function names, but if this is a general pattern, we can use that. Functions have a __name__ attribute and I think it always contains what you want in the list.
So how about:
return [fn.__name__ for fn in (self.taken, self.suggested, foo, bar, baz) if fn()]
If I understand the problem correctly, this works just as well for non-member functions as for member functions.
EDIT:
Okay, let's add a mapping dictionary. And split out the function names into a tuple or list.
fns_to_check = (self.taken, self.suggested, foo, bar, baz)
# This holds only the exceptions; if a function isn't in here,
# we will use the .__name__ attribute.
fn_name_map = {foo:'alternate', bar:'other'}
def fn_name(fn):
"""Return name from exceptions map, or .__name__ if not in map"""
return fn_name_map.get(fn, fn.__name__)
return [fn_name(fn) for fn in fns_to_check if fn()]
You could also just use #hcwhsa's mapping dictionary answer. The main difference here is I'm suggesting just mapping the exceptions.
In another instance (where a value will be defined but might be None - a Django model's fields in my case), I've found that just adding them and filtering works:
return filter(None, [self.user, self.partner])
If either of those is None, They'll be removed from the list. It's a little more intensive than just checking but still fairly easy way of cleaning the output without writing a book.
One option is to have a "sentinel"-style object to take the place of list entries that fail the corresponding condition. Then a function can be defined to filter out the missing items:
# "sentinel indicating a list element that should be skipped
Skip = object()
def drop_missing(itr):
"""returns an iterator yielding all but Skip objects from the given itr"""
return filter(lambda v: v is not Skip, itr)
With this simple machinery, we come reasonably close to list-comprehension style syntax:
return drop_skips([
'taken' if self.taken else Skip,
'suggested' if self.suggested else Skip,
100 if self.full else Skip,
// many other values and conditions
])
ret = [
*('taken' for _i in range(1) if self.taken()),
*('suggested' for _i in range(1) if self.suggested()),
]
The idea is to use the list comprehension syntax to construct either a single element list with item 'taken', if self.taken() is True, or an empty list, if self.taken() is False, and then unpack it.
i wrote a class inheriting from dict, i wrote a member method to remove objects.
class RoleCOList(dict):
def __init__(self):
dict.__init__(self)
def recyle(self):
'''
remove roles too long no access
'''
checkTime = time.time()-60*30
l = [k for k,v in self.items() if v.lastAccess>checkTime]
for x in l:
self.pop(x)
isn't it too inefficient? i used 2 list loops but i couldn't find other way
At the SciPy conference last year, I attended a talk where the speaker said that any() and all() are fast ways to do a task in a loop. It makes sense; a for loop rebinds the loop variable on each iteration, whereas any() and all() simply consume the value.
Clearly, you use any() when you want to run a function that always returns a false value such as None. That way, the whole loop will run to the end.
checkTime = time.time() - 60*30
# use any() as a fast way to run a loop
# The .__delitem__() method always returns `None`, so this runs the whole loop
lst = [k for k in self.keys() if self[k].lastAccess > checkTime]
any(self.__delitem__(k) for k in lst)
what about this?
_ = [self.pop(k) for k,v in self.items() if v.lastAccess>checkTime]
Since you don't need the list you generated, you could use generators and a snippet from this consume recipe. In particular, use collections.deque to run through a generator for you.
checkTime = time.time()-60*30
# Create a generator for all the values you will age off
age_off = (self.pop(k) for k in self.keys() if self[k].lastAccess>checkTime)
# Let deque handle iteration (in one shot, with little memory footprint)
collections.deque(age_off,maxlen=0)
Since the dictionary is changed during the iteration of age_off, use self.keys() which returns a list. (Using self.iteritems() will raise a RuntimeError.)
My (completly unreadable solution):
from operator import delitem
map(lambda k: delitem(self,k), filter(lambda k: self[k].lastAccess<checkTime, iter(self)))
but at least it should be quite time and memory efficient ;-)
If performance is an issue, and if you will have large volumes of data, you might want to look into using a Python front-end for a system like memcached or redis; those can handle expiring old data for you.
http://memcached.org/
http://pypi.python.org/pypi/python-memcached/
http://redis.io/
https://github.com/andymccurdy/redis-py