Develop Reference

Relationship between elements of two list: how to exploit it in Python?

SO here is my minimal working example:
# I have a list
list1 = [1,2,3,4]
#I do some operation on the elements of the list
list2 = [2**j for j in list1]
# Then I want to have these items all shuffled around, so for instance
list2 = np.random.permutation(list2)
#Now here is my problem: I want to understand which element of the new list2 came from which element of list1. I am looking for something like this:
list1.index(something)
# Basically given an element of list2, I want to understand from where it came from, in list1. I really cant think of a simple way of doing this, but there must be an easy way!
Can you please suggest me an easy solution? This is a minimal working example,however the main point is that I have a list, I do some operation on the elements and assign these to a new list. And then the items get all shuffled around and I need to understand where they came from.

enumerate, like everyone said is the best option but there is an alternative if you know the mapping relation. You can write a function that does the opposite of the mapping relation. (eg. decodes if the original function encodes.)
Then you use decoded_list = map(decode_function,encoded_list) to get a new list. Then by cross comparing this list with the original list, you can achieve your goal.
Enumerate is better if you are certain that the same list was modified using the encode_function from within the code to get the encoded list.
However, if you are importing this new list from elsewhere, eg. from a table on a website, my approach is the way to go.

You could use a permutation list/index :
# I have a list
list1 = [1,2,3,4]
#I do some operation on the elements of the list
list2 = [2**j for j in list1]
# Then I want to have these items all shuffled around, so for instance
index_list = range(len(list2))
index_list = np.random.permutation(index_list)
list3 = [list2[i] for i in index_list]
then,with input_element:
answer = index_list[list3.index(input_element)]

Based on your code:
# I have a list
list1 = [1,2,3,4]
#I do some operation on the elements of the list
list2 = [2**j for j in list1]
# made a recode of index and value
index_list2 = list(enumerate(list2))
# Then I want to have these items all shuffled around, so for instance
index_list3 = np.random.permutation(index_list2)
idx, list3 = zip(*index_list3)
#get the index of element_input in list3, then get the value of the index in idx, that should be the answer you want.
answer = idx[list3.index(element_input)]

def index3_to_1(index):
y = list3[index]
x = np.log(y)/np.log(2) # inverse y=f(x) for your operation
return list1.index(x)
This supposes that the operations you are doing on list2 are reversible. Also, it supposes that each element in list1 is unique.

Search through lists in Python to find matches?

I have Python lists with various strings in them such as:
List1 = ["a","b","c","d"]
List2 = ["b","d","e","f"]
List3 = []
List4 = ["d","f","g"]
I need to iterate through these lists, provided they are not blank, and finds items that are in all non-blank lists. In the above example, the exact matches list would be ["d"], since that is the only item that appears in all non-blank lists. List3 is blank, so it would not matter that it is not in that list.

Here's some functional programming beauty:
from operator import and_
from functools import reduce
Lists = List1, List2, List3, List4
result = reduce(and_, map(set, filter(None, Lists)))

I can't test this right now, but something like the following should work:
intersection(set(l) for l in [List1, List2, List3, List4] if l)
It uses Python's built-in set datatype to do the intersection operation.

for thing in list1: # iterate each item, you can check before hand if its not empty
if len(list2) > 0: # if not empty
if thing in list2: # in the list
# do the same thing for the other lists
something like that

Randomly chose an element of one list that's NOT in a second list

Say I have a list2 of randomly chosen elements from a large list1. Is there a clever way of choosing an element from list1 that's NOT already in list2?
For example:
list1 = range(20,100)
list2 = [37,49,22,35,72] # could be much longer
while True:
n = random.choice(list1)
if n not in list2:
break
# now n is an element of list1 that's not in list2
I feel like there must be a more efficient way of doing this than a guess-and-check while-loop.

You can subtract list2 of list1:
list3 = list(set(list1)-set(list2))
and choose from it randomly:
random.choice(list3)
Note: you need to reconvert the set to a list.

You could use:
import random
list1 = range(20,100)
list2 = [37,49,22,35,72]
not_in_list2 = [item for item in list1 if item not in list2]
n = random.choice(not_in_list2)
This uses a list comprehension to create a list of all elements in list1 that aren't inlist2. It then selects randomly from this list. Unlike when working with sets, this technique does not change the probability of items being selected, because it does not remove duplicate elements from list1.

In case that there are no repeating elements in list1, this is a pythonic way, working with set and -:
import random
list1 = range(20,100)
list2 = [37,49,22,35,72] # could be much longer
n = random.choice(tuple(set(list1)-set(list2)))
# now n is an element of list1 that's not in list2
The tuple call is needed to avoid a NotIndexable exception.

If you want to randomly select more than one item from a list, or select an item from a set, it's better to use random.sample instead of choice
import random
diff = set(list1)-set(list2)
num_to_select = 1 # set the number to select here.
list_of_random_items = random.sample(diff, num_to_select)

If you do not want the overhead of creating a new list (or a new list and two sets) which can become quite costly if list1 is very large, there is another option.
import random
list1 = range(20,100)
list2 = [37,49,22,35,72]
for i in list2:
while i in list1:
list1.remove(i)
random.choice(list1)
Just iterate through the items in list2 and remove them from list1. Since list.remove() only removes the first occurrence of an item, I added a while-loop to ensure that all occurences are removed.

Generate all combinations from multiple lists with repeat lists

I have multiple lists, some of them are repeats and I need all combinations, excluding ones where the same element from a repeated list is chosen. For example, I have
import itertools
list1 = [1,2,3]
list2 = [4,5,6]
list3 = [4,5,6]
list4 = [7,8,9]
a = [list1,list2,list3,list4]
print list(itertools.product(*a))
which outputs
(1,4,4,7)
(1,4,4,8)
(1,4,4,9)
(1,4,5,7)
.
.
.
etc, as you'd expect, but what I want it to do is output every combination, without repeating elements from lists 2 and 3. Like this:
(1,4,5,7)
(1,4,5,8)
(1,4,5,9)
(1,4,6,7)
(1,4,6,8)
(1,4,6,9)
(1,5,6,7)
(1,5,6,8)
(1,5,6,9)
(2,4,5,7)
.
.
.
I'd obviously like to avoid having to remove them manually after creating the list, but any help on how to do this efficiently is really appreciated. Thanks.

The easy way is a generator expression with a filter:
print list(item for item in itertools.product(*a) if item[1] != item[2])
If two items are considered the same if they contain the same elements, and if each item is guaranteed to contain no repeated elements, you can discard duplicates by changing them into sets and only adding them to your list if they're not already in it:
result = []
for item in itertools.product(*a):
if item[1]==item[2]:
continue
item = set(item)
if item not in result:
result.append(item)
print result

Making a new list in alphabetical order

My python instructions are: Write a function called merge
takes two lists of strings that are each in alphabetical order.
The function should return a list that has all the strings
in alphabetical order.
The function should return a list that has all the strings in alphabetical
order.
For example, if the function were given the two lists
[“cat”, “dog”, “pat”] and [ “bat”, “hamster”, “piglet”, “worm”]
it should return the list
[“bat”, “cat”, “dog “, hamster”, “pat”, “piglet”, “worm”].
Roughly speaking you will start with an empty list to hold the merged list
and an index set to 0 at the beginning of each list.
Compare the first words. Whichever comes first in alphabetical
order gets appended to the merged list and the index of that list
is increased. Continue until one of the lists is empty and copy
the remainder of the other list to the merged list.
Right now I have the code
list1 = ["cat", "dog", "pat"]
list2 = [ "bat", "hamster", "piglet", "worm"]
def merge (list1, list2):
newlist = []
newlist = list1 +list2
final = sorted(newlist)
return final
print merge (list1, list2)
It works, but it's not following instructions. I'm not really sure how to do it with comparing the two lists then appending them to the new list. I also didn't do it with an index set to 0. Can anyone help tweak my code so it fits the instructions?

lets start by writing a compare function
def cmp_lists(L1,L2):
'''returns the first item of the lesser list and the remaining lists'''
if L1 < L2:
return L1[0],(L1[1:],L2)
return L2[0],(L1,L2[1:])
now write a method to merge 2 lists
def merge(L1,L2):
if not L1: #if L1 is empty return L2
return L2
elif not L2: #if L2 is empty return L1
return L1
#otherwise take the first item from one of the lists
next_itm,(L1,L2) = cmp_lists(L1,L2)
# and recursively carry on
return [next_itm] + merge(L1,L2)

What you have been instructed is to write an implementation for a simple merge function that is popularly used in merge-sort sorting algorithm.
Since both lists are already sorted, you don't need to join and sort them again. Rather, just loop until one of the lists is empty and keep comparing the first elements of both lists. Pop the shorter element and add to the newer list.
Keep doing so until one or both of the lists become empty. If one of the lists have extra elements remaining, add all of them to the new list.
I encourage you to write the code once again.

It's a merge sort? Oh that's not tough. Just reverse both lists, compare each pair, and pop the loser. Once either list is empty, the remaining list is sorted in ascending order compared to the entire result, so just concatenate it.
from operator import itemgetter as iget
def merge(lst1, lst2):
lst1, lst2 = lst1[::-1], lst2[::-1]
# popping from the end is MASSIVELY better than popping from the front
result = []
while True:
if not all([lst1, lst2]): # one list is empty:
return result + lst1[::-1] + lst2[::-1] # so add them both and return
lst_to_pop_from = min(lst1,lst2, key=iget(-1))
# gets the list with the smallest final element
result.append(lst_to_pop_from.pop())
# DEMO
In [53]: list1 = ["cat", "dog", "pat"]
In [54]: list2 = [ "bat", "hamster", "piglet", "worm"]
In [55]: merge(list1, list2)
Out[55]: ['bat', 'cat', 'dog', 'hamster', 'pat', 'piglet', 'worm']

Merge sort works by looking at the top elements of the list and putting them in the newly created list.
def merge(list1, list2): # assume input lists are sorted
iter1 = iter(list1) # iterator on the list
iter2 = iter(list2)
top1 = iter1.next() # first element of the list
top2 = iter2.next()
newlist = [] # new list to fill in
while True: # loop will exit with break statement
if top1 <= top2:
newlist.append(top1) # put smaller element in new list
try:
top1 = iter1.next() # try to get next element in list1
except StopIteration:
newlist.append(top2) # if not, fill list with top2
newlist.extend(list(iter2)) # and rest of list2
break # and exit loop
else:
newlist.append(top2)
try:
top2 = iter2.next()
except StopIteration:
newlist.append(top1) # if not, fill list with top1
newlist.extend(list(iter1)) # and rest of list1
break
return newlist