Develop Reference

Sort list of strings numerically and filter duplicates?

Given a list of strings in the following format:
[
"464782,-100,4,3,1,100,0,0"
"465042,-166.666666666667,4,3,1,100,0,0",
"465825,-250.000000000001,4,3,1,100,0,0",
"466868,-166.666666666667,4,3,1,100,0,0",
"467390,-200.000000000001,4,3,1,100,0,0",
"469999,-100,4,3,1,100,0,0",
"470260,-166.666666666667,4,3,1,100,0,0",
"474173,-100,4,3,1,100,0,0",
"474434,-166.666666666667,4,3,1,100,0,0",
"481477,-100,4,3,1,100,0,1",
"531564,259.011439671919,4,3,1,60,1,0",
"24369,-333.333333333335,4,3,1,100,0,0",
"21082,410.958904109589,4,3,1,60,1,0",
"21082,-250,4,3,1,100,0,0",
"22725,-142.857142857143,4,3,1,100,0,0",
"23547,-166.666666666667,4,3,1,100,0,0",
"24369,-333.333333333335,4,3,1,100,0,0",
"27657,-200.000000000001,4,3,1,100,0,0",
"29301,-142.857142857143,4,3,1,100,0,0",
"30123,-166.666666666667,4,3,1,100,0,0",
"30945,-250,4,3,1,100,0,0",
"32588,-166.666666666667,4,3,1,100,0,0",
"34232,-250,4,3,1,100,0,0",
"35876,-142.857142857143,4,3,1,100,0,0",
"36698,-166.666666666667,4,3,1,100,0,0",
"37520,-250,4,3,1,100,0,0",
"42451,-142.857142857143,4,3,1,100,0,0",
"43273,-166.666666666667,4,3,1,100,0,0",
]
How can I sort the list based on the first number in each line with python?
And then, once sorted, remove all duplicates, if any are there?
The sorting criteria for the list is the number before the first comma in each line, which is always an integer.
I tried using list.sort() , however, this sorts the items in lexical order, not numerically.

You could use a dictionary for this. The key will be number before the first comma and the value the entire string. Duplicates will be eliminated, but only the last occurrence of a particular number's string is stored.
l = ['464782,-100,4,3,1,100,0,0',
'465042,-166.666666666667,4,3,1,100,0,0',
'465825,-250.000000000001,4,3,1,100,0,0',
'466868,-166.666666666667,4,3,1,100,0,0',
'467390,-200.000000000001,4,3,1,100,0,0',
...]
d = {int(s.split(',')[0]) : s for s in l}
result = [d[key] for key in sorted(d.keys())]

I would try one of these two methods:
def sort_list(lis):
nums = [int(num) if isdigit(num) else float(num) for num in lis]
nums = list(set(nums))
nums.sort()
return [str(i) for i in nums] # I assumed you wanted them to be strings.
The first will raise a TypeError if all items in lis are not ints, floats, or string representations of a number. The second method doesn't have that problem, but it's a bit wonkier.
def sort_list(lis):
ints = [int(num) for num in lis if num.isdigit()]
floats = [float(num) for num in lis if not num.isdigit()]
nums = ints.copy()
nums.extend(floats)
nums = list(set(nums))
nums.sort()
return [str(i) for i in nums] # I assumed you wanted them to be strings.
Hope this helps.

You can try this.
First we need to remove the duplicates inside the list using set()
removed_duplicates_list = list(set(listr))
Then we convert the list of strings in to a list of tuples
list_of_tuples = [tuple(i.split(",")) for i in removed_duplicates_list]
Then we sort it using the sort()
list_of_tuples.sort()
The complete code sample below:
listr = [
"464782,-100,4,3,1,100,0,0"
"465042,-166.666666666667,4,3,1,100,0,0",
"465825,-250.000000000001,4,3,1,100,0,0",
"466868,-166.666666666667,4,3,1,100,0,0",
"467390,-200.000000000001,4,3,1,100,0,0",
"469999,-100,4,3,1,100,0,0",
"470260,-166.666666666667,4,3,1,100,0,0",
"474173,-100,4,3,1,100,0,0",
"474434,-166.666666666667,4,3,1,100,0,0",
"481477,-100,4,3,1,100,0,1",
"531564,259.011439671919,4,3,1,60,1,0",
"24369,-333.333333333335,4,3,1,100,0,0",
"21082,410.958904109589,4,3,1,60,1,0",
"21082,-250,4,3,1,100,0,0",
"22725,-142.857142857143,4,3,1,100,0,0",
"23547,-166.666666666667,4,3,1,100,0,0",
"24369,-333.333333333335,4,3,1,100,0,0",
"27657,-200.000000000001,4,3,1,100,0,0",
"29301,-142.857142857143,4,3,1,100,0,0",
"30123,-166.666666666667,4,3,1,100,0,0",
"30945,-250,4,3,1,100,0,0",
"32588,-166.666666666667,4,3,1,100,0,0",
"34232,-250,4,3,1,100,0,0",
"35876,-142.857142857143,4,3,1,100,0,0",
"36698,-166.666666666667,4,3,1,100,0,0",
"37520,-250,4,3,1,100,0,0",
"42451,-142.857142857143,4,3,1,100,0,0",
"43273,-166.666666666667,4,3,1,100,0,0",
]
removed_duplicates_list = list(set(listr))
list_of_tuples = [tuple(i.split(",")) for i in removed_duplicates_list]
list_of_tuples.sort()
print(list_of_tuples) # the output is a list of tuples
OUTPUT:
[('21082', '-250', '4', '3', '1', '100', '0', '0'),
('21082', '410.958904109589', '4', '3', '1', '60', '1', '0'),
('22725', '-142.857142857143', '4', '3', '1', '100', '0', '0'),
('23547', '-166.666666666667', '4', '3', '1', '100', '0', '0'),
('24369', '-333.333333333335', '4', '3', '1', '100', '0', '0'),
('27657', '-200.000000000001', '4', '3', '1', '100', '0', '0'),
('29301', '-142.857142857143', '4', '3', '1', '100', '0', '0'),
('30123', '-166.666666666667', '4', '3', '1', '100', '0', '0'),
('30945', '-250', '4', '3', '1', '100', '0', '0'),
('32588', '-166.666666666667', '4', '3', '1', '100', '0', '0'),
('34232', '-250', '4', '3', '1', '100', '0', '0'),
('35876', '-142.857142857143', '4', '3', '1', '100', '0', '0'),
('36698', '-166.666666666667', '4', '3', '1', '100', '0', '0'),
('37520', '-250', '4', '3', '1', '100', '0', '0'),
('42451', '-142.857142857143', '4', '3', '1', '100', '0', '0'),
('43273', '-166.666666666667', '4', '3', '1', '100', '0', '0'),
('464782','-100','4','3','1','100','0'),
('465042','-166.666666666667','4','3','1','100','0','0'),
('465825', '-250.000000000001', '4', '3', '1', '100', '0', '0'),
('466868', '-166.666666666667', '4', '3', '1', '100', '0', '0'),
('467390', '-200.000000000001', '4', '3', '1', '100', '0', '0'),
('469999', '-100', '4', '3', '1', '100', '0', '0'),
('470260', '-166.666666666667', '4', '3', '1', '100', '0', '0'),
('474173', '-100', '4', '3', '1', '100', '0', '0'),
('474434', '-166.666666666667', '4', '3', '1', '100', '0', '0'),
('481477', '-100', '4', '3', '1', '100', '0', '1'),
('531564', '259.011439671919', '4', '3', '1', '60', '1', '0')]

I hope this will help to.
I place all your list elements in a separate file named lista.txt
In this example I will get your list from file... I like to be more organizated and to have separate files you can do in on python as well, but the idea is you need to get all elements from list one by one (while function or for function) and to add them to a temporary list by checking if the new items already exist, if is exist pass and then you can sample use .sort() because will do the trick and with numbers.
# Global variables
file = "lista.txt"
tempList = []
# Logic get items from file
def GetListFromFile(fileName):
# Local variables
showDoneMsg = True
# Try to run this code
try:
# Open file and try to read it
with open(fileName, mode="r") as f:
# Define line
line = f.readline()
# For every line in file
while line:
# Get out all end white space (\n, \r)
item = line.rstrip()
# Check if this item is not allready in the list
if item not in tempList:
# Append item to a temporar list
tempList.append(item)
# Show me if a itmes allready exist
else:
print("Dublicate >>", item)
# Go to new line
line = f.readline()
# This is optional because is callet automatical
# but I like to be shore
f.close()
# Execptions
except FileNotFoundError:
print("ERROR >> File do not exist!")
showDoneMsg = False
# Sort the list
tempList.sort()
# Show me when is done if file exist
if showDoneMsg == True:
print("\n>>> DONE <<<\n")
# Logic show list items
def ShowListItems(thisList):
if len(thisList) == 0:
print("Temporary list is empty...")
else:
print("This is new items list:")
for i in thisList:
print(i)
# Execute function
GetListFromFile(file)
# Testing if items was sorted
ShowListItems(tempList)
Out put:
========================= RESTART: D:\Python\StackOverflow\help.py =========================
Dublicate >> 43273,-166.666666666667,4,3,1,100,0,0
>>> DONE <<<
21082,-250,4,3,1,100,0,0
21082,410.958904109589,4,3,1,60,1,0
22725,-142.857142857143,4,3,1,100,0,0
...
474434,-166.666666666667,4,3,1,100,0,0
481477,-100,4,3,1,100,0,1
531564,259.011439671919,4,3,1,60,1,0
>>>

How to pair 2 list into 1 list

I have a code like this:
def datauji(self):
uji = []
for x in self.fiturs:
a = [x[0],x[-5:]] #I think the problem in this line
uji.append(a)
return uji
with open('DataUjiBaru.csv','wb') as dub:
testing = csv.writer(dub)
datatest = d.datauji()
datatest.pop(0)
for x in datatest:
testing.writerow(x)
I want to pair the value in self.fiturs, In self.fiturs:
F37,0,1,0,1,1,1,0,1,0,2,1,0,0,0,1
F10,8,4,3,3,3,6,8,5,8,4,8,4,5,6,4
F8,1,0,2,0,0,0,2,0,0,0,0,0,2,0,0
So i want to pair index[0] and index[-5:] and write it to the csv, and the output on the csv like this:
F37,"['1', '0', '0', '0', '1']"
F10,"['8', '4', '5', '6', '4']"
F8,"['0', '0', '2', '0', '0']"
My Expectation in that csv is like this:
F37,1,0,0,0,1
F10,8,4,5,6,4
F8,0,0,2,0,0
How can I fix that?

You were correct about the issue with your code, it is found in the line:
a = [x[0],x[-5:]]
This creates nested items that look like this:
['F37', ['1', '0', '0', '0', '1']]
Here are two ways to fix this:
Option 1 - Use the splat* operator:
a = [x[0],*x[-5:]]
Option 2 - Concatenate two slices of your list:
a = x[:1] + x[-5:]
Both of these will remove the nesting of your lists, and instead give you lines looking like:
['F37', '1', '0', '0', '0', '1']
Which you can then write to your output file.

Python, not able to get the index working in my list

I have a list that contains some values as given below:
PC1=list();
PC1=[57,49,41,33,25,17,9,1,58,50,42,34,26,18,10,2,59,51,43,35,27,19,11,3,60,52,44,36,63,55,47,39,31,23,15,7,62,54,46,38,30,22,14,6,61,53,45,37,29,21,13,5,28,20,12,4]
Thats ok. Now, I do have a string that I enter an "bitsequence" (well, its a sequence of characters 1 and 0...) to:
print ("The main key entered is: " + KEY)
#test key that I enter: 0001001100110100010101110111100110011011101111001101111111110001
Yepp, thats the first step in DES when initial preparing for creating the 16 subkeys I tries to achieve here. So when using the items in the PC1 list I thought that the corresponding character from my KEY string should be picked, I iterate according to:
KEY_PERM = list();
i=0
for i in range(0,56):
print ("index", i, "PC1 ", PC1[i], "value from KEY ", KEY[PC1[i]])
KEY_PERM.insert(i, KEY[PC1[i]])
mm, by this the KEY_PERM list that is now populated based upon the PC1 index should be: 11110000110011001010101011110101010101100110011110001111
but.... it is not, it gives me the list:
['1', '1', '0', '0', '1', '1', '0', '0', '1', '0', '1', '0', '1', '0', '1', '0', '1', '1', '1', '1', '1', '1', '1', '1', '0', '1', '1', '1', '1', '1', '0', '1', '1', '1', '0', '1', '0', '1', '0', '1', '0', '1', '0', '1', '0', '1', '1', '0', '0', '1', '1', '0', '1', '0', '0', '0']
And that is incorrect, for instance, the character in place 41 and 33 in the KEY string is not 0....but 1!
11110000110011001010101011110101010101100110011110001111 and not, as now, 0 and 0 as returned characters.
Please help me out! Yepp, I am a total beginner in Python, but wants to learn. I guess that the initial zeroes in the string might be treated wrong? or that the index runs crazy due to me not looping correctly, or something else... like wrong types etc. I dont now.

I think you got confused what position 33 means. When not in the context of programming, then you would look at the 33rd entry in your list, which in your case is 1.
In Python and many other languages arrays and lists are indexed, starting at 0. So the 1st entry in your list has index 0. So when you tell python key[33] it actually looks up the 34th entry in your list. That is 0 as the script also outputs. What you want is the 33rd entry in the list. Which has index 33-1=32. So you need to do:
for i in range(0,56):
print ("index", i, "PC1 ", PC1[i], "value from KEY ", KEY[PC1[i]-1])
KEY_PERM.insert(i, KEY[PC1[i]-1])
print(''.join(KEY_PERM))
>>> '11110000110011001010101011110101010101100110011110001111'
Notice PC1[i]-1 when using the entry from PC1[i] to acces KEY
Complete Test Script:
PC1=[57,49,41,33,25,17,9,1,58,50,42,34,26,18,10,2,59,51,43,35,27,19,11,3,60,52,44,36,63,55,47,39,31,23,15,7,62,54,46,38,30,22,14,6,61,53,45,37,29,21,13,5,28,20,12,4]
KEY='0001001100110100010101110111100110011011101111001101111111110001'
KEY_PERM=[]
for i in range(0,56):
KEY_PERM.insert(i, KEY[PC1[i]-1])
print(''.join(KEY_PERM))
prints: 11110000110011001010101011110101010101100110011110001111

Filtering out a generator

Whats the best way to filter out some subsets from a generator. For example I have a string "1023" and want to produce all possible combinations of each of the digits. All combinations would be:
['1', '0', '2', '3']
['1', '0', '23']
['1', '02', '3']
['1', '023']
['10', '2', '3']
['10', '23']
['102', '3']
['1023']
I am not interested in a subset that contains a leading 0 on any of the items, so the valid ones are:
['1', '0', '2', '3']
['1', '0', '23']
['10', '2', '3']
['10', '23']
['102', '3']
['1023']
I have two questions.
1) If using a generator, whats the best way to filter out the ones with leading zeroes. Currently, I generate all combinations then loop through it afterwards and only continuing if the subset is valid. For simplicity I am only printing the subset in the sample code. Assuming the generator that was created is very long or if it constains a lot of invalid subsets, its almost a waste to loop through the entire generator. Is there a way to stop the generator when it sees an invalid item (one with leading zero) then filter it off 'allCombinations'
2) If the above doesn't exist, whats a better way to generate these combinations (disregarding combinations with leading zeroes).
Code using a generator:
import itertools
def isValid(subset): ## DIGITS WITH LEADING 0 IS NOT VALID
valid = True
for num in subset:
if num[0] == '0' and len(num) > 1:
valid = False
break
return valid
def get_combinations(source, comb):
res = ""
for x, action in zip(source, comb + (0,)):
res += x
if action == 0:
yield res
res = ""
digits = "1023"
allCombinations = [list(get_combinations(digits, c)) for c in itertools.product((0, 1), repeat=len(digits) - 1)]
for subset in allCombinations: ## LOOPS THROUGH THE ENTIRE GENERATOR
if isValid(subset):
print(subset)

Filtering for an easy and obvious condition like "no leading zeros", it can be more efficiently done at the combination building level.
def generate_pieces(input_string, predicate):
if input_string:
if predicate(input_string):
yield [input_string]
for item_size in range(1, len(input_string)+1):
item = input_string[:item_size]
if not predicate(item):
continue
rest = input_string[item_size:]
for rest_piece in generate_pieces(rest, predicate):
yield [item] + rest_piece
Generating every combination of cuts, so long it's not even funny:
>>> list(generate_pieces('10002', lambda x: True))
[['10002'], ['1', '0002'], ['1', '0', '002'], ['1', '0', '0', '02'], ['1', '0', '0', '0', '2'], ['1', '0', '00', '2'], ['1', '00', '02'], ['1', '00', '0', '2'], ['1', '000', '2'], ['10', '002'], ['10', '0', '02'], ['10', '0', '0', '2'], ['10', '00', '2'], ['100', '02'], ['100', '0', '2'], ['1000', '2']]
Only those where no fragment has leading zeros:
>>> list(generate_pieces('10002', lambda x: not x.startswith('0')))
[['10002'], ['1000', '2']]
Substrings that start with a zero were never considered for the recursive step.

One common solution is to try filtering just before using yield. I have given you an example of filtering just before yield:
import itertools
def my_gen(my_string):
# Create combinations
for length in range(len(my_string)):
for my_tuple in itertools.combinations(my_string, length+1):
# This is the string you would like to output
output_string = "".join(my_tuple)
# filter here:
if output_string[0] != '0':
yield output_string
my_string = '1023'
print(list(my_gen(my_string)))
EDIT: Added in a generator comprehension alternative
import itertools
my_string = '1023'
my_gen = ("".join(my_tuple)[0] for length in range(len(my_string))
for my_tuple in itertools.combinations(my_string, length+1)
if "".join(my_tuple)[0] != '0')

Doctest not running tests

import doctest
def create_grid(size):
grid = []
for i in range(size):
row = ['0']*size
grid.append(row)
"""
>>> create_grid(4)
[['0', '0', '0', '0'], ['0', '0', '0', '0'],
['0', '0', '0', '0'], ['0', '0', '0', '0']]
"""
return grid
if __name__ == '__main__':
doctest.testmod()
Running the above with python Test_av_doctest.py -v gives the following message:
2 items had no tests:
__main__
__main__.create_grid
0 tests in 2 items.
0 passed and 0 failed.
Test passed.
Any idea why this error occurs?

The issue is that your doctest-formatted string isn't a docstring.
Which docstrings are examined?
The module docstring, and all function, class and method docstrings are searched.
If you move the testing string below the function definition, it will become a function docstring, and thus will be targeted by doctest:
def create_grid(size):
"""
>>> create_grid(4)
[['0', '0', '0', '0'], ['0', '0', '0', '0'],
['0', '0', '0', '0'], ['0', '0', '0', '0']]
"""
grid = []
for i in range(size):
row = ['0']*size
grid.append(row)
return grid
if __name__ == '__main__':
doctest.testmod()
$ python Test_av_doctest.py -v
...
1 passed and 0 failed.
Test passed.