Develop Reference

Open and read all binary files from directory an output to lists - Python [duplicate]

This question already has answers here:
How can I use `return` to get back multiple values from a loop? Can I put them in a list?
(2 answers)
Apply function to each element of a list
(4 answers)
Closed 6 months ago.
I'm trying to output the entire contents of multiple .pkt files, containing binary data, as a series of hexadecimal lists: (Adapted from How to open every file in a folder)
import glob
path = 'filepath'
for filename in glob.glob(os.path.join(path, '*.pkt')):
with open(os.path.join(os.getcwd(), filename), 'rb') as f:
pair_hex = ["{:02x}".format(c) for c in f.read()]
print(pair_hex)
Which outputs:
['09', '04', '04', '04', '04', '04', '04', '04', '04', '0b', '09']
['09', '04', 'bb']
['09', 'bb']
['bb']
['09', '04', '0b', '09']
That makes sense, because i'm looping through the files, but what I need is:
[['09', '04', '04', '04', '04', '04', '04', '04', '04', '0b', '09'],['09', '04', 'bb'],['09', 'bb'],['bb'],['09', '04', '0b', '09']]
So I can manipulate all the data.
I have tried to apply append(), "".join(), map() as well as the advice at How to merge multiple lists into one list in python?, but nothing changes the output. How can I get the desired list of lists?

untested but try
import glob, os
path = 'filepath'
ret = []
for filename in glob.glob(os.path.join(path, '*.pkt')):
with open(os.path.join(os.getcwd(), filename), 'rb') as f:
pair_hex = ["{:02x}".format(c) for c in f.read()]
ret.append(pair_hex)
print(ret)
the above prints the following on my console which is the same as your "desired output"
[['09', '04', '04', '04', '04', '04', '04', '04', '04', '0b', '09'], ['09', '04', 'bb'], ['09', 'bb'], ['bb'], ['09', '04', '0b', '09']]
and this is what I used to create the .pkt files on my machine with out set to a copy--paste of your "desired output"
out = [['09', '04', '04', '04', '04', '04', '04', '04', '04', '0b', '09'],['09', '04', 'bb'],['09', 'bb'],['bb'],['09', '04', '0b', '09']]
for i, a in enumerate(out):
with open(f"{i}.pkt", 'w') as f:
f.write(''.join(map(lambda s: chr(int(s, 16)), a)))

Generating all digit strings of constant length

What would be the easiest way to generate all the digit strings of a constant length?
Lets say length == 3, so all the strings would be:
000
001
002
...
010
011
etc...
Thanks!

Try this, it works for any given n - the expected length of the output:
def generate(n):
return [str(x).zfill(n) for x in xrange(10**n)]
For example:
generate(2)
=> ['00', '01', '02', ... , '97', '98', '99']

Just create all numbers of that length (0 to 10**length). Then convert them to strings with string.format(). The option 0:<length> makes sure they get appended to the correct number of zeroes.
def fun(length):
form = '{:0'+str(length)+'}'
return [form.format(n) for n in range(10**length)]
In [1]: fun(2)
Out[1]:
['00',
'01',
'02',
'03',
'04',
'05',
'06',
'07',
'08',
'09',
...

python set manipulation :the simple script does not output right set after unioning or disjoining?

all_tags = ['24', '02', '26', '03', '33', '32', '31', '30', '29', '68', '11']
ref_tag = str('24')
union_tags = set(all_tags) | set(ref_tag)
left_tags = set(all_tags) - set(ref_tag)
print(union_tags)
print(left_tags)
The above is the simple code which I expect elements in union_tags should be the same as those in all_tags. However, the result is
set
(['24', '02', '26', '03', '33', '32', '31', '30', '29', '68', '2', '4', '11'])
The union_tags instead contains two extra elements '2' and '4', which I think it is the result splitting the str '24'.
Again, left_tags should exclude element '24'. However, the result still have the '24'.
Please let me know why. I use the python 2.7 as the interpreter.

Set function accept an iterable with hashable items and convert it to a set object, and since strings are iterables when you pass the string 24 to your set function it converts your string to following set:
{'2', '4'}
And at last the unioin of this set with all_tags would contain items 2 and 4.
If you want to put the 24 in a set as one item you can use {} in order to create your expected set:
>>> ref_tag = {'24'}
set(['24'])

Check whether a list of hexadecimal values is sequential or not

Want a function / statement, to check whether all the values of mylist are sequential or not, which is hexadecimal list.
For example:
def checkmylist(mylist):
#code returns True or False
mylist1 = ['03', '04', '05', '06', '07', '08', '09', '0a', '0b', '0c','0d', '0e', '0f']
mylist2 = ['03', '05', '06', '07', '08', '09', '0a', '0b', '0c','0d', '0e', '0f']
checkmylist(mylist1)
#expected to returns pass
checkmylist(mylist2)
#expected to returns fail

def checkmylist(mylist):
it = (int(x, 16) for x in mylist)
first = next(it)
return all(a == b for a, b in enumerate(it, first + 1))
With the first statement we convert the hex numbers to a generator of integers. With next(it) we take the first element of the generator. Then we enumerate the rest of the elements starting the numbering from first + 1. We conclude that we have a sequential list if the numbering of each element is the same as the element itself.

One hack to do it is. This finds the total elements in the list. As you mention that it has t be sequential, the last element must be length of the list more than the first element.
>>> def checkmylist(l):
... a = [int(i,16) for i in sorted(set(l))]
... return (len(a) == (a[-1]-a[0]+1))
...
>>> checkmylist(mylist1)
True
>>> checkmylist(mylist2)
False
>>> checkmylist(['2', '2', '4'])
False

You can use iter to create an iterator of your list (from second index to end) and then use all function to check if you have a sequence, note that int(next(it),16) (or as a more efficient way as mentioned in comment use functools.partial(int, base=16))will convert your string to integer with base 16 then you can do operation on them :
>>> import functools
>>> int16 = functools.partial(int, base=16)
>>> def checker(li):
... it=iter(li[1:])
... return all(int16(next(it))-int16(i)==1 for i in li[:-1])
Demo:
mylist1 = ['03', '04', '05', '06', '07', '08', '09', '0a', '0b', '0c','0d', '0e', '0f']
mylist2 = ['03', '05', '06', '07', '08', '09', '0a', '0b', '0c','0d', '0e', '0f']
>>> checker(mylist1)
True
>>> checker(mylist2)
False

mylist1 = ['03', '04', '05', '06', '07', '08', '09', '0a', '0b', '0c','0d', '0e', '0f']
mylist2 = ['03', '05', '06', '07', '08', '09', '0a', '0b', '0c','0d', '0e', '0f']
def checkmylist(li):
start=int(li[0],16)
for e in li[1:]:
a=int(e,16)
if a==start+1:
start=a
else:
return False
return True
assert checkmylist(mylist1)==True
#expected to returns pass
assert checkmylist(mylist2)==False
#expected to returns fail

Similar to #JuniorCompressor but using a list comprehension,
your base construct would be:
bool_match_to_myList = [1 if int(i, 16) else 0 for i in mylist]
You could check to see if that is true/false easy enough:
myList_is_sequential = sum([1 if int(i, 16) else 0 for i in mylist]) < 1
Or you can find the indexes where it is off (not sequential) with the help of numpy
import numpy as np
bool_match_to_myList_np = np.array([1 if int(i, 16) else 0 for i in mylist])
np.where(bool_match_to_myList == 0)[0]

Another option, using the "power" of Pandas:
def is_sequential(mylist):
serie = pd.Series(mylist).apply(lambda x: int(x,16))
return (serie.diff() > 1).sum() == 0
Transform the mylist variables in a int Series
Use the diff() function, to get the difference between the last and the current value
Sum all values more than 1 (the step that we wanted), so if the sum() is equal 0, all values are sequential
is_sequential(mylist1) # returns True
is_sequential(mylist2) # returns False

Convert range(r) to list of strings of length 2 in python

I just want to change a list (that I make using range(r)) to a list of strings, but if the length of the string is 1, tack a 0 on the front.
I know how to turn the list into strings using
ranger= map(str,range(r))
but I want to be able to also change the length of those strings.
Input:
r = 12
ranger = range(r)
ranger = magic_function(ranger)
Output:
print ranger
>>> ['00','01','02','03','04','05','06','07','08','09','10','11']
And if possible, my final goal is this:
I have a matrix of the form
numpy.array([[1,2,3],[4,5,6],[7,8,9]])
and I want to make a set of strings such that the first 2 characters are the row, the second two are the column and the third two are '01', and have matrix[row,col] of each one of these.
so the above values would look like such:
000001 since matrix[0,0] = 1
000101 since matrix[0,1] = 2
000101 since matrix[0,1] = 2
000201
000201
000201
etc

Use string formatting and list comprehension:
>>> lst = range(11)
>>> ["{:02d}".format(x) for x in lst]
['00', '01', '02', '03', '04', '05', '06', '07', '08', '09', '10']
or format:
>>> [format(x, '02d') for x in lst]
['00', '01', '02', '03', '04', '05', '06', '07', '08', '09', '10']

zfill does exactly what you want and doesn't require you to understand an arcane mini-language as with the various types of string formatting. There's a place for that, but this is a simple job with a ready-made built-in tool.
ranger = [str(x).zfill(2) for x in range(r)]

Here's my take on it:
>>> map('{:02}'.format, xrange(12))
['00', '01', '02', '03', '04', '05', '06', '07', '08', '09', '10', '11']
For your own enlightenment, try reading about the format string syntax.

Use string formatting:
>>> sr = []
>>> for r in range(11):
... sr.append('%02i' % r)
...
>>> sr
['00', '01', '02', '03', '04', '05', '06', '07', '08', '09', '10']

One liner in Python 3.x
>>> ranger = [f'{i:>02}' for i in range(1, 12)]
>>> print (ranger)
['01', '02', '03', '04', '05', '06', '07', '08', '09', '10', '11']