Develop Reference

How to implement a stopping condition for splitting an image matrix into pixels?

I'm currently using Python to split a binary list into quadrants. For example:
11110000 1111 0000 11 11 00 00
11110000 1111 0000 11 11 00 00
00011100 0001 1100
00011100 ==> 0001 1100 00 01 11 00
11110000 00 01 11 00
11110000 1111 0000 ==>
11110011 1111 0000 11 11 00 00
11110011 1111 0011 11 11 00 00
1111 0011
11 11 00 11
11 11 00 11
My program receives an input n which signifies the size of the n x n matrix. In the example above, n = 8.
The function that I've written to do that is:
def divide_to_quadrants(original):
quad_size = len(original) // 2
q1 = [i[:quad_size] for i in original][:quad_size]
q2 = [i[quad_size:] for i in original][:quad_size]
q3 = [i[:quad_size] for i in original][quad_size:]
q4 = [i[quad_size:] for i in original][quad_size:]
return [q1, q2, q3, q4]
I want to ultimately end up with quadrants that are only a single unit, but I'm having trouble coming up with a way to implement that with varying values of n.
I thought of indexing one of the quadrants and checking to see if the length is 1, but since each image size will have varying number of times until they reach single-unit quadrants, that doesn't seem like a viable solution either.
Is there any way that I would be able to do this? Thanks.
Edit
n is an exponent of 2, and the quadrants should be split into equal sizes. That means that an image of 8 x 8 would be split into four quadrants of 4 x 4, and an image of size 32 x 32 would be split into four 16 x 16 and so on.
The image is a Python list which contains strings, so it looks like ['11110000', '11110000', '00011100' ...]. I believe that it wouldn't be a problem if these strings were separate integers though.

It may actually be easier to do this iteratively than recursively:
def pair(values):
iterator = iter(values)
return [(x, next(iterator)) for x in iterator]
def format_block(lines):
formatted_lines = [" ".join("".join(unit) for unit in pair(line)) for line in lines]
formatted_output = "\n\n".join("\n".join(line_pair) for line_pair in pair(formatted_lines))
return formatted_output
Result
>>> lines = ["11111111"] * 8
>>> print(format_block(lines))
11 11 11 11
11 11 11 11
11 11 11 11
11 11 11 11
11 11 11 11
11 11 11 11
11 11 11 11
11 11 11 11

How to keep the header and trailer while zlib decompress and compress

I have raw data extracted from PDF and I decompressed the raw data and compressed it again.
I expected the same header and trailer, but the header was changed.
Original Hex Header
48 89 EC 57 ....
Converted Hex Header
78 9C EC BD ...
I dug into zlib compression and got header 48 also is one of zlib.header.
But mostly 78 is used for zlib compression.
It's my code which decompress and compress:
decompress_wbit = 12
compress_variable = 6
output_data = zlib.decompress(open(raw_data, "rb").read(), decompress_wbit)
output_data = zlib.compress(output_data, 6)
output_file = open(raw_data + '_', "wb")
output_file.write(output_data)
output_file.close()
I changed the decompress_wbit and compress_variable but still keeps 78.
So not sure how to get 48 as header.
Here is the short description about zlib.header.
CINFO (bits 12-15)
Indicates the window size as a power of two, from 0 (256 bytes) to 7 (32768 bytes). This will usually be 7. Higher values are not allowed.
CM (bits 8-11)
The compression method. Only Deflate (8) is allowed.
FLEVEL (bits 6-7)
Roughly indicates the compression level, from 0 (fast/low) to 3 (slow/high)
FDICT (bit 5)
Indicates whether a preset dictionary is used. This is usually 0. 1 is technically allowed, but I don't know of any Deflate formats that define preset dictionaries.
FCHECK (bits 0-4)
A checksum (5 bits, 0..31), whose value is calculated such that the entire value divides 31 with no remainder.
Typically, only the CINFO and FLEVEL fields can be freely changed, and FCHECK must be calculated based on the final value.* Assuming no preset dictionary, there is no choice in what the other fields contain, so a total of 32 possible headers are valid. Here they are:
FLEVEL: 0 1 2 3
CINFO:
0 08 1D 08 5B 08 99 08 D7
1 18 19 18 57 18 95 18 D3
2 28 15 28 53 28 91 28 CF
3 38 11 38 4F 38 8D 38 CB
4 48 0D 48 4B 48 89 48 C7
5 58 09 58 47 58 85 58 C3
6 68 05 68 43 68 81 68 DE
7 78 01 78 5E 78 9C 78 DA
Please let me know how to keep the zlib.header while decompression & compression
Thanks for your time.

I will first note that it doesn't matter. The data will be decompressed fine with that zlib header. Why do you care?
You are giving zlib.compress a small amount of data that permits a smaller window. Since it is permitted, the Python library is electing to compress with a smaller window.
A way to avoid that would be to use zlib.compressobj instead. Upon initiation, it doesn't know how much data you will be feeding it and will default to the largest window size.

Python: convert hex bytestream to “int16"

So I'm working with incoming audio from Watson Text to Speech. I want to play the sound immediately when data arrives to Python with a websocket from nodeJS.
This is a example of data I'm sending with the websocket:
<Buffer e3 f8 28 f9 fa f9 5d fb 6c fc a6 fd 12 ff b3 00 b8 02 93 04 42 06 5b 07 e4 07 af 08 18 0a 95 0b 01 0d a2 0e a4 10 d7 12 f4 12 84 12 39 13 b0 12 3b 13 ... >
So the data arrives as a hex bytestream and I try to convert it to something that Sounddevice can read/play. (See documentation: The types 'float32', 'int32', 'int16', 'int8' and 'uint8' can be used for all streams and functions.) But how can I convert this?
I already tried something, but when I run my code I only hear some noise, nothing recognizable.
Here you can read some parts of my code:
def onMessage(self, payload, isBinary):
a = payload.encode('hex')
queue.put(a)
After I receive the bytesstream and convert to hex, I try to send the incoming bytestream to Sounddevice:
def stream_audio():
with sd.OutputStream(channels=1, samplerate=24000, dtype='int16', callback=callback):
sd.sleep(int(20 * 1000))
def callback(outdata, frames, time, status):
global reststuff, i, string
LENGTH = frames
while len(reststuff) < LENGTH:
a = queue.get()
reststuff += a
returnstring = reststuff[:LENGTH]
reststuff = reststuff[LENGTH:]
for char in returnstring:
i += 1
string += char
if i % 2 == 0:
print string
outdata[:] = int(string, 16)
string = ""

look at your stream of data:
e3 f8 28 f9 fa f9 5d fb 6c fc a6 fd 12 ff b3 00
b8 02 93 04 42 06 5b 07 e4 07 af 08 18 0a 95 0b
01 0d a2 0e a4 10 d7 12 f4 12 84 12 39 13 b0 12
3b 13
you see here that every two bytes the second one is starting with e/f/0/1 which means near zero (in two's complement).
So that's your most significant bytes, so your stream is little-endian!
you should consider that in your conversion.
If I have more data I would have tested but this is worth some miliseconds!

Assigning strings to a variable in python

Consider the below string which will be given as the input to a function.
01 02 01 0D A1 D6 72 02 00 01 00 00 00 00 53 73 F2
The highlighted part is the address I need.
If the preceding byte is 1 then I have to take only 6 octet and assign it to a variable.
If it is more than 1 the I should read 6 * Num(preceding value) and assign 6 octets for each variable.
Currently I am assigning it statically.
def main(line_input):
Device = ' '.join(line_input[9:3:-1])
Length = line_input[2]
var1 = line_input[3]
main("01 02 02 0D A1 D6 72 02 00 01 00 00 00 00 53 73 F2")
Can this be done?

Here I think this does it, let me know if there is anything that needs changing:
import string
def address_extract(line_input):
line_input = string.split(line_input, ' ')
length = 6 * int(line_input[2])
device_list = []
for x in range(3, 3+length, 6):
if x+6 > len(line_input):
print "Length multiplier too long for input string"
else:
device_list.append(' '.join(line_input[x:x+6]))
return device_list
print address_extract("01 02 02 0D A1 D6 72 02 00 01 00 00 00 00 53 73 F2")
#output = ['0D A1 D6 72 02 00', '01 00 00 00 00 53']

Here is some code that I hope will help you. I tried to add many comments to explain what is happening
import binascii
import struct
#note python 3 behaves differently and won't work with this code (personnaly I find it easyer for strings convertion to bytes)
def main(line_input):
formated_line = line_input.split(" ") #I start by cutting the input on each space character
print formated_line #the output is a list. Each element is composed of 2 chars
formated_line = [binascii.unhexlify(xx) for xx in formated_line] #create a list composed of unhelified bytes of each elements of the original list
print formated_line #the output is a list of bytes char
#can be done in one step but I try to be clearer as you are nee to python (moereover this is easyer in python-3.x)
formated_line = map(ord, formated_line) #convert to a list of int (this is not needed in python 3)
print formated_line
Length = formated_line[2] #this is an int
unformated_var1 = formated_line[3:3+(6*length)] #keep only interesting data
#now you can format your address
main("01 02 02 0D A1 D6 72 02 00 01 00 00 00 00 53 73 F2")
#if the input comes from a machine and not a human, they could exchange 17bytes instead of (17x3)characters
#main("\x01\x02\x02\x0D\xA1\xD6\x72\x02\x00\x01\x00\x00\x00\x00\x53\x73\xF2")
#then the parsing could be done with struct.unpack

Python error reading from file buffer instance

I am trying to read from the powerball winning numbers file:
http://www.powerball.com/powerball/winnums-text.txt
I am trying to get it line by line and I have this code:
import urllib.request
with urllib.request.urlopen("http://www.powerball.com/powerball/winnums-text.txt") as file:
next(file)
for line in file:
line.lstrip("b'")
line.rstrip(" \r\n'")
print(line)
Each line in the file prints out like this:
b'12/06/1997 15 26 28 08 43 36 \r\n'
b'12/03/1997 18 09 14 47 42 32 \r\n'
b'11/29/1997 11 27 13 02 31 23 \r\n'
b'11/26/1997 15 46 34 23 40 35 \r\n'
b'11/22/1997 22 31 03 07 14 02 \r\n'
I am getting the error:
File "powerball.py", line 5, in <module>
line.lstrip("b'")
TypeError: 'str' does not support the buffer interface
I am trying to get rid of the excess characters and make the line like this:
12/06/1997 15 26 28 08 43 36
How do I fix this?

As someone already mentioned, the file is being read in binary mode. You need to convert the string to a text encoding format.
You can solve this with:
line = line.decode("utf-8","ignore")
This should give you the behaviour you expect.

I highly recommend use use pandas for this kind of IO, it will handle the http request, the parsing, everything in a single line of code; as a bonus you can use it for your data analysis too:
import pandas as pd
df = pd.read_csv('http://www.powerball.com/powerball/winnums-text.txt')
print(df)
Draw Date WB1 WB2 WB3 WB4 WB5 PB PP
0 02/24/2016 67 21 65 31 64 05 3
1 02/20/2016 54 12 11 16 15 25 5
2 02/17/2016 29 27 07 40 17 25 2
3 02/13/2016 07 15 36 18 19 20 2
4 02/10/2016 02 62 40 50 03 05 2
5 02/06/2016 13 04 36 31 52 08 3
6 02/03/2016 26 60 67 31 28 23 3
7 01/30/2016 16 05 12 31 43 18 4
8 01/27/2016 40 52 03 67 12 21 2
9 01/23/2016 32 22 40 69 34 19 4
10 01/20/2016 44 05 39 69 47 24 5
11 01/16/2016 61 52 51 64 03 06 2

line is a byte sequence, not a string. Convert it to a string using the str function.
import urllib.request
with urllib.request.urlopen("http://www.powerball.com/powerball/winnums-text.txt") as file:
next(file)
for bline in file:
line = str(bline, "utf-8")
print(line)

for line in file:
line.lstrip("b'")
line.rstrip(" \r\n'")
print(line)
You have mistaken the representation of the value, versus the value itself.
The values coming from that file are not text (str); they are byte sequences (bytes), which Python's programmer representation communicates to you by enclosing the string in b'…'. Those enclosing characters are not part of the value; you won't succeed in removing them.
Instead, you need to create a text string from the bytes. You do this by telling the byte sequence to decode itself:
for line_bytes in file:
line = line_bytes.decode("utf-8")
print(line)
This requires knowing the text codec for that byte sequence (the above code assumes “utf-8”). You can interrogate the HTTP response to ask the codec, unless you know how to get it elsewhere.
An alternative would be to open the file such that it knows its own text codec; then the items you retrieve from it would already be text.