Develop Reference

I'm reading into a 256 byte string. I want to skip it, if it's all binary zeros (\x00) Is there a single test?

Totally new to python. Trying to parse a file but not all records contain data. I want to skip the records that are all hex 00.
if record == ('\x00' * 256): from a sample of print("-"*80))
gave a Syntax error, hey I said I was new. :)
Thanks for the reply, I'm using 2.7 and reading like this....
with open(testfile, "rb") as f:
counter = 0
while True:
record = f.read(256)
counter += 1

Your example looks to be very close. I'm not sure about Python 2, but in Python 3 you should specify that a string is binary.
I would do something like:
empty = b'\x00' * 256
if record == empty:
print('skipped this line')
Remember that Python 2 uses print statements, so you should do print 'skipped this line' instead.

python from hex to shellcode format

I try to convert a hex string to shellcode format
For example: I have a file in hex string like aabbccddeeff11223344
and I want to convert that through python to show this exact format:
"\xaa\xbb\xcc\xdd\xee\xff\x11\x22\x33\x44" including the quotes "".
My code is:
with open("file","r") as f:
a = f.read()
b = "\\x".join(a[i:i+2] for i in range(0, len(a), 2))
print b
so my output is aa\xbb\xcc\xdd\xee\xff\x11\x22\x33\x44\x.
I understand I can do it via sed command but I wonder how I may accomplish this through python.

The binascii standard module will help here:
import binascii
print repr(binascii.unhexlify("aabbccddeeff11223344"))
Output:
>>> print repr(binascii.unhexlify("aabbccddeeff11223344"))
'\xaa\xbb\xcc\xdd\xee\xff\x11"3D'

Data reading - csv

I have some datas in a .dfx file and I trying to read it as a csv with pandas. But it has some special characters which are not read by pandas. They are separators as well.I attached one line from it
The "DC4" is being removed when I print the file. The SI is read as space, correctly. I tried some encoding (utf-8, latin1 etc), but no success.
I attached the printed first line as well. I marked the place where the characters should be.
My code is simple:
import pandas
file_log = pandas.read_csv("file_log.DFX", header=None)
print(file_log)
I hope I was clear and someone has an idea.
Thanks in advance!
EDIT:
The input. LINK: drive.google.com/open?id=0BxMDhep-LHOIVGcybmsya2JVM28
The expected output:
88.4373 0 12.07.2014/17:05:22 38.0366 38.5179 1.3448 31.9839
30.0070 0 12.07.2014/17:14:27 38.0084 38.5091 0.0056 0.0033

By examining the example.DFX in hex (with xxd), the two separators are 0x14 and 0x0f accordingly.
Read the csv with multiple separators using python engine:
import pandas
sep1 = chr(0x14) # the one shows dc4
sep2 = chr(0x0f) # the one shows si
file_log = pandas.read_csv('example.DFX', header=None, sep='{}|{}'.format(sep1, sep2), engine='python')
print file_log
And you get:
0 1 2 3 4 5 6 7
0 88.4373 0 12.07.2014/17:05:22 38.0366 38.5179 1.3448 31.9839 NaN
1 30.0070 0 12.07.2014/17:14:27 38.0084 38.5091 0.0056 0.0033 NaN
It seems it has an empty column at the end. But I'm sure you can handle that.

The encoding seems to be ASCII here. DC4 stands for "device control 4" and SI for "Shift In". These are control characters in an ASCII file and not printable. Thus you cannot see them when you issue a "print(file_log)", although it might do something depending on your terminal to view this (like \n would do a new-line).
Try typing file_log in your interpreter to get the representation of that variable and check if those special characters are included. Chances are that you'll see DC4 in the representation as '\x14' which means hexadecimal 14.
You may then further process these strings in your program by using string manipulation like replace.

Raspberry Pi - How to print hex value?

I m using Python in Raspberry Pi and my packet is in hexadecimal value like "0x750x010x010x060x000x08". I want serial communication between UART and Raspberry Pi so I wrote a program using Python in Raspberry Pi and I'm checking data on terminal but when I selected ASCII option in terminal it showing below output:
75
01
01
06
00
08
And when I selected hex option in terminal it is not showing above output. Now I want above output when I will select hex option but not ASCII option. So how to get that? If I need to convert it into hex or byte or any other than tell me the code in Python.

import serial
port=serial.Serial("/dev/ttyAMA0",baudrate=115200,timeout=3.0)
while True:
a="0x750x010x010x060x000x08"
b=a.replace("0x"," ")
#print b
alist = b.split(" ")
for element in alist
port.write("\r\n"+str(element))
this gives the desired formatted data you want

First of all your for loop and if statment are used wrong here, and they are not required. The while loop can be equivently rewriten as:
a = "0x750x010x010x060x000x08"
b = a.replace("0x", " ")
while True:
port.write("\r\n"+b)
Your main misanderstanding is that you assume Python understands you want to iterate over hex numbers. But it doesn't. In you original loop it just iterates the a string letter by letter. And in fact at each iteration of for loop you just changes the orignal a string to " 75 01 01 06 00 08" and send it as a string.
If you need to send bytes, you should split your string into separate records with each byte's infromation and convert these records to bytes.
Here is the code for that
a = "0x750x010x010x060x000x08"
b1 = a.split("0x")
# b1 is ["", "75", "01", "01", "06", "00", "08"], the values are still strings
b2 = [int(x, 16) for x in b1[1:]]
# b2 is a list of integers, the values calculated from strings assuming they are hex (16-bit) numbers
# use b1[1:] to cut the first blank string
b = str(bytearray(b2))
#b joins values list into a bytes string (each interger transformed into one byte)
while True:
port.write("\r\n" + b)
Update for question in comments:
If a format is like this "0x750101060008", you just split it by 2 letters:
b1 = [a[2*i:2*i+2] for i in range(len(a)/2)]

import serial
port=serial.Serial("/dev/ttyAMA0",baudrate=115200,timeout=3.0)
while True:
a="0x750x010x010x060x000x08"
b=a.replace("0x"," ")
#print b
alist = b.split(" ")
for element in alist
port.write("\r\n"+str(element))
this one

Ignore newline character in binary file with Python?

I open my file like so :
f = open("filename.ext", "rb") # ensure binary reading with b
My first line of data looks like this (when using f.readline()):
'\x04\x00\x00\x00\x12\x00\x00\x00\x04\x00\x00\x00\xb4\x00\x00\x00\x01\x00\x00\x00\x08\x00\x00\x00\x00\x00\x00\x00\x18\x00\x00\x00\x01\x00\x00\x00\x02\x00\x00\x00\x03\x00\x00\x00\x04\x00\x00\x00\x05\x00\x00\x00\x06\x00\x00\x00:\x00\x00\x00;\x00\x00\x00<\x00\x00\x007\x00\x00\x008\x00\x00\x009\x00\x00\x00\x07\x00\x00\x00\x08\x00\x00\x00\t\x00\x00\x00\n'
Thing is, I want to read this data byte by byte (f.read(4)). While debugging, I realized that when it gets to the end of the first line, it still takes in the newline character \n and it is used as the first byte of the following int I read. I don't want to simply use .splitlines()because some data could have an n inside and I don't want to corrupt it. I'm using Python 2.7.10, by the way. I also read that opening a binary file with the b parameter "takes care" of the new line/end of line characters; why is not the case with me?
This is what happens in the console as the file's position is right before the newline character:
>>> d = f.read(4)
>>> d
'\n\x00\x00\x00'
>>> s = struct.unpack("i", d)
>>> s
(10,)

(Followed from discussion with OP in chat)
Seems like the file is in binary format and the newlines are just mis-interpreted values. This can happen when writing 10 to the file for example.
This doesn't mean that newline was intended, and it is probably not. You can just ignore it being printed as \n and just use it as data.

You should just be able to replace the bytes that indicate it is a newline.
>>> d = f.read(4).replace(b'\x0d\x0a', b'') #\r\n should be bytes b'\x0d\x0a'
>>> diff = 4 - len(d)
>>> while diff > 0: # You can probably make this more sophisticated
... d += f.read(diff).replace(b'\x0d\x0a', b'') #\r\n should be bytes b'\x0d\x0a'
... diff = 4 - len(d)
>>>
>>> s = struct.unpack("i", d)
This should give you an idea of how it will work. This approach could mess with your data's byte alignment.
If you really are seeing "\n" in your print of d then try .replace(b"\n", b"")