I have a sensor system. The sensors receive commands from me, do something and then send a response to me.
The response is like this:
seq. number | net_id | opcode_group | opcode | payloadlength | val
where I have these values delimited by a space character.
Now I want to take the last value named val. In this part, I have all the information I want to know to elaborate the response from the sensors.
For example, I have this response for the command that wants to know the IEEE MAC address of the sensor:
In this case val is all the fields after Length in the response. There are not separation, but I have a sort of string.
All I have to do is to split this array/string of numbers, just knowing only the length of every field. For ex. the status is 1 byte, the MAC address 8 byte, and so on...
My code is this:
if response.error:
ret['error'] = 'Error while retrieving unregistered sensors'
else:
for line in response.body.split("\n"):
if line != "":
value = int(line.split(" ")[6])
ret['response'] = value
self.write(tornado.escape.json_encode(ret))
self.finish()
if command == 'IDENTIFY':
status = value.split(" ")[0]
IEEEAddrRemoteDev = value.split(" ")[1]
NWKAddrRemoteDev = value.split(" ")[2]
NumOfAssociatedDevice = value.split(" ")[3]
StartIndex = value.split(" ")[4]
ListOfShortAddress = value.split(" ")[5]
if status == 0x00:
ret['success'] = "The %s command has been succesfully sent! \
IEEE address: %s" % % (command.upper(), IEEEAddrRemoteDev)
self.write(tornado.escape.json_encode(ret))
elif status == 0x80:
ret['success'] = "Invalid Request Type"
self.write(tornado.escape.json_encode(ret))
elif status == 0x81:
ret['success'] = "Device Not Found"
self.write(tornado.escape.json_encode(ret))
where in the first part I take the 6th value from the entire response and I put this in the variable value. After this I want to split this variable in every component.
For ex. this status = value.split(" ")[0] in which way I have to split????
Thank you very much for the help!
What is the exact format of the val field (i.e. the contents of value in your code)? Is it a string? A sequence of bytes?
If it's a string, you could use:
status = value[0]
IEEEAddrRemoteDev = value[1:8]
NWKAddrRemoteDev = value[9:2]
NumOfAssociatedDevice = value[11:1]
StartIndex = value[12:1]
ListOfShortAddress = value[13:2*NumOfAssociatedDevice]
If it's a sequence of bytes, then you could use the struct.unpack() - see here.
The problem is that you're treating this as a Python string, but it's just going to be a bunch of bits.
You need to use struct.unpack to split these up.
Related
i am very new to coding and I am not familiar with python, could you guys give me maybe a small example of how you would solve this problem.
Basically this new device i will be working on has a 2D code(its sort of a barcode kkind of thing) and when i scan the code witha 2D scanner a string like this shows up on my notepad for example: 58183#99AF0M000F9EF3F800
the last 12 characters are the MAC address and the first 5 characters are the order number.
i need to compare that(58183#99AF0M000F9EF3F800) with the MAC address value i get from the XML page.
here is the terminal output for more reference:
####################################################################################################
Current device information:
Order-number: 58184 Software-version: 1.0.0 ( Build : 1 ) Hardware version: 1.00 MAC address: 00:0F:9E:F4:1A:80
desired-Order-number: 58183 desired-Softwareversion: 1.0.0 ( Build : 1 ) desired-hardwareversion: 1.00 pc-praefix: 7A2F7
PASS
PS C:\Users\Aswin\Python Project>
The MAC address from the XML page has looks like this "00:0F:9E:F4:1A:80" and the 2D scancode looks like this "58183#99AF0M000F9EF3F800". how can i take the last 12 characters of this scan code and compare it with the mac address from the XML page to see if they match.
Any example of code blocks would be much appreciated guys.
try:
preflash = urllib.request.urlopen("http://10.10.10.2", timeout=3).getcode()
print("Web page status code:", preflash, "FAIL")
sys.exit(0)
except urllib.error.URLError:
correct = urllib.request.urlopen("http://192.168.100.5", timeout=10).getcode()
print("Web page status code:", correct)
print("IP address: 192.168.100.5 is reachable")
print(100*"#")
# Declare url String
url_str = 'http://192.168.100.2/globals.xml'
# open webpage and read values
xml_str = urllib.request.urlopen(url_str).read()
# Parses XML doc to String for Terminal output
xmldoc = minidom.parseString(xml_str)
# prints the order_number from the xmldoc
order_number = xmldoc.getElementsByTagName('order_number')
ord_nmr = order_number[0].firstChild.nodeValue
# prints the firmware_version from the xmldoc
firmware_version = xmldoc.getElementsByTagName('firmware_version')
frm_ver = firmware_version[0].firstChild.nodeValue
# prints the hardware_version from the xmldoc
hardware_version = xmldoc.getElementsByTagName('hardware_version')
hrd_ver = hardware_version[0].firstChild.nodeValue
v = hrd_ver.split()[-1]
# prints the mac_address from the xmldoc
mac_address = xmldoc.getElementsByTagName('mac_address')
mac_addr = mac_address[0].firstChild.nodeValue
print("Current device information: ")
print("Order-number: ",ord_nmr, "Software-version: ",frm_ver, "Hardware version: ",v, "MAC address: ",mac_addr)
d_ordernum = "58183"
d_hw_version = "1.00"
d_sf_version = "1.0.0 ( Build : 1 )"
pc_praefix = "7A2F7"
print("desired-Order-number: 58183 desired-Softwareversion: 1.0.0 ( Build : 1 ) desired-hardwareversion: 1.00 pc-praefix: 7A2F7")
if d_sf_version == frm_ver:
print("PASS")
else:
print("FAIL")
You could take the string from the scan code and slice it
scan_code_cropped = scancode_string[11:]
This will get you the last 12 characters of the scan code.
Now to get the MAC address in a format to be able to compare it to the scan code, split it on the basis of ":"
list_of_chars = mac_address_string.split(":")
this will get you the character list, which can be concatenated using
mac_address_string_joined = ''.join(list_of_chars)
and finally to compare the two strings
if scan_code_cropped == mac_address_string_joined:
print("Mac address & Scan code matched !")
If needed in a function format, here you go:
def match_scan_with_mac(scancode_string, mac_address_string):
# get the last 12 characters of the scan code
scan_code_cropped = scancode_string[11:]
# get the mac address without the ":"
list_of_chars = mac_address_string.split(":")
mac_address_string_joined = ''.join(list_of_chars)
# compare the MAC address and the scan string
if scan_code_cropped == mac_address_string_joined:
print("Mac address & Scan code matched !")
return True
return False
I am not sure what I am doing wrong here but I am trying to open a file, trace1.flow, read the header information then throw the source IP and destination IP into dictionaries. This is done in Python running on a Fedora VM. I am getting the following error:
(secs, nsecs, booted, exporter, mySourceIP, myDestinationIP) = struct.unpack('IIIIII',myBuf)
struct.error: unpack requires a string argument of length 24
Here is my code:
import struct
import socket
#Dictionaries
uniqSource = {}
uniqDestination = {}
def int2quad(i):
z = struct.pack('!I', i)
return socket.inet_ntoa(z)
myFile = open('trace1.flow')
myBuf = myFile.read(8)
(magic, endian, version, headerLen) = struct.unpack('HBBI', myBuf)
print "Magic: ", hex(magic), "Endian: ", endian, "Version: ", version, "Header Length: ", headerLen
myFile.read(headerLen - 8)
try:
while(True):
myBuf = myFile.read(24)
(secs, nsecs, booted, exporter, mySourceIP, myDestinationIP) = struct.unpack('IIIIII',myBuf)
mySourceIP = int2quad(mySourceIP)
myDestinationIP = int2quad(myDestinationIP)
if mySourceIP not in uniqSource:
uniqSource[mySourceIP] = 1
else:
uniqSource[mySourceIP] += 1
if myDestinationIP not in uniqDestination:
uniqDestination[myDestinationIP] = 1
else:
uniqDestination[myDestinationIP] += 1
myFile.read(40)
except EOFError:
print "END OF FILE"
You seem to assume that file.read will raise EOFError on end of file, but this error is only raised by input() and raw_input(). file.read will simply return a string that's shorter than requested (possibly empty).
So you need to check the length after reading:
myBuf = myFile.read(24)
if len(myBuf) < 24:
break
Perhaps your have reached end-of-file. Check the length of myBuf:
len(myBuf)
It's probably less than 24 chars long. Also you don't need those extra parenthesis, and try to specify duplicated types using 'nI' like this:
secs, nsecs, booted, exporter, mySourceIP, myDestinationIP = struct.unpack('6I',myBuf)
I am telneting to my server, which answers to me with messages and at the end of each message is appended hex00 (null character) which cannot be read. I tried searching through and through, but can't seem to make it work, a simple example:
from telnetlib import Telnet
connection = Telnet('localhost', 5001)
connection.write('aa\n')
connection.read_eager()
This returns an output:
'Fail - Command aa not found.\n\r'
whereas there should be sth like:
'Fail - Command aa not found.\n\r\0'
Is there any way to get this end of string character? Can I get bytes as an output if the character is missed on purpose?
The 00 character is there:
I stumbled in this same problem when trying to get data from an RS232-TCP/IP Converter using telnet - the telnetlib would suppress every 0x00 from the message. As Fredrik Johansson well answered, it is the way telnetlib was implemented.
One solution would be to override the process_rawq() function from telnetlib's Telnet class that doesn't eat all the null characters:
import telnetlib
from telnetlib import IAC, DO, DONT, WILL, WONT, SE, NOOPT
def _process_rawq(self):
"""Alteração da implementação desta função necessária pois telnetlib suprime 0x00 e \021 dos dados lidos
"""
buf = ['', '']
try:
while self.rawq:
c = self.rawq_getchar()
if not self.iacseq:
# if c == theNULL:
# continue
# if c == "\021":
# continue
if c != IAC:
buf[self.sb] = buf[self.sb] + c
continue
else:
self.iacseq += c
elif len(self.iacseq) == 1:
# 'IAC: IAC CMD [OPTION only for WILL/WONT/DO/DONT]'
if c in (DO, DONT, WILL, WONT):
self.iacseq += c
continue
self.iacseq = ''
if c == IAC:
buf[self.sb] = buf[self.sb] + c
else:
if c == SB: # SB ... SE start.
self.sb = 1
self.sbdataq = ''
elif c == SE:
self.sb = 0
self.sbdataq = self.sbdataq + buf[1]
buf[1] = ''
if self.option_callback:
# Callback is supposed to look into
# the sbdataq
self.option_callback(self.sock, c, NOOPT)
else:
# We can't offer automatic processing of
# suboptions. Alas, we should not get any
# unless we did a WILL/DO before.
self.msg('IAC %d not recognized' % ord(c))
elif len(self.iacseq) == 2:
cmd = self.iacseq[1]
self.iacseq = ''
opt = c
if cmd in (DO, DONT):
self.msg('IAC %s %d',
cmd == DO and 'DO' or 'DONT', ord(opt))
if self.option_callback:
self.option_callback(self.sock, cmd, opt)
else:
self.sock.sendall(IAC + WONT + opt)
elif cmd in (WILL, WONT):
self.msg('IAC %s %d',
cmd == WILL and 'WILL' or 'WONT', ord(opt))
if self.option_callback:
self.option_callback(self.sock, cmd, opt)
else:
self.sock.sendall(IAC + DONT + opt)
except EOFError: # raised by self.rawq_getchar()
self.iacseq = '' # Reset on EOF
self.sb = 0
pass
self.cookedq = self.cookedq + buf[0]
self.sbdataq = self.sbdataq + buf[1]
telnetlib.Telnet.process_rawq = _process_rawq
then override the Telnet class' method:
telnetlib.Telnet.process_rawq = _process_rawq
This solved the problem for me.
This code (http://www.opensource.apple.com/source/python/python-3/python/Lib/telnetlib.py) seems to just ignore null characters. Is that really correct behavior?
def process_rawq(self):
"""Transfer from raw queue to cooked queue.
Set self.eof when connection is closed. Don't block unless in
the midst of an IAC sequence.
"""
buf = ''
try:
while self.rawq:
c = self.rawq_getchar()
if c == theNULL:
continue
:
:
process_rawq is then in turn called by e.g. read_until
def read_until(self, match, timeout=None):
"""Read until a given string is encountered or until timeout.
When no match is found, return whatever is available instead,
possibly the empty string. Raise EOFError if the connection
is closed and no cooked data is available.
"""
n = len(match)
self.process_rawq()
:
:
I also want to receive the null character. In my particular case it marks the end of a multiline message.
So the answer seems to be that this is expected behavior as the library code is written.
FWIW https://support.microsoft.com/en-us/kb/231866 states:
Communication is established using TCP/IP and is based on a Network
Virtual Terminal (NVT). On the client, the Telnet program is
responsible for translating incoming NVT codes to codes understood by
the client's display device as well as for translating
client-generated keyboard codes into outgoing NVT codes.
The NVT uses 7-bit codes for characters. The display device, referred
to as a printer in the RFC, is only required to display the standard
printing ASCII characters represented by 7-bit codes and to recognize
and process certain control codes. The 7-bit characters are
transmitted as 8-bit bytes with the most significant bit set to zero.
An end-of-line is transmitted as a carriage return (CR) followed by a
line feed (LF). If you want to transmit an actual carriage return,
this is transmitted as a carriage return followed by a NUL (all bits
zero) character.
and
Name Code Decimal Value
Function NULL NUL 0 No operation
I have a URL which gives me the below JSON String if I hit them on the browser -
Below is my URL, let's say it is URL-A and I have around three URL's like this -
http://hostnameA:1234/Service/statistics?%24format=json
And below is my JSON String which I get back from the url -
{
"description": "",
"statistics": {
"dataCount": 0,
}
}
Now I have written a Python script which is scanning all my 3 URL's and then parse then JSON String to extract the value of dataCount from it. And it should keep on running every few seconds to scan the URL and then parse it.
Below are my URL's
hostnameA http://hostnameA:1234/Service/statistics?%24format=json
hostnameB http://hostnameB:1234/Service/statistics?%24format=json
hostnameC http://hostnameC:1234/Service/statistics?%24format=json
And the data which I am seeing on the console is like this after running my python script -
hostnameA - dataCount
hostnameB - dataCount
hostnameC - dataCount
Below is my python script which works fine
def get_data_count(url):
try:
req = requests.get(url)
except requests.ConnectionError:
return 'could not get page'
try:
data = json.loads(req.content)
return int(data['statistics']['dataCount'])
except TypeError:
return 'field not found'
except ValueError:
return 'not an integer'
def send_mail(data):
sender = 'user#host.com'
receivers = ['some_name#host.com']
message = """\
From: user#host.com
To: some_name#host.com
Subject: Testing Script
"""
body = '\n\n'
for item in data:
body = body + '{name} - {res}\n'.format(name=item['name'], res=item['res'])
message = message + body
try:
smtpObj = smtplib.SMTP('some_server_name' )
smtpObj.sendmail(sender, receivers, message)
print "Mail sent"
except smtplib.SMTPException:
print "Mail sending failed!"
def main():
urls = [
('hostnameA', 'http://hostnameA:1234/Service/statistics?%24format=json'),
('hostnameB', 'http://hostnameB:1234/Service/statistics?%24format=json'),
('hostnameC', 'http://hostnameC:1234/Service/statistics?%24format=json')
]
count = 0
while True:
data = []
print('')
for name, url in urls:
res = get_data_count(url)
print('{name} - {res}'.format(name=name, res=res))
data.append({'name':name, 'res':res})
if len([item['res'] for item in data if item['res'] >= 20]) >= 1: count = count+1
else: count = 0
if count == 2:
send_mail(data)
count = 0
sleep(10.)
if __name__=="__main__":
main()
What I am also doing with above script is, suppose if any of the machines dataCount value is greater than equal to 20 for two times continuously, then I am sending out an email and it also works fine.
One issue which I am noticing is, suppose hostnameB is down for whatever reason, then it will print out like this for first time -
hostnameA - 1
hostnameB - could not get page
hostnameC - 10
And second time it will also print out like this -
hostnameA - 5
hostnameB - could not get page
hostnameC - 7
so my above script, sends out an email for this case as well since could not get page was two times continuously but infact, hostnameB dataCount value is not greater than equal to 20 at all two times? Right? So there is some bug in my script and not sure how to solve that?
I just need to send out an email, if any of the hostnames dataCount value is greater than equal to 20 for two times continuously. if the machine is down for whatever reason, then I will skip that case but my script should keep on running.
Without changing the get_data_count function:
I took the liberty to make data a dictionary with the server name as index, this makes looking up the last value easier.
I store the last dictionary and then compare the current and old values to 20. Most strings are > 19, so I create an int object from the result, this throws an exception when the result is a string, which I can then again catch to prevent shut-down servers from being counted.
last = False
while True:
data = {}
hit = False
print('')
for name, url in urls:
res = get_data_count(url)
print('{name} - {res}'.format(name=name, res=res))
data[name] = res
try:
if int(res) > 19:
hit = True
except ValueError:
continue
if hit and last:
send_mail(data)
last = hit
sleep(10.)
Pong Wizard is right, you should not handle errors like that. Either return False or None and reference the value later, or just throw an exception.
You should use False for a failed request, instead of the string "could not get page". This would be cleaner, but a False value will also double as a 0 if it is treated as an int.
>>> True + False
1
Summing two or more False values will therefore equal 0.
I'm receiving the following message trough TCP:
{"message": "Start", "client": "134.106.74.21", "type": 1009}<EOM>
but when I'm trying to partition that
msg.partition( "<EOM>" )
I'm getting the following array:
('{\x00\x00\x00"\x00\x00\x00m\x00\x00\x00e\x00\x00\x00s\x00\x00\x00s\x00\x00\x00a\x00
\x00\x00g\x00\x00\x00e\x00\x00\x00"\x00\x00\x00:\x00\x00\x00 \x00\x00\x00"\x00\x00\x00#
\x00\x00\x00B\x00\x00\x00E\x00\x00\x00G\x00\x00\x00I\x00\x00\x00N\x00\x00\x00;\x00\x00
\x00A\x00\x00\x00l\x00\x00\x00l\x00\x00\x00;\x00\x00\x000\x00\x00\x00;\x00\x00\x001\x00\x00
\x00;\x00\x00\x000\x00\x00\x00;\x00\x00\x001\x00\x00\x003\x00\x00\x004\x00\x00\x00.\x00\x00
\x001\x00\x00\x000\x00\x00\x006\x00\x00\x00.\x00\x00\x007\x00\x00\x004\x00\x00\x00.\x00\x00
\x001\x00\x00\x002\x00\x00\x005\x00\x00\x00:\x00\x00\x003\x00\x00\x000\x00\x00\x000\x00\x00
\x000\x00\x00\x000\x00\x00\x00;\x00\x00\x00#\x00\x00\x00E\x00\x00\x00N\x00\x00\x00D\x00\x00
\x00"\x00\x00\x00,\x00\x00\x00 \x00\x00\x00"\x00\x00\x00c\x00\x00\x00l\x00\x00\x00i\x00\x00
\x00e\x00\x00\x00n\x00\x00\x00t\x00\x00\x00"\x00\x00\x00:\x00\x00\x00 \x00\x00\x00"\x00
\x00\x001\x00\x00\x003\x00\x00\x004\x00\x00\x00.\x00\x00\x001\x00\x00\x000\x00\x00\x006
\x00\x00\x00.\x00\x00\x007\x00\x00\x004\x00\x00\x00.\x00\x00\x001\x00\x00\x002\x00\x00
\x005\x00\x00\x00"\x00\x00\x00,\x00\x00\x00 \x00\x00\x00"\x00\x00\x00t\x00\x00\x00y\x00
\x00\x00p\x00\x00\x00e\x00\x00\x00"\x00\x00\x00:\x00\x00\x00 \x00\x00\x002\x00\x00\x000
\x00\x00\x000\x00\x00\x005\x00\x00\x00}\x00\x00\x00<\x00\x00\x00E\x00\x00\x00O\x00\x00\x00M
\x00\x00\x00>\x00\x00\x00{"message": "Start", "client": "134.106.74.21", "type": 1009}',
'', '')
Updated
try:
#Check if there are messages, if don't than throwing an exception otherwise continue
ans = self.request.recv( 20480 )
if( ans ):
recv = self.getMessage( recv + ans )
else:
#Master client disconnected
break
except:
...
def getMessage( self, msg ):
print( "masg:" + msg );
aSplit = msg.partition( "<EOM>" )
while( aSplit[ 1 ] == "<EOM>" ):
self.recvMessageHandler( json.loads( aSplit[ 0 ] ) )
#Get the new message id any
msg = aSplit[ 3 ]
aSplit = msg.partition( "<EOM>" )
return msg;
The problem has occurred when I'm trying to add two strings.
recv + ans
If you print msg.encode("hex") then you will likely see that this is exactly what is in the string.
In any case, you may have noticed that every 4th byte of the result is one of the characters that you expected. This suggests that you have a UCS4 Unicode string that you are not handling properly.
Did you receive UCS4 encoded bytes? If so then you should be stuffing them into a unicode string u"".append(stuff). But if you are receiving UCS4-encoded bytes and you have any influence over the sender, you really should get things changed to transmit and receive UTF-8 encoded strings since that is more normal over network connections.
Are you sure that the 5 literal bytes < E O M > are indeed the delimiter that you need to use for partitioning. Or is it supposed to be the single byte ASCII code named EOM? Or is it a UCS4 encoded u"<EOM>" ?