Develop Reference

How to send a Python dictionary consisting of ndarray and None using ZeroMQ PUB/SUB?

I'm trying to pass a python dictionary of 3 images (stored as ndarray) using ZeroMQ to pass it to another program as a consumer and parse back the data to original form. Followed three ways, but couldn't achieve success in anyone of the ways.
Below is the sample minimal reproduced code:
import pickle
import zmq
# Adding ZMQ Context
def zmq_context():
# Creating ZMQ context starts
context = zmq.Context()
footage_socket = context.socket(zmq.PUB)
footage_socket.connect('tcp://localhost:5002')
return footage_socket
wagon_dtl, ctr1_dtl, ctr2_dtl = NdArrays of images
socket_ctx = zmq_context()
# Trying two different ways of formatting the image before creating the dict, the below approach works for all three ndarrays
# 1st way
wagon_dtl = image_np_save # image_np_save is the original image
# 2nd way (this I tried because an ndarray isn't JSON serializable)
encoded, buffer = cv2.imencode('.jpg', image_np_save)
wagon_dtl = base64.b64encode(buffer)
if cond == "fulfilled":
full_wgn_dict = {"wagon_dtl": wagon_dtl, "ctr1_dtl": ctr1_dtl, "ctr2_dtl": ctr2_dtl}
# 1st way
dict_as_b64text = base64.b64encode(full_wgn_dict)
socket_ctx.send(dict_as_b64text)
# 2nd way
myjson = json.dumps(full_wgn_dict)
socket_ctx.send(myjson)
# 3rd way
dict_as_text = pickle.dumps(full_wgn_dict).encode('base64', 'strict')
socket_ctx.send(dict_as_text)
How to solve this?
I've followed these Q/As while working on this solution: 1, 2, 3, 4, 5

Q : "How to send a Python dictionary consisting of ndarray and None using ZeroMQ PUB/SUB?"
Easy, one may best use the ready-made .send_pyobj()-method for doing right this.
The sending side,the PUB shall be doing a call to the socket.send_pyobj( full_wgn_dict )-method, and that's basically all on this side.
A receiving side,each of the potential SUB-s shall reuse the .recv_pyobj()-method.
Yet all the SUB-s have to also do one more step, to actively subscribe to receive any message at all.
For details on socket.setsockopt( zmq.SUBSCRIBE, "" ) see the ZeroMQ documented API, or do not hesitate to sip from many examples here.
Some additional tricks (not needed for trivial dict-s) may help with the pickle-phase of the SER/DES stage. Yet these go way beyond of the scope of this Question, and may introduce advantages in controlled environments but problems in open, uncontrolled environments, where you lack zero chances to meet the required prerequisites - in my apps, I prefer to use import dill as pickle for having way higher robustness of the pickle.dumps()-SER/DES processing of objects and many more advances, like storing a full-session snapshot. Credits go to #MikeMcKearns
Feel free to re-read the documentation present for all syntax-related details in the __doc__ strings:
>>> print zmq.Socket.send_pyobj.__doc__
Send a Python object as a message using pickle to serialize.
Parameters
----------
obj : Python object
The Python object to send.
flags : int
Any valid flags for :func:`Socket.send`.
protocol : int
The pickle protocol number to use. The default is pickle.DEFAULT_PROTOCOL
where defined, and pickle.HIGHEST_PROTOCOL elsewhere.
>>> print zmq.Socket.recv_pyobj.__doc__
Receive a Python object as a message using pickle to serialize.
Parameters
----------
flags : int
Any valid flags for :func:`Socket.recv`.
Returns
-------
obj : Python object
The Python object that arrives as a message.
Raises
------
ZMQError
for any of the reasons :func:`~Socket.recv` might fail

Python Tornado respond to GET request

I am misunderstanding something very basic probably. I am new to tornado and web servers in general. I used some tutorials and a lot of googling to get started but I still find myself stuck at the basics.
The Situation
I am using python 3.6.9 on an Ubuntu 18.04 server with tornado 6.0.4.
I have a tornado server that accepts GET requests via a tornado.web.RequestHandler class get () function and does some computation on it. This all works properly.
I need the tornado server to return the results (a numpy array) to the client that sent the request.
To my knowledge everything I am doing is synchronous as I did not add any async code myself.
My code in a nutshell:
class MainHandler(tornado.web.RequestHandler):
def get(self):
base_data = self.get_argument("base_data")
compute_data(base_data)
#Here I want to return the data back to the client
application = tornado.web.Application(handlers=[ (r"/calculator", MainHandler)])
if __name__ == "__main__":
http_server=tornado.httpserver.HTTPServer(application)
http_server.listen(__PORT__)
tornado.ioloop.IOLoop.instance().start()
The problem
I do not have info about the client.
I do not have any idea and cannot find any tutorial explaining how to respond back to a client from a GET request.
What I tried
I tried simply returning the np.array at the end of my get() function but I got:
TypeError: object numpy.ndarray can't be used in 'await' expression
I thought what I need to do is make a POST request back to the client, but I do not (that I know of) have the IP and port of the client.
I also found randomly maybe I should use tornado.ioloop.IOLoop.current().spawn_callback(data) but that wasn't right I guess because it asked me for a callable function.
What I want to happen
I want to send back the computed data to the client that requested it.
Thanks in advance for any help available. I know I am probably misunderstanding the very basics of what tornado is meant to do or how it works, but I can't find any place addressing this question specifically.

See official documentation:
Many methods in RequestHandler are designed to be overridden in
subclasses and be used throughout the application. It is common to
define a BaseHandler class that overrides methods such as write_error
and get_current_user and then subclass your own BaseHandler instead of
RequestHandler for all your specific handlers.
So in your example it is also possible to write a write_response method that could make it easier to write responses in MainHandler as well as in other handlers.
See a simple example:
from tornado.web import RequestHandler
from http import HTTPStatus
import json
class BaseHandler(RequestHandler):
def write_response(self, status_code, result=None, message=None):
self.set_status(status_code)
if result:
self.finish(json.dumps(result))
elif message:
self.finish(json.dumps({
"message": message
}))
elif status_code:
self.set_status(status_code)
self.finish()
class MainHandler(BaseHandler):
def get(self):
self.write_response(status_code=HTTPStatus.OK, message='Hello calculator!')
If the data you return to the client is in the form below, then use write_response with the result argument
data = ['foo', {'bar': ('baz', None, 1.0, 2)}]
self.write_response(status_code=HTTPStatus.OK, result=data)
# and so you will send to the client:
["foo", {"bar": ["baz", null, 1.0, 2]}]
# or
your_numpy_list = your_numpy_object.tolist()
self.write_response(status_code=HTTPStatus.OK, result=your_numpy_list)

So I was missing the most basic thing.
Apparently in Tornado self.write({"data_name":data}) in the get() function will return the data.
Now I am still running into an issue of not being able to return byte data (my circumstances have changed and now I need to turn the numpy array into a wav file and send the wav file over) and I am getting a different error that Object of type 'bytes' is not JSON serializable but if I wont be able to figure it out I will open a new question for it.

server name reachability in dnspython

I am currently trying to find a way to check whether or not the name servers can respond to either TCP or UDP packets.
My idea behind that was, to get all the name servers from a website (for example google.com), store them in a list, and then try to send TCP and UDP messages to all of them.
Although I am getting the name servers, my interpreter shows a problem when I am trying to make a query on udp(check udpPacket on the code) saying:
"TypeError: coercing to Unicode: need string or buffer, NS found"
I am new in Python(coming from C and C++) and I am guessing this is just incompatible types.
I checked dnspython's documentation and could not find what kind of type NS is (probably it's a type by itself) and why it cannot be passed as an argument.
What do you think the problem is? Is there maybe a better way to solve that kind of problem?
def getNSResults(url):
#create an empty list where we can store all the nameservers we found
nameServers = []
nameServers = dns.resolver.query(url,dns.rdatatype.NS, raise_on_no_answer=False)
#create a dictionary where based on all the nameservers.
#1st label refers to the ns name of our url that we inserted.
#2nd label shows wether or not we received a UDP response or not.
#3rd label shows wether or not we received a TCP response or not.
results = {}
for nameServer in nameServers:
#make a dns ns query, acts as a dumb message since whatever we send we just care of what we get back
query = dns.message.make_query(dns.name.from_text(url), dns.rdatatype.ANY)
query.flags |= dns.flags.AD
query.find_rrset(query.additional, dns.name.root, 65535, dns.rdatatype.OPT, create=True, force_unique=True)
#try sending a udp packet to see if it's listening on UDP
udpPacket = dns.query.udp(query,nameServer)
#try sending a tcp packet to see if it's listening on TCP
tcpPacket = dns.query.tcp(None,nameServer)
#add the results in a dictionary and return it, to be checked later by the user.
results.update({"nsName" == nameServer, "receivedUDPPacket" == isNotNone(udpPacket),"receivedTCPPacket" == isNotNone(tcpPacket)})
Thanks in advance!

Looking at your code, I see some DNS problems, some Python problems, and some dnspython problems. Let's see if we can't learn something together.
DNS
First, the parameter to your function getNSResults is called url. When you send DNS queries, you query for a domain name. A URL is something totally different (e.g. https://example.com/index.html). I would rename url to something like domain_name, domain, or name. For more on the difference between URLs and domain names, see https://www.copahost.com/blog/domain-vs-url/.
Second, let's talk about what you're trying to do.
i am currently trying to find a way to check wether or not the name servers can respond to either tcp or udp packets.
My idea behind that was, to get all the name servers from a website (for example google.com), store them in a list, and then, try to send tcp and udp messages to all of them.
That sounds like a great approach. I think you might be missing a few details here. so let me explain the steps you can take to do this:
Do an NS query for a domain name. You already have this step in your code. What you'll actually get from that query is just another domain name (or multiple domain names). For example, if you run dig +short NS google.com, you'll get this output:
ns3.google.com.
ns1.google.com.
ns4.google.com.
ns2.google.com.
At this step, we have a list of one or more names of authoritative servers. Now we need an IP address to use to send them queries. So we'll do a type A query for each of the names we got from step 1.
Now we have a list of IP addresses. We can send a DNS query over UDP and one over TCP to see if they're supported.
Python
For the most part, your Python syntax is okay.
The biggest red flag I see is the following code:
results.update({"nsName" == nameServer,
"receivedUDPPacket" == isNotNone(udpPacket),
"receivedTCPPacket" == isNotNone(tcpPacket)})
Let's break this down a bit.
First, you have results, which is a dict.
Then you have this:
{"nsName" == nameServer,
"receivedUDPPacket" == isNotNone(udpPacket),
"receivedTCPPacket" == isNotNone(tcpPacket)}
which is a set of bools.
What I think you meant to do was something like this:
results.update({
"nsName": nameServer,
"receivedUDPPacket": true,
"receivedTCPPacket": true
})
Function and variables names in Python are usually written in lowercase, with words separated by underscores (e.g. my_variable, def my_function()). Class names are usually upper camel case (e.g. class MyClass).
None of this is required, you can name your stuff however you want, plenty of super popular libraries and builtins break this convention, just figured I'd throw it out there because it can be helpful when reading Python code.
dnspython
When you're not sure about the types of things, or what attributes things have, remember these four friends, all builtin to Python:
1. pdb
2. dir
3. type
4. print
pdb is a Python debugger. Just import pdb, and the put pdb.set_trace() where you want to break. Your code will stop there, and then you can check out the values of all the variables.
dir will return the attributes and methods of whatever you pass to it. Example: print(dir(udpPacket)).
type will return the type of an object.
print as you probably already know, will print out stuff so you can see it.
I'm going to leave this part for you to test out.
Run dir() on everything if you don't know what it is.
I also should probably mention help(), which is super useful for built-in stuff.
The summary for this section is that sometimes documentation isn't all there, or hard to find, especially when you're new to a language/library/whatever.
So you have to figure stuff out on your own, and that means using all the tools I've just mentioned, looking at the source code, things like that.
Summary
I hope this was helpful. I know it's a lot, it's probably too much, but just be patient and know that DNS and Python are some very useful and fun things to learn about.
I went ahead and wrote something up that is a start at what I think you're hoping to achieve.
I recommend walking through the whole thing and making sure you understand what's going on.
If you don't understand something, remember pdb and dir (and there's always Google, SO, etc).
import dns.resolver
import dns.message
import dns.rdatatype
import json
import sys
def check_tcp_and_udp_support(name):
# this will give me the first default system resolver from /etc/resolv.conf
# (or Windows registry)
where = dns.resolver.Resolver().nameservers[0]
q = dns.message.make_query(name, dns.rdatatype.NS)
ns_response = dns.query.udp(q, where)
ns_names = [t.target.to_text() for ans in ns_response.answer for t in ans]
# this code is the same as the one-liner above
# ns_names = []
# for ans in ns_response.answer:
# for t in ans:
# ns_names.append(t.target.to_text())
results = {}
for ns_name in ns_names:
# do type A lookup for nameserver
q = dns.message.make_query(ns_name, dns.rdatatype.A)
response = dns.query.udp(q, where)
nameserver_ips = [item.address for ans in response.answer for item in ans.items if ans.rdtype == dns.rdatatype.A]
# now send queries to the nameserver IPs
for nameserver_ip in nameserver_ips:
q = dns.message.make_query('example.com.', dns.rdatatype.A)
try:
udp_response = dns.query.udp(q, nameserver_ip)
supports_udp = True
except dns.exception.Timeout:
supports_udp = False
try:
tcp_response = dns.query.tcp(q, nameserver_ip)
supports_tcp = True
except dns.exception.Timeout:
supports_tcp = True
results[nameserver_ip] = {
'supports_udp': supports_udp,
'supports_tcp': supports_tcp
}
return results
def main():
results = check_tcp_and_udp_support('google.com')
# this is just fancy JSON printing
# you could do print(results) instead
json.dump(results, sys.stdout, indent=4)
if __name__ == '__main__':
main()
Again, I hope this is helpful. It's hard when I don't know exactly what's going on in your head, but this is what I've got for you.

Bad request type PUT when sending PUT request through Qt, works with curl

I have subclassed BaseHTTPRequestHandler and implemented do_GET(), do_POST() and do_PUT().
I thought that everything works fine until I put my Qt application to the test. The GET request works and the server sends an XML document with some data that the application processes. The POST request also works and is used to generate test cases through REST.
As for the PUT things are looking weird.
Here is the PUT handler on the server side:
def do_PUT(self):
"""
Processes PUT request. It can be tested using wget, curl or any
other tool with support for http PUT. Use this to send reports
about the current status of device and all of its slave devices
Example:
curl -X PUT -d "status=downloading" http://127.0.0.1:8090/so/updateProgress
"""
print('Processing PUT request...')
params_parsed = urlparse(self.path)
params = parse_qs(params_parsed.query)
if len(params) > 0 or '/so/updateProgress' not in params_parsed.path:
print('Use "/so/updateProgress" to report on update progress')
self.__set_headers(data_type='text/html')
self.send_error(501)
return
self.__set_headers(data_type='text/xml')
report_raw = self.rfile.read(int(self.headers.getheader('Content-length')))
print('Received report')
print('Parsing report...')
# Generate XML from the raw data and validate it
report = SoRequestHandler.Report.from_xml(report_raw)
print('Parsing of report complete')
report.write_to_file(log_path='report_log', append=True)
self.send_response(200)
In my Qt application I have a class called ReportPublisher, which takes some reports from all the devices it is connected to (through MQTT), aggregates the reports into a single one and sends it to the server, which logs it in a file:
void ReportPublisher::publishHttpReport(HttpReport report)
{
QString reportXml = report.xml().toString();
if (reportXml.isEmpty())
{
LOG(ERROR) << "Something went wrong with the generation of HTTP report";
return;
}
LOG(INFO) << "Generated report for SO server: " << reportXml;
QByteArray reportRaw = reportXml.toUtf8();
QUrl target(this->urlServer);
target.setPath(this->urlPath);
QNetworkRequest req(target);
req.setHeader(QNetworkRequest::ContentTypeHeader, QString("application/xml"));
this->reply = this->nam->put(req, reportRaw);
// TODO Read back response (expected code 200)?
}
I have to be honest. I have never done a PUT request in Qt and all I've done so far were GET requests so there might be some really fundamental yet easy to spot error in the code above.
The data that the server receives looks like this:
<updateProgress xmlns='service.so.de/so'>
<deviceTypeId>...</deviceTypeId>
<packageId>...</packageId>
<version>...</version>
<status>...</status>
</updateProgress>
If I use curl like this
root#obunit:~$ curl -X PUT -i 'http://192.168.120.61:8090/so/updateProgress' -d "<updateProgress xmlns='service.so.de/so'><deviceTypeId>...</deviceTypeId><packageId>...</packageId><version>...</version><status>...</status></updateProgress>"
where 192.168.120.61:8090 is the IP address and the port the server is located at/listening to for incoming requests I have no problem.
However with data coming from my Qt application I get
192.168.120.172 - - [11/Apr/2018 15:32:37] code 400, message Bad HTTP/0.9 request type ('PUT')
192.168.120.172 - - [11/Apr/2018 15:32:37] "PUT HTTP/1.1" 400 -
in my log (with 192.168.120.172 being the IP address of the system where my software is running.
From my scarce knowledge code 400 means invalid syntax, which can be due to following 2 reasons (at least what I can think of right now):
malformed data (invalid XML, JSON etc.)
incorrect encoding of the data, which is basically equivalent to the malformed data but coming from a different source.
I have tried converting the XML document that I'm generating to QByteArray using QDomDocument::toByteArray(int i). I have also tried (as you can see in the code) to convert the document to QString and then to a UTF-8 QByteArray but I can't get my server to process the data.
What's even stranger is that (as you can see in my do_PUT()) implementation my PUT handler starts with printing a message, which never appears in the logs hence the issue arises from somewhere deeper in the code of BaseHTTPRequestHandler.

The problem was rather simple yet very annoying.
Following the advice of #MrEricSir I did use Wireshark just to find out that get some weird TCP retramsission. Retransmission usually occurs due to network congestion, which in my case was not a possibility however it did indicate a network issue that has nothing to do with the XML content I was sending.
So I started my remote debugging session again and look closer to where I was emitting the PUT request. And there it was! There was a / missing between the base URL (IP:PORT) and the path. So instead of
PUT 192.168.120.61:8090/so/updateProgress
I was doing
PUT 192.168.120.61:8090so/updateProgress
which ultimately led to the PUT request not succeeding at all (hence the code 400).
Just a hint for anyone reading this in the future - whenever you manually set that path using QUrl::setPath(QString)
QUrl target(this->urlServer);
target.setPath(this->urlPath);
ALWAYS make sure that the string you pass as a path starts with / because Qt isn't going to add one!

Making all attributes and methods available for a socket server in Python

I use a Raspberry Pi to collect sensor data and set digital outputs, to make it easy for other applications to set and get values I'm using a socket server. But I am having some problems finding an elegant way of making all the data available on the socket server without having to write a function for each data type.
Some examples of values and methods I have that I would like to make available on the socket server:
do[2].set_low() # set digital output 2 low
do[2].value=0 # set digital output 2 low
do[2].toggle() # toggle digital output 2
di[0].value # read value for digital input 0
ai[0].value # read value for analog input 0
ai[0].average # get the average calculated value for analog input 0
ao[4].value=255 # set analog output 4 to byte value 255
ao[4].percent=100 # set analog output 4 to 100%
I've tried eval() and exec():
self.request.sendall(str.encode(str(eval('item.' + recv_string)) + '\n'))
eval() works unless I am using equal sign (=), but I'm not to happy about the solution because of dangers involved. exec() does the work but does not return any value, also dangerous.
I've also tried getattr():
recv_string = bytes.decode(self.data).lower().split(';')
values = getattr(item, recv_string[0])
self.request.sendall(str.encode(str(values[int(recv_string[1])].value) + '\n'))
^^^^^
This works for getting my attributes, and the above example works for getting the value of the attribute I am getting with getattr(). But I can not figure out how to use getattr() on the value attribute as well.
The semi-colon (;) is used to split the incoming command, I've experimented with multiple ways of formatting the commands:
# unit means that I want to talk to a I/O interface module,
# and the name specified which one
unit;unit_name;get;do;1
unit;unit_name;get;do[1]
unit;unit_name;do[1].value
I am free to choose the format since I am also writing the software that uses these commands. I have not yet found a good format which covers all my needs.
Any suggestions how I can write an elegant way of accessing and returning the data above? Preferably with having to add new methods to the socket server every time a new value type or method is added to my I/O ports.
Edit: This is not public, it's only available on my LAN.

Suggestions
Make your API all methods so that eval can always be used:
def value_m(self, newValue=None):
if newValue is not None:
self.value = newValue
return self.value
Then you can always do
result = str(eval(message))
self.request.sendall(str.encode(result + '\n'))
For your message, I would suggest that your messages are formatted to include the exact syntax of the command exactly so that it can be evaled as-is, e.g.
message = 'do[1].value_m()' # read a value, alternatively...
message = 'do[1].value_m(None)'
or to write
message = 'do[1].value_m(0)' # write a value
This will make it easy to keep your messages up-to-date with your API, because they must match exactly, you won't have a second DSL to deal with. You really don't want to have to maintain a second API, on top of your IO one.
This is a very simple scheme, suitable for a home project. I would suggest some error handling in evaluation, like so:
import traceback
try:
result = str(eval(message))
except Exception:
result = traceback.format_exc()
self.request.sendall(str.encode(result + '\n'))
This way your caller will receive a printout of the exception traceback in the returned message. This will make it much, much easier to debug bad calls.
NOTE If this is public-facing, you cannot do this. All input must be sanitised. You will have to parse each instruction and compare it to the list of available (and desirable) commands, and verify input validity and validity ranges for everything. For such a scenario you are better off simply using one of the input validation systems used for web services, where this problem receives a great deal of attention.