Develop Reference

issue in sending realtime data from esp32 to Django webpage

I am having trouble in sending data from tcp client in my esp32 board to my python django server,I am not familiar with setting channels in Django ,is there a way so that i can send the data and display in my page?

in order for your microcontroller (esp32) communicate with your own server side code first you need to define protocol you're going to use:
A. TCP:
TCP relies on IP which provides address to communicate between computers. TCP/IP is a basis for internet and other networks.
B. HTTP:
HTTP mostly used by browser (IE, Google Chrome). It rides on top of TCP which provides a safe and reliable link between two computers because if packet get lost - it can be safely re-transmitted.
After deciding protocol that you're going to use now you need suitable server side code. In python there are several library / framework that you can use:
A. HTTP:
Django, Flask, AIOHTTP (all of this supports sending and receiving JSON (REST)), I preferably use one of this framework for my IoT Projects.
B. TCP: If your microcontroller is very minimal and doesn't support HTTP/JSON, you can use a simple SocketServer or Tornado TCP Server. Don't worry even though communication between your board and server done through TCP you can still import django's libraries and use django's ORM.

Referring a currently open XMPP connection from another python script present inside a django server

My application server's requirements are as follows:
Receive sensor data from mobile phones (using HTTP)
Process them (python libraries)
Send notification to the mobile devices (rendered as notifications on Android devices)
Implementation Setup:
In order to do the above, my server has three modules:
Django app module: Provides an HTTP interface to the inference library to cater to the HTTP requests sent by the Android devices to the server.
Python Inference Library: Processes the sensor data received from the phones
GCM App Server Module: explained below
GCM App Server Module: I have implemented GCM Message App Server using CCS that talks to the Google's servers (that sits between the app server and Android devices) for delivering messages to/from mobile devices running Android. Following is from their official website (previous link):
The GCM Cloud Connection Server (CCS) is an XMPP endpoint that provides a persistent, asynchronous, bidirectional connection to Google servers. The connection can be used to send and receive messages between your server and your users' GCM-connected devices.
In the documentation, they have given a sample python script that I have referred and used to implement my GCM App server. This implementation is executed as a standalone script that runs forever.
Python Inference Library and Django app module: I have implemented the inference library in python that processes sensor data received from the phones. It has a Django interface to talk to the Android devices. The inference library resides inside the Django app server.
PROBLEM:
The GCM App Server script contains a few functions, one of them being send_message(), that sends messages to the Android devices. I need to refer this function in my inference library scripts when some processed data is available to be sent to the devices. Or I need to refer to the persistent open XMPP connection client to send messages. I want to avoid putting the processing code in the GCM app server script. I have been stuck for weeks to find a way to do the above.
Is there a way to do this with my current setup or do I need to add some other layer/module?
Any help or suggestions would be greatly appreciated.
Thanks.

I think your idea here is valid. That you want clear separation between processing code and communication code. There are many ways to solve this problem, one simple way I can think of is to have a Queue object in your GCMApp Server and make a thread block on the Queue.get() method. Have the same Queue object shared with the processing django app, and whenever processed data is available push it in the Queue. The blocked thread would wake up and send it to the devices. Other way are instead of using the Queue you can use socket. Another way is having a eventloop, https://docs.python.org/3/library/asyncio-eventloop.html , this is available in python 3.0 but you can look at eventloops in general. I would suggest you to start from something simple and get it working and then start making it beautiful.
Let me know if it makes sense.

Is there a way to do this with my current setup?
Yes! Using multiprocessing. See answer of this question - Accessing python class variable defined inside the main module of a script

Can pyzmq pub/sub sockets be used bidirectionally?

I'm using a pyzmq pub/sub socket for a server to advertise notifications to client subscribers. It works nicely but I have a question:
Is there any way to use the same socket to send information back to the server? Or do I need a separate socket for that?
Use case: I just want to allow the server to see who's actively subscribing to notifications, so I was hoping I could allow clients to send back periodic "heartbeat" messages. I have a use case where if no clients are listening, I want the server to spawn one. (This is a multiprocess system that uses localhost only.)

You need a separate socket. From the ZMQ guide (http://zguide.zeromq.org/page:all#Pros-and-Cons-of-Pub-Sub):
Killing back-chatter is essential to real scalability. With pub-sub, it's how the pattern can map cleanly to the PGM multicast protocol, which is handled by the network switch. In other words, subscribers don't connect to the publisher at all, they connect to a multicast group on the switch, to which the publisher sends its messages.
In order for this to work, the PUB socket will not send back data to the subscribers (at least not in a way visible to the user. The heartbeating problem was discussed in-depth in the guide: http://zguide.zeromq.org/page:all#The-Asynchronous-Client-Server-Pattern
Also, check out the 7/MDP and 18/MDP protocols (http://rfc.zeromq.org/spec:7 -- this is also discussed in the guide) if you want to keep track of clients.

How can I setup an Autobahn Pub/Sub Server and a Autobahn Webserver listening on the same port

I recently discovered autobahn python and js as a comfortable method to establish a pub/sub server and corresponding client even with rpc-calls.
After looking through the tutorials, I set up a test version with a websocket server and a webserver running on the same port. The server sends periodically data to the client via websockets. The html the user gets lies on the localhost root. All that works fine.
However, what I want to accomplish is: Setup a pub/sub server and a webserver listening on the same port.
The tutorials show only how to setup these on two different ports (as shown at http://autobahn.ws/python/tutorials/pubsub).
Im very new to python in general and autobahn and twisted especially.
Any advice would be really nice!
Thanks very much!
Marc

Sure. You can run a WAMP/WebSocket server and a plain old Web server on one port using Autobahn. Here is an example for pure WebSocket and here is one for WAMP.
Disclaimer: I am author of Autobahn and work for Tavendo.

When using WAMP while having HTTP and WS servers listening on the same port you will need to start your instance of WampServerFactory manually as explained here.
factory = WampServerFactory("ws://localhost:8080")
factory.protocol = YourServerProtocolClass
factory.startFactory() # <--- need to call this manually
resource = WebSocketResource(factory)
root = File(".")
root.putChild("ws", resource)
For more details please see this complete example.

I would put nginx as a frontend that forwards each call either to pubsub or to web... Recent Nginx supports WebSocket forwarding.
Or you man write something similar with Twisted :)
Another alternative would be to adapt autobahn.websocket.WebSocketServerProtocol and its subclass autobahn.wamp.WampServerProtocol to Twisted.web. It should be possible.

Implement a Web based Client that interacts with a TCP Server

EDIT:Question Updated. Thanks Slott.
I have a TCP Server in Python.
It is a server with asynchronous behaviour. .
The message format is Binary Data.
Currently I have a python client that interacts with the code.
What I want to be able to do eventually implement a Web based Front End to this client.
I just wanted to know , what should be correct design for such an application.

Start with any WSGI-based web server. werkzeug is a choice.
The Asynchronous TCP/IP is a seriously complicated problem. HTTP is synchronous. So using the synchronous web server presenting some asynchronous data is always a problem. Always.
The best you can do is to buffer things and have two processes in your web application.
TCP/IP process that collects data from the remove server and buffers it in a file (or files) somewhere.
WSGI web process which handles GET/POST processing.
GET requests will fetch some or all of the buffer and display it.
POST requests will send a message to the TCP/IP server.

For Web-based, talk HTTP. Use JSON or XML as data formats.
Be standards-compliant and make use of the vast number of libraries out there. Don't reinvent the wheel. This way you have less headaches in the long run.

if you need to maintain a connection to a backend server across multiple HTTP requests, Twisted's HTTP server is an ideal choice, since it's built to manage multiple connections easily.