Develop Reference

Subscribe to Google PubSub from Azure Function App

I'm trying to figure out the best way to subscribe to a Google PubSub subscription from Azure Function App. My solution below "works" -- meaning I can start it up locally and it will pull messages that are published to the subscribed-to topic. But this can't be the right way of doing this, so I'm looking for better ideas.
I'm kicking off a timer and then listening for messages. But would a Durable app be better? The described patterns in the documentation don't fit this.
Another approach might be using the timer, but then pulling instead of listening asynchronously?
I want to log these messages and publish them to Azure ESB.
Any advice or examples would be appreciated. I'm happy to use C# or Python -- or any language for that matter.
using System;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Host;
using Microsoft.Extensions.Logging;
using Google.Cloud.PubSub.V1;
using System.Collections.Generic;
using System.Threading.Tasks;
namespace NetEaiDemo
{
public class Function1
{
[FunctionName("Function1")]
public async Task Run([TimerTrigger("*/30 * * * * *")]TimerInfo myTimer, ILogger log)
{
log.LogInformation($"C# Timer trigger function executed at: {DateTime.Now}");
string projectId = "project1";
string subscriptionId = "subscription1";
SubscriptionName subscriptionName = new SubscriptionName(projectId, subscriptionId);
SubscriberClient subscriber = await SubscriberClient.CreateAsync(subscriptionName);
List<PubsubMessage> receivedMessages = new List<PubsubMessage>();
// Start the subscriber listening for messages.
await subscriber.StartAsync((msg, cancellationToken) =>
{
receivedMessages.Add(msg);
Console.WriteLine($"Received message {msg.MessageId} published at {msg.PublishTime.ToDateTime()}");
Console.WriteLine($"Text: '{msg.Data.ToStringUtf8()}'");
// Stop this subscriber after one message is received.
// This is non-blocking, and the returned Task may be awaited.
subscriber.StopAsync(TimeSpan.FromSeconds(15));
// Return Reply.Ack to indicate this message has been handled.
return Task.FromResult(SubscriberClient.Reply.Ack);
});
}
}
}

Is there a way to broadcast a message to all (or filtered) WebSocket clients connected to a WebSocket server? [duplicate]

I'm assuming this isn't possible, but wanted to ask in case it is. If I want to provide a status information web page, I want to use WebSockets to push the data from the server to the browser. But my concerns are the effect a large number of browsers will have on the server. Can I broadcast to all clients rather than send discrete messages to each client?

WebSockets uses TCP, which is point to point, and provides no broadcast support.

Not sure how is your client/server setup, but you can always just keep in the server a collection of all connected clients - and then iterate over each one and send the message.
A simple example using Node's Websocket library:
Server code
var WebSocketServer = require('websocket').server;
var clients = [];
var socket = new WebSocketServer({
httpServer: server,
autoAcceptConnections: false
});
socket.on('request', function(request) {
var connection = request.accept('any-protocol', request.origin);
clients.push(connection);
connection.on('message', function(message) {
//broadcast the message to all the clients
clients.forEach(function(client) {
client.send(message.utf8Data);
});
});
});

As noted in other answers, WebSockets don't support multicast, but it looks like the 'ws' module maintains a list of connected clients for you, so it's pretty easy to iterate through them. From the docs:
const WebSocketServer = require('ws').Server;
const wss = new WebSocketServer({ port: 8080 });
wss.broadcast = function(data) {
wss.clients.forEach(client => client.send(data));
};

Yes, it is possible to broadcast messages to multiple clients.
In Java,
#OnMessage
public void onMessage(String m, Session s) throws IOException {
for (Session session : s.getOpenSessions()) {
session.getBasicRemote().sendText(m);
}
}
and here it is explained.
https://blogs.oracle.com/PavelBucek/entry/optimized_websocket_broadcast.

It depends on the server-side really. Here's an example of how it's done using Tomcat7:
Tomcat 7 Chat Websockets Servlet Example
and an explanation of the how it's constructed here.

Yes you can and there are many socket servers out there written in various scripting languages that are doing it.

The Microsoft.Web.WebSockets namespace has a WebSocketCollection with Broadcast capability. Look for the assembly in Nuget. The name is Microsoft.WebSockets.

zeromq REP node will only get the first message (Req,Rep pattern)

I have a nodejs app which listen for messages from clients (python app).
the pattern i used for communication over zmq is REQ/REP pattern.
the Main app should get messages from many clients. it will not reply to them, just get messages.
the problem is the main app will only get the first message and the next messages are not shown in nodejs app console.
in other words every time i start nodejs app i only get one message.
here is my code:
Nodejs app
var responder = zmq.socket('rep');
responder.on('message', function(request) {
console.log(request);
//here, it seems this function will be called just once!
});
responder.bind('tcp://127.0.0.1:8000', function(err) {
if (err) {
console.log(err);
} else {
console.log('Listening on 8000...');
}
});
python (client) part:
socket = context.socket(zmq.REQ)
socket.connect("tcp://127.0.0.1:8000")
socket.send('blaaaa')
print 'message sent!'
python part is inside a function. i could see the output of "message sent!" in python console(i mean many 'message sent!').
but i could not see the messages in nodejs app.just the first message is seen in the console of nodejs.

When using the REQ/REP-pattern you actually need to respond to a request before you are given the next request - you will only handle one request at the time.
var responder = zmq.socket('rep');
responder.on('message', function(request) {
console.log(request);
responder.send('Here comes the reply!');
});
Respond, and you will receive the next one. If you do not wish to respond, then you need to choose some other socket pair than req/rep - ex: push/pull or maybe look at xreq/xrep (router/dealer) if you wish to handle multiple requests at the same time.
If in doubt, look up the send/receive pattern for each socket type at http://api.zeromq.org/2-1:zmq-socket

Channel API python can't understand one part

I am trying to build a chat application using channel API in Google App Engine.
When going through http://developers.google.com/appengine/docs/python/channel/overview
I could not understand the opening the socket part using the XMLHttpRequest();
Any help will be appreciated!!
Thanks

Basically, when the socket is opened, the code below updates the user interface of an example Tic Tac Toe game, and sends a POST message to the server asking for the latest game state.
The code is not Python, it's client-side Javascript. I have commented it below:
sendMessage = function(path, opt_param) {
/* path variable is part of a URL being maintained */
path += '?g=' + state.game_key; /* Saving game key in URL */
if (opt_param) {
path += '&' + opt_param; /* Adding optional parameters to the path */
}
var xhr = new XMLHttpRequest(); /* Used for Ajax in Javascript */
xhr.open('POST', path, true); /* Asynchronously POST, via HTTP, the path */
xhr.send(); /* Start the POST above */
};
onOpened = function() {
connected = true; /* Set boolean value, which lets us know we're connected */
sendMessage('opened'); /* We can now send messages to the server */
updateBoard(); /* Update user interface to reflect that socket is open */
};
Note that the application defines sendMessage() as a wrapper around XmlHttpRequest, which the client uses to send messages to the server.

The socket is "opened" when the channel between client and server is established. At that point the OnOpened callback is called. The callback does a POST request back to the server to get the current state of the game. So the XMLHttpRequest is not related to the opening of the socket but rather is just a common coding pattern to have a two way communication since the channel is just one way (server to client). The other route (client to server) is done through these HTTP requests. Almost every time you get a message in the channel from the server you will want to send something back to the server (a response, an update, etc.).
Hope this helps.

Node.js as a custom (streaming) upload handler for Django

I want to build an upload-centric app using Django. One way to do this is with nginx's upload module (nonblocking) but it has its problems. Node.js is supposed to be a good candidate for this type of application. But how can I make node.js act as an upload_handler() for Django? I'm not sure where to look for examples?

Okay I'm not like an expert on the subject or anything, but the way I think I understand it, nginx is a proxy that runs in front of the webserver that serves your django app, right?
You could build a simple node.js server that does the same - listen on port 80, wait until the request is completely sent to the server, and then forward it to the webserver that serves the Django app. If your problem is that the webservers threads are being used up by long running uploads, then I think this would solve that problem.
Here's some code - just off the top of my head
var http = require('http');
var server = http.createServer(function (req, res) {
var headers = req.headers;
var url = req.url;
var method = req.method;
var body = '';
req.addListener('data', function(chunk) {
body += chunk;
});
req.addListener('end', function() {
// at this point the request is completely uploaded (including files)
// so it can be forwarded to the django webserver
var dj_client = http.createClient(8000, 'localhost');
var dj_req = dj_client.request(method, url, headers);
dj_req.addListener('response', function (dj_res) {
// here the response have been received from the django server
// so we can return that to the waiting client
res.writeHead(dj_res.status, dj_res.headers);
dj_res.addListener('data', res.write);
dj_res.addListener('end', res.close);
});
// send the request to the django webserver
dj_req.write(body);
dj_req.close();
});
});
server.listen(80);

Make node.js write the file to disk and proxy the upload POST to Django along with a '_node_file' value so your Django view knows where to get the file.