I'm writing a thrift service in python and I would like to understand how I can
get the client's IP address in the handler functions context.
Thanks,
Love.
You need to obtain the transport, and get the data from there. Not sure how to do this exactly in Python, but there's a mailing list thread and there's this JIRA-ticket THRIFT-1053 describing a solution for C++/Java.
This is the relevant part from the mailing list thread:
I did it by decorating the TProcessor like this psuedo-code.
-craig
class TrackingProcessor implements TProcessor {
TrackingProcessor (TProcessor processor) {this.processor=processor;}
public boolean process(TProtocol in, TProtocol out) throws TException {
TTransport t = in.getTransport();
InetAddress ia = t instanceof TSocket ?
((TSocket)t).getSocket().getInetAddress() : null;
// Now you have the IP address, so what ever you want.
// Delegate to the processor we are decorating.
return processor.process(in,out);
}
}
This is a bit old but I'm currently solving the same problem.
Here's my solution with thriftpy:
import thriftpy
from thriftpy.thrift import TProcessor, TApplicationException, TType
from thriftpy.server import TThreadedServer
from thriftpy.protocol import TBinaryProtocolFactory
from thriftpy.transport import TBufferedTransportFactory, TServerSocket
class CustomTProcessor(TProcessor):
def process_in(self, iprot):
api, type, seqid = iprot.read_message_begin()
if api not in self._service.thrift_services:
iprot.skip(TType.STRUCT)
iprot.read_message_end()
return api, seqid, TApplicationException(TApplicationException.UNKNOWN_METHOD), None # noqa
args = getattr(self._service, api + "_args")()
args.read(iprot)
iprot.read_message_end()
result = getattr(self._service, api + "_result")()
# convert kwargs to args
api_args = [args.thrift_spec[k][1] for k in sorted(args.thrift_spec)]
# get client IP address
client_ip, client_port = iprot.trans.sock.getpeername()
def call():
f = getattr(self._handler, api)
return f(*(args.__dict__[k] for k in api_args), client_ip=client_ip)
return api, seqid, result, call
class PingPongDispatcher:
def ping(self, param1, param2, client_ip):
return "pong %s" % client_ip
pingpong_thrift = thriftpy.load("pingpong.thrift")
processor = CustomTProcessor(pingpong_thrift.PingService, PingPongDispatcher())
server_socket = TServerSocket(host="127.0.0.1", port=12345, client_timeout=10000)
server = TThreadedServer(processor,
server_socket,
iprot_factory=TBinaryProtocolFactory(),
itrans_factory=TBufferedTransportFactory())
server.serve()
Remember that every method in the dispatcher will be called with extra parameter client_ip
The only way I found to get the TProtocol at the service handler is to extend the processor and create one handler instance for each client related by transport/protocol. Here's an example:
public class MyProcessor implements TProcessor {
// Maps sockets to processors
private static final Map<, Processor<ServiceHandler>> PROCESSORS = Collections.synchronizedMap(new HashMap<String, Service.Processor<ServiceHandler>>());
// Maps sockets to handlers
private static final Map<String, ServiceHandler> HANDLERS = Collections.synchronizedMap(new HashMap<String, ServiceHandler>());
#Override
public boolean process(final TProtocol in, final TProtocol out)
throws TException {
// Get the socket for this request
final TTransport t = in.getTransport();
// Note that this cast will fail if the transport is not a socket, so you might want to add some checking.
final TSocket socket = (TSocket) t;
// Get existing processor for this socket if any
Processor<ServiceHandler> processor = PROCESSORS.get(socket);
// If there's no processor, create a processor and a handler for
// this client and link them to this new socket
if (processor == null) {
// Inform the handler of its socket
final ServiceHandler handler = new ServiceHandler(socket);
processor = new Processor<ServiceHandler>(handler);
PROCESSORS.put(clientRemote, processor);
HANDLERS.put(clientRemote, handler);
}
return processor.process(in, out);
}
}
Then you need to tell Thrift to use this processor for incoming requests. For a TThreadPoolServer it goes like this:
final TThreadPoolServer.Args args = new TThreadPoolServer.Args(new TServerSocket(port));
args.processor(new MyProcessor());
final TThreadPoolServer server = new TThreadPoolServer(args);
The PROCESSORS map might look superfluous, but it is not since there's no way to get the handler for a processor (i.e. there's no getter).
Note that it is your ServiceHandler instance that needs to keep which socket it is associated to. Here I pass it on the constructor but any way will do. Then when the ServiceHandler's IFace implementation is called, it will already have the associated Socket.
This also means you will have an instance of MyProcessor and ServiceHandler for each connected client, which I think is not the case with base Thrift where only one instance of each of the classes are created.
This solution also has a quite annoying drawback: you need to figure out a method to remove obsolete data from PROCESSORS and HANDLERS maps, otherwise these maps will grow indefinitely. In my case each client has a unique ID, so I can check if there are obsolete sockets for this client and remove them from the maps.
PS: the Thrift guys need to figure out a way to let the service handler get the used protocol for current call (for example by allowing to extend a base class instead of implementing an interface). This is very useful in so many scenarios.
Related
I'm trying to understand if grpc server using streams is able to wait for all client messages to be read in prior to sending responses.
I have a trivial application where I send in several numbers I'd like to add and return.
I've set up a basic proto file to test this:
syntax = "proto3";
message CalculateRequest{
int64 x = 1;
int64 y = 2;
};
message CalculateReply{
int64 result = 1;
}
service Svc {
rpc CalculateStream (stream CalculateRequest) returns (stream CalculateReply);
}
On my server-side I have implemented the following code which returns the answer message as the message is received:
class CalculatorServicer(contracts_pb2_grpc.SvcServicer):
def CalculateStream(self, request_iterator, context):
for request in request_iterator:
resultToOutput = request.x + request.y
yield contracts_pb2.CalculateReply(result=resultToOutput)
def serve():
server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))
contracts_pb2_grpc.add_SvcServicer_to_server(
CalculatorServicer(), server)
server.add_insecure_port('localhost:9000')
server.start()
server.wait_for_termination()
if __name__ == '__main__':
print( "We're up")
logging.basicConfig()
serve()
I'd like to tweak this to first read in all the numbers and then send these out at a later stage - something like the following:
class CalculatorServicer(contracts_pb2_grpc.SvcServicer):
listToReturn = []
def CalculateStream(self, request_iterator, context):
for request in request_iterator:
listToReturn.append (request.x + request.y)
# ...
# do some other stuff first before returning
for item in listToReturn:
yield contracts_pb2.CalculateReply(result=resultToOutput)
Currently, my implementation to write out later doesn't work as the code at the bottom is never reached. Is this by design that the connection seems to "close" before reaching there?
The grpc.io website suggests that this should be possible with BiDirectional streaming:
for example, the server could wait to receive all the client messages before writing its responses, or it could alternately read a message then write a message, or some other combination of reads and writes.
Thanks in advance for any help :)
The issue here is the definition of "all client messages." At the transport level, the server has no way of knowing whether the client has finished independent of the client closing its connection.
You need to add some indication of the client's having finished sending requests to the protocol. Either add a bool field to the existing CalculateRequest or add a top-level oneof with one of the options being something like a StopSendingRequests
I recently have acquired an ACS Linear Actuator (Tolomatic Stepper) that I am attempting to send data to from a Python application. The device itself communicates using Ethernet/IP protocol.
I have installed the library cpppo via pip. When I issue a command
in an attempt to read status of the device, I get None back. Examining the
communication with Wireshark, I see that it appears like it is
proceeding correctly however I notice a response from the device indicating:
Service not supported.
Example of the code I am using to test reading an "Input Assembly":
from cpppo.server.enip import client
HOST = "192.168.1.100"
TAGS = ["#4/100/3"]
with client.connector(host=HOST) as conn:
for index, descr, op, reply, status, value in conn.synchronous(
operations=client.parse_operations(TAGS)):
print(": %20s: %s" % (descr, value))
I am expecting to get a "input assembly" read but it does not appear to be
working that way. I imagine that I am missing something as this is the first
time I have attempted Ethernet/IP communication.
I am not sure how to proceed or what I am missing about Ethernet/IP that may make this work correctly.
clutton -- I'm the author of the cpppo module.
Sorry for the delayed response. We only recently implemented the ability to communicate with simple (non-routing) CIP devices. The ControlLogix/CompactLogix controllers implement an expanded set of EtherNet/IP CIP capability, something that most simple CIP devices do not. Furthermore, they typically also do not implement the *Logix "Read Tag" request; you have to struggle by with the basic "Get Attribute Single/All" requests -- which just return raw, 8-bit data. It is up to you to turn that back into a CIP REAL, INT, DINT, etc.
In order to communicate with your linear actuator, you will need to disable these enhanced encapsulations, and use "Get Attribute Single" requests. This is done by specifying an empty route_path=[] and send_path='', when you parse your operations, and to use cpppo.server.enip.getattr's attribute_operations (instead of cpppo.server.enip.client's parse_operations):
from cpppo.server.enip import client
from cpppo.server.enip.getattr import attribute_operations
HOST = "192.168.1.100"
TAGS = ["#4/100/3"]
with client.connector(host=HOST) as conn:
for index, descr, op, reply, status, value in conn.synchronous(
operations=attribute_operations(
TAGS, route_path=[], send_path='' )):
print(": %20s: %s" % (descr, value))
That should do the trick!
We are in the process of rolling out a major update to the cpppo module, so clone the https://github.com/pjkundert/cpppo.git Git repo, and checkout the feature-list-identity branch, to get early access to much better APIs for accessing raw data from these simple devices, for testing. You'll be able to use cpppo to convert the raw data into CIP REALs, instead of having to do it yourself...
...
With Cpppo >= 3.9.0, you can now use much more powerful cpppo.server.enip.get_attribute 'proxy' and 'proxy_simple' interfaces to routing CIP devices (eg. ControlLogix, Compactlogix), and non-routing "simple" CIP devices (eg. MicroLogix, PowerFlex, etc.):
$ python
>>> from cpppo.server.enip.get_attribute import proxy_simple
>>> product_name, = proxy_simple( '10.0.1.2' ).read( [('#1/1/7','SSTRING')] )
>>> product_name
[u'1756-L61/C LOGIX5561']
If you want regular updates, use cpppo.server.enip.poll:
import logging
import sys
import time
import threading
from cpppo.server.enip import poll
from cpppo.server.enip.get_attribute import proxy_simple as device
params = [('#1/1/1','INT'),('#1/1/7','SSTRING')]
# If you have an A-B PowerFlex, try:
# from cpppo.server.enip.ab import powerflex_750_series as device
# parms = [ "Motor Velocity", "Output Current" ]
hostname = '10.0.1.2'
values = {} # { <parameter>: <value>, ... }
poller = threading.Thread(
target=poll.poll, args=(device,), kwargs={
'address': (hostname, 44818),
'cycle': 1.0,
'timeout': 0.5,
'process': lambda par,val: values.update( { par: val } ),
'params': params,
})
poller.daemon = True
poller.start()
# Monitor the values dict (updated in another Thread)
while True:
while values:
logging.warning( "%16s == %r", *values.popitem() )
time.sleep( .1 )
And, Voila! You now have regularly updating parameter names and values in your 'values' dict. See the examples in cpppo/server/enip/poll_example*.py for further details, such as how to report failures, control exponential back-off of connection retries, etc.
Version 3.9.5 has recently been released, which has support for writing to CIP Tags and Attributes, using the cpppo.server.enip.get_attribute proxy and proxy_simple APIs. See cpppo/server/enip/poll_example_many_with_write.py
hope this is obvious, but accessing HOST = "192.168.1.100" will only be possible from a system located on the subnet 192.168.1.*
How to use python as a backend for an Android App that is built using C#? The Python Backend is written using the Flask framework. The Android app is built using xamarin.
No matter what type of technology your server or the client use if they can communicate with each other using some sort of standard "protocol".
There are many ways to communicate both sides (client and server) like sockets, xml, json, etc. They just need to understand each other.
In your particular case I suggest to build a REST or RESTful API (https://flask-restful.readthedocs.org/en/0.3.3/) on the server and a REST client library on the client.
There are many ways and libraries to call REST APIs from C#:
The built-in method would be using HttpWebRequest as you can see on this link:
private async Task<JsonValue> FetchWeatherAsync (string url)
{
// Create an HTTP web request using the URL:
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create (new Uri (url));
request.ContentType = "application/json";
request.Method = "GET";
// Send the request to the server and wait for the response:
using (WebResponse response = await request.GetResponseAsync ())
{
// Get a stream representation of the HTTP web response:
using (Stream stream = response.GetResponseStream ())
{
// Use this stream to build a JSON document object:
JsonValue jsonDoc = await Task.Run (() => JsonObject.Load (stream));
Console.Out.WriteLine("Response: {0}", jsonDoc.ToString ());
// Return the JSON document:
return jsonDoc;
}
}
}
But I don´t recommend it if you don´t want your app to be full of crap (boiler plate code) everywhere.
A helper library could be, for example, RESTSharp. It allows you to build REST calls easily and cast the response to your typed objects. Here´s and example:
var client = new RestClient("http://example.com");
// client.Authenticator = new HttpBasicAuthenticator(username, password);
var request = new RestRequest("resource/{id}", Method.POST);
request.AddParameter("name", "value"); // adds to POST or URL querystring based on Method
request.AddUrlSegment("id", "123"); // replaces matching token in request.Resource
// easily add HTTP Headers
request.AddHeader("header", "value");
// add files to upload (works with compatible verbs)
request.AddFile(path);
// execute the request
RestResponse response = client.Execute(request);
var content = response.Content; // raw content as string
// or automatically deserialize result
// return content type is sniffed but can be explicitly set via RestClient.AddHandler();
RestResponse<Person> response2 = client.Execute<Person>(request);
var name = response2.Data.Name;
// easy async support
client.ExecuteAsync(request, response => {
Console.WriteLine(response.Content);
});
// async with deserialization
var asyncHandle = client.ExecuteAsync<Person>(request, response => {
Console.WriteLine(response.Data.Name);
});
// abort the request on demand
asyncHandle.Abort();
You can search "C# REST client" on google and judge by yourself. But IMHO, the easier and nicer to code REST client I´ve ever used is Refit.
Why? you define API calls and responses with just an interface. No coding required at all! Even more, all your API calls will be async by default, something needed for mobile apps to be responsive. From the author´s readme:
public interface IGitHubApi
{
[Get("/users/{user}")]
Task<User> GetUser(string user);
}
var gitHubApi = RestService.For<IGitHubApi>("https://api.github.com");
var octocat = await gitHubApi.GetUser("octocat");
I´ve used this library on Xamarin Android/iOS projects and it works well. No issues at all.
Hope it helps
I am using autobahnpython with twisted (wamp) on server side and autobahnjs in browser. Is there a straight-forward way to allow/restrict subscriptions on a per session basis? For example, a client should not be able to subscribe to topics relavant to other users.
While I am NOT using crossbar.io, I tried using the Python code shown in the 'Example' section at the end of this page http://crossbar.io/docs/Authorization/ where a RPC call is first used to give authorization to a client. Of course, I am using my own authorization logic. Once this authorization is successful, I'd like to give the client privileges to subscribe to topics related only to this client, like 'com.example.user_id'. My issue is that even if auth passes, however, I have not found a way to limit subscription requests in the ApplicationSession class which is where the authorization takes place. How can I prevent a client who authorizes with user_id=user_a from subscribing to 'com.example.user_b'?
You can authorize by creating your own router. To do that, subclass Router() and override (at a minumum) the authorize() method:
def authorize(self, session, uri, action):
return True
This method is pretty simple, if you return a True then the session is authorized to do whatever it is attempting. You could make a rule that all subscriptions must start with 'com.example.USER_ID', so, your python code would split the uri, take the third field, and compare it to the current session id, returning True if they match, false otherwise. This is where things get a little weird though. I have code that does a similar thing, here is my authorize() method:
#inlineCallbacks
def authorize(self, session, uri, action):
authid = session._authid
if authid is None:
authid = 1
log.msg("AuthorizeRouter.authorize: {} {} {} {} {}".format(authid,
session._session_id, uri, IRouter.ACTION_TO_STRING[action], action))
if authid != 1:
rv = yield self.check_permission(authid, uri, IRouter.ACTION_TO_STRING[action])
else:
rv = yield True
log.msg("AuthorizeRouter.authorize: rv is {}".format(rv))
if not uri.startswith(self.svar['topic_base']):
self.sessiondb.activity(session._session_id, uri, IRouter.ACTION_TO_STRING[action], rv)
returnValue(rv)
return
Note that I dive into the session to get the _authid, which is bad karma (I think) because I should not be looking at these private variables. I don't know where else to get it, though.
Also, of note, this goes hand in hand with Authentication. In my implementation, the _authid is the authenticated user id, which is similar to a unix user id (positive unique integer). I am pretty sure this can be anything, like a string, so you should be ok with your 'user_b' as the _auth_id if you wish.
-g
I found a relatively simple solution using a Node guest. Here's the code:
// crossbar setup
var autobahn = require('autobahn');
var connection = new autobahn.Connection({
url: 'ws://127.0.0.1:8080/ws',
realm: 'realm1'
}
);
// Websocket to Scratch setup
// pull in the required node packages and assign variables for the entities
var WebSocketServer = require('websocket').server;
var http = require('http');
var ipPort = 1234; // ip port number for Scratch to use
// this connection is a crossbar connection
connection.onopen = function (session) {
// create an http server that will be used to contain a WebSocket server
var server = http.createServer(function (request, response) {
// We are not processing any HTTP, so this is an empty function. 'server' is a wrapper for the
// WebSocketServer we are going to create below.
});
// Create an IP listener using the http server
server.listen(ipPort, function () {
console.log('Webserver created and listening on port ' + ipPort);
});
// create the WebSocket Server and associate it with the httpServer
var wsServer = new WebSocketServer({
httpServer: server
});
// WebSocket server has been activated and a 'request' message has been received from client websocket
wsServer.on('request', function (request) {
// accept a connection request from Xi4S
//myconnection is the WS connection to Scratch
myconnection = request.accept(null, request.origin); // The server is now 'online'
// Process Xi4S messages
myconnection.on('message', function (message) {
console.log('message received: ' + message.utf8Data);
session.publish('com.serial.data', [message.utf8Data]);
// Process each message type received
myconnection.on('close', function (myconnection) {
console.log('Client closed connection');
boardReset();
});
});
});
};
connection.open();
I am trying to produce an amqp message from python and consume that same message from java/spring.
Here is my producer code (python):
import pika, sys, pickle
sys.path.append("trc/suivi/amqp")
from Person import Person
connection = pika.BlockingConnection()
channel = connection.channel()
me = Person("Juliano", 38)
pickled_me = pickle.dumps(me)
channel.basic_publish(exchange='',
routing_key="myqueue",
body=pickled_me,
properties=pika.BasicProperties(delivery_mode=1))
Here is my consumer code (java):
ApplicationContext context = new GenericXmlApplicationContext("classpath:/applicationContext.xml");
AmqpTemplate template = context.getBean(AmqpTemplate.class);
Person me = (Person) template.receiveAndConvert("myqueue");
System.out.println("Me: " + me.getName() + ":" + me.getAge());
Here is the java class for Person:
package trc.suivi.amqp;
import java.io.Serializable;
public class Person implements Serializable {
private static final long serialVersionUID = 1L;
private String name;
private int age;
public String getName() {
return name;
}
public void setName(String name) {
this.name = name;
}
public int getAge() {
return age;
}
public void setAge(int age) {
this.age = age;
}
}
and the corresponding python class:
class Person:
def __init__(self, name, age):
self.name = name
self.age = age
(located in trc/suivi/amqp directory structure)
I get a class cast exception. I am pretty sure this has to do either with the package/module name or with some serialization issue...
EDIT: I am now using JSon and I get this upon java deserialization:
Exception in thread "main" org.springframework.amqp.support.converter.MessageConversionException: failed to convert Message content. Could not resolve
__TypeId__ in header
at org.springframework.amqp.support.converter.DefaultJavaTypeMapper.retrieveHeader(DefaultJavaTypeMapper.java:104)
at org.springframework.amqp.support.converter.DefaultJavaTypeMapper.toJavaType(DefaultJavaTypeMapper.java:53)
at org.springframework.amqp.support.converter.JsonMessageConverter.fromMessage(JsonMessageConverter.java:118)
at org.springframework.amqp.rabbit.core.RabbitTemplate.receiveAndConvert(RabbitTemplate.java:425)
at trc.suivi.amqp.Consumer.main(Consumer.java:12)
Are you using a common serialization library? Take a look at this answer for more details Is there any library to deserialize with python which is serialized with java.
You could try serializing into json and then it will be easy to deserialize that in your python code
The key to this is using a standard message format which can be encoded and decoded in many languages. Many people just use JSON in UTF-8-encoded strings, but I have found MessagePack to be almost as flexible as JSON, and more bandwidth efficient. http://msgpack.org/
I like to use an object-style format because it is easy to add data to a message and then send it on to another queue. I have done that to deal with handling errors (by adding an error reason code and dispatching a message to a queue where the handler will examine the reason and either retry or fix and retry). Also, adding timestamps has been useful for charting transit times through the system. After handling a message (which will have timestamps from several previous steps), it is sent to a grapher which records the time in queue and time to handle for each step.