Develop Reference

How to make each client get their state if there is class instance in grpc-python server side?

I want to use grpc-python in the following scenario, but I don' t know how to realize it.
The scenario is that, in the python server, it uses class to calculate and update the instance' s state, then sends such state to corresponding client; in the client side, more than one clients need to communicate with the server to get its one result and not interfered by others.
Specifically, suppose there is a class with initial value self.i =0, then each time the client calls the class' s update function, it does self.i=self.i+1 and returns self.i. Actually there are two clients call such update function simultaneously, like when client1 calls update at third time, client2 calls update at first time.
I think this may can be solved by creating thread for each client to avoid conflict. If the new client calls, new thead will be created; if existing client calls, existing thread will be used. But I don' t know how to realize it?
Hope you can help me. Thanks in advance.

I think I solved this problem by myself. If you have any other better solutions, you can post here.
I edited helloworld example in grpc-python introduction to explain my aim.
For helloworld.proto
syntax = "proto3";
option java_multiple_files = true;
option java_package = "io.grpc.examples.helloworld";
option java_outer_classname = "HelloWorldProto";
option objc_class_prefix = "HLW";
package helloworld;
// The greeting service definition.
service Greeter {
// Sends a greeting
rpc SayHello (HelloRequest) returns (HelloReply) {}
rpc Unsubscribe (HelloRequest) returns (HelloReply) {}
}
// The request message containing the user's name.
message HelloRequest {
string name = 1;
}
// The response message containing the greetings
message HelloReply {
string message = 1;
}
I add Unsubsribe function to allow one specific client to diconnect from server.
In hello_server.py
import grpc
import helloworld_pb2
import helloworld_pb2_grpc
import threading
from threading import RLock
import time
from concurrent import futures
import logging
class Calcuate:
def __init__(self):
self.i = 0
def add(self):
self.i+=1
return self.i
class PeerSet(object):
def __init__(self):
self._peers_lock = RLock()
self._peers = {}
self.instances = {}
def connect(self, peer):
with self._peers_lock:
if peer not in self._peers:
print("Peer {} connecting".format(peer))
self._peers[peer] = 1
a = Calcuate()
self.instances[peer] = a
output = a.add()
return output
else:
self._peers[peer] += 1
a = self.instances[peer]
output = a.add()
return output
def disconnect(self, peer):
print("Peer {} disconnecting".format(peer))
with self._peers_lock:
if peer not in self._peers:
raise RuntimeError("Tried to disconnect peer '{}' but it was never connected.".format(peer))
del self._peers[peer]
del self.instances[peer]
def peers(self):
with self._peers_lock:
return self._peers.keys()
class Greeter(helloworld_pb2_grpc.GreeterServicer):
def __init__(self):
self._peer_set = PeerSet()
def _record_peer(self, context):
return self._peer_set.connect(context.peer())
def SayHello(self, request, context):
output = self._record_peer(context)
print("[thread {}] Peers: {}, output: {}".format(threading.currentThread().ident, self._peer_set.peers(), output))
time.sleep(1)
return helloworld_pb2.HelloReply(message='Hello, {}, {}!'.format(request.name, output))
def Unsubscribe(self, request, context):
self._peer_set.disconnect(context.peer())
return helloworld_pb2.HelloReply(message='{} disconnected!'.format(context.peer()))
def serve():
server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))
helloworld_pb2_grpc.add_GreeterServicer_to_server(Greeter(), server)
server.add_insecure_port('[::]:50051')
server.start()
server.wait_for_termination()
if __name__ == '__main__':
logging.basicConfig()
serve()
The use of context.peer() is adapted from Richard Belleville' s answer in this post. You can change add() function to any other functions that can be used to update instance' s state.
In hello_client.py
from __future__ import print_function
import logging
import grpc
import helloworld_pb2
import helloworld_pb2_grpc
def run():
# NOTE(gRPC Python Team): .close() is possible on a channel and should be
# used in circumstances in which the with statement does not fit the needs
# of the code.
with grpc.insecure_channel('localhost:50051') as channel:
stub = helloworld_pb2_grpc.GreeterStub(channel)
response = stub.SayHello(helloworld_pb2.HelloRequest(name='you'))
print("Greeter client received: " + response.message)
response = stub.SayHello(helloworld_pb2.HelloRequest(name='Tom'))
print("Greeter client received: " + response.message)
response = stub.SayHello(helloworld_pb2.HelloRequest(name='Jerry'))
print("Greeter client received: " + response.message)
stub.Unsubscribe(helloworld_pb2.HelloRequest(name="end"))
if __name__ == '__main__':
logging.basicConfig()
run()
If we run serveral hello_client.py simultaneously, the server can distinguish the different clients and send correct corresponding info to them.

handling async streaming request in grpc python

I am trying to understand how to handle a grpc api with bidirectional streaming (using the Python API).
Say I have the following simple server definition:
syntax = "proto3";
package simple;
service TestService {
rpc Translate(stream Msg) returns (stream Msg){}
}
message Msg
{
string msg = 1;
}
Say that the messages that will be sent from the client come asynchronously ( as a consequence of user selecting some ui elements).
The generated python stub for the client will contain a method Translate that will accept a generator function and will return an iterator.
What is not clear to me is how would I write the generator function that will return messages as they are created by the user. Sleeping on the thread while waiting for messages doesn't sound like the best solution.

This is a bit clunky right now, but you can accomplish your use case as follows:
#!/usr/bin/env python
from __future__ import print_function
import time
import random
import collections
import threading
from concurrent import futures
from concurrent.futures import ThreadPoolExecutor
import grpc
from translate_pb2 import Msg
from translate_pb2_grpc import TestServiceStub
from translate_pb2_grpc import TestServiceServicer
from translate_pb2_grpc import add_TestServiceServicer_to_server
def translate_next(msg):
return ''.join(reversed(msg))
class Translator(TestServiceServicer):
def Translate(self, request_iterator, context):
for req in request_iterator:
print("Translating message: {}".format(req.msg))
yield Msg(msg=translate_next(req.msg))
class TranslatorClient(object):
def __init__(self):
self._stop_event = threading.Event()
self._request_condition = threading.Condition()
self._response_condition = threading.Condition()
self._requests = collections.deque()
self._last_request = None
self._expected_responses = collections.deque()
self._responses = {}
def _next(self):
with self._request_condition:
while not self._requests and not self._stop_event.is_set():
self._request_condition.wait()
if len(self._requests) > 0:
return self._requests.popleft()
else:
raise StopIteration()
def next(self):
return self._next()
def __next__(self):
return self._next()
def add_response(self, response):
with self._response_condition:
request = self._expected_responses.popleft()
self._responses[request] = response
self._response_condition.notify_all()
def add_request(self, request):
with self._request_condition:
self._requests.append(request)
with self._response_condition:
self._expected_responses.append(request.msg)
self._request_condition.notify()
def close(self):
self._stop_event.set()
with self._request_condition:
self._request_condition.notify()
def translate(self, to_translate):
self.add_request(to_translate)
with self._response_condition:
while True:
self._response_condition.wait()
if to_translate.msg in self._responses:
return self._responses[to_translate.msg]
def _run_client(address, translator_client):
with grpc.insecure_channel('localhost:50054') as channel:
stub = TestServiceStub(channel)
responses = stub.Translate(translator_client)
for resp in responses:
translator_client.add_response(resp)
def main():
server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))
add_TestServiceServicer_to_server(Translator(), server)
server.add_insecure_port('[::]:50054')
server.start()
translator_client = TranslatorClient()
client_thread = threading.Thread(
target=_run_client, args=('localhost:50054', translator_client))
client_thread.start()
def _translate(to_translate):
return translator_client.translate(Msg(msg=to_translate)).msg
translator_pool = futures.ThreadPoolExecutor(max_workers=4)
to_translate = ("hello", "goodbye", "I", "don't", "know", "why",)
translations = translator_pool.map(_translate, to_translate)
print("Translations: {}".format(zip(to_translate, translations)))
translator_client.close()
client_thread.join()
server.stop(None)
if __name__ == "__main__":
main()
The basic idea is to have an object called TranslatorClient running on a separate thread, correlating requests and responses. It expects that responses will return in the order that requests were sent out. It also implements the iterator interface so that you can pass it directly to an invocation of the Translate method on your stub.
We spin up a thread running _run_client which pulls responses out of TranslatorClient and feeds them back in the other end with add_response.
The main function I included here is really just a strawman since I don't have the particulars of your UI code. I'm running _translate in a ThreadPoolExecutor to demonstrate that, even though translator_client.translate is synchronous, it yields, allowing you to have multiple in-flight requests at once.
We recognize that this is a lot of code to write for such a simple use case. Ultimately, the answer will be asyncio support. We have plans for this in the not-too-distant future. But for the moment, this sort of solution should keep you going whether you're running python 2 or python 3.

Python - Poloniex Push API

I am trying to get live data in Python 2.7.13 from Poloniex through the push API.
I read many posts (including How to connect to poloniex.com websocket api using a python library) and I arrived to the following code:
from autobahn.twisted.wamp import ApplicationSession
from autobahn.twisted.wamp import ApplicationRunner
from twisted.internet.defer import inlineCallbacks
import six
class PoloniexComponent(ApplicationSession):
def onConnect(self):
self.join(self.config.realm)
#inlineCallbacks
def onJoin(self, details):
def onTicker(*args):
print("Ticker event received:", args)
try:
yield self.subscribe(onTicker, 'ticker')
except Exception as e:
print("Could not subscribe to topic:", e)
def main():
runner = ApplicationRunner(six.u("wss://api.poloniex.com"), six.u("realm1"))
runner.run(PoloniexComponent)
if __name__ == "__main__":
main()
Now, when I run the code, it looks like it's running successfully, but I don't know where I am getting the data. I have two questions:
I would really appreciate if someone could walk me through the process of subscribing and getting ticker data, that I will elaborate in python, from step 0: I am running the program on Spyder on Windows. Am I supposed to activate somehow Crossbar?
How do I quit the connection? I simply killed the process with Ctrl+c and now when I try to run it agan, I get the error: ReactorNonRestartable.

I ran into a lot of issues using Poloniex with Python2.7 but finally came to a solution that hopefully helps you.
I found that Poloniex has pulled support for the original WAMP socket endpoint so I would probably stray from this method altogether. Maybe this is the entirety of the answer you need but if not here is an alternate way to get ticker information.
The code that ended up working best for me is actually from the post you linked to above but there was some info regarding currency pair ids I found elsewhere.
import websocket
import thread
import time
import json
def on_message(ws, message):
print(message)
def on_error(ws, error):
print(error)
def on_close(ws):
print("### closed ###")
def on_open(ws):
print("ONOPEN")
def run(*args):
# ws.send(json.dumps({'command':'subscribe','channel':1001}))
ws.send(json.dumps({'command':'subscribe','channel':1002}))
# ws.send(json.dumps({'command':'subscribe','channel':1003}))
# ws.send(json.dumps({'command':'subscribe','channel':'BTC_XMR'}))
while True:
time.sleep(1)
ws.close()
print("thread terminating...")
thread.start_new_thread(run, ())
if __name__ == "__main__":
websocket.enableTrace(True)
ws = websocket.WebSocketApp("wss://api2.poloniex.com/",
on_message = on_message,
on_error = on_error,
on_close = on_close)
ws.on_open = on_open
ws.run_forever()
I commented out the lines that pull data you don't seem to want, but for reference here is some more info from that previous post:
1001 = trollbox (you will get nothing but a heartbeat)
1002 = ticker
1003 = base coin 24h volume stats
1010 = heartbeat
'MARKET_PAIR' = market order books
Now you should get some data that looks something like this:
[121,"2759.99999999","2759.99999999","2758.000000‌​00","0.02184376","12‌​268375.01419869","44‌​95.18724321",0,"2767‌​.80020000","2680.100‌​00000"]]
This is also annoying because the "121" at the beginning is the currency pair id, and this is undocumented and also unanswered in the other stack overflow question referred to here.
However, if you visit this url: https://poloniex.com/public?command=returnTicker it seems the id is shown as the first field, so you could create your own mapping of id->currency pair or parse the data by the ids you want from this.
Alternatively, something as simple as:
import urllib
import urllib2
import json
ret = urllib2.urlopen(urllib2.Request('https://poloniex.com/public?command=returnTicker'))
print json.loads(ret.read())
will return to you the data that you want, but you'll have to put it in a loop to get constantly updating information. Not sure of your needs once the data is received so I will leave the rest up to you.
Hope this helps!

I made, with the help of other posts, the following code to get the latest data using Python 3.x. I hope this helps you:
#TO SAVE THE HISTORICAL DATA (X MINUTES/HOURS) OF EVERY CRYPTOCURRENCY PAIR IN POLONIEX:
from poloniex import Poloniex
import pandas as pd
from time import time
import os
api = Poloniex(jsonNums=float)
#Obtains the pairs of cryptocurrencies traded in poloniex
pairs = [pair for pair in api.returnTicker()]
i = 0
while i < len(pairs):
#Available candle periods: 5min(300), 15min(900), 30min(1800), 2hr(7200), 4hr(14400), and 24hr(86400)
raw = api.returnChartData(pairs[i], period=86400, start=time()-api.YEAR*10)
df = pd.DataFrame(raw)
# adjust dates format and set dates as index
df['date'] = pd.to_datetime(df["date"], unit='s')
df.set_index('date', inplace=True)
# Saves the historical data of every pair in a csv file
path=r'C:\x\y\Desktop\z\folder_name'
df.to_csv(os.path.join(path,r'%s.csv' % pairs[i]))
i += 1

How to make sure that the object gets updated from module to the downstream?

I'm not sure if my title is misleading but I'm a python noob. I'm doing an interface for Vault. What it's doing is just renewing the keys every 30s so the client doesn't have to do it. Otherwise the secret will expire or change and the client using the key will get an error. I'm using python threading to renew the keys. Here's my code.
from __future__ import print_function
import hvac
import time
import threading
import os
VAULT_URL = os.environ['VAULT_ADDR']
VAULT_TOKEN = os.environ['VAULT_TOKEN']
class Client:
def __init__(self, *keys):
self.keys = keys
self.data_dict = {}
self.client = hvac.Client(
url=VAULT_URL,
token=VAULT_TOKEN)
self.__renew()
def read(self):
for key in self.keys:
self.data_dict[key] = self.client.read(key)
return self.data_dict
def __renew(self):
self.client.renew_token()
threading.Timer(30, self.__renew).start()
self.read()
And this is how it's being used.
from cnvault.cnvault import Client
data_dict = Client('secret/key').read()
// This is for web.py just to test
class ping:
def GET(self):
print(data_dict)
return 'pong'
Now if I change the data in Vault using Vault CLI and I call /ping, I'm still seeing old data.

Can not connect to Twitter using Tweepy Streaming

Trying to check whether connection established or not, but nothing happened
I used on_connect to understand but got nothing:
import tweepy
import time
class InOutStreamListener(tweepy.StreamListener):
def on_connect(self):
print 'Connected'
def disconnect(self):
if self.running is False:
return
self.running = False
def on_friends(self, friends):
print friends[0]
auth = tweepy.OAuthHandler('code', 'code')
auth.set_access_token('code', 'code')
l = InOutStreamListener()
streamer = tweepy.Stream(auth, l)
time.sleep(15)
streamer.disconnect()

You only created a Stream, you didn't start it, see the docs.
In this example we will use filter to stream all tweets containing the
word python. The track parameter is an array of search terms to
stream.
myStream.filter(track=['python'])