I want to implement an IoT application. I will give here a toy version of what I want to do.
Say I have two clients : 'client1' and 'client2' on REMOTE COMPUTERS, and a server 'server', that regulates the computations. The hard thing for me is the fact that the computations can't be made at the same place.
We have : clients_list = ['client1', 'client2']
I want to simulate an algorithm that looks like this:
The server starts with an initial value server_value
for round in range(R):
client_values_dict = {}
for client_id in clients_list:
server broadcasts server_value to the client 'client_id' # via http
client_value = action(server_value) # executed on clients computer
client broadcasts its value to the server # via http
at the meantime, server waits for the response
server fills dictionary with keys clients_list, values client values obtained with 'action' :
client_values_dict[client_id]
server_value = aggregate(client_values_dict) # executed on server computer
On the client side (in client.py), I have a function:
import time
def action(server_value):
time.sleep(10*random.random())
return server_value + random.random()-0.5
On the server side (in server.py), I have a function:
def aggregate(client_values_dict):
return sum(client_values_dict.values())/len(client_values_dict.values())
I want to implement that : I want to write a loop at server level that performs this. I think what I need is an API to handle client-server interactions and parallel computing.
I thought of using Flask for this but I'm afraid that the loop at server level will be blocked by the app.run(debug=True) loop, and that my code won't run until I break the app with CTRL+C.
I want the computations to be made in parallel by the two clients.
I am not familiar with web developpement, my problem might seem trivial and help is probably to be found everywhere on internet, but I don't know where to look at. Any help is cheerfully welcomed.
Here is an example ofa script that simulates what I want, but online.
# -*- coding: utf-8 -*-
import time
import random
server_value = 0
R = 10
clients_list = ['client1', 'client2']
def action(server_value):
time.sleep(3*random.random())
return server_value + random.random()-0.5
def aggregate(client_values_dict):
return sum(client_values_dict.values())/len(client_values_dict.values())
for round in range(R):
client_values_dict = {}
for client_id in clients_list:
client_value = action(server_value) # executed on clients computer
client_values_dict[client_id] = client_value
server_value = aggregate(client_values_dict)
print(server_value)
Have you tried network zero? It's an amazing networking library that I use all the time.
Install:
pip install networkzero
PyPI link: https://pypi.org/project/networkzero/
Docs: https://networkzero.readthedocs.io/en/latest/
Code sample (from their doc page):
Machine/process A:
import networkzero as nw0
address = nw0.advertise("hello")
while True:
name = nw0.wait_for_message_from(address)
nw0.send_reply_to(address, "Hello " + name)
Machine/process B:
import networkzero as nw0
hello = nw0.discover("hello")
reply = nw0.send_message_to(hello, "World!")
print(reply)
reply = nw0.send_message_to(hello, "Tim")
print(reply)
This library also supports more than just 2 connections on the local WiFi, read the docs for more info.
NOTE: I've used this answer before. You can see it here: How to set up a server for a local wifi multiplayer game for python
Related
I am currently underway with my Senior Capstone project, in which I am to write a somewhat basic program which allows a custom interface on my iPhone6 device to remotely control or issue critical commands to a NIDS (Suricata) established at my home RaspberryPi(3B+) VPN. My question, however, is whether it's feasible to write said program which can allow remote access control of basic functions/response options on the Pi's IDS, given that I am utilizing it as a device within the VPN network. The main issue would be establish remote signaling to the iOS device whenever there is an anomaly and allowing it to respond back and execute root-level commands on the NIDS.
If it is of any good use, I am currently using Pythonista as a runtime environment on my mobile device and have set my VPN's connection methods to UDP, but I'm not sure if enabling SSH would assist me. I have a rather basic understanding of how to operate programming in regards to network connectivity. I very much appreciate any and all the help given!
from tkinter import *
window=Tk()
window.geometry("450x450")
window.title("IDS Response Manager")
label1=Label(window,text="Intrusion Response Options",fg= 'black',bg ='white',relief="solid",font=("times new roman",12,"bold"))
label1.pack()
button1=Button(window,text="Terminate Session",fg='white', bg='brown',relief=RIDGE,font=("arial",12,"bold"))
button1.place(x=50,y=110) #GROOVE ,RIDGE ,SUNKEN ,RAISED
button2=Button(window,text="Packet Dump",fg='white', bg='brown',relief=RIDGE,font=("arial",12,"bold"))
button2.place(x=220,y=110) #GROOVE ,RIDGE ,SUNKEN ,RAISED
button3=Button(window,text="Block Port",fg='white', bg='brown',relief=RIDGE,font=("arial",12,"bold"))
button3.place(x=110,y=170) #GROOVE ,RIDGE ,SUNKEN ,RAISED
Very basic options as are shown here.
You can use a flask server with an API, which you can send post requests to. You can then send get requests to receive the commands. To host your API, look at Heroku (free tier available, and very much functional, with already configured app_name.herokuapp.com).
Search up to send a post request with the technologies you are using to build your app. Send keyword command with the command to the /send_commands along with the password, "password_here" (changeable to anything you want).
Python:
Modules: Flask (server), request (client)
Server Code:
from flask import Flask
app = Flask(__name__)
commands = []
#app.route('/get_commands', methods=['GET'])
def get_commands():
tmp_commands = commands[::]
commands = []
return {'commands': tmp_commands}
#app.route('/send_commands', methods=['POST'])
def send_commands():
if request.json['password'] == "password_here":
commands.append(request.json['command'])
return {'worked': True}
else:
return {'worked': False}
if __name__ == '__main__':
app.run(debug=True)
Client Code:
import requests
URL = "url_here/get_commands"
commands = requests.get(url = URL)
for command in commands:
os.system(command)
I want an Expert Advisor to open an Trade triggerd by a Telegram-Message.
I succesfully set up an Hello-World application using MQ4 as Server and Python/Telegram-bot as Client.
When the Telegram-Bot recieves a Message, he will send a request to MQ4 and gets a simple response without executing a trade.
Running Code below.
# Hello World client in Python
# Connects REQ socket to tcp://localhost:5555
import zmq
context = zmq.Context()
# Socket to talk to server
print("Connecting to trading server…")
socket = context.socket(zmq.REQ)
socket.connect("tcp://localhost:5555")
print("Connecting to trading server succeed")
#################################################################################
# Use your own values from my.telegram.org
api_id = ######
api_hash = '#####'
bot_token = '#####'
#################################################################################
from telethon import TelegramClient, events
client = TelegramClient('anon', api_id, api_hash)
#client.on(events.NewMessage)
async def my_event_handler(event):
if "Ascending" in event.raw_text:
if "AUDUSD" in event.raw_text:
await event.reply("AUDUSD sell")
# Do 1 request, waiting for a response
for request in range(1):
print("Telegram: AUDUSD sell execution requested %s …" % request)
socket.send(b"AUDUSD Sell execute")
#Send 2 variables (Ordertype // Symbol)
# Get the reply. -> Not neccesary for final application
# Apülication just needs to send 2 Varianles to MQ4 and trigger the open_order()
message = socket.recv()
print("Received reply %s [ %s ]" % (request, message))
client.start()
client.run_until_disconnected()
// Hello World server in MQ4
#include <Zmq/Zmq.mqh>
//+------------------------------------------------------------------+
void OnTick()
{
Context context("helloworld");
Socket socket(context,ZMQ_REP);
socket.bind("tcp://*:5555");
while(!IsStopped())
{
ZmqMsg request;
// Wait for next request from client
// MetaTrader note: this will block the script thread
// and if you try to terminate this script, MetaTrader
// will hang (and crash if you force closing it)
socket.recv(request);
Print("Receive: AUDUSD Sell execute");
Sleep(1000);
ZmqMsg reply("Trade was executed");
// Send reply back to client
socket.send(reply);
Print("Feedback: Trade was executed");
}
}
//+------------------------------------------------------------------+
Now I want to send 2 variables from Python to MQ4.
1. Ordertype: buy/sell
2. Symbol: EURUSD, AUDUSD,...
Send "Sell" if message contains "Ascending" -
Send "Buy" if message contains "Descending"
Send "AUDUSD" if message contains "AUDUSD",...
To do so I found a Library from Darwinex and want to combine it (interpretation of message, sending value as an array) with my already functioning telegram-bot.
For testing I wanted to try the example-code from Darwinex by itself.
I found the Code v2.0.1:
Python:
https://github.com/darwinex/DarwinexLabs/blob/master/tools/dwx_zeromq_connector/v2.0.1/Python/DWX_ZeroMQ_Connector_v2_0_1_RC8.py
MQ4: (Note: This Library code may replace the whole MQ4 code above in final app.)
https://github.com/darwinex/DarwinexLabs/blob/master/tools/dwx_zeromq_connector/v2.0.1/MQL4/DWX_ZeroMQ_Server_v2.0.1_RC8.mq4
When I copy the Code without changing I get an error in Python:
NameError: name '_zmq' is not defined
After running: _zmq._DWX_ZeroMQ_Connector() - in the Kernel of Spyder.
What can I do to fix that error?
In the final state I want to run the Python-Code and the Expert Advisor on the same Windows Server 2012 R2.
Is it enough if I run the .py file in the powershell from the server or should I host the file with the Webside?
I expect to get the whole system/examplecode running on my VPS or Webside-Host-Server and get an testing environment for further coding action, but currenty I cant get the Library Code in Python to run properly.
Also the MT4 ceeps crashing with the current code - but should be fixed if I combine my application with the Library-Codeexample.
(running everything on my local PC with WIN 10).
Q : I think it is a connection-problem between MT4 and Python.
Without a fully reproducible MCVE-code this is undecideable.
Having used a ZeroMQ-based bidirectional signalling/messaging between a QuantFX in python and trading ecosystem MetaTrader 4 Terminal implemented in MQL4, there is a positive experience of using this architecture.
Details decide.
The Best Next Step :
Start with a plain PUSH/PULL archetype python-PUSH-es, MQL4-script-PULL-s, preferably using tcp:// transport-class ( win platforms need not be ready to use an even simpler, protocol-less, ipc:// transport-class.
Once you have posack'd this trivial step, move forwards.
Q : How do I need to setup my Server to get a connection betwen those two - since it should be the same as on my local PC?
It is normal to use ZeroMQ on the same localhost during prototyping, so you may test and debug the integration. For details on ZeroMQ, feel free to read all details in other posts.
Q : Is it enough if I run the .py file in the powershell from the server or should I host the file with the Webside I already have and use that as "Python-Server"?
Yes, in case the .py file was designed that way. No code, no advice. That simple.
Possible issues :
Versions - ZeroMQ, since 2.11.x till the recent 4.3.+, has made a lot of changes
Installation DLL-details matter.
MQL4 has similarly gone through many changes ( string ceased to be a string and become struct to name a biggest impacting one ), so start with simple scenarios and integrate the target architecture in steps / phases with due testing whether the completed phases work as expected.
to fix that problem you need this:
from DWX_ZeroMQ_Connector_v2_0_1_RC8 import DWX_ZeroMQ_Connector
_zmq = DWX_ZeroMQ_Connector()
(adjust your version of the connector as appropriate).
should fix that problem.
So I'm using Flask_Socket to try to implement a websocket on Flask. Using this I hope to notify all connected clients whenever a piece of data has changed. Here's a simplification of my routes/index.py. The issue that I have is that when a websocket connection is opened, it will stay in the notify_change loop until the socket is closed, and in the meantime, other routes like /users can't be accessed.
from flask_sockets import Sockets
sockets = Sockets(app)
#app.route('/users',methods=['GET'])
def users():
return json.dumps(get_users())
data = "Some value" # the piece of data to push
is_dirty = False # A flag which is set by the code that changes the data
#sockets.route("/notifyChange")
def notify_change(ws):
global is_dirty, data
while not ws.closed:
if is_dirty:
is_dirty = False
ws.send(data)
This seems a normal consequence of what is essentially a while True: however, I've been looking online for a way to get around this while still using flask_sockets and haven't had any luck. I'm running the server on GUnicorn
flask/bin/gunicorn -b '0.0.0.0:8000' -k flask_sockets.worker app:app
I tried adding threads by doing --threads 12 but no luck.
Note: I'm only serving up to 4 users at a time, so scaling is not a requirement, and the solution doesn't need to be ultra-optimized.
I want to make a little update script for a software that runs on a Raspberry Pi and works like a local server. That should connect to a master server in the web to get software updates and also to verify the license of the software.
For that I set up two python scripts. I want these to connect via a TLS socket. Then the client checks the server certificate and the server checks if it's one of the authorized clients. I found a solution for this using twisted on this page.
Now there is a problem left. I want to know which client (depending on the certificate) is establishing the connection. Is there a way to do this in Python 3 with twisted?
I'm happy with every answer.
In a word: yes, this is quite possible, and all the necessary stuff is
ported to python 3 - I tested all the following under Python 3.4 on my Mac and it seems to
work fine.
The short answer is
"use twisted.internet.ssl.Certificate.peerFromTransport"
but given that a lot of set-up is required to get to the point where that is
possible, I've constructed a fully working example that you should be able to
try out and build upon.
For posterity, you'll first need to generate a few client certificates all
signed by the same CA. You've probably already done this, but so others can
understand the answer and try it out on their own (and so I could test my
answer myself ;-)), they'll need some code like this:
# newcert.py
from twisted.python.filepath import FilePath
from twisted.internet.ssl import PrivateCertificate, KeyPair, DN
def getCAPrivateCert():
privatePath = FilePath(b"ca-private-cert.pem")
if privatePath.exists():
return PrivateCertificate.loadPEM(privatePath.getContent())
else:
caKey = KeyPair.generate(size=4096)
caCert = caKey.selfSignedCert(1, CN="the-authority")
privatePath.setContent(caCert.dumpPEM())
return caCert
def clientCertFor(name):
signingCert = getCAPrivateCert()
clientKey = KeyPair.generate(size=4096)
csr = clientKey.requestObject(DN(CN=name), "sha1")
clientCert = signingCert.signRequestObject(
csr, serialNumber=1, digestAlgorithm="sha1")
return PrivateCertificate.fromCertificateAndKeyPair(clientCert, clientKey)
if __name__ == '__main__':
import sys
name = sys.argv[1]
pem = clientCertFor(name.encode("utf-8")).dumpPEM()
FilePath(name.encode("utf-8") + b".client.private.pem").setContent(pem)
With this program, you can create a few certificates like so:
$ python newcert.py a
$ python newcert.py b
Now you should have a few files you can use:
$ ls -1 *.pem
a.client.private.pem
b.client.private.pem
ca-private-cert.pem
Then you'll want a client which uses one of these certificates, and sends some
data:
# tlsclient.py
from twisted.python.filepath import FilePath
from twisted.internet.endpoints import SSL4ClientEndpoint
from twisted.internet.ssl import (
PrivateCertificate, Certificate, optionsForClientTLS)
from twisted.internet.defer import Deferred, inlineCallbacks
from twisted.internet.task import react
from twisted.internet.protocol import Protocol, Factory
class SendAnyData(Protocol):
def connectionMade(self):
self.deferred = Deferred()
self.transport.write(b"HELLO\r\n")
def connectionLost(self, reason):
self.deferred.callback(None)
#inlineCallbacks
def main(reactor, name):
pem = FilePath(name.encode("utf-8") + b".client.private.pem").getContent()
caPem = FilePath(b"ca-private-cert.pem").getContent()
clientEndpoint = SSL4ClientEndpoint(
reactor, u"localhost", 4321,
optionsForClientTLS(u"the-authority", Certificate.loadPEM(caPem),
PrivateCertificate.loadPEM(pem)),
)
proto = yield clientEndpoint.connect(Factory.forProtocol(SendAnyData))
yield proto.deferred
import sys
react(main, sys.argv[1:])
And finally, a server which can distinguish between them:
# whichclient.py
from twisted.python.filepath import FilePath
from twisted.internet.endpoints import SSL4ServerEndpoint
from twisted.internet.ssl import PrivateCertificate, Certificate
from twisted.internet.defer import Deferred
from twisted.internet.task import react
from twisted.internet.protocol import Protocol, Factory
class ReportWhichClient(Protocol):
def dataReceived(self, data):
peerCertificate = Certificate.peerFromTransport(self.transport)
print(peerCertificate.getSubject().commonName.decode('utf-8'))
self.transport.loseConnection()
def main(reactor):
pemBytes = FilePath(b"ca-private-cert.pem").getContent()
certificateAuthority = Certificate.loadPEM(pemBytes)
myCertificate = PrivateCertificate.loadPEM(pemBytes)
serverEndpoint = SSL4ServerEndpoint(
reactor, 4321, myCertificate.options(certificateAuthority)
)
serverEndpoint.listen(Factory.forProtocol(ReportWhichClient))
return Deferred()
react(main, [])
For simplicity's sake we'll just re-use the CA's own certificate for the
server, but in a more realistic scenario you'd obviously want a more
appropriate certificate.
You can now run whichclient.py in one window, then python tlsclient.py a;
python tlsclient.py b in another window, and see whichclient.py print out
a and then b respectively, identifying the clients by the commonName
field in their certificate's subject.
The one caveat here is that you might initially want to put that call to
Certificate.peerFromTransport into a connectionMade method; that won't
work.
Twisted does not presently have a callback for "TLS handshake complete";
hopefully it will eventually, but until it does, you have to wait until you've
received some authenticated data from the peer to be sure the handshake has
completed. For almost all applications, this is fine, since by the time you
have received instructions to do anything (download updates, in your case) the
peer must already have sent the certificate.
I am new to Python and Tornado WebServer.
I am trying to figure out the number of request and number of requests/second in my server side code. I am using Tornadio2 to implement websockets.
Kindly take a look at the following code and let me know, what modification can be done to it.
I am using the RequestHandler.prepare() to bottleneck all the requests and using a list as it is immutable to store the count.
Consider all modules are included
count=[0]
class IndexHandler(tornado.web.RequestHandler):
"""Regular HTTP handler to serve the chatroom page"""
def prepare(self):
count[0]=count[0]+1
def get(self):
self.render('index1.html')
class SocketIOHandler(tornado.web.RequestHandler):
def get(self):
self.render('../socket.io.js')
partQue=Queue.Queue()
class ChatConnection(tornadio2.conn.SocketConnection):
participants = set()
def on_open(self, info):
self.send("Welcome from the server.")
self.participants.add(self)
def on_message(self, message):
partQue.put(message)
time.sleep(10)
self.qmes=partQue.get()
for p in self.participants:
p.send(self.qmes+" "+str(count[0]))
partQue.task_done()
def on_close(self):
self.participants.remove(self)
partQue.join()
# Create tornadio server
ChatRouter = tornadio2.router.TornadioRouter(ChatConnection)
# Create socket application
sock_app = tornado.web.Application(
ChatRouter.urls,
flash_policy_port = 843,
flash_policy_file = op.join(ROOT, 'flashpolicy.xml'),
socket_io_port = 8002)
# Create HTTP application
http_app = tornado.web.Application(
[(r"/", IndexHandler), (r"/socket.io.js", SocketIOHandler)])
if __name__ == "__main__":
import logging
logging.getLogger().setLevel(logging.DEBUG)
# Create http server on port 8001
http_server = tornado.httpserver.HTTPServer(http_app)
http_server.listen(8001)
# Create tornadio server on port 8002, but don't start it yet
tornadio2.server.SocketServer(sock_app, auto_start=False)
# Start both servers
tornado.ioloop.IOLoop.instance().start()
Also, I am confused about every Websocket messages. Does each Websocket event got to server in the form of an HTTP request? or a Socket.IO request?
I use Siege - excellent tool for testing requests if your running on linux. Example
siege http://localhost:8000/?q=yourquery -c10 -t10s
-c10 = 10 concurrent users
-t10s = 10 seconds
Tornadio2 has built-in statistics module, which includes incoming connections/s and other counters.
Check following example: https://github.com/MrJoes/tornadio2/tree/master/examples/stats
When testing applications, always approach performance testing with a healthy appreciation for the uncertainty principle..
If you want to test a server, hook up two PCs to a HUB where you can monitor traffic from one going to the other. Then bang the hell out of the server. There are a variety of tools for doing this, just look for web load testing tools.
Normal HTTP requests in Tornado create a new RequestHandler instance, which persists until the connection is terminated.
WebSockets use persistent connections. One WebSocketHandler instance is created, and each message sent by the browser to the server calls the on_message method.
From what I understand, Socket.IO/Tornad.IO will use WebSockets if supported by the browser, falling back to long polling.