In my project, I want to call an action on a Raspberry Pi from the Internet. Somehow like that:
Visit webpage
Hit a button on the webpage
Raspberry pi executes a script
The difficult part is, that the raspberry pi only has a mobile Internet connection without a flatrate. So, sending and receiving data from/to the Raspberry Pi is costly.
So this is my question:
Which technologies and patterns can I use to send push notifications to the raspberry pi with minimal data usage?
Start off by creating or starting something that accepts incoming connections on your RPi. Something small as the below example would do:
#!/usr/bin/python
from socket import *
s = socket()
s.bind(('', 8001))
s.listen(4)
while 1:
ns, na = s.accept()
print(na,'sent you:',ns.recv(8192))
Now, the above example will only open a port on 8001 and then print whatever the other send sent to it.
On the server end, I suggest you setup a web server or something else that is easily accessible that can store an IP in a database/register somewhere.
Then, during boot of your RPi (pref after the networking service is started) go ahead and schedule:
curl https://your-server.eu/reg_ip.php > /dev/null 2>&1
Now, on the server your-server.eu:443, make sure req_ip.php saves the clients IP (client being your RPi) in a database somewhere.
Now, which ever application that you have that need to send out PUSH notifications can lookup the current IP of clients in the database and try to connect to port 8001 on those IP's and send whatever data you need.
Now two things:
A listening TCP socket (on the RPi) won't use up any data at all but will allow for incomming connections when needed.
If your IP changes on the RPi (which it probably will on a moving GSM/3G/4G network for instance), you need to do another curl request to your server. This however could easily be tied to a for instance ip monitor command and perform the curl request then.
tl;dr
Here's your chain:
Pi opens a listening socket -> Pi connects to HTTP(S) -> servers saves IP -> when server has something to send -> connect to last known IP on port X -> send data -> close connection
Enhancing further
Now, the HTTP header alone is quite big, in fact it's default 78 bytes of package data (TCP and IP headers usually isn't included in data rates, only the data being transferred is - in this case, HTTP data.). So what you could expand on is if you could use the simple example of a socket above on the server as well and just take the na[0] variable and send that to a database, that way you use literally 0 data-rate on your data subscription.
Only the actual data sent then later on from the server application as a "PUSH notification" would use up data.
In case your RPi is on a NAT based network (private network)
Seeing as the OP gets a 10.X.X.X address it's unlikely that a listening socket will do the trick.
Instead, what you could do is you could simply try and establish a connection and keep it open and have the server send data over any open socket when it has data.
Here's an extremely rough idea of how you could achieve it.
I kept it simple and stupid just to give an idea without solving the entire thing.
Again, the open socket between the client (RPi) and the Server won't use up any data until you actually send data over the channel.
You could in your server fetch data from a database that you want to send to the clients, you could do anything really. Since I don't know your goal I'll just leave it as it is for the time being.
Server:
#!/usr/bin/python
from socket import *
from threading import *
from time import time, sleep
from random import randint
class ClientThread(Thread)
def __init__(self, clients):
self.clients = clients
self.messages = []
Thread.__init__(self)
self.start()
def notify(self, message):
if type(message) is str: message = bytes(message, 'UTF-8')
self.messages.append(message)
def run(self):
while 1:
if len(self.messages) > 0:
m = self.messages.pop(0)
for client in self.clients:
self.clients[client]['sock'].send(m)
class RandomMessageGenerator(Thread):
def __init__(self, notify_func):
self.notify_func = notify_func
self.last = time()
Thread.__init__(self)
self.start()
def run(self):
while 1:
self.notify_func('Time since last update: ' + str(time() - self.last))
self.last = time()
sleep(randint(0,30))
client_list = {}
client_thread_handle = ClientThread(client_list)
random_generator = RandomMessageGenerator(client_thread_handle.nofity)
s = socket()
s.bind(('', 8001))
s.listen(4)
while 1:
ns, na = s.accept()
client_list[na] = {'sock' : 'ns'}
Client:
from socket import *
s = socket()
s.connect(('server', 8001))
while 1:
print(s.recv(8192))
Related
Suppose we have an echo server and a file transfer app.
There is a sender (Client) to send files and a receiver (Server) to receive files. The echo server will echo whatever received from the Client and the Server.
However, Client and Server cannot communicate directly, i.e., all packets have to go through the echo server. For example, the Client sends a UDP packet to the Echo server, and the Echo server echo that packet to Server and Server send Acknowledgment to the Echo server, and the Echo server echo that ack packet to the Client.
The objective is to implement a reliable UDP for file transfer. And we have only one UDP socket.
This figure demonstrates what the setup is
Client, Server and Echo Server
I have tried to use multi-thread and select.select and both do not work perfectly
The issue with multi-thread is that since Client and Server cannot communicate internally, and we have only one socket, it is difficult to choose who should send or receive now.
The issue with select.select is that the return list always has writeable non-empty, which makes the Client continues to send a bunch of packets before the readable is ready.
Here is the implementation for both Client and Server inside one file (say transceiver.py) what I do not use select.select (instead using send bool variable) but it seems to work fine. But I do believe this is bad practice, so I wonder what can I do to improve my design.
def create_transceiver(ip, port):
address = (ip, port)
udp = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
udp.settimeout(1)
inout = [udp]
client = Client(udp, address)
server = Server(udp, address)
client_to_server = True
send = True
while True:
# infds, outfds, errfds = select.select(inout, inout, [])
if not send: # len(infds) != 0
if client_to_server:
server.start_recv()
client_to_server = False
else:
client.start_recv()
client_to_server = True
send = True
elif send: # len(outfds) != 0
if client_to_server:
if client.has_ack_all():
print(server.write_content())
break
client.start_send()
client_to_server = True
else:
server.start_send()
client_to_server = False
send = False
Here is the implementation of Echo Server:
import socket
ip = "10.10.1.100"
port = 8888
address = (ip, port)
udp_socket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
udp_socket.bind(address)
while True:
data, address = udp_socket.recvfrom(2048)
udp_socket.sendto(data, address)
And we have only one UDP socket.
As far as I understand this objective, you have a server process somewhere (you call it echo server) that is listening on a specific port. Also there is a client that wants to send data to some sort of server.
You provided code shows an implementation of what is called a state machine that (in your case) switches between receiving and sending.
However, Client and Server cannot communicate directly
The scenario you are describing makes your echo server a classic server that handles different types of clients. In your case this would be your "client" and your "server". I would like to call these just client-A and client-B. Most tutorials on the internet would call them Alice and Bob, I guess.
The objective is to implement a reliable UDP for file transfer.
So you want to transfer files between different clients using UDP as the base protocol.
UDP is not very well suited for this purpose. It does not garantee delivery of each packet transmitted. It is possible that packets arrive in different order than they were sent.
Usually you would use TCP for this kind of transmission. UDP is usually used for live streaming data like audio/video calls and stuff like that.
For more information on the differences between UDP and TCP you might check out the wikipedia pages for each:
https://en.wikipedia.org/wiki/User_Datagram_Protocol
https://en.wikipedia.org/wiki/Transmission_Control_Protocol
It is possible to use UDP for your transfers but you would have to implement all the safeties provided by TCP yourself.
I assume, your client and your server are actually different programs. Otherwise there would be a way they could communicate directly.
If that is the case, this tutorial might give you a starting point:
https://pythonprogramming.net/server-chatroom-sockets-tutorial-python-3/
I am trying to implement a simple chat program that uses sockets to transmit data via a UDP connection. However, I can't figure out how to correctly set it up so that people from outside my local network can access it if I am hosting it on my laptop. I am utilizing port 5000, and have port-forwarded that port on my router for my laptop. The port-forwarding doesn't seem to be the issue; at least the "Port Forward Network Utilities" from portforward.com seems to detect it as properly forwarded. Maybe I am mixing up the IP addresses I need to host from and connect with? The code in question is below:
import socket
import threading
import sys
class Server:
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
connections = []
def __init__(self):
self.sock.bind(('192.168.1.5', 5000))
self.sock.listen(1)
def handler(self, c, a):
while True:
data = c.recv(1024)
for connection in self.connections:
print(data.decode())
connection.send(data)
if not data:
break
def run(self):
while True:
c, a = self.sock.accept()
cThread = threading.Thread(target=self.handler, args=(c, a))
cThread.daemon = True
cThread.start()
self.connections.append(c)
print(self.connections)
class Client:
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
usr_name = ""
def sendMsg(self):
while True:
self.sock.send(bytes(self.usr_name + ": " + input("-> "), 'utf-8'))
def __init__(self, address):
self.sock.connect((address, 5000))
self.usr_name = input("Enter a username: ")
iThread = threading.Thread(target=self.sendMsg)
iThread.daemon = True
iThread.start()
while True:
data = self.sock.recv(1024)
if not data:
break
print(data.decode())
if len(sys.argv) > 1:
client = Client(sys.argv[1])
else:
server = Server()
server.run()
As you can see, I have my current local IP address inputted for hosting the server, while the client asks for an IP to connect to. I'm not sure what to do now for hosting this over the internet, but I have tried every IP combination I can think of and it returns a number of errors.
Thanks in advance.
Edit:
The two main errors I was getting are:
Timeout Error [WinError 10060]
My friend received this when trying to connect from another network
[WinError 10061]
I would receive this when trying to connect using my public IP from the same computer
I'm sorry that I can't be more detailed in my errors and provide a full printout, and I will try to update this if I'm able to replicate them.
Edit:
I was able to rewrite it and get it to work, I don't need anymore help with this.
Thanks.
You're port-forwarding UDP port 5000 to 5000.
But you're opening TCP streams, not UDP. That's what SOCK_STREAM means. If you want UDP, you need to use SOCK_DGRAM.
So, you need to make these two consistent. The only problem is, I'm not sure which one you actually want here.
On the one hand, your code is doing connection-oriented recv, and seems to be assuming reliable transmission, which means you probably want TCP.
On the other hand, your code seems to be assuming that each recv(1024) is going to get exactly one send from the other side, which is only true for UDP; TCP sockets are byte streams, not message streams. When you do a recv(1024), you could easily get just the first 12 bytes of an 80-byte line, which means it could end in the middle of a UTF-8 character, which means decode will throw an exception.
I think you want TCP, but with a framing protocol on top of it. The simplest protocol that would probably make sense here is lines of text. Which is pretty easy to do on your own, but even easier to do with socket.makefile, given that you're dedicating a thread to each connection.
Good afternoon everyone reading this, I am new to programming with sockets, as well as new to asynchronous coding (I feel async may be part of the solution to my problem), so forgive me for any silly mistakes I make.
To start, I have a UDP Echo server that acts as a game server. Anytime it gets a ping sent to it, it adds the source ip and port to a list of "connected clients", and sends that exact ping out to everyone on the list, excluding the sender. This works fairly well, because it reacts upon receiving a message, so it can always just listen. The problem with the client however, is that I need to be constantly sending pings, while also listening.
This is currently what my client looks like:
import socket
from time import sleep
from contextlib import contextmanager
UDP_IP_ADDRESS = "127.0.0.1"
UDP_PORT_NO = 14004
Message = b"Hello World, From Client B"
#contextmanager
def socket_ctx():
""" Context Manager for the socket. Makes sure the socket will close regardless of why it exited."""
my_socket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
# Assign IP address and a RANDOM available port number to socket
my_socket.bind(('127.0.0.1', 0))
try:
# Let the rest of the app use the socket and wait for it to finish
yield my_socket
finally:
my_socket.close()
def send_data(client_sock):
client_sock.sendto(Message, (UDP_IP_ADDRESS, UDP_PORT_NO))
def listen(client_sock):
print(client_sock.recvfrom(100))
with socket_ctx() as sock:
while True:
send_data(sock)
listen(sock)
sleep(2)
Currently, it sends a ping once, then just idles as it presumably is listening. If it does happen to get a ping back, say, another client send a ping to the server, and the server sent the ping to this client, it hears it, prints it, and starts the loop again. The issue is, without another client sending something to jolt this one out of the listen, it doesn't send it's pings.
I think async might be my solution, but I would have no clue how to go about that. Does anyone have a solution for this problem?
Here's how I would implement a server with "receive and handle incoming UDP sockets, plus do some packet-sending once per second" behavior. Note that this uses the select() function to multiplex the two tasks, rather than asynchronous I/O; hopefully that is okay.
import socket
import select
import time
UDP_IP_ADDRESS = "127.0.0.1"
UDP_PORT_NO = 14004
Message = b"Hello World, From Client B"
udp_socket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
udp_socket.bind(('127.0.0.1', 0))
print "UDP socket is listening for incoming packets on port", udp_socket.getsockname()[1]
# When we want to send the next periodic-ping-message out
nextPingTime = time.time()
while True:
secondsUntilNextPing = nextPingTime - time.time();
if (secondsUntilNextPing < 0):
secondsUntilNextPing = 0
# select() won't return until udp_socket has some data
# ready-for-read, OR until secondsUntilNextPing seconds
# have passed, whichever comes first
inReady, outReady, exReady = select.select([udp_socket], [], [], secondsUntilNextPing)
if (udp_socket in inReady):
# There's an incoming UDP packet ready to receive!
print(udp_socket.recvfrom(100))
now = time.time()
if (now >= nextPingTime):
# Time to send out the next ping!
print "Sending out scheduled ping at time ", now
udp_socket.sendto(Message, (UDP_IP_ADDRESS, UDP_PORT_NO))
nextPingTime = now + 1.0 # we'll do it again in another second
I am trying to broadcast a Udp packet to all the device on my subnet and then i would like to wait for incoming clients to connect to my ip address. I already did a program where it broadcasts its ip address but before that i want to simply send a msg. My problem is both of them is not happening together say if broadcasting is occurred server doesn't wait for client. If i add the line listen to client first then it happens first but i want both to happen. I am new to multithreading and i think there is a small change needed for both to happen, Please post your suggestions and thanks
This is my server side program
import threading
import socket
from socket import *
class Broker():
def broadcast(self,msg):
self.sock=socket(AF_INET,SOCK_DGRAM)
host='192.168.1.255'
port=10000
address=host,port
self.sock.setsockopt(SOL_SOCKET,SO_BROADCAST,1)
msg='hi broadcast'
self.sock.sendto(msg.encode(),(address))
def __init__(self):
self.sock = socket(AF_INET,SOCK_DGRAM)
self.sock.bind(('192.168.1.60', 10000))
self.clients_list = []
def talkToClient(self, ip):
self.sock.sendto("ok".encode('utf-8'), ip)
def listen_clients(self):
while True:
msg, client = self.sock.recvfrom(1024)
print (msg.decode())
t = threading.Thread(None, self.talkToClient, None, (client,), None)
t.start()
b = Broker()
b.listen_clients()
b.broadcast(msg='this os bros')
I'm developing a Flask/gevent WSGIserver webserver that needs to communicate (in the background) with a hardware device over two sockets using XML.
One socket is initiated by the client (my application) and I can send XML commands to the device. The device answers on a different port and sends back information that my application has to confirm. So my application has to listen to this second port.
Up until now I have issued a command, opened the second port as a server, waited for a response from the device and closed the second port.
The problem is that it's possible that the device sends multiple responses that I have to confirm. So my solution was to keep the port open and keep responding to incoming requests. However, in the end the device is done sending requests, and my application is still listening (I don't know when the device is done), thereby blocking everything else.
This seemed like a perfect use case for a thread, so that my application launches a listening server in a separate thread. Because I'm already using gevent as a WSGI server for Flask, I can use the greenlets.
The problem is, I have looked for a good example of such a thing, but all I can find is examples of multi-threading handlers for a single socket server. I don't need to handle a lot of connections on the socket server, but I need it launched in a separate thread so it can listen for and handle incoming messages while my main program can keep sending messages.
The second problem I'm running into is that in the server, I need to use some methods from my "main" class. Being relatively new to Python I'm unsure how to structure it in a way to make that possible.
class Device(object):
def __init__(self, ...):
self.clientsocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.serversocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
def _connect_to_device(self):
print "OPEN CONNECTION TO DEVICE"
try:
self.clientsocket.connect((self.ip, 5100))
except socket.error as e:
pass
def _disconnect_from_device(self):
print "CLOSE CONNECTION TO DEVICE"
self.clientsocket.close()
def deviceaction1(self, ...):
# the data that is sent is an XML document that depends on the parameters of this method.
self._connect_to_device()
self._send_data(XMLdoc)
self._wait_for_response()
return True
def _send_data(self, data):
print "SEND:"
print(data)
self.clientsocket.send(data)
def _wait_for_response(self):
print "WAITING FOR REQUESTS FROM DEVICE (CHANNEL 1)"
self.serversocket.bind(('10.0.0.16', 5102))
self.serversocket.listen(5) # listen for answer, maximum 5 connections
connection, address = self.serversocket.accept()
# the data is of a specific length I can calculate
if len(data) > 0:
self._process_response(data)
self.serversocket.close()
def _process_response(self, data):
print "RECEIVED:"
print(data)
# here is some code that processes the incoming data and
# responds to the device
# this may or may not result in more incoming data
if __name__ == '__main__':
machine = Device(ip="10.0.0.240")
Device.deviceaction1(...)
This is (globally, I left out sensitive information) what I'm doing now. As you can see everything is sequential.
If anyone can provide an example of a listening server in a separate thread (preferably using greenlets) and a way to communicate from the listening server back to the spawning thread, it would be of great help.
Thanks.
EDIT:
After trying several methods, I decided to use Pythons default select() method to solve this problem. This worked, so my question regarding the use of threads is no longer relevant. Thanks for the people who provided input for your time and effort.
Hope it can provide some help, In example class if we will call tenMessageSender function then it will fire up an async thread without blocking main loop and then _zmqBasedListener will start listening on separate port untill that thread is alive. and whatever message our tenMessageSender function will send, those will be received by client and respond back to zmqBasedListener.
Server Side
import threading
import zmq
import sys
class Example:
def __init__(self):
self.context = zmq.Context()
self.publisher = self.context.socket(zmq.PUB)
self.publisher.bind('tcp://127.0.0.1:9997')
self.subscriber = self.context.socket(zmq.SUB)
self.thread = threading.Thread(target=self._zmqBasedListener)
def _zmqBasedListener(self):
self.subscriber.connect('tcp://127.0.0.1:9998')
self.subscriber.setsockopt(zmq.SUBSCRIBE, "some_key")
while True:
message = self.subscriber.recv()
print message
sys.exit()
def tenMessageSender(self):
self._decideListener()
for message in range(10):
self.publisher.send("testid : %d: I am a task" %message)
def _decideListener(self):
if not self.thread.is_alive():
print "STARTING THREAD"
self.thread.start()
Client
import zmq
context = zmq.Context()
subscriber = context.socket(zmq.SUB)
subscriber.connect('tcp://127.0.0.1:9997')
publisher = context.socket(zmq.PUB)
publisher.bind('tcp://127.0.0.1:9998')
subscriber.setsockopt(zmq.SUBSCRIBE, "testid")
count = 0
print "Listener"
while True:
message = subscriber.recv()
print message
publisher.send('some_key : Message received %d' %count)
count+=1
Instead of thread you can use greenlet etc.