i have a bunch of devices that send NMEA sentences to a URL/ip.
that look like this
"$GPGGA,200130.0,3447.854659,N,11014.636735,W,1,11,0.8,41.4,M,-24.0,M,,*53"
i want to read this data in, parse it and upload the key parts to a database. i know how to parse it and upload it to the DB but i am at a complete loss on how to "read"/accept/get the data into a python program so that i can parse and upload.
my first thought was to point it at a Django page and then have Djanog parse it and upload to the database (data will be accessed from Django site) but its a NMEA sentence not a HTTP request so Django rejects it as "message Bad request syntax"
what is the best (python) way to read NMEA sentences sent to a url/IP?
thanks
I assume you have some hardware that has an ethernet connection, and it pipes out the NMEA string over its ethernet connection. this probably defaults to having some random 192.168.0.x ip address and spitting out data over port 12002 or something
you would typically create a socket to listen for this incomming data
server.py
import socket
host = "" #Localhost
port = 12002
PACKET_SIZE=1024 # how many characters to read at a time
sock = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
sock.bind((host,port))
sock.listen(5) # we should never have more than one client
def work_thread(client):
while True: #continuously read and handle data
data = client.recv(PACKET_SIZE)
if not data:
break # done with this client
processData(data)
while True:
client,addr = sock.accept() # your script will block here waiting for a connection
t = threading.Thread(target=work_thread,args=(client,))
t.start()
sometimes however you need to ping the device to get the data
client.py
import socket
host = "192.168.0.204" #Device IP
port = 12002
PACKET_SIZE=1024 # how many characters to read at a time
sock = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
sock.connect((host,port)) #connect to the device
while True: #continuously read and handle data
data = sock.recv(PACKET_SIZE)
processData(data)
Related
I have the following problem: I want a sever to send the contents of a textfile
when requested to do so. I have writen a server script which sends the contents to the client and the client script which receives all the contents with a revcall loop. The recvall works fine when
I run the server and client from the same device for testing.
But when I run the server from a different device in the same wifi network to receive the textfile contents from the server device, the recvall doesn't work and I only receive the first 1460 bytes of the text.
server script
import socket
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind(("", 5000))
server.listen(5)
def send_file(client):
read_string = open("textfile", "rb").read() #6 kilobyte large textfile
client.send(read_string)
while True:
client, data = server.accept()
connect_data = client.recv(1024)
if connect_data == b"send_string":
send_file(client)
else:
pass
client script
import socket
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client.connect(("192.168.1.10", 5000))
connect_message = client.send(b"send_string")
receive_data = ""
while True: # the recvall loop
receive_data_part = client.recv(1024).decode()
receive_data += receive_data_part
if len(receive_data_part) < 1024:
break
print(receive_data)
recv(1024) means to receive at least 1 and at most 1024 bytes. If the connection has closed, you receive 0 bytes, and if something goes wrong, you get an exception.
TCP is a stream of bytes. It doesn't try to keep the bytes from any given send together for the recv. When you make the call, if the TCP endpoint has some data, you get that data.
In client, you assume that anything less than 1024 bytes must be the last bit of data. Not so. You can receive partial buffers at any time. Its a bit subtle on the server side, but you make the same mistake there by assuming that you'll receive exactly the command b"send_string" in a single call.
You need some sort of a protocol that tells receivers when they've gotten the right amount of data for an action. There are many ways to do this, so I can't really give you the answer. But this is why there are protocols out there like zeromq, xmlrpc, http, etc...
I am running a simple closing socket listener to ingest a UDP stream containing json directed to port 5001 (in this instance) on localhost:
import socket
import json
from contextlib import closing
def monitor_stream():
with closing(socket.socket(socket.AF_INET, socket.SOCK_DGRAM)) as s:
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind(('', 5001))
while True:
data, addr = s.recvfrom(4096)
try:
data = data.decode('ascii')
data = json.loads(data)
except:
print 'Not valid JSON: ', data
I need to re-broadcast the stream to another local port (arbitrarily 5002 in this example) so that a second program can access the data in as near as real time. Low latency is crucial. Is the following (using socket.sendto() within the while) an acceptable method:
while True:
data, addr = s.recvfrom(4096)
s.sendto(data, ('localhost', 5002))
If not, how else might I achieve the same result?
I have assumed that there is no way of multiple programs ingesting the original stream in a simultaneous fashion as they originate as unicast packets so only the first bound socket receives them.
Secondly, how might I cast the same stream to multiple ports (local or not)?
I am unable to change the incoming stream port / settings.
This was a simple misuse of socket.sendto() which can only be called on a non-connected socket.
I was trying to use the bound-listener to send-on the ingested stream.
I seems you need two socket objects to re-broadcast to a different address. The second socket is an un-bound echo client as the address is specified in .sendto(string, address)
I am writing a simple client server program using Sockets in python. In my example, 1st I want to send a simple "text" in a packet for example, 2nd I will send an image for example.
In the client I want that if I receive a text, I print it on my console. But, if I receive a file (like an image), I save it on my hard drive.
How can I differentiate between the packets I receive on the client side?
#Server Code
import socket
host = socket.gethostname()
s_client=socket.socket()
port_server = 8081
s_client.bind((host, port_server))
s_client.listen(1)
print("Waiting for any incoming connections from clients ...")
conn_client, addr_client = s_client.accept()
#Send a text
conn_client.send("text".encode())
#Send a file (image for example)
f = open("image", "rb")
l = f.read(1024)
while (l):
conn_client.send(l)
l = f.read(1024)
#Client code
import socket
s = socket.socket()
host = socket.gethostname()
port_server = 8081
s.connect((host, port_server))
print("Connected...")
while True:
data = s.recv(1024)
#I WANT TO DIFFERENTIATE HERE
if data:
print(data)
Python's socket object is designed for low-level network communication. In other words, you can use it to send some raw data from one communication endpoint to another one. The socket object itself does not have any means to differentiate between data types or content. This typically takes place on a higher protocol layer.
For example, when a browser receives some data from a server via http, usually the content type is transmitted so the browser knows how to handle the data.
For your purpose, it might be perfectly sufficient if the server would in a similar way send some kind of prefix before the actual data - for example the string 'TXT' if the subsequent data is a text, and 'IMG' if the subsequent data is an image. The client could then evaluate these first three characters and interpret the data appropriately.
I am trying to send commands to my remote server using Python sockets. My remote server is listening on port 50000 (server socket is not configured with python). When I log onto the server and echo the command I want I get the correct results back as followed:
echo mycommand | nc 127.0.0.1 50000
I get back three lines to stdout:
>ServerOn=
>Command_received
>Cmd=mycommand
I want to achieve the same on the client side using python3. I know Iam able to connect to the server, however, I am not getting the full response like above. I tried the following:
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(('myhost', 50000))
s.sendall('mycommand')
data = s.recv(1024)
s.close()
print(repr(data))
but I only get back the first line
>ServerOn=
Any ideas why I am missing part of the response?
The socket recv() returns when a chunk of data has arrived. There's no guarantee on what size that chunk is, other than that it's less than the passed size, 1024 in this case. Here it looks like the client has sent part of the data when it saw an end-of-line.
If you want to check/parse what you have received, it will be necessary to read it into a buffer until you have the amount of data you expect.
In the short term, keep reading until the socket closes:
connected = True
while connected:
data = s.recv(2048)
if not data:
connected = False
print("### Client Disconnected")
else:
print("Received data: %u octets" % (len(data)))
try:
as_string = str(data,'utf-8')
print(as_string)
except:
pass
I need to create a communication between a client and a server with TCP. But I'd like to send and work with "headers". So from the client I'd like to send a header "COMMAND1" and the server returns me something.
I have the following code:
Server
import socket
import threading
bind_ip = '0.0.0.0'
bind_port = 9998
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind((bind_ip, bind_port))
server.listen(5) # max backlog of connections
print ('Listening on {}:{}'.format(bind_ip, bind_port))
def handle_client_connection(client_socket):
request = client_socket.recv(1024)
print ('Received {}'.format(request))
client_socket.send('Response1!'.encode('utf-8'))
client_socket.close()
while True:
client_sock, address = server.accept()
print ('Accepted connection from {}:{}'.format(address[0], address[1]))
client_handler = threading.Thread(
target=handle_client_connection,
args=(client_sock,) # without comma you'd get a... TypeError: handle_client_connection() argument after * must be a sequence, not _socketobject
)
client_handler.start()
Client
import socket
hostname, sld, tld, port = 'www', 'integralist', 'co.uk', 80
target = '{}.{}.{}'.format(hostname, sld, tld)
# create an ipv4 (AF_INET) socket object using the tcp protocol (SOCK_STREAM)
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
# connect the client
# client.connect((target, port))
client.connect(('0.0.0.0', 9998))
# send some data (in this case a HTTP GET request)
client.send('hi'.encode('utf-8'))
# receive the response data (4096 is recommended buffer size)
response = client.recv(4096)
print (response)
Anyone knows the best way to return "Response1!" when the header is "COMMAND1" and "Response2!" when the header is "COMMAND2"?
I can't find examples on how to use headers
EDIT: It doesn't have to be "COMMAND1" or "COMMAND2" it can be a "0" or "1", or anything else.
If you want to add your own header, you just have to:
Make sure your programm finds the start of your message (like, every message beginns "!?&")
Send your own header-data just after the start-symbol of your message.
Maybe mark the end of your message with something or pass a length in your header.
Since TCP will give you a stream of data, it might come to a case, where it just gives you 2 or 3 messages at once. You have to separate these messages by yourself (e.g. by using "?!&" as start of every message).
You can always create your own protocoll as payload of another protocoll. Just as TCP is just payload from the ethernet point of view.
You can do something i have done with my program to accept such headers
Use pickle library to encode a dict headers and send it through socket.
Code will look something like this.
import pickle
def handleSocket(headers:dict):
message = pickle.dumps(headers)
socket.send(message)
For server side, you will be handling it
Gonna initialise the socket recv to 100 kb
def handleReceivedSocket(socket):
message:dict = pickle.loads(socket.recv(102400))
Another way to do this. Is sending a raw json string to the server (just change pickle.dumps,pickle.loads by json.dumps,json.loads
But it will be in raw and less secure.
Last way you can do it is uri encoding. Check w3schools