Develop Reference

How to save file in Raspberry pi before stream to Influxdb and visualize in Grafana?

I have created a python code for send my GPS data to influxdb by using influxdb.clients. Is that can anyone help me to give any ways to save the data in the raspberry pi if there is no internet connection, then after getting the connection send straightway all the data to the Influxdb which i have created already?
from datetime import datetime
from influxdb_client import InfluxDBClient, Point, WritePrecision
from influxdb_client.client.write_api import SYNCHRONOUS
import serial
# Setup database
token = "<mytoken>"
org = "<myorg>"
bucket = "<mybucket>"
with InfluxDBClient(url="<influxurl>", token=token, org=org) as client:
write_api = client.write_api(write_options=SYNCHRONOUS)
# Setup dataload
json_dataload = []
ser = serial.Serial("/dev/ttyS0")
gpgga_info = "$GPGGA,"
GPGGA_buffer = 0
NMEA_buff = 0
def convert_to_degrees(raw_value):
decimal_value = raw_value / 100.00
degrees = int(decimal_value)
mm_mmmm = (decimal_value - int(decimal_value)) / 0.6
position = degrees + mm_mmmm
position = "%.4f" % position
return position
while True:
received_data = str(ser.readline()) # read NMEA string received
GPGGA_data_available = received_data.find(gpgga_info) # check for NMEA>
if (GPGGA_data_available > 0):
GPGGA_buffer = received_data.split("$GPGGA,", 1)[1] # store data com>
NMEA_buff = (GPGGA_buffer.split(','))
nmea_latitude = []
nmea_longitude = []
extract_latitude = NMEA_buff[1] # extract latitude from >
extract_longitude = NMEA_buff[3] # extract longitude from>
lat = float(extract_latitude)
lat = convert_to_degrees(lat)
longi = float(extract_longitude)
longi = convert_to_degrees(longi)
point = Point("latest GPS") \
.field("latitude", lat) \
.field("longitude", longi) \
.time(datetime.utcnow(), WritePrecision.NS)
json_dataload.append(point)
# Send our payload
write_api.write(bucket, org,json_dataload)

Just setup Telegraf service on your Raspberry PI and write all data to it instead of InfluxDB. Telegraf has a buffer which stores not written data and perform a flush when InfluxDB is available again. What is more, you will be able to batch your data within Telegraf and collect some additional metrics such as CPU/MEM usage of your Raspberry PI.
To do so, install Telegraf Telegraf downloads (remember to choose appropiate CPU arch and OS) and edit config: /etc/telegraf/telegraf.conf
In this file you should config http input for InfluxDB data [[inputs.http_listener]] or [[inputs.http_listener_v2]].
Then config [[outputs.influxdb_v2]] for InfluxDB v2.x data output. After that, restart telegraf service and check its status. If it is "running", you should be able to write data to Telegraf.
Telegraf has many options/plugins which may allow you to collect, buffer and send data to InfluxDB.
Check Telegraf repository for plugin documentation and some configuration examples: Telegraf plugins repository
Ad1:
In Telegraf plugins there is also [[inputs.tail]], using this plugin, you can configure data input from file/directory. This option may be good if you need some persistence during no internet connection period (in case of power outages). If power outages are not a problem, you should use http listener plugin.

How to implement simple client-server interactions in pure python?

I want to implement an IoT application. I will give here a toy version of what I want to do.
Say I have two clients : 'client1' and 'client2' on REMOTE COMPUTERS, and a server 'server', that regulates the computations. The hard thing for me is the fact that the computations can't be made at the same place.
We have : clients_list = ['client1', 'client2']
I want to simulate an algorithm that looks like this:
The server starts with an initial value server_value
for round in range(R):
client_values_dict = {}
for client_id in clients_list:
server broadcasts server_value to the client 'client_id' # via http
client_value = action(server_value) # executed on clients computer
client broadcasts its value to the server # via http
at the meantime, server waits for the response
server fills dictionary with keys clients_list, values client values obtained with 'action' :
client_values_dict[client_id]
server_value = aggregate(client_values_dict) # executed on server computer
On the client side (in client.py), I have a function:
import time
def action(server_value):
time.sleep(10*random.random())
return server_value + random.random()-0.5
On the server side (in server.py), I have a function:
def aggregate(client_values_dict):
return sum(client_values_dict.values())/len(client_values_dict.values())
I want to implement that : I want to write a loop at server level that performs this. I think what I need is an API to handle client-server interactions and parallel computing.
I thought of using Flask for this but I'm afraid that the loop at server level will be blocked by the app.run(debug=True) loop, and that my code won't run until I break the app with CTRL+C.
I want the computations to be made in parallel by the two clients.
I am not familiar with web developpement, my problem might seem trivial and help is probably to be found everywhere on internet, but I don't know where to look at. Any help is cheerfully welcomed.
Here is an example ofa script that simulates what I want, but online.
# -*- coding: utf-8 -*-
import time
import random
server_value = 0
R = 10
clients_list = ['client1', 'client2']
def action(server_value):
time.sleep(3*random.random())
return server_value + random.random()-0.5
def aggregate(client_values_dict):
return sum(client_values_dict.values())/len(client_values_dict.values())
for round in range(R):
client_values_dict = {}
for client_id in clients_list:
client_value = action(server_value) # executed on clients computer
client_values_dict[client_id] = client_value
server_value = aggregate(client_values_dict)
print(server_value)

Have you tried network zero? It's an amazing networking library that I use all the time.
Install:
pip install networkzero
PyPI link: https://pypi.org/project/networkzero/
Docs: https://networkzero.readthedocs.io/en/latest/
Code sample (from their doc page):
Machine/process A:
import networkzero as nw0
address = nw0.advertise("hello")
while True:
name = nw0.wait_for_message_from(address)
nw0.send_reply_to(address, "Hello " + name)
Machine/process B:
import networkzero as nw0
hello = nw0.discover("hello")
reply = nw0.send_message_to(hello, "World!")
print(reply)
reply = nw0.send_message_to(hello, "Tim")
print(reply)
This library also supports more than just 2 connections on the local WiFi, read the docs for more info.
NOTE: I've used this answer before. You can see it here: How to set up a server for a local wifi multiplayer game for python

How to configure in Google Core IoT?

I've made a python program in Raspberry Pi 3 Model B that's supposed to connect to Google Cloud IoT Core with MQTT-protocol and get configurations. Sending data to Core has worked so far, but I just can't figure out how configuring works! Here's a code that's just supposed to get a config:
import time
import datetime
import jwt
import ssl
import json
import paho.mqtt.client as mqtt
time_now = datetime.datetime.utcnow()
#make key
token = {'iat' : time_now ,
'exp' : time_now + datetime.timedelta(minutes=60),
'aud' : "[PROJECT]"}
r = open("[PRIVATE KEY]", 'r')
pub_key = r.read()
jwt_key = jwt.encode(token, pub_key, algorithm='RS256')
#connect to iot-core
client = mqtt.Client(client_id='projects/[PROJECT]/locations/[LOCATION]/registries/[REGISTER]/devices/[DEVICE]')
client.username_pw_set(username='unused', password=jwt_key)
client.tls_set(ca_certs='/home/pi/.local/lib/python2.7/site-packages/grpc/_cython/_credentials/roots.pem', tls_version=ssl.PROTOCOL_TLSv1_2)
client.connect("mqtt.googleapis.com", 8883)
#configure and change state
state = 0
print state #naturally shows 0
print client.subscribe([DEVICE]/config, qos=1) #doesn't print custom config either, just (0,1)
print state #still shows 0
configuration in Iot Core device id is:
{ "state": 1 }
Even after running the program, the device's "state"-variable stays at 0 and the Core's Configuration & State History state that the CONFIG is "Not yet acknowledged by the device"
How do I get the device's "state" variable change from 0 to 1 from Core?

You have done the half of the job. 2 remarks.
It's maybe a detail, but you name your private key pub_key. And you don't close the file.
Based on this tutorial, you only subscribe to the MQTT. The return tuple (0,1) means MQTT_ERR_SUCCESS on QOS = 1. Thus your are connected. Great! Now, do the second part: consume the messages in the channel, and make your logic (according with the received message, change the state in your app if it's your use case)

The pymodbus connection sometimes doesn't respond?

I'm use "pymodbus" lib to connect PLC devices. The device is used Modbus RTU over TCP that devices will return the temperature and humidity of the environment.
map address list
0001: temperature
0002: humidity
I performed once to get value and it can succeed.
But I'm using while loop sometimes get error.
I don't know why.
code:
from time import sleep
from pymodbus.client.sync import ModbusTcpClient
from pymodbus.framer.rtu_framer import ModbusRtuFramer
from pymodbus.register_read_message import ReadHoldingRegistersResponse
client = ModbusTcpClient(host='192.168.1.1', port=5000, framer=ModbusRtuFramer)
client.connect()
while True:
rr = client.read_holding_registers(0, 2, unit=1)
if isinstance(rr, ReadHoldingRegistersResponse):
temp = rr.registers
print(temp)
else:
print('error')
sleep(1)
client.close()
output:
> ...
> [189, 444]
> [189, 443]
> [189]
> error
> error
> ...
We can see that sometimes the result is obtained normally, sometimes the result is incomplete, and sometimes the result is not available.
What should I do to solve this problem, I want to monitor this device. Thank you.

Yes, I see this all the time in my pymodbus code. I suspect there's something wrong with the implementation when doing succesive reads. What I do, is quite simply, to retry the failed read after a slight delay. And that usually gets it working again. Alternatively, try closing and re-connecting the client and re-attempt the reading. Also try increasing the sleep time. Let me know how it goes!

You can try to print what you have in your temp variable in case it's not instance of ReadHoldingRegisterResponse - it may help.
What I use to have sometimes when the device didn't sent the response yet is:
Modbus Error: [Input/Output] No Response received from the remote unit

SMTP - Fast and reliable connection probing without auth?

Briefing
I am currently building a python SMTP Mail sender program.
I added a feature so that the user would not be able to log in if there was no active internet connection, I tried many solutions/variations to make the real time connection checking as swift as possible, there were many problems such as:
The thread where the connection handler was running suddenly lagged when I pulled out the ethernet cable ( to test how it would handle the sudden disconnect )
The whole program crashed
It took several seconds for the program to detect the change
My current solution
I set up a data handling class which would contain all the necessary info ( the modules needed to share info effectively )
import smtplib
from socket import gaierror, timeout
class DataHandler:
is_logged_in = None
is_connected = None
server_conn = None
user_address = ''
user_passwd = ''
#staticmethod
def try_connect():
try:
DataHandler.server_conn = smtplib.SMTP('smtp.gmail.com', 587, timeout=1) # The place where the connection is checked
DataHandler.is_connected = True
except (smtplib.SMTPException, gaierror, timeout):
DataHandler.is_connected = False # Connection status changed upon a connection error
I put a connection handler class on a second thread, the server connection process slowed down the gui when it was all on one thread.
from root_gui import Root
import threading
from time import sleep
from data_handler import DataHandler
def handle_conn():
DataHandler.try_connect()
smtp_client.refresh() # Refreshes the gui according to the current status
def conn_manager(): # Working pretty well
while 'smtp_client' in globals():
sleep(0.6)
try:
handle_conn() # Calls the connection
except NameError: # If the user quits the tkinter gui
break
smtp_client = Root()
handle_conn()
MyConnManager = threading.Thread(target=conn_manager)
MyConnManager.start()
smtp_client.mainloop()
del smtp_client # The connection manager will detect this and stop running
My question is:
Is this a good practice or a terrible waste of resources? Is there a better way to do this because no matter what I tried, this was the only solution that worked.
From what I know the try_connect() function creates a completely new smtp object each time it is run ( which is once in 0.6 seconds! )
Resources/observations
The project on git: https://github.com/cernyd/smtp_client
Observation: the timeout parameter when creating the smtp object improved response times drastically, why is that so?