I want to read a set of sensors in parallel with concurrent loops, but each system has a different set of sensors.
I currently have:
with open(inputfilelocation, 'r') as f:
sensorlist = json.load(f)
while True:
for sensor in sensorlist:
H, T = read_sensor(sensor['model'], sensor['address'])
send_data(H, T)
time.sleep(60)
which reads them all, then sleeps for a minute. But now I want to specify how frequently to sample each sensor.
I could do:
from multiprocessing import Process
def loop_a():
while True:
#Sample and send data
time.sleep(sensor_a_delay)
def loop_b():
while True:
#Sample and send data
time.sleep(sensor_b_delay)
Process(target=loop_a).start()
Process(target=loop_b).start()
but then I would need to know at the very least how many sensors I have.
Is there any way to define these loops on the fly?
Edit: I've tried this:
def loop_a(sleeptime, string):
while 1:
print(string)
time.sleep(sleeptime)
Process(target=loop_a(5,"foo")).start()
Process(target=loop_a(2,"bar")).start()
but only the "foo" loop runs.
OK, I've figured out what Klaus meant. This solves my problem:
import time
import json
import threading
with open('sensorlist.json', 'r') as f:
sensorlist = json.load(f)
def sensorloop(model,address,delay):
while True:
H, T = read_sensor(model, address)
send_data(H, T)
time.sleep(delay)
for sensor in sensorlist:
x = threading.Thread(target=sensorloop,
args = (sensor['model'],
sensor['address'],
sensor['delay'])
)
x.start()
Related
I am making a File Sharing Program using sockets in python. I wanna show the transfer progress by making use of progress bar in rich. But the progress bar is not properly synced with the transfer progress
sender script-
import socket, os, time
from rich.console import Console
from rich.progress import Progress
HOST = socket.gethostbyname(socket.gethostname())
PORT = 12345
ADDR = (HOST, PORT)
BUFSIZ = 4096
FORMAT = "utf-8"
SEPARATOR = "<SEPARATOR>"
console = Console()
FILENAMES = ["file.txt", "lol.txt"]
FILSIZ = [str(os.path.getsize(x)) for x in FILENAMES]
def send():
"""main function to send files"""
console.clear()
# creating a client socket
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client.connect(ADDR)
print(client.recv(BUFSIZ).decode(FORMAT))
# sending file data
client.send(SEPARATOR.join(FILENAMES).encode(FORMAT))
print(client.recv(BUFSIZ).decode(FORMAT))
client.send(SEPARATOR.join(FILSIZ).encode(FORMAT))
print(client.recv(BUFSIZ).decode(FORMAT))
# sending files
for idx, files in enumerate(FILENAMES):
with open(files, "rb") as f, Progress() as progress:
task = progress.add_task(f"Sending {files}", total=int(FILSIZ[idx]))
client.send(f.read(int(FILSIZ[idx])))
while not progress.finished:
progress.update(task, advance="<AMOUNT_OF_DATA_OR_CHUNKS_SENT>")
time.sleep(0.1)
f.close()
# closing connection
client.close()
send()
receiver script - https://www.toptal.com/developers/hastebin/avomadisox.py
afaik advance value must be amount of data or chunks sent(might be wrong here)... how do i calculate the amount of data sent?
Rich's progress bars are nice!
For many use-cases, the track function that wraps a Sequence or Iterable will suffice:
import time
from rich.progress import track
for i in track(range(100)):
time.sleep(0.05)
To increment progress by a variable amount at each step, use rich.progress.Progress.
This example might show something in the spirit of the original question. Just for fun, let's customize the progress bar while we're at it.
import time
import random
from rich.progress import (
BarColumn,
Progress,
SpinnerColumn,
TaskProgressColumn,
TimeElapsedColumn,
TimeRemainingColumn,
)
def process(chunks):
for chunk in chunks:
time.sleep(0.1)
yield chunk
chunks = [random.randint(1,20) for _ in range(100)]
progress_columns = (
SpinnerColumn(),
"[progress.description]{task.description}",
BarColumn(),
TaskProgressColumn(),
"Elapsed:",
TimeElapsedColumn(),
"Remaining:",
TimeRemainingColumn(),
)
with Progress(*progress_columns) as progress_bar:
task = progress_bar.add_task("[blue]Downloading...", total=sum(chunks))
for chunk in process(chunks):
progress_bar.update(task, advance=chunk)
Note: The generator process(chunks) is a generic stand-in for the file sizes in original question. This answer is mostly for the benefit of those brought here by searching on something like "python rich.progress_bar.ProgressBar example".
Your main question asks for
How to calculate the amount of data sent?
From Real Python's Socket Programming in Python (Guide):
The .send() method also behaves this way. It returns the number of bytes sent, which may be less than the size of the data passed in.
This means that you have to pass the returned int of socket.send() to the parameter advance of progress.update() function (compare your "<AMOUNT_OF_DATA_OR_CHUNKS_SENT>"):
# (1) define the transfer function returning the bytes_sent
def transfer_to_socket(from_file, to_socket, size):
bytes_to_read = int(size)
chunk = from_file.read(bytes_to_read)
# <AMOUNT_OF_DATA_OR_CHUNKS_SENT>
return to_socket.send(chunk) # may be: bytes_sent < len(chunk) < bytes_to_read
# (2) replace the progress-loop with:
while not progress.finished:
bytes_sent = transfer_to_socket(f, client, FILSIZ[idx])
progress.update(task, advance=bytes_sent)
time.sleep(0.1)
I want to use the bleak library in Python to receive data from a Bluetooth Low Energy device. This part is working. My problem is now, that I don't know how to run this code in the background or parallel.
Eventually, I want to build a tiny python app which is processing the data from the Bluetooth device. So bleak is looping all the time fetching data from a bluetooth device and sending it to the main process where it is processed and displayed.
For some reason, bleak does not run in a thread. Is it possible to use asyncio for this (since it is already used by bleak maybe a good way to go)?
I checked out threads and multiprocessing but somehow I found only examples without processes which loop infinitely and send data. I'm totally new to the topic of parallelization and/or asynchronous processes. Maybe one of you can give a hint where to look for a proper solution for this case.
Below is my code so far (for now I just loop and print data).
from bleak import BleakClient
import json
import time
current_index = 0
time_array = [0] * 20
def TicTocGenerator():
# Generator that returns time differences
ti = 0 # initial time
tf = time.time() # final time
while True:
ti = tf
tf = time.time()
yield tf-ti # returns the time difference
TicToc = TicTocGenerator() # create an instance of the TicTocGen generator
# This will be the main function through which we define both tic() and toc()
def toc(tempBool=True):
# Prints the time difference yielded by generator instance TicToc
tempTimeInterval = next(TicToc)
global current_index
if tempBool:
#print( "Elapsed time: %f seconds.\n" %tempTimeInterval )
time_array[current_index] = tempTimeInterval
if current_index == 19:
current_index = 0
else:
current_index += 1
def tic():
# Records a time in TicToc, marks the beginning of a time interval
toc(False)
def Average(lst):
return sum(lst) / len(lst)
#address = "30:ae:a4:5d:bc:ba"
address = "CCA9907B-10EA-411E-9816-A5E247DCA0C7"
MODEL_NBR_UUID = "beb5483e-36e1-4688-b7f5-ea07361b26a8"
async def run(address, loop):
async with BleakClient(address, loop=loop) as client:
while True:
tic()
model_number = await client.read_gatt_char(MODEL_NBR_UUID)
toc()
json_payload=json.loads(model_number)
print()
print(json_payload)
print("Temp [°C]: "+"{:.2f}".format(json_payload["Temp"]))
print("Volt [V]: "+"{:.2f}".format(json_payload["Volt"]))
print("AngX: "+str(json_payload["AngX"]))
print("AngY: "+str(json_payload["AngY"]))
print("AngZ: "+str(json_payload["AngZ"]))
#print("Millis: {0}".format("".join(map(chr, model_number))))
print("Average [ms]: {:.1f}".format(Average(time_array)*1000))
loop = asyncio.get_event_loop()
loop.run_until_complete(run(address, loop))
I had to make GUI for app that automates FUOTA on multiple BLE devices so my solution was to put bleak loop in separate thread in order to be able to use tkinter mainloop in main thread. You need to use asyncio.run_coroutine_threadsafe to schedule a new task from main thread.
from threading import Thread
import tkinter as tk
from Bleak import BleakScanner
async def scan():
device = await BleakScanner.discover()
for device in devices:
print(device)
def startScan():
# call startScan() from main thread
asyncio.run_coroutine_threadsafe(scan(), loop)
if __name__ == "__main__":
window = tk.Tk()
# ...
loop = asyncio.get_event_loop()
def bleak_thread(loop):
asyncio.set_event_loop(loop)
loop.run_forever()
t = Thread(target=bleak_thread, args=(loop,))
t.start()
window.mainloop()
loop.call_soon_threadsafe(loop.stop)
I am new to python. I am writing a python program to write to a JSON file if the website is unreachable. The multiple websites will be stored in hosts variable. It will be scheduled to check every 5 seconds. I have used pool from multiprocessing to process the website at the same time without delay. After that, i will write the data to the json file. But in here, it is writing only one website data to json file. So how to make this to write two data at the same time.
Here's the sample code:
import os
from multiprocessing import Pool
from datetime import datetime
import time
import json
hosts = ["www.google.com","www.smackcoders.com"]
n = len(hosts)
def write(hosts):
u = "down"
name = "stack.json"
if not os.path.exists(name):
with open(name, 'w') as f:
f.write('{}')
result = [(timestamp, {'monitor.status': u,
"monitor.id": "tcp-tcp#"+hosts
})]
with open(name, 'rb+') as f:
f.seek(-1, os.SEEK_END)
f.truncate()
for entry in result:
_entry = '"{}":{},\n'.format(entry[0], json.dumps(entry[1]))
_entry = _entry.encode()
f.write(_entry)
f.write('}'.encode('ascii'))
def main(hosts):
p = Pool(processes= n)
result = p.map(write, hosts)
while True:
timestamp = datetime.now().strftime("%B %d %Y, %H:%M:%S")
main(hosts)
time.sleep(5)
My output:
""March 13 2019, 10:49:03":{"monitor.id": "tcp-tcp#www.smackcoders.com", "monitor.status": "down"},
}
Required Output:
{"March 13 2019, 10:49:03":{"monitor.id": "tcp-tcp#www.smackcoders.com", "monitor.status": "down"},"March 13 2019, 10:49:03":{"monitor.id": "tcp-tcp#www.google.com", "monitor.status": "down"},
}
Ive made some minor changes to your code and implemented a Lock.
import os
from multiprocessing import Pool,RLock
from datetime import datetime
import time
import json
file_lock=RLock()
hosts = ["www.google.com","www.smackcoders.com"]
n = len(hosts)
def write(hosts):
u = "down"
name = "stack.json"
if not os.path.exists(name):
with open(name, 'w') as f:
f.write('{}')
result = [(timestamp, {'monitor.status': u,
"monitor.id": "tcp-tcp#"+hosts
})]
with file_lock:
with open(name, 'rb+') as f:
f.seek(-1, os.SEEK_END)
f.truncate()
for entry in result:
_entry = '"{}":{},\n'.format(entry[0], json.dumps(entry[1]))
_entry = _entry.encode()
f.write(_entry)
f.write('}'.encode('ascii'))
def main(hosts):
p = Pool(processes= n)
result = p.map(write, hosts)
while True:
timestamp = datetime.now().strftime("%B %d %Y, %H:%M:%S")
main(hosts)
time.sleep(5)
However, for a long running process that constantly has to read and write a file for logging seems like a poor implementation as the code will have to read a bulky file and completely rewrite it on every process. Consider writing the log in a database instead.
Here's a different option that will use Thread over Pool.
Created a class to get the return of join()
# Class that overwrite Thread to get the return of join()
class ThreadWithReturnValue(Thread):
def __init__(self, group=None, target=None, name=None, args=None, kwargs=None, Verbose=None):
if args is None:
args = ()
if kwargs is None:
kwargs = {}
super().__init__(group, target, name, args, kwargs)
self._return = None
def run(self):
print(type(self._target))
if self._target is not None:
self._return = self._target(*self._args, **self._kwargs)
def join(self, *args):
Thread.join(self, *args)
return self._return
I have changed the code to get the status of each hosts first, then writing the result to your file. Also fixed the way the JSON file is written.
import os
from datetime import datetime
import time
import json
from threading import Thread
hosts = ["www.google.com","www.smackcoders.com"]
filepath = os.path.join(os.getcwd(), "stack.json")
n = len(hosts)
def perform_ping(host_ip):
"""
You have hardcoded down, this method will ping to check if we get an ICMP response
"""
response = os.system("ping -c 1 " + host_ip)
if response == 0:
return 'UP'
else:
return 'DOWN'
def write_result(timestamp, results):
# u = "down" Using perform_ping to get the status
if not os.path.exists(filepath):
current_file = {}
else:
# If file exist, reading the current output
with open(filepath, 'r') as f_read:
current_file = json.loads(f_read.read())
inner_result = []
for result in results:
host, status = result
inner_result.append({'monitor.status': status,
"monitor.id": "tcp-tcp#"+host
})
current_file[timestamp] = inner_result
# writing the file with new input
with open(filepath, 'w') as f_write:
f_write.write(json.dumps(current_file))
def main():
while True:
thread_list = []
for host_ip in hosts:
thread_list.append(ThreadWithReturnValue(target=perform_ping, name=host_ip, args=(host_ip, )))
results = []
timestamp = datetime.now().strftime("%B %d %Y, %H:%M:%S")
for thread in thread_list:
thread.start()
for thread in thread_list:
results.append((thread.name, thread.join()))
# Ping is done in parallel, writing the result at the end to avoid thread collision and reading/writing the file to many times if you increase the number of host
write_result(timestamp, results)
time.sleep(5)
if __name__ == '__main__':
main()
***Python code:***
import serial
import pandas as pd
import time
import re
import xlrd
from msvcrt import getch
import numpy as np
i = 0
x = 0
y = 0
df = pd.read_excel(r'C:\Users\lynchfamily\Desktop\mlglovesdata.xls')
# Read COM9
# Read from COM10 as well
# Readline() only works with a timeout (IMPORTANT)
serHC = serial.Serial('COM9', 115200,timeout=.250,parity=serial.PARITY_NONE,rtscts=1) # This is the JY
serRN = serial.Serial('COM10', 115200,timeout=.250,parity=serial.PARITY_NONE,rtscts=1) # This is the silvermate
def serialin():
# Sensor lists
sensor_names = list()
sensor_values = list()
global i
# Read a certain amount of bytes from serial and then continue
# Regular expressions for finding the proper data
while i < 6:
# print(i) for debugging
global serHC
global serRN
#searchObj = re.search(r'(A\d?\d*)?(\d*)?',serHC.read(4).decode(),re.I)
#searchObjRN = re.search(r'(A\d?\d*)?(\d*)?',serRN.read(4).decode(),re.I)
# Serial data stops while in loop
# The if statements keep the false values out of the program
#if searchObj.group(1):
sensor_names.append(serHC.read(2))
#if searchObj.group(2):
sensor_values.append(serHC.read(2))
#if searchObjRN.group(1):
sensor_names.append(serRN.read(2))
#if searchObjRN.group(2):
sensor_values.append(serRN.read(2))
i = i + 1
while 1:
# Get the key from the msvcrt module
key = getch().decode('ASCII')
# If key is pressed, do something
if key:
print(key)
# Zip them together
# Final 2D list
final_2d_list = zip(sensor_names,sensor_values)
print(list(sorted(final_2d_list)))
#vals = df.Dataframe([
#df.append(vals)
#print(sorted_array_1stdim[r])
#sensor_values = [0] * 10
# Thread for reading definition
break
# Fancy recursion
sensor_values.clear()
sensor_names.clear()
i = 0
serialin()
serialin()
Arduino Code:
// The device with green colored wires
void setup() {
Serial.begin(115200);
}
void loop() {
// It won't work with the I2C while loop for some reason. Perhaps it is getting stuck up on it
Serial.print("A4");
Serial.print(analogRead(0)); // Read the local analog signal
delay(5);
Serial.print("A5");
Serial.print(analogRead(1)); // Read the local analog signal
delay(5);
Serial.print("A6");
Serial.print(analogRead(2)); // Read the local analog signal
delay(5);
Serial.print("A7");
Serial.print(analogRead(3)); // Read the local analog signal
}
I'm trying to send analog data from sensors over through bluetooth silver mate from sparkfun, and HC-06 modules to python.
I have to read the analog data at a delay of 5 seconds between each, so that the readings aren't conflicted.
The data comes through serial ports COM9 and COM10. I know that serial in python can be blocking, that's why I attempted to read it first, and then put it in a list.
I also knows that once serial has been read through, it appears to be non-blocking. When I was using serHC.readline() and serRN.readline(), I was getting something like what I'd expect to see.
However, the data in the list were not updating according to the change in the sensors. I have to admit python is not my main programming language, so that is why I'm asking for help.
I thought maybe using multiple threads might work, but I wasn't able to get the serHC and serRN variables in the main thread.
Any help will be appreciated!!
As you have discovered it is not possible to read sequentially from serial ports: a blocking read over one port implies a loss of data simultaneous sent over the other port.
Use a thread based approach.
The following sketch should be enough to get started:
import serial
import time
import re
import threading
BYTES_TO_READ = 6
# read from serial port
def read_from_serial(board, port):
print("reading from {}: port {}".format(board, port))
payload = b''
ser = serial.Serial(port, 115200,timeout=.250, parity=serial.PARITY_NONE, rtscts=1)
bytes_count = 0
while bytes_count < BYTES_TO_READ:
read_bytes = ser.read(2)
# sum number of bytes returned (not 2), you have set the timeout on serial port
# see https://pythonhosted.org/pyserial/pyserial_api.html#serial.Serial.read
bytes_count = bytes_count + len(read_bytes)
payload = payload + read_bytes
# here you have the bytes, do your logic
# ...
print("READ from {}: [{}]".format(board, payload))
return
def main():
board = {
'JY': 'COM9',
'SILVER': 'COM10'
}
threads = []
for b in board:
t = threading.Thread(target=read_from_serial, args=(b, board[b],))
threads.append(t)
t.start()
# wait for all threads termination
for t in threads:
t.join()
main()
For learning about threading: https://pymotw.com/3/threading/
Periodic read from serials
Below a sketch for reading each TIME_PERIOD seconds.
A parte the infinite while loop around the read there is a "thread" loop with a nested try/catch block
for catching serials communication problems and retrying to connect after TIME_PERIOD.
Take it just as a starting example!
import serial
import time
import re
import threading
BYTES_TO_READ = 6
TIME_PERIOD = 5
def read_message(board, port, handle):
payload = b''
bytes_count = 0
while bytes_count < BYTES_TO_READ:
read_bytes = handle.read(2)
bytes_count = bytes_count + len(read_bytes)
payload = payload + read_bytes
# here you have the bytes, do your logic
# ...
print("READ from {}: [{}]".format(board, payload))
def serial_thread(board, port):
print("reading from {}: port {}".format(board, port))
while True:
try:
handle = serial.Serial(port, 115200,timeout=.250, parity=serial.PARITY_NONE, rtscts=1)
while True:
read_message(board, port, handle)
time.sleep(TIME_PERIOD)
except Exception as e:
print("ERROR: {}".format(e))
print("retrying in {} seconds".format(TIME_PERIOD))
handle.close()
time.sleep(TIME_PERIOD)
def main():
board = {
'JY': '/dev/ttyUSB0',
'SILVER': '/dev/ttyACM0'
}
threads = []
for b in board:
t = threading.Thread(target=serial_thread, args=(b, board[b],))
threads.append(t)
t.start()
# wait for all threads termination
for t in threads:
t.join()
main()
I am trying to read from two serial ports at once. Each connected device spits out a line of data. I read the data from each port as a list and then concatenate the list and print it out as one line.
If I read each port individually, the data updates fine. But the second I attempt to read from both, it lags up and the data stops changing in the print output. The timestamp updates fine, but the data itself is what starts to lag.
Below is my code, should I be doing some sort of threading? I am reading from an Arduino and a Teensy.
import serial
import time
serA = serial.Serial('/dev/arduino', 230400)
serT = serial.Serial('/dev/teensy', 9600)
while 1 :
timestamp = "%f" % time.time()
print(timestamp)
arduino = serA.readline().rstrip('\n')
data_listA = arduino.split('$')
teensy = serT.readline().rstrip('\n')
data_listT = teensy.split('$')
data_list = data_listA + data_listT
print(data_list)
just check to see if your serial port has bytes to read before you try to read it ...
while 1 :
timestamp = "%f" % time.time()
print(timestamp)
if serA.inWaiting(): # only read if there is something waiting to be read
arduino = serA.readline().rstrip('\n')
data_listA = arduino.split('$')
print("GOT ARDUINO:",data_listA)
if serB.inWaiting():
teensy = serT.readline().rstrip('\n')
data_listT = teensy.split('$')
print("GOT TEENSY:",data_listT)
Using inwaiting() unfortunately did not work for me. I ended up having to use threading. A basic example for people who might encounter my problem is shown below.
import serial
import Queue
import threading
queue = Queue.Queue(1000)
serA = serial.Serial('/dev/arduino', 230400)
serT = serial.Serial('/dev/teensy', 9600)
def serial_read(s):
while 1:
line = s.readline()
queue.put(line)
threadA = threading.Thread(target=serial_read, args=(serA,),).start()
threadT = threading.Thread(target=serial_read, args=(serT,),).start()
while 1:
line = queue.get(True, 1)
print line
I based my code on the last answer from this question.