Develop Reference

DS18B20, W1ThermSensor, Raspberry pi Zero W, and Python3.9 - Does not consistently read sensor

Before I start, I have browsed the "similar questions" section before writing this and could not see one that matched a situation like mine. If one is found, please let me know and I will mark it as "answered" if it is in fact similar. I am a .net full stack developer by profession, i only recently started dabbling in Python and Electrical Engineering as a hobby.
I am creating an Automated Aquaponics Control system, a part of the project reads the temp of the grow bed media and with the input of various other sensors, recalculates the frequency at which the pump cycles to flood the bed. I am using a DS18B20 with Python3.9 and the W1ThermSensor v2.0.0a2 library. Here is the init and first of several functions for the sensor. I have the w1thermsensor as a property of the class instead of inheritance just during the initial testing, since it is easier to manipulate the code this way for me.
#!/usr/bin/env python3
from w1thermsensor import W1ThermSensor, Sensor, Unit
from datetime import datetime
import os
import numpy
import traceback
class DS18B20:
def __init__(self, min_temp=18, max_temp=26):
self.sensor = W1ThermSensor()
self.temp_string = "{dt} : Sensor: {id} :: {temp_c}C - {temp_f}F"
self.temp_c = 0.00
self.temp_f = 0.00
self.is_active = False
self.is_alert = False
self.min_temp = min_temp
self.max_temp = max_temp
self.values = [0.00, 0.00, 0.00, 0.00, 0.00]
self.value = 0.00
def start(self):
if self.sensor is None:
return False
os.system('modprobe w1-gpio')
os.system('modprobe w1-therm')
# Set baseline for values Average
self.is_active = True
self.monitor()
self.values = [self.temp_c, self.temp_c, self.temp_c, self.temp_c, self.temp_c]
self.value = numpy.average(self.values)
This issue that I am running into is that it will have one of 3 issues:
Raises w1thermsensor.errors.NoSensorFoundError
Raises w1thermsensor.errors.SensorNotReadyError
Returns no value in the temp_c property after calling get_temperature()
I looked into this a bit more, If i load up the IDLE in Terminal using the 'sudo python3' command I can enter the following commands and it works no problem:
sudo python3
>>> from w1thermsensor import W1ThermSensor, Sensor
>>> import time
>>> temp_sensor = W1ThermSensor(Sensor.DS18B20)
>>> while True:
... print(str(round(temp_sensor.get_temperature()))
... time.sleep(2)
and it works without issue. I also try the 'cat' command
cd /sys/bus/w1/devices
cd 28-3c01d607414b
cat w1_slave
94 01 55 05 7f a5 81 66 5b : crc=5b YES
94 01 55 05 7f a5 81 66 5b t=25250
The stacktrace shows that it is throwing the Errors when it is calling the W1Termsensor() function in "init()". My question is, is it my code or implementation that is causing the issue, or is it something else. My sleep is set to 2 seconds in the hope that I am just catching it in the middle of an update. Any help would be a big help.
Addtional Info:
the DS18B20 is wired to a separate 5v power source, the capacitor it to stableize the voltage since there is a 5v relay and a LED array on the same 5v power rail of the power supply.
5v+ -------------+---------VCC------
| |
4.7 Kohm |
| |
GPIO4 ---------------------DQ = 1uf polCap
|
|
|
GND ----------------------GND-------
I have double-checked that I have 1-wire enabled.

Python libusb : lacking privilege for USB access

I am trying to connect to my USB bar-code reader, but bump into a lack of rights to open it under Mac OS 10.14.6.
I am trying to use this library: https://github.com/vpelletier/python-libusb1 the lack of documentation is a bit annoying.
import usb1
VENDOR_ID = 8208
PRODUCT_ID = 30264
INTERFACE = 0
with usb1.USBContext() as context:
handle = context.openByVendorIDAndProductID(
VENDOR_ID,
PRODUCT_ID,
skip_on_error=False,
)
if handle is None:
# Device not present, or user is not allowed to access device.
print("no handle")
with handle.claimInterface(INTERFACE):
print("handle = ", handle)
# Do stuff with endpoints on claimed interface.
while True:
data = handle.bulkRead(ENDPOINT, BUFFER_SIZE)
# Process data...
Is there anyone having a clue on what to do ?
The error is :
USBErrorAccess Traceback (most recent call last)
<ipython-input-1-217d9879b039> in <module>
12 # Device not present, or user is not allowed to access device.
13 print("no handle")
---> 14 with handle.claimInterface(INTERFACE):
15 print("handle = ", handle)
16 # Do stuff with endpoints on claimed interface.
/Applications/anaconda3/lib/python3.7/site-packages/usb1/__init__.py in claimInterface(self, interface)
1307 """
1308 mayRaiseUSBError(
-> 1309 libusb1.libusb_claim_interface(self.__handle, interface),
1310 )
1311 return _ReleaseInterface(self, interface)
/Applications/anaconda3/lib/python3.7/site-packages/usb1/__init__.py in mayRaiseUSBError(value, __raiseUSBError)
131 ):
132 if value < 0:
--> 133 __raiseUSBError(value)
134 return value
135
/Applications/anaconda3/lib/python3.7/site-packages/usb1/__init__.py in raiseUSBError(value, __STATUS_TO_EXCEPTION_DICT, __USBError)
123 __USBError=USBError,
124 ):
--> 125 raise __STATUS_TO_EXCEPTION_DICT.get(value, __USBError)(value)
126
127 def mayRaiseUSBError(
USBErrorAccess: LIBUSB_ERROR_ACCESS [-3]
There is some hints :
seems like we should set up rights somewhere, but these pages are about ubuntu :
=> https://askubuntu.com/questions/978552/how-do-i-make-libusb-work-as-non-root
=> https://github.com/vpelletier/python-libusb1/issues/34
here are some things about Mac OS, but it is unclear to me :
=> IOCreatePlugInInterfaceForService returns mysterious error
=> OpenCV command line app can't access camera under macOS Mojave

Is there a way to find out which function is printing to the terminal?

I was debugging a really complex function (python function that calls some C++ modules from Boost/Python), and there's a print or cout or another similar function that I forgot to delete (can't seem to find it with grep), and now the program just keeps printing a bunch of 0s. Is there a way to tell which function was printing to the terminal, or is deleting the print statement one by one the only way? I'm on Fedora linux if that helps
Thanks

I usually augment my complex designs with various forms of output tracing. The particular item that applies here is the trace_prefix. For instance:
TRACE = True
# TRACE = False # Uncomment this line for production runs.
def func(param):
trace_prefix = "TRACE func" if TRACE else ""
if TRACE:
print("ENTER", func, "param =", param)
...
print(trace_prefix, func_data)
...
This produces output I can identify by "grep TRACE", as well as picking out the stream from a given routine or package. This is easy enough to parameterize (grab the function name); I've done it with a decorator.

If printing happens from Python, setting up logger with elaborate formatters can help you precisely view the module, line number, func name, timestamp, etc.
Here's a sample formatter that I use in my project.
%(asctime)-s.%(msecs)03d %(levelname)-6.6s %(filename)-18.18s line:%(lineno)-4.4d %(funcName)-18s %(message)s
And the results:
2020-08-05 00:28:36.232 DEBUG connectionpool.py line:0959 _new_conn The fancy thing that gets printed.
2020-08-05 00:28:38.177 DEBUG connectionpool.py line:0437 _make_request This is your message but you have more info before this.
2020-08-05 00:28:38.180 DEBUG item.py line:0159 start_indexing Loggers with proper formatters can help you see
2020-08-05 00:28:38.180 DEBUG indexer.py line:0036 start Extensive information
Code:
import logging
import sys
LOGGER = logging.getLogger(__name__)
HANDLER = logging.StreamHandler(sys.stdout)
HANDLER.setFormatter(logging.Formatter("%(asctime)-s.%(msecs)03d %(levelname)-6.6s %(filename)-18.18s line:%(lineno)-4.4d %(funcName)-18s %(message)s"))
LOGGER.addHandler(HANDLER)
LOGGER.error("Here we go!")

1. If your program runs in multi-process, you need to know which process writes "0" to terminal.
#!/usr/bin/env python3
'''
a simple program to stimulate writing terminal in multiple processes.
'''
prog name: ttywriter.py
import time
import sys
import multiprocessing
def print_stderr(s):
sys.stderr.write(f'{s}\n')
while True:
print("I'm writing to my terminal")
p = multiprocessing.Process(target=print_stderr,args=("I am a CPP program, writing to termianl",))
p.start()
time.sleep(1)
Pseudo terminal is usually opened as file descriptor 1 and 2. you can strace them.
# tty
/dev/pts/0
# ps -ef
UID PID PPID C STIME TTY TIME CMD
root 2640 2427 0 07:07 pts/0 00:00:00 python3 ./ttywriter.py
# lsof -p `pgrep -f ttywriter` | grep pts
python3 2640 root 0u CHR 136,0 0t0 3 /dev/pts/0
python3 2640 root 1u CHR 136,0 0t0 3 /dev/pts/0
python3 2640 root 2u CHR 136,0 0t0 3 /dev/pts/0
# strace -s 65536 -e trace=write -e write=1,2 -fp `pgrep -f ttywriter`
strace: Process 2640 attached
write(1, "I'm writing to my terminal\n", 27) = 27
| 00000 49 27 6d 20 77 72 69 74 69 6e 67 20 74 6f 20 6d I'm writing to m |
| 00010 79 20 74 65 72 6d 69 6e 61 6c 0a y terminal. |
strace: Process 2695 attached
[pid 2695] write(2, "I am a CPP program, writing to termianl\n", 40) = 40
| 00000 49 20 61 6d 20 61 20 43 50 50 20 70 72 6f 67 72 I am a CPP progr |
| 00010 61 6d 2c 20 77 72 69 74 69 6e 67 20 74 6f 20 74 am, writing to t |
| 00020 65 72 6d 69 61 6e 6c 0a ermianl. |
[pid 2695] +++ exited with 0 +++
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=2695, si_uid=0, si_status=0, si_utime=0, si_stime=0} ---
2. if it's Python, you can hijack sys.out or print to print call stack.
3. if it's C/C++, you can set a conditional breakpoint for a specific string being written to terminal.

modbus-tk getting Modbus Error: Exception code = 3

I am trying to use modbus-tk to serially communicate with a device via Modbus RTU over a RS-485 network.
To understand how to use modbus-tk, I am trying to query:
Input register: Year of Manufacture
Register address: 0x7543 = 30019
Device: 1
Data type: 16-bit integer
Value = 2019 (verified using a free master modbus simulator).
Here is the code I use, based primarily on rtumaster_example.py.
Input:
#!/usr/bin/env python3
import time
from collections import namedtuple
from logging import Logger
from serial import Serial
from modbus_tk import modbus_rtu
import modbus_tk.defines as cst # cst = constants
from modbus_tk.utils import create_logger
PORT = "COM3"
SLAVE_NUM = 1
MODBUS_MASTER_TIMEOUT_SEC = 5.0
ModbusReg = namedtuple("ModbusInputRegister", ["name", "block_type", "address"])
year_of_manuf = ModbusReg(
"year of manufacture", cst.HOLDING_REGISTERS, 18
) # 0x7543 = 30019
logger = create_logger(name="console") # type: Logger
serial_ = Serial(PORT)
modbus_master = modbus_rtu.RtuMaster(serial_)
modbus_master.set_timeout(MODBUS_MASTER_TIMEOUT_SEC)
modbus_master.set_verbose(True)
time.sleep(2) # Per https://github.com/ljean/modbus-tk/issues/73#issuecomment-284800980
logger.info(
modbus_master.execute(
slave=SLAVE_NUM,
function_code=cst.READ_INPUT_REGISTERS,
starting_address=year_of_manuf.address,
)
)
Output:
2020-01-21 10:38:09,031 INFO modbus_rtu.__init__ MainThread RtuMaster COM3 is opened
2020-01-21 10:38:11,048 DEBUG modbus.execute MainThread -> 1-4-0-18-0-0-80-15
2020-01-21 10:38:11,077 DEBUG modbus.execute MainThread <- 1-132-3-3-1
---------------------------------------------------------------------------
ModbusError Traceback (most recent call last)
<ipython-input-2-9afaebcf3a35> in <module>
7 slave=SLAVE_NUM,
8 function_code=cst.READ_INPUT_REGISTERS,
----> 9 starting_address=year_of_manuf.address,
10 )
c:\path\to\venv\lib\site-packages\modbus_tk\utils.py in new(*args, **kwargs)
37 ret = fcn(*args, **kwargs)
38 except Exception as excpt:
---> 39 raise excpt
40 finally:
41 if threadsafe:
c:\path\to\venv\lib\site-packages\modbus_tk\utils.py in new(*args, **kwargs)
35 lock.acquire()
36 try:
---> 37 ret = fcn(*args, **kwargs)
38 except Exception as excpt:
39 raise excpt
c:\path\to\venv\lib\site-packages\modbus_tk\modbus.py in execute(self, slave, function_code, starting_address, quantity_of_x, output_value, data_format, expected_length)
312 # the slave has returned an error
313 exception_code = byte_2
--> 314 raise ModbusError(exception_code)
315 else:
316 if is_read_function:
ModbusError: Modbus Error: Exception code = 3
It looks like this exception is for if the slave has returned an error
What do you think I am doing wrong? I am new to this library.
What I Have Read
Can't connect to slave with Python's modbus_tk, I am using pyserial >= 3.1
And every other question under the modbus-tk tag, some posts in the Google Groups, and the repo's examples + README.md
Device Specifics
Device: SST Sensing's OXY-LC-485
Modbus RTU, 9600/8-N-1
User Guide (section 7.1.2.1 contains set of input registers)
Device is plugged into Windows machine that I run this Python script
Packages
I am using Python 3.6 on Windows 10.
pyserial==3.4
modbus-tk==1.1.0
**EDIT 1**
Per #Brits comment, I updated my registry addresses to be correct function codes and data frame addresses.
**EDIT 2**
Updated question since I am getting a different error after more correct library usage.

My problem was a lack of understanding of the Master.execute method. I did not realize that the quantity_of_x parameter needed to be made non-zero. A special thanks to #Brits for helping point out several key problems I had with my original code.
When the arg function_code == defines.READ_INPUT_REGISTERS, quantity_of_x means the amount of H (C type, unsigned short, length = 2-byte = 16-bit, source) to read (source code modbus.py, line 138).
Since the device's data type for the year_of_manuf register is 16-bit unsigned integer, the correct change was to add quantity_of_x=1 as an argument.
Functional Input:
logger.info(
modbus_master.execute(
slave=SLAVE_NUM,
function_code=cst.READ_INPUT_REGISTERS,
starting_address=year_of_manuf.address,
quantity_of_x=1,
)
)
Output:
2020-01-21 18:42:05,520 DEBUG modbus.execute MainThread -> 1-4-0-18-0-1-145-207
2020-01-21 18:42:05,560 DEBUG modbus.execute MainThread <- 1-4-2-7-227-250-137
2020-01-21 18:42:05,562 INFO <ipython-input-1-2d4d0280e33d>.<module> MainThread (2019,)

How to profile performance in a QGIS Python plugin?

Is it possible to use kernprof.py, line_profiler.py, or something similar to profile a QGIS plugin? I can't run the plugin outside of QGIS because the plugin requires state from QGIS & will make calls to the QGIS API.
It seems like I might be able to modify the plugin's initializer to call kernprof, to call back to the plugin and pass the state all the way through, but I can't wrap my head around it.
Does anyone have experience with running a Python profiler from inside another tool?

I used a simpler way to profile my plugin using cProfile. In the constructor of main class of plugin (that is returned in classFactory), I used this code:
self.pr = cProfile.Profile()
self.pr.enable()
and in unload method of the class or any where some one needs print the profile stats:
self.pr.disable()
s = io.StringIO()
sortby = SortKey.CUMULATIVE
ps = pstats.Stats(self.pr, stream=s).sort_stats(sortby)
ps.print_stats()
remember to use following code for imports:
import cProfile, pstats, io
from pstats import SortKey

It's possible to use line_profiler while running your script inside QGIS.
You need to import it inside the main file of your plugin along your other imports, then add profile = line_profiler.LineProfiler() before your main class, add the decorator #profile just before your main function to profile and finally add profile.print_stats(stream=stream) just before the return of the function.
I suppose there is other ways to do it, but it's the way I found that works good enough for me.
Below is an example for a Processing plugin:
import os
import line_profiler
profile = line_profiler.LineProfiler()
class processingScriptExample(QgsProcessingAlgorithm):
INPUT_directory = 'INPUT_directory'
def initAlgorithm(self, config):
self.addParameter(QgsProcessingParameterNumber(self.INPUT_directory,
self.tr('Output directory'),
QgsProcessingParameterFile.Folder))
#profile
def processAlgorithm(self, parameters, context, feedback):
directory = self.parameterAsInt(parameters, self.INPUT_directory, context)
ls = []
for ii in range(1000000):
ls.append(ii)
ls = [ii for ii in range(1000000)]
path_profiling = os.path.join(directory, "line_profiling.txt")
with open(path_profiling, 'w') as stream:
profile.print_stats(stream=stream)
return {'Profiling file': path_profiling}
The resulting file:
Timer unit: 1e-07 s
Total time: 1.31260 s
File: C:\OSGeo4W\profiles\default/python/plugins\test\algo_test.py
Function: processAlgorithm at line 70
Line # Hits Time Per Hit % Time Line Contents
==============================================================
70 #profile
71 def processAlgorithm(self, parameters, context, feedback):
72 1 248.0 248.0 0.0 directory = self.parameterAsInt(parameters, self.INPUT_directory, context)
73
74 1 8.0 8.0 0.0 ls = []
75 1000001 5054594.0 5.1 38.5 for ii in range(1000000):
76 1000000 6633146.0 6.6 50.5 ls.append(ii)
77
78 1 1418416.0 1418416.0 10.8 ls = [ii for ii in range(1000000)]
79
80 1 561.0 561.0 0.0 path_profiling = os.path.join(directory, "line_profiling.txt")
81 1 19001.0 19001.0 0.1 with open(path_profiling, 'w') as stream:
82 profile.print_stats(stream=stream)
83
84 return {"Profiling file":path_profiling}