Develop Reference

Python writing twice but randomly?

So I am currently working on a program that was handed down to me from a previous coworker and I am working through a strange bug. When reading data output from 2 separate serial sources byte by byte, python will write to the same cell in the .csv file as well as the console.
import serial
from datetime import datetime
import os
pressure_passed = False
arduino_passed = False
file_passed = False
BAUD_RATE = 115200
GARBAGE_CYCLES = 3 # how many cycles to ignore before logging data
garbage_cycle = 0
# Save data to log file
def LogData(startTime, pressureData, arduinoData, file):
global garbage_cycle
if garbage_cycle < GARBAGE_CYCLES:
garbage_cycle += 1
else:
delta = datetime.now() - startTime
ms = delta.total_seconds() * 1000
dataString = "{:0.2f}, {}, {}\n".format(ms, pressureData, arduinoData)
file.write(dataString)
file.flush()
print(dataString, end = "")
# Get the COM port for the Mark-10 Series 5
while not pressure_passed:
try:
pressure_com = input("Enter Mark-10 Series 5 COM Port #: ")
pressure_ser = serial.Serial("COM" + str(pressure_com), BAUD_RATE)
pressure_passed = True
except:
print("Invalid COM Port, please enter a valid port.\n-----")
# Get the COM port for the Arduino
while not arduino_passed:
try:
arduino_com = input("Enter Ardunio COM Port #: ")
arduino_ser = serial.Serial("COM" + str(arduino_com), BAUD_RATE)
arduino_passed = True
except:
print("Invalid COM Port, please enter a valid port.\n-----")
# Get the name for the log file
while not file_passed:
try:
file_name = input("Enter log file name: ")
# Add extension if not already given
if "." not in file_name:
file_name += ".csv"
log_file = open(file_name, "a")
# Add header row to log file
if os.stat(log_file.name).st_size == 0:
log_file.write("time (ms), pressure, rate (deg/ms)")
file_passed = True
except:
print("Invalid file, or could not open the file specified.\n-----")
start = datetime.now()
# Variables to read serial input
pressure_data = ""
last_pressure = ""
arduino_data = ""
last_arduino = ""
# Main program loop
# Serial is read from byte by byte to better sync the two devices
while True:
try:
x_changed = False
y_changed = False
# Read from Mark-10 serial if available
# x is a byte read from the serial line, converted to ascii
if pressure_ser.in_waiting > 0:
x = pressure_ser.read().decode('ascii')
x_changed = True
# Read from Arduino serial if available
# y is a byte read from the serial line, converted to ascii
if arduino_ser.in_waiting > 0:
y = arduino_ser.read().decode('ascii')
y_changed = True
# If new data received, check if we should log it
if x_changed:
if x == '\n': # New line detected, log the accumulated data
if last_pressure != pressure_data:
LogData(start, last_pressure, last_arduino, log_file)
last_pressure = pressure_data
pressure_data = ""
elif x != '\r': # Otherwise, add the read character to the string
pressure_data += x
if y_changed:
if y == '\n': # New line detected, log the accumulated data
if last_arduino != arduino_data:
LogData(start, last_pressure, last_arduino, log_file)
last_arduino = arduino_data
arduino_data = ""
elif y != '\r': # Otherwise, add the read character to the string
arduino_data += y
except Exception as e:
print(e)
if arduino_ser.isOpen():
arduino_ser.close()
if pressure_ser.isOpen():
pressure_ser.close()
log_file.close()
break
Here is what the file is spitting out, IE the double printing to a single cell. Sample of the data
Any advice is much appreciated, thank you all!

It looks like when a new pressure is read in, but the value has not changed from last time, then it's not resetting the string that's accumulating all the characters and it doubles up. Then on the next pass, when the REAL pressure hasn't changed, it compares the doubled to the non-doubled and writes again, and vice-versa.
Try unindenting the line that resets the string to remove it from the if clause:
# If new data received, check if we should log it
if x_changed:
if x == '\n': # New line detected, log the accumulated data
if last_pressure != pressure_data:
LogData(start, last_pressure, last_arduino, log_file)
last_pressure = pressure_data
pressure_data = ""
elif x != '\r': # Otherwise, add the read character to the string
pressure_data += x
Then the same thing for the arduino value block.
Your logs will probably be much shorter now.

I like your username! My guess is that it is reading from the serial too quickly and going through the loop twice before the arduino has time to change the value of in_waiting.
At the top of your code add:
import time
And in the LogData function add:
time.sleep(0.1)
Give that a shot and let me know if it helps. 0.1s may be too long for your application but it is a good test to see if this is the issue. If it is, you can play around with the amount of time it sleeps.

Based on the sample output provided, I think it's not writing twice but rather the following specific condition is met occasionally which calls two identical LogData() lines.
Only when Condition 1 AND Condition 2 are met - the data is written "twice". Note that the LogData() call is same in both conditions.
Condition 1:
# If new data received, check if we should log it
if x_changed:
if x == '\n': # New line detected, log the accumulated data
if last_pressure != pressure_data:
LogData(start, last_pressure, last_arduino, log_file)
Condition 2:
if y_changed:
if y == '\n': # New line detected, log the accumulated data
if last_arduino != arduino_data:
LogData(start, last_pressure, last_arduino, log_file)

Python datetime from gps to local timezone not changing

So I've been looking for a working solution, but I can't just find it...
I currently have a little program which fetches data from my serial device. This works 100%, but I have trouble with some datetime stuff....
The date and time is in UTC, but I want it in another timezone (Europe/Brussels).
Now, I have written some code that works to convert the input strings, but when I try to replace the data in my json with new data, it doesn't do anything...
Full code:
import serial
import string
import pynmea2
import datetime
import pytz
import math
import time
def utc_to_local(utc_date, utc_time):
tz_eastern = pytz.timezone('UTC')
tz_brussels = pytz.timezone('Europe/Brussels')
return tz_eastern.localize(datetime.datetime.strptime(str(utc_date) + str(utc_time), '%Y-%m-%d%H:%M:%S')).astimezone(tz_brussels).strftime("%Y-%m-%d %H:%M:%S")
def knots_to_km(knots):
return round(math.floor(knots * 1.852))
def open_serial_connection():
ser = serial.Serial()
ser.port = "/dev/ttyS0"
ser.baudrate = 9600
ser.timeout = 1
ser.open()
return ser
def readGPS(serialObject):
try:
json = dict()
latitude = 0.0
longitude = 0.0
speed = 0.0
datetimestamp = ''
sentence = serialObject.readline().decode('utf-8')
if sentence.startswith('$GPGGA'):
data = pynmea2.parse(sentence)
latitude = data.latitude
longitude = data.longitude
if sentence.startswith('$GPVTG'):
data = pynmea2.parse(sentence)
speed = data.spd_over_grnd_kmph
if sentence.startswith('$GPRMC'):
data = pynmea2.parse(sentence)
datetimestamp = utc_to_local(data.datestamp, data.timestamp)
json["lat"] = latitude
json["lon"] = longitude
json["speed"] = speed
json["time"] = datetimestamp
if json["lat"] != 0.0 and json["lon"] != 0.0:
return json
except:
pass
# Test code on pi
ser = open_serial_connection()
while True:
data = readGPS(ser)
if data is not None:
print(data)
time.sleep(0.1)
The output here is the following:
{'lat': xx.xxxxxxxxxxxxxx, 'lon': x.xxxx, 'speed': 0.0, 'time': ''}
The X-es are the correct Latitude and Longitude, they also change.
As you can see, the time is empty, but should change according to the code, right?
What am I missing here?

So the issue here was, as mentioned before, the utc_to_local, which wanted to parse "wrong" data and thus crashed the whole program...
Now the working code stays the same as before, except for the readGPS(), this has now a loop for the data, using a set():
def readGPS(serialObject):
try:
result = {}
result_set = set()
request_set = {'GPGGA', 'GPVTG', 'GPRMC'}
while not request_set.issubset(result_set) and len(result_set) != 10: # 10 == max sentences
sentence = serialObject.readline().decode('utf-8')
key = sentence[1:6]
result_set.add(key)
data = pynmea2.parse(sentence)
if key == 'GPGGA':
result["lat"] = data.latitude
result["lon"] = data.longitude
if key == 'GPVTG':
result["speed"] = data.spd_over_grnd_kmph
if key == 'GPRMC':
result["datetime"] = utc_to_local(data.datestamp, data.timestamp)
return result
except:
pass
It is also good practice to not just pass on errors, but log them or some kind of handling, but for my case, I don't really need to have error loggging (except in debugging), since I'll have alot of "wrong" responses.

Coffee roasting program, how do I write a program that sorts samples of temperature readings and only return an average of the lowest 10?

I am using Python to build a script for a program to be ran by Artisan-Scope roasting software. The program already works with my device (the Phidgets 1045_1B) but I need to do more filtering on the temperature readings. I would like the program to sample at 32ms and organize those 30 samples per second in ascending order. I would then like the lowest 10 samples to be average and returned to Artisan software to be graphed.
This is what I have so far but I need help figuring out how to organize the samples and average them before giving one temperature reading to Artisan.
import sys
import time
import traceback
from Phidget22.Devices.TemperatureSensor import *
from Phidget22.PhidgetException import *
from Phidget22.Phidget import *
from Phidget22.Net import *
try:
from PhidgetHelperFunctions import *
except ImportError:
sys.stderr.write("\nCould not find PhidgetHelperFunctions. Either add PhdiegtHelperFunctions.py to your project folder "
"or remove the import from your project.")
sys.stderr.write("\nPress ENTER to end program.")
readin = sys.stdin.readline()
sys.exit()
def onAttachHandler(self):
ph = self
try:
#If you are unsure how to use more than one Phidget channel with this event, we recommend going to
#www.phidgets.com/docs/Using_Multiple_Phidgets for information
print("\nAttach Event:")
channelClassName = ph.getChannelClassName()
serialNumber = ph.getDeviceSerialNumber()
channel = ph.getChannel()
ph.setDataInterval(32)
ph.setTemperatureChangeTrigger(0)
except PhidgetException as e:
print("\nError in Attach Event:")
DisplayError(e)
traceback.print_exc()
return
def onDetachHandler(self):
ph = self
try:
except PhidgetException as e:
print("\nError in Detach Event:")
DisplayError(e)
traceback.print_exc()
return
def onErrorHandler(self, errorCode, errorString):
sys.stderr.write("[Phidget Error Event] -> " + errorString + " (" + str(errorCode) + ")\n")
"""
* Outputs the TemperatureSensor's most recently reported temperature.
* Fired when a TemperatureSensor channel with onTemperatureChangeHandler registered meets DataInterval and ChangeTrigger criteria
*
* #param self The TemperatureSensor channel that fired the TemperatureChange event
* #param temperature The reported temperature from the TemperatureSensor channel
"""
def onTemperatureChangeHandler(self, temperature):
#If you are unsure how to use more than one Phidget channel with this event, we recommend going to
#www.phidgets.com/docs/Using_Multiple_Phidgets for information
print("[Temperature Event] -> Temperature: " + str(temperature))
"""
* Prints descriptions of how events related to this class work
"""
def PrintEventDescriptions():
print("\n--------------------\n"
"\n | Temperature change events will call their associated function every time new temperature data is received from the device.\n"
" | The rate of these events can be set by adjusting the DataInterval for the channel.\n"
" | Press ENTER once you have read this message.")
readin = sys.stdin.readline(1)
print("\n--------------------")
"""
* Creates, configures, and opens a TemperatureSensor channel.
* Displays Temperature events for 10 seconds
* Closes out TemperatureSensor channel
*
* #return 0 if the program exits successfully, 1 if it exits with errors.
"""
def main():
try:
ch = TemperatureSensor()
ch.setOnAttachHandler(onAttachHandler)
ch.setDeviceSerialNumber(424909)
ch.setChannel(0)
ch.openWaitForAttachment(5000)
ch.setTemperatureChangeTrigger(0)
ch.setOnDetachHandler(onDetachHandler)
ch.setOnErrorHandler(onErrorHandler)
#This call may be harmlessly removed
PrintEventDescriptions()
ch.setOnTemperatureChangeHandler(onTemperatureChangeHandler)
try:
ch.openWaitForAttachment(5000)
except PhidgetException as e:
PrintOpenErrorMessage(e, ch)
raise EndProgramSignal("Program Terminated: Open Failed")
time.sleep(1)
return 0
except PhidgetException as e:
sys.stderr.write("\nExiting with error(s)...")
DisplayError(e)
traceback.print_exc()
print("Cleaning up...")
ch.close()
return 1
except EndProgramSignal as e:
print(e)
print("Cleaning up...")
ch.close()
return 1
except RuntimeError as e:
sys.stderr.write("Runtime Error: \n\t" + e)
traceback.print_exc()
return 1
finally:
print("Press ENTER to end program.")
readin = sys.stdin.readline()
main()

The first thing that you need is some kind of buffer to hold recorded values until you have enough of them to process them. You can for instance use a python list::
# in onAttachHandler:
# init buffer
global buffer
buffer = []
In onTemperatureChangeHandler, store the values in the buffer. Once the buffer is full, calculate your average, then pass that value on.
# in onTEmperatureChangeHandler
global buffer
buffer.append(temperature)
if len(buffer) > 30:
buffer.sort()
mean_temperature = sum(buffer[:10]) / 10.0
buffer = []
# Do something with mean_temperature here
That said, global variables as used here are considered bad style for good reasons. The code should be improved by defining a class, which has the buffer and all the handlers as attributes. There are plenty of Python tutorials about this.

Dronekit Python didn't go to specific location

I have problem about Dronekit on Python
My project about receive GPS Location from Android application
and Launch drone flying to that Position, Everything work fine but problem is Drone can takeoff but drone didn't go to that Location(Testing 10+ times work only 1 time)
Here is my code (I think problem is GlobalRelative)
# Import DroneKit-Python
from dronekit import connect, VehicleMode, LocationGlobalRelative
import time
import urllib2
import simplejson
import json
import requests
#Wait for json data
response = urllib2.urlopen("http://localhost:3000/posts")
data = simplejson.load(response)
print(data)
json_string = data[0]["Information"]
gps = json.loads(json_string)
x=gps['lat']
y=gps['lon']
r = requests.delete('http://localhost:3000/posts/1')
# Connect to the Vehicle.
print("Connecting")
vehicle = connect('com4', wait_ready=False, baud=57600)#; vehicle.wait_ready(True, timeout=300)
print("Connected")
# Get some vehicle attributes (state)
print "Get some vehicle attribute values:"
print " GPS: %s" % vehicle.gps_0
print " Battery: %s" % vehicle.battery
print " Last Heartbeat: %s" % vehicle.last_heartbeat
print " Is Armable?: %s" % vehicle.is_armable
print " System status: %s" % vehicle.system_status.state
print " Mode: %s" % vehicle.mode.name # settable
# Takeoff Function
def arm_and_takeoff(tgt_altitude):
print("Arming motors")
# while not vehicle.is_armable:
# time.sleep(1)
vehicle.mode = VehicleMode("GUIDED")
vehicle.armed = True
print("Takeoff")
vehicle.simple_takeoff(tgt_altitude)
# wait reach tgt_altitude
while True:
altitude = vehicle.location.global_relative_frame.alt
if altitude >= tgt_altitude -1:
print("Altitude Reached")
break
time.sleep(1)
# Main
arm_and_takeoff(10)
# Speed
vehicle.airspeed = 7
# Go to wp
wp1 = LocationGlobalRelative(x, y, 10)
# Close vehicle object before exiting script
vehicle.mode = VehicleMode("RTL")
vehicle.close()
print("Completed")
Alternative, If I can't fix this problem I want to use MissionPlanner( I test on it , and it work) But I want to wait for GPS Location from Phone, and Launch the mission( every thing must automatic ) I have no idea how to bypass MissionPlanner

The line: wp1 = LocationGlobalRelative(x, y, 10) only assign wp1 variable with a location coordinate. You can use vehicle.simple_goto(wp1). simple_goto is a built in function in dronekit to command the vehicle to specific coordinate you can read more about it here http://python.dronekit.io/automodule.html?highlight=simple_goto#dronekit.Vehicle.simple_goto

How to use flow control in pyserial

I need help is using serial port, I have 3DR radio telemetry connected to my python and other side to my windows PC ,I have small python code which continiously writes the data to serial port and reads , reading might not be an issue, or it might be later anyway...
The issue is i am afraid the too many writes might cause some buffer overflow, every time i search the solution is to enable rts/cts flow control, I dont know how to use it ?? what will happen if i set these then pyserial will do what and how can i control my write ?? its really confusing ..
hardware flow ccontol, I am not sure it might work, becuase I have just connected rx tx ground and power to my raspberry pi, even if try to connect the other flow control pins to pi, i am not sure it works or supported by 3dr radio telemetry.. I believe software flow control will be good and simple solution for now.
here is my code ..
for channel in list(self.__channelDict.values()):
# Addition for channel priority later
# We check if the channels in the list is active
if channel.getChannelActive() is True:
# Check if we have reached the max count
if (messageCount >= (self.__NoOfMessagesInUARTStream - 1)) or UARTForceSend:
self.sendUARTStream(UARTCacheBuffer, messageCount, UARTStreamCRC)
# Reset
messageCount = 0
UARTStreamCRC = 0
UARTCacheBuffer.emptyBuffer()
message = channel.RetriveMessage(queueType = 1, raw = True)
# # there is no TX message in this channel
if message is None:
continue # continue with next channel
else:
UARTStreamCRC = binascii.crc32(message, UARTStreamCRC)
UARTCacheBuffer.append(message, raw = True)
messageCount +=1
and the function to write to serial port
def sendUARTStream(self, UARTCacheBuffer, messageCount, UARTStreamCRC):
# retrieve all the data from the buffer and create a stream packet
UARTFrame = None # Used to forward the data
UARTStreamHeader = None
# Create the message header
if messageCount == 0:
# looks like all channels are empty
return 0
else:
messageArray = UARTCacheBuffer.getBuffer()
print(messageArray)
print('messageCount = ' + str(messageCount) + 'crc = ' + str(UARTStreamCRC))
UARTFrame[:self.__UARTStreamHeaderFormat.size] = self.createHeader(messageCount, UARTStreamCRC)
UARTFrame[self.__UARTStreamHeaderFormat.size : self.__UARTStreamHeaderFormat.size + self.__messageFormat * messageCount] = messageArray
# Its time to finally send the data
print('UARTFrame = ##' + str(UARTFrame))
self.__txPort.write(UARTFrame)
return messageCount