I'm trying to get date and time flowing into Azure IoT hub to enable me to analyze using Azure DX as time series. I can get the temperature and humidity (humidity at the moment is just a random number). If I use this code, all works well and the JSON is well formatted and flows into IoT hub and onto Azure DX:
The basis for the code is taken from the Microsoft examples here - https://github.com/Azure-Samples/azure-iot-samples-python/blob/master/iot-hub/Quickstarts/simulated-device/SimulatedDeviceSync.py
import asyncio
import random
from azure.iot.device import Message
from azure.iot.device.aio import IoTHubDeviceClient
import time
from datetime import datetime
from w1thermsensor import W1ThermSensor
sensor = W1ThermSensor()
import json
CONNECTION_STRING = "xxxxx"
HUMIDITY = 60
MSG_TXT = '{{"temperature": {temperature},"humidity": {humidity}}}'
async def main():
try:
# Create instance of the device client
client = IoTHubDeviceClient.create_from_connection_string(CONNECTION_STRING)
print("Simulated device started. Press Ctrl-C to exit")
while True:
humidity = round(HUMIDITY + (random.random() * 20), 2)
temperature = sensor.get_temperature()
msg_txt_formatted = MSG_TXT.format(temperature=temperature, humidity=humidity)
message = Message(msg_txt_formatted)
# Send a message to the IoT hub
print(f"Sending message: {message}")
await client.send_message(message)
await asyncio.sleep(1)
except KeyboardInterrupt:
print("Simulated device stopped")
if __name__ == '__main__':
asyncio.run(main())
The JSON format is valid and works well -
{ "temperature": 7, "humidity": 66.09 }
If I try to add a date/time field like this:
import asyncio
import random
from azure.iot.device import Message
from azure.iot.device.aio import IoTHubDeviceClient
import time
from datetime import datetime
from w1thermsensor import W1ThermSensor
sensor = W1ThermSensor()
import json
CONNECTION_STRING = "xxxxx"
HUMIDITY = 60
x = datetime.now()
timesent = str(x)
MSG_TXT = '{{"temperature": {temperature},"humidity": {humidity},"timesent": {timesent}}}'
async def main():
try:
# Create instance of the device client
client = IoTHubDeviceClient.create_from_connection_string(CONNECTION_STRING)
print("Simulated device started. Press Ctrl-C to exit")
while True:
humidity = round(HUMIDITY + (random.random() * 20), 2)
temperature = sensor.get_temperature()
msg_txt_formatted = MSG_TXT.format(temperature=temperature, humidity=humidity, timesent=timesent)
message = Message(msg_txt_formatted)
# Send a message to the IoT hub
print(f"Sending message: {message}")
await client.send_message(message)
await asyncio.sleep(1)
except KeyboardInterrupt:
print("Simulated device stopped")
if __name__ == '__main__':
asyncio.run(main())
The output from the JSON is no longer valid and Azure DX will not map. The invalid JSON I get is:
"{\"temperature\": 7,\"humidity\": 72.88, \"timesent\": 2022-11-08 14:21:04.021812}"
I suspect this is something to do with the date/time being formatted as a string, but I'm totally lost.
Would anyone have any ideas how I can send this data?
#JoeHo, thank you for pointing the sources that helped you resolve the issue. I am posting the solution here so that other community members facing similar issue would benefit. Making the below modifications to the code helped me resolve the issue.
def json_serial(obj):
if isinstance(obj, (datetime, date)):
return obj.isoformat()
raise TypeError ("Type %s not serializable" % type(obj))
x = datetime.now().isoformat();
timesent = dumps(datetime.now(), default=json_serial);
MSG_TXT = '{{"temperature": {temperature},"humidity": {humidity}, "timesent": {timesent}}}'
My table on the Azure data explorer has the following filed definitions defined.
.create table jsondata (temperature: real, humidity: real, timesent: datetime)
My data mapping query is as below
.create table jsondata ingestion json mapping 'jsonMapping' '[{"column":"humidity","path":"$.humidity","datatype":"real"},{"column":"temperature","path":"$.temperature","datatype":"real"}, {"column":"timesent","path":"$.timesent","datatype":"datetime"}]'
I then connected the Azure Data Explorer table to IoT Hub using the steps outlined in the following resource Connect Azure Data Explorer table to IoT hub
When I execute the program, I could see the Azure IoT Hub telemetry data flow getting bound to the Azure Data explorer table without any issues.
Related
I am trying to copy the behaviour of a software which control a device using HID interface, the work flow is send an idle request then send a bunch of data to the device, I can send the data, but I don't know how to send the idle request. I found the HID document online and saw this and I can find similar question using Pyusb, but I can't use Pyusb due to the driver issue.
import pywinusb.hid as HID
import time
import datetime
v_id = 0x047a
p_id = 0x1004
i=0
count_line=0
data_2_list=[]
all_devices = HID.HidDeviceFilter(vendor_id = v_id).get_devices()
target_usage = HID.get_full_usage_id(0xFF00, 0x01)
print(all_devices)
device = all_devices[0]
device.open()
data=[0]*64
# print(data)
my_file=open("demofile2.txt","r")
data_lines=my_file.readlines()
for data_line in data_lines:
count_line+=1
data=data_line.replace("\n","")
data_2_str_list=data.split(",")
while i<len(data_2_str_list):
data_2_list.append(int(data_2_str_list[i],16))
i+=1
print(data_2_list)
device.send_output_report(data_2_list)
time.sleep(0.05)
print(count_line)
Is there a way to send Idle request using pywinusb?
I found out that this is a dumb question I simply send the code below
idle=[0x21,0x0A,0x200]
device.send_output_report(idle)
but now it occur another question is that I can send idle request, but the code give me error 87
import pywinusb.hid as hid
import time
import datetime
v_id = 0x047a
p_id = 0x1004
i=0
count_line=0
data_2_list=[]
all_devices = hid.HidDeviceFilter(vendor_id = v_id).get_devices()
target_usage = hid.get_full_usage_id(0xFF00, 0x01)
print(all_devices)
device = all_devices[0]
device.open()
data=[0]*64
# print(data)
my_file=open("demofile2.txt","r")
data_lines=my_file.readlines()
idle=[0x21,0x0A,0x200]
device.send_output_report(idle)
time.sleep(0.05)
for data_line in data_lines:
count_line+=1
data=data_line.replace("\n","")
data_2_str_list=data.split(",")
while i<len(data_2_str_list):
data_2_list.append(int(data_2_str_list[i],16))
i+=1
print(data_2_list)
device.send_output_report(data_2_list)
time.sleep(0.05)
print(count_line)
I have a async_receive method of Eventhub developed in python and also has a checkpoint with it. The code was taken from the official Eventhub sample github repo.
Problem- Using the above-mentioned code, I am just able to receive 20-35 messages per minute if I keep the receiver on for the whole day whereas my Eventhub has a lot of stream data ingested (~200 messages per Minute). The enqueued time at eventhub for a message is now lagging behind by 90 minutes due to poor throughput at the receiver's end which means that the data that got enqueued at X minute in the Eventhub got pulled out of it at X+90 minutes
Investigation- I tried to look at the receive subclass in the Eventhub python SDK and came across a prefetch parameter (line 318) which is set to 300 by default. If this is already set to 300 then I should be able to pull more than 30-35 messages by default.
Any idea on how can I increase my pull capacity? I'm stuck at this point and have no direction forward, any help is highly appreciated.
EDIT 1-
I'm now attaching my Python Code as shown below-
import asyncio
import json
import logging
import os
import sys
import time
from datetime import date
import requests
from azure.eventhub.aio import EventHubConsumerClient
from azure.eventhub.extensions.checkpointstoreblobaio import BlobCheckpointStore
import log_handler
import threading
import traceback
try:
## Set env variables
CONNECTION_STR = os.environ["ECS"].strip()
EVENTHUB_NAME = os.environ['EN'].strip()
EVENTHUB_CONSUMER = os.environ["EC"].strip()
API = os.environ['API_variable'].strip()
AZURE_BLOB_CONNECTION_STR = os.environ["ACS"].strip()
BLOB_CONTAINER_NAME = os.environ["BCN"].strip()
BLOB_ACCOUNT_URL = os.environ["BAU"].strip()
PREFETCH_COUNT = int(os.environ["PREFETCH_COUNT"])
MAX_WAIT_TIME = float(os.environ["MAX_WAIT_TIME"])
except Exception as exception:
logging.debug(traceback.format_exc())
logging.warning(
"*** Please check the environment variables for {}".format(str(exception)))
sys.exit()
def API_CALL(event_data):
"""
Sends the request to the API
"""
try:
url = event_data['image_url']
payload = {"url": url}
## API call to the server
service_response = requests.post(API, json=payload)
logging.info(f"*** service_response.status_code : {service_response.status_code}")
cloud_response = json.loads(
service_response.text) if service_response.status_code == 200 else None
today = date.today()
response_date = today.strftime("%B %d, %Y")
response_time = time.strftime("%H:%M:%S", time.gmtime())
response_data = {
"type": "response_data",
"consumer_group": EVENTHUB_CONSUMER,
'current_date': response_date,
'current_time': response_time,
'image_url': url,
'status_code': service_response.status_code,
'response': cloud_response,
'api_response_time': int(service_response.elapsed.total_seconds()*1000),
"eventhub_data": event_data
}
logging.info(f"*** response_data {json.dumps(response_data)}")
logging.debug(f"*** response_data {json.dumps(response_data)}")
except Exception as exception:
logging.debug(traceback.format_exc())
logging.error(
"**** RaiseError: Failed request url %s, Root Cause of error: %s", url, exception)
async def event_operations(partition_context, event):
start_time = time.time()
data_ = event.body_as_str(encoding='UTF-8')
json_data = json.loads(data_)
## forming data payload
additional_data = {
"type": "event_data",
"consumer_group": EVENTHUB_CONSUMER,
"image_name": json_data["image_url"].split("/")[-1]
}
json_data.update(additional_data)
logging.info(f"*** Data fetched from EH : {json_data}")
logging.debug(f"*** Data fetched from EH : {json_data}")
API_CALL(json_data)
logging.info(f"*** time taken to process an event(ms): {(time.time()-start_time)*1000}")
def between_callback(partition_context, event):
"""
Loop to create threads
"""
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
loop.run_until_complete(event_operations(partition_context, event))
loop.close()
async def on_event(partition_context, event):
"""
Put your code here.
Do some sync or async operations.
If the operation is i/o intensive, async will have better performanceself.
"""
t1 = time.time()
_thread = threading.Thread(target=between_callback, args=(partition_context, event))
_thread.start()
logging.info(f"*** time taken to start a thread(ms): {(time.time()-t1)*1000}")
logging.info("*** Fetching the next event")
## Update checkpoint per event
t2 = time.time()
await partition_context.update_checkpoint(event)
logging.info(f"*** time taken to update checkpoint(ms): {(time.time()-t2)*1000}")
async def main(client):
"""
Run the on_event method for each event received
Args:
client ([type]): Azure Eventhub listener client
"""
try:
async with client:
# Call the receive method. Only read current data (#latest)
logging.info("*** Listening to event")
await client.receive(on_event=on_event,
prefetch=PREFETCH_COUNT,
max_wait_time=MAX_WAIT_TIME)
except KeyboardInterrupt:
print("*** Stopped receiving due to keyboard interrupt")
except Exception as err:
logging.debug(traceback.format_exc())
print("*** some error occured :", err)
if __name__ == '__main__':
## Checkpoint initialization
checkpoint_store = BlobCheckpointStore(
blob_account_url=BLOB_ACCOUNT_URL,
container_name=BLOB_CONTAINER_NAME,
credential=AZURE_BLOB_CONNECTION_STR
)
## Client initialization
client = EventHubConsumerClient.from_connection_string(
CONNECTION_STR,
consumer_group=EVENTHUB_CONSUMER,
eventhub_name=EVENTHUB_NAME,
checkpoint_store=checkpoint_store, #COMMENT TO RUN WITHOUT CHECKPOINT
logging_enable=True,
on_partition_initialize=on_partition_initialize,
on_partition_close=on_partition_close,
idle_timeout=10,
on_error=on_error,
retry_total=3
)
logging.info("Connecting to eventhub {} consumer {}".format(
EVENTHUB_NAME, EVENTHUB_CONSUMER))
logging.info("Registering receive callback.")
loop = asyncio.get_event_loop()
try:
loop.run_until_complete(main(client))
except KeyboardInterrupt as exception:
pass
finally:
loop.stop()
Execution-flow main()-->on_event()-->Thread(between_callback-->API_CALL)-->update_checkpoint
Change the function for the receiver in the below format when we can able to get the events and run them until they complete.
import asyncio
from azure.eventhub.aio import EventHubConsumerClient
from azure.eventhub.extensions.checkpointstoreblobaio import BlobCheckpointStore
async def on_event(partition_context, event):
# Print the event data.
print("Received the event: \"{}\" from the partition with ID: \"{}\"".format(event.body_as_str(encoding='UTF-8'), partition_context.partition_id))
# Update the checkpoint so that the program doesn't read the events
# that it has already read when you run it next time.
await partition_context.update_checkpoint(event)
async def main():
# Create an Azure blob checkpoint store to store the checkpoints.
checkpoint_store = BlobCheckpointStore.from_connection_string("AZURE STORAGE CONNECTION STRING", "BLOB CONTAINER NAME")
# Create a consumer client for the event hub.
client = EventHubConsumerClient.from_connection_string("EVENT HUBS NAMESPACE CONNECTION STRING", consumer_group="$Default", eventhub_name="EVENT HUB NAME", checkpoint_store=checkpoint_store)
async with client:
# Call the receive method. Read from the beginning of the partition (starting_position: "-1")
await client.receive(on_event=on_event, starting_position="-1")
if __name__ == '__main__':
loop = asyncio.get_event_loop()
# Run the main method.
loop.run_until_complete(main())
Also as per the suggest in the comment, call the API on an interval bases.
Newbie to Azure Stream analytics. I am sending events to Azure Event Hub with below Python program by serializing the respone\data to JSON format so that it can be read by the Stream Analytics job input. However, there is something either wrong or I am missing because of which I can not view the test data in the Stream Analytics job.
import asyncio
import time
import requests
import json
from azure.eventhub.aio import EventHubProducerClient
from azure.eventhub import EventData
connection_str = 'XXX'
consumer_group = 'XXX'
eventhub_name = 'XXX'
async def run():
# create a producer client to send messages to the event hub
# specify connection string to your event hubs namespace and
# the event hub name
producer = EventHubProducerClient.from_connection_string(connection_str, consumer_group=consumer_group, eventhub_name=eventhub_name)
async with producer:
# create a batch
event_data_batch = await producer.create_batch(partition_id='1')
while True:
# add events to the batch
response = requests.get("https://jsonplaceholder.typicode.com/todos/1")
data = json.loads(response.text)
event_data_batch.add(EventData(data))
time.sleep(5)
# send the batch of events to the event hub
await producer.send_batch(event_data_batch)
loop = asyncio.get_event_loop()
loop.run_until_complete(run())
I am trying to get live data in Python 2.7.13 from Poloniex through the push API.
I read many posts (including How to connect to poloniex.com websocket api using a python library) and I arrived to the following code:
from autobahn.twisted.wamp import ApplicationSession
from autobahn.twisted.wamp import ApplicationRunner
from twisted.internet.defer import inlineCallbacks
import six
class PoloniexComponent(ApplicationSession):
def onConnect(self):
self.join(self.config.realm)
#inlineCallbacks
def onJoin(self, details):
def onTicker(*args):
print("Ticker event received:", args)
try:
yield self.subscribe(onTicker, 'ticker')
except Exception as e:
print("Could not subscribe to topic:", e)
def main():
runner = ApplicationRunner(six.u("wss://api.poloniex.com"), six.u("realm1"))
runner.run(PoloniexComponent)
if __name__ == "__main__":
main()
Now, when I run the code, it looks like it's running successfully, but I don't know where I am getting the data. I have two questions:
I would really appreciate if someone could walk me through the process of subscribing and getting ticker data, that I will elaborate in python, from step 0: I am running the program on Spyder on Windows. Am I supposed to activate somehow Crossbar?
How do I quit the connection? I simply killed the process with Ctrl+c and now when I try to run it agan, I get the error: ReactorNonRestartable.
I ran into a lot of issues using Poloniex with Python2.7 but finally came to a solution that hopefully helps you.
I found that Poloniex has pulled support for the original WAMP socket endpoint so I would probably stray from this method altogether. Maybe this is the entirety of the answer you need but if not here is an alternate way to get ticker information.
The code that ended up working best for me is actually from the post you linked to above but there was some info regarding currency pair ids I found elsewhere.
import websocket
import thread
import time
import json
def on_message(ws, message):
print(message)
def on_error(ws, error):
print(error)
def on_close(ws):
print("### closed ###")
def on_open(ws):
print("ONOPEN")
def run(*args):
# ws.send(json.dumps({'command':'subscribe','channel':1001}))
ws.send(json.dumps({'command':'subscribe','channel':1002}))
# ws.send(json.dumps({'command':'subscribe','channel':1003}))
# ws.send(json.dumps({'command':'subscribe','channel':'BTC_XMR'}))
while True:
time.sleep(1)
ws.close()
print("thread terminating...")
thread.start_new_thread(run, ())
if __name__ == "__main__":
websocket.enableTrace(True)
ws = websocket.WebSocketApp("wss://api2.poloniex.com/",
on_message = on_message,
on_error = on_error,
on_close = on_close)
ws.on_open = on_open
ws.run_forever()
I commented out the lines that pull data you don't seem to want, but for reference here is some more info from that previous post:
1001 = trollbox (you will get nothing but a heartbeat)
1002 = ticker
1003 = base coin 24h volume stats
1010 = heartbeat
'MARKET_PAIR' = market order books
Now you should get some data that looks something like this:
[121,"2759.99999999","2759.99999999","2758.00000000","0.02184376","12268375.01419869","4495.18724321",0,"2767.80020000","2680.10000000"]]
This is also annoying because the "121" at the beginning is the currency pair id, and this is undocumented and also unanswered in the other stack overflow question referred to here.
However, if you visit this url: https://poloniex.com/public?command=returnTicker it seems the id is shown as the first field, so you could create your own mapping of id->currency pair or parse the data by the ids you want from this.
Alternatively, something as simple as:
import urllib
import urllib2
import json
ret = urllib2.urlopen(urllib2.Request('https://poloniex.com/public?command=returnTicker'))
print json.loads(ret.read())
will return to you the data that you want, but you'll have to put it in a loop to get constantly updating information. Not sure of your needs once the data is received so I will leave the rest up to you.
Hope this helps!
I made, with the help of other posts, the following code to get the latest data using Python 3.x. I hope this helps you:
#TO SAVE THE HISTORICAL DATA (X MINUTES/HOURS) OF EVERY CRYPTOCURRENCY PAIR IN POLONIEX:
from poloniex import Poloniex
import pandas as pd
from time import time
import os
api = Poloniex(jsonNums=float)
#Obtains the pairs of cryptocurrencies traded in poloniex
pairs = [pair for pair in api.returnTicker()]
i = 0
while i < len(pairs):
#Available candle periods: 5min(300), 15min(900), 30min(1800), 2hr(7200), 4hr(14400), and 24hr(86400)
raw = api.returnChartData(pairs[i], period=86400, start=time()-api.YEAR*10)
df = pd.DataFrame(raw)
# adjust dates format and set dates as index
df['date'] = pd.to_datetime(df["date"], unit='s')
df.set_index('date', inplace=True)
# Saves the historical data of every pair in a csv file
path=r'C:\x\y\Desktop\z\folder_name'
df.to_csv(os.path.join(path,r'%s.csv' % pairs[i]))
i += 1
Im new to python and trying to insert the date/time into my firebase.put Im unsure of what this should be and would appreciate some help. Ive tried this, strip() and a couple other unsuccessful things. Please help! Thank you
from firebase import firebase
import time
import datetime
whattime = datetime.datetime.now()
firebase = firebase.FirebaseApplication('my_firebase_link',None)
result = firebase.put('/user',str(whattime),'testing')
There are 2 approaches for timestamps in Firebase, the local machine one and Firebase's own timestamp, the following examples demonstrate each one.
Both examples will require to import the following modules:
import urllib.request
import urllib.error
import json
import time #For the first example only
In this example we use the built in time module from Python (the datetime module works the same). The value from time.time() will be stored as a float and you should convert it back into a date object when you read it back from Firebase.
def send_message(message, username):
my_data = dict()
my_data["message"] = message
my_data["username"] = username
my_data["timestamp"] = time.time()
json_data = json.dumps(my_data).encode()
try:
loader = urllib.request.urlopen("https://<YOUR-PROJECT-ID>.firebaseio.com/messages.json", data=json_data)
except urllib.error.URLError as e:
message = json.loads(e.read())
print(message["error"])
else:
print(loader.read())
This example will use Firebase internal timestamp. The value {".sv": "timestamp"} is interpreted by Firebase and it's converted into a timestamp when the request is processed.
def send_message(message, username):
my_data = dict()
my_data["message"] = message
my_data["username"] = username
my_data["timestamp"] = {".sv": "timestamp"}
json_data = json.dumps(my_data).encode()
try:
loader = urllib.request.urlopen("https://<YOUR-PROJECT-ID>.firebaseio.com/messages.json", data=json_data)
except urllib.error.URLError as e:
message = json.loads(e.read())
print(message["error"])
else:
print(loader.read())
This code solved the problem for me:
import firebase_admin
from firebase_admin import credentials
cred = credentials.Certificate("/home/pi/Documents/gpioZero/iot-maison-firebase- adminsdk-a0c7t-666fdeb9bd.json")
firebase_admin.initialize_app(cred)
db = firestore.client()
pir = MotionSensor(4)
led = LED(16)
def motionOn():
doc_ref = db.collection(u'motion').document(u'test1')
dateNow = datetime.datetime.now()
doc_ref.set({u'started': True,u'date': dateNow })
led.on()
print("On")
pir.when_activated=motionOn