I am looking to send sensor data through an ESP32 board using micropython to Google IoT Cloud.
I found an implementation example here: https://nihal-pasham.medium.com/you-could-try-freezing-the-pyasn-1-9afd6533cd7a
But it is throwing the following error message:
Traceback (most recent call last):
File "main.py", line 50, in <module>
File "main.py", line 44, in get_mqtt_client
File "umqtt/simple.py", line 99, in connect
MQTTException: 4
That's the board I am using:
https://www.amazon.com/gp/product/B0718T232Z/ref=ppx_yo_dt_b_asin_title_o00_s00?ie=UTF8&psc=1
That's the micropython version I am using:
esp32-idf3-20200902-v1.13.bin
and that's my code:
from third_party import rsa
import config
from third_party import string
from umqtt.simple import MQTTClient
from ubinascii import b2a_base64
import ujson
import utime
def on_message(topic, message):
print((topic,message))
def b42_urlsafe_encode(payload):
return string.translate(b2a_base64(payload)[:-1].decode('utf-8'),{ ord('+'):'-', ord('/'):'_' })
def create_jwt(project_id, private_key, algorithm, token_ttl):
print("Creating JWT...")
private_key = rsa.PrivateKey(*private_key)
# Epoch_offset is needed because micropython epoch is 2000-1-1 and unix is 1970-1-1. Adding 946684800 (30 years)
epoch_offset = 946684800
claims = {
# The time that the token was issued at
'iat': utime.time() + epoch_offset,
# The time the token expires.
'exp': utime.time() + epoch_offset + token_ttl,
# The audience field should always be set to the GCP project id.
'aud': project_id
}
#This only supports RS256 at this time.
header = { "alg": algorithm, "typ": "JWT" }
content = b42_urlsafe_encode(ujson.dumps(header).encode('utf-8'))
content = content + '.' + b42_urlsafe_encode(ujson.dumps(claims).encode('utf-8'))
signature = b42_urlsafe_encode(rsa.sign(content,private_key,'SHA-256'))
return content+ '.' + signature #signed JWT
def get_mqtt_client(project_id, cloud_region, registry_id, device_id, jwt):
"""Create our MQTT client. The client_id is a unique string that identifies
this device. For Google Cloud IoT Core, it must be in the format below."""
client_id = 'projects/{}/locations/{}/registries/{}/devices/{}'.format(project_id, cloud_region, registry_id, device_id)
print('Sending message with password {}'.format(jwt))
client = MQTTClient(client_id.encode('utf-8'),server=config.google_cloud_config['mqtt_bridge_hostname'],port=config.google_cloud_config['mqtt_bridge_port'],user=b'ignored',password=jwt.encode('utf-8'),ssl=True)
client.set_callback(on_message)
client.connect()
client.subscribe('/devices/{}/config'.format(device_id), 1)
client.subscribe('/devices/{}/commands/#'.format(device_id), 1)
return client
jwt = create_jwt(config.google_cloud_config['project_id'], config.jwt_config['private_key'], config.jwt_config['algorithm'], config.jwt_config['token_ttl'])
client = get_mqtt_client(config.google_cloud_config['project_id'], config.google_cloud_config['cloud_region'], config.google_cloud_config['registry_id'], config.google_cloud_config['device_id'], jwt)
I have checked my credentials to Google Cloud with other examples, and I am able to publish/receive messages with my computer, so it should be ok from that side.
As anyone encountered similar issues? Could you someone help on this please?
thx
Related
I'm trying to upgrade a legacy mail bot to authenticate via Oauth2 instead of Basic authentication, as it's now deprecated two days from now.
The document states applications can retain their original logic, while swapping out only the authentication bit
Application developers who have built apps that send, read, or
otherwise process email using these protocols will be able to keep the
same protocol, but need to implement secure, Modern authentication
experiences for their users. This functionality is built on top of
Microsoft Identity platform v2.0 and supports access to Microsoft 365
email accounts.
Note I've explicitly chosen the client credentials flow, because the documentation states
This type of grant is commonly used for server-to-server interactions
that must run in the background, without immediate interaction with a
user.
So I've got a python script that retrieves an Access Token using the MSAL python library. Now I'm trying to authenticate with the IMAP server, using that Access Token. There's some existing threads out there showing how to connect to Google, I imagine my case is pretty close to this one, except I'm connecting to a Office 365 IMAP server. Here's my script
import imaplib
import msal
import logging
app = msal.ConfidentialClientApplication(
'client-id',
authority='https://login.microsoftonline.com/tenant-id',
client_credential='secret-key'
)
result = app.acquire_token_for_client(scopes=['https://graph.microsoft.com/.default'])
def generate_auth_string(user, token):
return 'user=%s\1auth=Bearer %s\1\1' % (user, token)
# IMAP time!
mailserver = 'outlook.office365.com'
imapport = 993
M = imaplib.IMAP4_SSL(mailserver,imapport)
M.debug = 4
M.authenticate('XOAUTH2', lambda x: generate_auth_string('user#mydomain.com', result['access_token']))
print(result)
The IMAP authentication is failing and despite setting M.debug = 4, the output isn't very helpful
22:56.53 > b'DBDH1 AUTHENTICATE XOAUTH2'
22:56.53 < b'+ '
22:56.53 write literal size 2048
22:57.84 < b'DBDH1 NO AUTHENTICATE failed.'
22:57.84 NO response: b'AUTHENTICATE failed.'
Traceback (most recent call last):
File "/home/ubuntu/mini-oauth.py", line 21, in <module>
M.authenticate("XOAUTH2", lambda x: generate_auth_string('user#mydomain.com', result['access_token']))
File "/usr/lib/python3.10/imaplib.py", line 444, in authenticate
raise self.error(dat[-1].decode('utf-8', 'replace'))
imaplib.IMAP4.error: AUTHENTICATE failed.
Any idea where I might be going wrong, or how to get more robust information from the IMAP server about why the authentication is failing?
Things I've looked at
Note this answer no longer works as the suggested scopes fail to generate an Access Token.
The client credentials flow seems to mandate the https://graph.microsoft.com/.default grant. I'm not sure if that includes the scope required for the IMAP resource
https://outlook.office.com/IMAP.AccessAsUser.All?
Verified the code lifted from the Google thread produces the SASL XOAUTH2 string correctly, per example on the MS docs
import base64
user = 'test#contoso.onmicrosoft.com'
token = 'EwBAAl3BAAUFFpUAo7J3Ve0bjLBWZWCclRC3EoAA'
xoauth = "user=%s\1auth=Bearer %s\1\1" % (user, token)
xoauth = xoauth.encode('ascii')
xoauth = base64.b64encode(xoauth)
xoauth = xoauth.decode('ascii')
xsanity = 'dXNlcj10ZXN0QGNvbnRvc28ub25taWNyb3NvZnQuY29tAWF1dGg9QmVhcmVyIEV3QkFBbDNCQUFVRkZwVUFvN0ozVmUwYmpMQldaV0NjbFJDM0VvQUEBAQ=='
print(xoauth == xsanity) # prints True
This thread seems to suggest multiple tokens need to be fetched, one for graph, then another for the IMAP connection; could that be what I'm missing?
Try the below steps.
For Client Credentials Flow you need to assign “Application permissions” in the app registration, instead of “Delegated permissions”.
Add permission “Office 365 Exchange Online / IMAP.AccessAsApp” (application).
Grant admin consent to you application
Service Principals and Exchange.
Once a service principal is registered with Exchange Online, administrators can run the Add-Mailbox Permission cmdlet to assign receive permissions to the service principal.
Use scope 'https://outlook.office365.com/.default'.
Now you can generate the SALS authentication string by combining this access token and the mailbox username to authenticate with IMAP4.
#Python code
def get_access_token():
tenantID = 'abc'
authority = 'https://login.microsoftonline.com/' + tenantID
clientID = 'abc'
clientSecret = 'abc'
scope = ['https://outlook.office365.com/.default']
app = ConfidentialClientApplication(clientID,
authority=authority,
client_credential = clientSecret)
access_token = app.acquire_token_for_client(scopes=scope)
return access_token
def generate_auth_string(user, token):
auth_string = f"user={user}\1auth=Bearer {token}\1\1"
return auth_string
#IMAP AUTHENTICATE
imap = imaplib.IMAP4_SSL(imap_host, 993)
imap.debug = 4
access_token = get_access_token_to_authenticate_imap()
imap.authenticate("XOAUTH2", lambda x:generate_auth_string(
'useremail',
access_token['access_token']))
imap.select('inbox')
The imaplib.IMAP4.error: AUTHENTICATE failed Error occured because one point in the documentation is not that clear.
When setting up the the Service Principal via Powershell you need to enter the App-ID and an Object-ID. Many people will think, it is the Object-ID you see on the overview page of the registered App, but its not!
At this point you need the Object-ID from "Azure Active Directory -> Enterprise Applications --> Your-App --> Object-ID"
New-ServicePrincipal -AppId <APPLICATION_ID> -ServiceId <OBJECT_ID> [-Organization <ORGANIZATION_ID>]
Microsoft says:
The OBJECT_ID is the Object ID from the Overview page of the
Enterprise Application node (Azure Portal) for the application
registration. It is not the Object ID from the Overview of the App
Registrations node. Using the incorrect Object ID will cause an
authentication failure.
Ofcourse you need to take care for the API-permissions and the other stuff, but this was for me the point.
So lets go trough it again, like it is explained on the documentation page.
Authenticate an IMAP, POP or SMTP connection using OAuth
Register the Application in your Tenant
Setup a Client-Key for the application
Setup the API permissions, select the APIs my organization uses tab and search for "Office 365 Exchange Online" -> Application permissions -> Choose IMAP and IMAP.AccessAsApp
Setup the Service Principal and full access for your Application on the mailbox
Check if IMAP is activated for the mailbox
Thats the code I use to test it:
import imaplib
import msal
import pprint
conf = {
"authority": "https://login.microsoftonline.com/XXXXyourtenantIDXXXXX",
"client_id": "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXXX", #AppID
"scope": ['https://outlook.office365.com/.default'],
"secret": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX", #Key-Value
"secret-id": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX", #Key-ID
}
def generate_auth_string(user, token):
return f"user={user}\x01auth=Bearer {token}\x01\x01"
if __name__ == "__main__":
app = msal.ConfidentialClientApplication(conf['client_id'], authority=conf['authority'],
client_credential=conf['secret'])
result = app.acquire_token_silent(conf['scope'], account=None)
if not result:
print("No suitable token in cache. Get new one.")
result = app.acquire_token_for_client(scopes=conf['scope'])
if "access_token" in result:
print(result['token_type'])
pprint.pprint(result)
else:
print(result.get("error"))
print(result.get("error_description"))
print(result.get("correlation_id"))
imap = imaplib.IMAP4('outlook.office365.com')
imap.starttls()
imap.authenticate("XOAUTH2", lambda x: generate_auth_string("target_mailbox#example.com", result['access_token']).encode("utf-8"))
After setting up the Service Principal and giving the App full access on the mailbox, wait 15 - 30 minutes for the changes to take effect and test it.
Try with this script:
import json
import msal
import requests
client_id = '***'
client_secret = '***'
tenant_id = '***'
authority = f"https://login.microsoftonline.com/{tenant_id}"
app = msal.ConfidentialClientApplication(
client_id=client_id,
client_credential=client_secret,
authority=authority)
scopes = ["https://graph.microsoft.com/.default"]
result = None
result = app.acquire_token_silent(scopes, account=None)
if not result:
print(
"No suitable token exists in cache. Let's get a new one from Azure Active Directory.")
result = app.acquire_token_for_client(scopes=scopes)
# if "access_token" in result:
# print("Access token is " + result["access_token"])
if "access_token" in result:
userId = "***"
endpoint = f'https://graph.microsoft.com/v1.0/users/{userId}/sendMail'
toUserEmail = "***"
email_msg = {'Message': {'Subject': "Test Sending Email from Python",
'Body': {'ContentType': 'Text', 'Content': "This is a test email."},
'ToRecipients': [{'EmailAddress': {'Address': toUserEmail}}]
},
'SaveToSentItems': 'true'}
r = requests.post(endpoint,
headers={'Authorization': 'Bearer ' + result['access_token']}, json=email_msg)
if r.ok:
print('Sent email successfully')
else:
print(r.json())
else:
print(result.get("error"))
print(result.get("error_description"))
print(result.get("correlation_id"))
Source: https://kontext.tech/article/795/python-send-email-via-microsoft-graph-api
Initially, I generate my own X509 Certificate that is CA-signed by following this tutorial (Powershell variant) - https://learn.microsoft.com/en-us/azure/iot-hub/tutorial-x509-scripts
Then, I made the following two scenarios:
Communication from my own laptop (Windows 10) to Azure IoT Hub Device using a .NET Framework app. Here is my simple code:
static void Main(string[] args)
{
try
{
// Create an X.509 certificate object.
var cert = new X509Certificate2(#"..\test-device-auth\test-device-auth.pfx", "pass", X509KeyStorageFlags.UserKeySet);
Console.WriteLine("cert: ");
Console.WriteLine(cert);
// Create an authentication object using your X.509 certificate.
var auth = new DeviceAuthenticationWithX509Certificate(deviceId, cert);
// Create the device client.
var deviceClient = DeviceClient.Create("Arduino-IoT-Hub-Temperature.azure-devices.net", auth, TransportType.Mqtt);
if (deviceClient == null)
{
Console.WriteLine("Failed to create DeviceClient!");
}
else
{
Console.WriteLine("Successfully created DeviceClient!");
SendEvent(deviceClient).Wait();
}
Console.WriteLine("Exiting...\n");
}
catch (Exception ex)
{
Console.WriteLine("Error in sample: {0}", ex.Message);
}
}
In this case, the program works fine when passing the correct pfx and the correct pass phrase. Additionally, when I pass incorrect pass phrase or incorrect pfx, it fails - this is perfectly fine.
Communication directly from my Raspberry Pi 3B to the Azure IoT Hub Device using a python script. Here is the code:
# -------------------------------------------------------------------------
# Copyright (c) Microsoft Corporation. All rights reserved.
# Licensed under the MIT License. See License.txt in the project root for
# license information.
# --------------------------------------------------------------------------
import os
import uuid
from azure.iot.device.aio import IoTHubDeviceClient
from azure.iot.device import Message, X509
import asyncio
messages_to_send = 10
async def main():
hostname = "Arduino-IoT-Hub-Temperature.azure-devices.net"
# The device that has been created on the portal using X509 CA signing or Self signing capabilities
device_id = "test-device-auth"
x509 = X509(
cert_file="../test-device-auth/test-device-auth-public.pem",
key_file="../test-device-auth/test-device-auth-private.pem",
pass_phrase="pass",
)
# The client object is used to interact with your Azure IoT hub.
device_client = IoTHubDeviceClient.create_from_x509_certificate(
hostname=hostname, device_id=device_id, x509=x509
)
# Connect the client.
await device_client.connect()
async def send_test_message(i):
print("sending message #" + str(i))
msg = Message("test wind speed " + str(i))
msg.message_id = uuid.uuid4()
msg.correlation_id = "correlation-1234"
# msg.custom_properties["tornado-warning"] = "yes"
msg.content_encoding = "utf-8"
msg.content_type = "application/json"
await device_client.send_message(msg)
print("done sending message #" + str(i))
# send `messages_to_send` messages in parallel
await asyncio.gather(*[send_test_message(i) for i in range(1, messages_to_send + 1)])
# Finally, shut down the client
await device_client.shutdown()
if __name__ == "__main__":
asyncio.run(main())
# If using Python 3.6 use the following code instead of asyncio.run(main()):
# loop = asyncio.get_event_loop()
# loop.run_until_complete(main())
# loop.close()
In this case, the .pem files are not secured with the pass_phrase and it does not matter if I will set correct, incorrect or no pass_phrase at all.
Does anyone know why it is like this and how it can be still secured with the pass_phrase?
When test-device-auth-private.pem was created it wasn't created as an encrypted key blob, so no passphrase is needed. You can encrypt it via something like openssl pkcs8 -in test-device-auth-private.pem -out test-device-auth-private-enc.pem -topk8 and give a password at the prompt.
I'm attempting to write a GCP Cloud Function in Python that calls the API for creating an IoT device. The initial challenge seems to be getting the appropriate module (specifically iot_v1) loaded within Cloud Functions so that it can make the call.
Example Python code from Google is located at https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/iot/api-client/manager/manager.py. The specific call desired is shown in "create_es_device". Trying to repurpose that into a Cloud Function (code below) errors out with "ImportError: cannot import name 'iot_v1' from 'google.cloud' (unknown location)"
Any thoughts?
import base64
import logging
import json
import datetime
from google.auth import compute_engine
from apiclient import discovery
from google.cloud import iot_v1
def handle_notification(event, context):
#Triggered from a message on a Cloud Pub/Sub topic.
#Args:
# event (dict): Event payload.
# context (google.cloud.functions.Context): Metadata for the event.
#
pubsub_message = base64.b64decode(event['data']).decode('utf-8')
logging.info('New device registration info: {}'.format(pubsub_message))
certData = json.loads(pubsub_message)['certs']
deviceID = certData['device-id']
certKey = certData['certificate']
projectID = certData['project-id']
cloudRegion = certData['cloud-region']
registryID = certData['registry-id']
newDevice = create_device(projectID, cloudRegion, registryID, deviceID, certKey)
logging.info('New device: {}'.format(newDevice))
def create_device(project_id, cloud_region, registry_id, device_id, public_key):
# from https://cloud.google.com/iot/docs/how-tos/devices#api_1
client = iot_v1.DeviceManagerClient()
parent = client.registry_path(project_id, cloud_region, registry_id)
# Note: You can have multiple credentials associated with a device.
device_template = {
#'id': device_id,
'id' : 'testing_device',
'credentials': [{
'public_key': {
'format': 'ES256_PEM',
'key': public_key
}
}]
}
return client.create_device(parent, device_template)
You need to have the google-cloud-iot project listed in your requirements.txt file.
See https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/iot/api-client/manager/requirements.txt
We are busy upgrading from the Marketing API to the Business API (not graph API and not marketing API) for Facebook. The documentation states that one should be able to create a test.py with the following code:
import sys
sys.path.append('/opt/homebrew/lib/python2.7/site-packages')
sys.path.append('/opt/homebrew/lib/python2.7/site-packages/facebook_business-3.0.0-py2.7.egg-info')
from facebook_business.api import FacebookAdsApi
from facebook_business.adobjects.adaccount import AdAccount
my_app_id = 'your-app-id'
my_app_secret = 'your-appsecret'
my_access_token = 'your-access-token'
FacebookAdsApi.init(my_app_id, my_app_secret, my_access_token)
my_account = AdAccount('your-adaccount-id')
campaigns = my_account.get_campaigns()
print(campaigns)
This requires an app secret proof which I am obtaining with the following hash:
facebook_app_id = 'xxxxx'
facebook_app_secret = 'xxxxx'
facebook_app_token = 'xxxxx|xxxxx'.format(facebook_app_id,facebook_app_secret)
import hmac,hashlib
app_secret_proof = hmac.new(facebook_app_secret.encode('utf-8'),
msg=facebook_app_token.encode('utf-8'),
digestmod=hashlib.sha256).hexdigest()
print(app_secret_proof)
The is the error I get:
Traceback (most recent call last):
File "test.py", line 12, in <module>
FacebookAdsApi.init(my_app_id,my_app_secret,my_access_token,my_appsecret_proof)
File "/Users/facebook_business/api.py", line 202, in init
cls.set_default_account_id(account_id)
File "/Users/facebook_business/api.py", line 225, in set_default_account_id
"Account ID provided in FacebookAdsApi.set_default_account_id "
ValueError: Account ID provided in FacebookAdsApi.set_default_account_id expects a string that begins with 'act_'
None of the ID's start with act_, I am currently using the App ID that is at the top left of the app dashboard which one should I be using?
Marketing API
The Ad account number in business manager is obtained on the front log in screen.
From Business Manager
I've been trying to set up a connection to the FIX API for the GDAX crpyto exchange, but I can't seem to logon properly. I'm using the code below to generate the message:
import time
import simplefix
import socket
import base64
import hmac
import hashlib
from datetime import datetime
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(("127.0.0.1", 4197))
API_KEY = "KEY_GOES_HERE"
PASSPHRASE = "PASSPHRASE_GOES_HERE"
API_SECRET = "SECRET_GOES_HERE"
seq_num = "1"
sendingTime = str(datetime.utcnow()).replace("-","").replace(" ", "-")[:-3]
rawSig = "\x01".join([sendingTime, "A", seq_num, API_KEY, "Coinbase", PASSPHRASE]).encode("utf-8")
hmac_key = base64.b64decode(API_SECRET)
signature = hmac.new(hmac_key, rawSig, hashlib.sha256)
sign_b64 = base64.b64encode(signature.digest()).decode()
msg = simplefix.FixMessage()
msg.append_pair(8, "FIX.4.2")
msg.append_pair(35, "A")
msg.append_pair(49, API_KEY)
msg.append_pair(52, sendingTime)
msg.append_pair(56, "Coinbase")
msg.append_pair(98, "0")
msg.append_pair(108, "30")
msg.append_pair(554, PASSPHRASE)
msg.append_pair(96, sign_b64)
msg.append_pair(8013, "Y")
print(msg.encode())
s.sendall(msg.encode("ascii"))
print(s.recv(4096))
And I'm getting 0 bytes of response from the server. As far as I can tell, the stunnel is working properly (connects and validates certificates successfully, but disconnects after sending my logon message).
Have just tried with a newly generated API key, passphrase and secret but to no avail.
For reference, I was working from the question asked here: How to send FIX logon message with Python to GDAX but I'm not allowed to comment there.
If anyone has any ideas, would be appreciated. Below is an example of the fix message generated, passwords removed in post:
8=FIX.4.2\x019=161\x0135=A\x0149=[KEY_REMOVED]\x0152=20180113-18:24:07.889\x0156=Coinbase\x0198=0\x01108=30\x01554=[PASSPHRASE_REMOVED]\x0196=jueSJHoSNQM2BOCN3KM0mgB2/9tXpICbg4amqDKc2wY=\x018013=Y\x0110=053\x01