Develop Reference

Google API Multi-Processing

I'm trying to grab specific information from emails under my Gmail account (Subject, From, Date, Message Body) and was able to do so succesfully using the Google API and relevant libraries, however, I've noticed the more emails you have the longer it takes to parse, so much so that parsing 34 emails takes nearly 15 seconds, which is bad if you tried to scale that to parsing 1000 emails. My aim was to utilise concurrency/multi-processing on the parse_messages() function, however, I've had no luck and keep returning an empty list. The aim is to process all the emails, then append them all to a the combined list.
Apologies for the sloppyness, it's yet to be cleaned up, there's less than 100 lines in total.
from __future__ import print_function
import os.path
from google.auth.transport.requests import Request
from google.oauth2.credentials import Credentials
from google_auth_oauthlib.flow import InstalledAppFlow
from googleapiclient.discovery import build
from concurrent.futures import ProcessPoolExecutor
import base64
import re
combined = []
def authenticate():
# If modifying these scopes, delete the file token.json.
SCOPES = ['https://www.googleapis.com/auth/gmail.readonly']
creds = None
if os.path.exists('token.json'):
creds = Credentials.from_authorized_user_file('token.json', SCOPES)
if not creds or not creds.valid:
if creds and creds.expired and creds.refresh_token:
creds.refresh(Request())
else:
flow = InstalledAppFlow.from_client_secrets_file(
'creds.json', SCOPES)
creds = flow.run_local_server(port=0)
with open('token.json', 'w') as token:
token.write(creds.to_json())
return creds
def get_messages(creds):
# Get the messages
days = 31
service = build('gmail', 'v1', credentials=creds)
results = service.users().messages().list(userId='me', q=f'newer_than:{days}d, in:inbox').execute()
messages = results.get('messages', [])
message_count = len(messages)
print(f"You've received {message_count} email(s) in the last {days} days")
if not messages:
print(f'No Emails found in the last {days} days.')
return messages
def parse_message(msg):
# Call the Gmail API
service = build('gmail', 'v1', credentials=creds)
txt = service.users().messages().get(userId='me', id=msg['id']).execute()
payload = txt['payload']
headers = payload['headers']
#Grab the Subject Line, From and Date from the Email
for d in headers:
if d['name'] == 'Subject':
subject = d['value']
if d['name'] == 'From':
sender = d['value']
try:
match = re.search(r'<(.*)>', sender).group(1)
except:
match = sender
if d['name'] == "Date":
date_received = d['value']
def get_body(payload):
if 'body' in payload and 'data' in payload['body']:
return payload['body']['data']
elif 'parts' in payload:
for part in payload['parts']:
data = get_body(part)
if data:
return data
else:
return None
data = get_body(payload)
data = data.replace("-","+").replace("_","/")
decoded_data = base64.b64decode(data).decode("UTF-8")
decoded_data = (decoded_data.encode('ascii', 'ignore')).decode("UTF-8")
decoded_data = decoded_data.replace('\n','').replace('\r','').replace('\t', '')
# Append parsed message to shared list
return combined.append([date_received, subject, match, decoded_data])
if __name__ == '__main__':
creds = authenticate()
messages = get_messages(creds)
# Create a process pool with 4 worker processes
with ProcessPoolExecutor(max_workers=4) as executor:
# Submit the parse_message function for each message in the messages variable
executor.map(parse_message, messages)
print(f"Combined: {combined}")
When running the script, my output is normally.
You've received 34 email(s) in the last 31 days
combined: []

Thanks to the help of simpleApp, I made their changes along with a few others to get this working.
# Append parsed message to shared list
return [date_received, subject, match, decoded_data]
if __name__ == '__main__':
creds = authenticate()
messages, service = get_messages(creds)
# Create a process pool with default worker processes
with ProcessPoolExecutor() as executor:
combined = []
# Submit the parse_message function for each message in the messages variable
all_pools = executor.map(parse_message, messages, [service]*len(messages))
for e_p in all_pools:
combined.append(e_p)

Azure communication services send email

I am using azure communication services in my react app to send email.
But I want to send bulk messages via a text file or an excel file
import time
from azure.communication.email import EmailClient, EmailContent, EmailAddress, EmailMessage, EmailRecipients
def main():
try:
connection_string = "<ACS_CONNECTION_STRING>"
client = EmailClient.from_connection_string(connection_string)
sender = "<SENDER_EMAIL>"
content = EmailContent(
subject="Test email from Python",
plain_text="This is plaintext body of test email.",
html= "<html><h1>This is the html body of test email.</h1></html>",
)
recipient = EmailAddress(email="<RECIPIENT_EMAIL>", display_name="<RECIPIENT_DISPLAY_NAME>")
message = EmailMessage(
sender=sender,
content=content,
recipients=EmailRecipients(to=[recipient])
)
response = client.send(message)
if (not response or response.message_id=='undefined' or response.message_id==''):
print("Message Id not found.")
else:
print("Send email succeeded for message_id :"+ response.message_id)
message_id = response.message_id
counter = 0
while True:
counter+=1
send_status = client.get_send_status(message_id)
if (send_status):
print(f"Email status for message_id {message_id} is {send_status.status}.")
if (send_status.status.lower() == "queued" and counter < 12):
time.sleep(10) # wait for 10 seconds before checking next time.
counter +=1
else:
if(send_status.status.lower() == "outfordelivery"):
print(f"Email delivered for message_id {message_id}.")
break
else:
print("Looks like we timed out for checking email send status.")
break
except Exception as ex:
print(ex)
main()
How to solve this issue?
I tried to get emails from a text file, but it failed for me

GitHub cannot authorize with oauth

I'm trying to make a GitHub bot that posts issues when a complementary app throws errors, but I'm having trouble trying to get an access token. This is my code so far:
import asyncio
import random
import webbrowser
import aiohttp.web
randgen = random.SystemRandom()
# The task is relatively simple. Query for the client ID, client secret, and then produces a URL.
# It opens this URL in a web-browser, then sets up a server on port 12345, waiting for an appropriate response.
# Using this code, it finishes the rest of the flow.
class Authorizer:
def __init__(self):
self.client_id = self.client_secret = ""
self.redirect_url = "http://localhost:12345/callback"
self.scopes = ["public_repo"]
self.state = 0
self.done = False
async def handler(self, request: aiohttp.web.Request):
code = request.query["code"]
async with aiohttp.ClientSession() as client:
async with client.post("https://github.com/login/oauth/access_token",
data=dict(client_id=self.client_id, client_secret=self.client_secret, code=code, state=self.state),
headers=dict(Accept="application/json")) as request:
json = await request.json()
assert str(json["state"]) == self.state
resp = "Access Token: " + json["access_token"]
print(resp)
self.done = True
return aiohttp.web.Response(text=resp)
async def start(self):
self.client_id = input("App Client ID: ")
self.client_secret = input("App Client Secret: ")
self.state = randgen.randint(1, 1000)
scope_str = ' '.join(self.scopes)
url = f"https://github.com/login/oauth/authorize?client_id={self.client_secret}&redirect_uri={self.redirect_url}&scope=" \
f"{scope_str}&state={self.state}"
print("Opening URL: " + url)
webbrowser.open(url)
self.server = aiohttp.web.Server(self.handler)
self.runner = aiohttp.web.ServerRunner(self.server)
await self.runner.setup()
self.site = aiohttp.web.TCPSite(self.runner, 'localhost', 12345)
await self.site.start()
while not self.done:
await asyncio.sleep(1)
await self.site.stop()
input("Complete! Make sure you take your access token with you! Now hit enter to exit.")
if __name__ == '__main__':
auth = Authorizer()
loop = asyncio.get_event_loop()
loop.run_until_complete(auth.start())
I get an error, where the constructed URL gives me a 404 error. Getting rid of the state, redirect URL, and scopes don't change the error. I looked at the official docs when making this, so I don't understand why the constructed error is giving me a 404.

i want to send an auto reply to a new chat with whom i havent had a conversation earlier on Telegram

I'm new to Python and Telethon.
I've gone through the documentation of Telethon module.
I would like to send auto reply to a new person if they send me a message on Telegram private message.
I've found one (in this gist)
which sends an autoreply to every incoming private chat. but i want to send a reply to new people only.
please guide
i want to check if user is present in the df below and send a reply accordingly.
code to get list of dialog id's is as below:
import time
from telethon import TelegramClient
from telethon import events
import pandas as pd
api_id = 123456
api_hash = 'enterownapihash'
# fill in your own details here
phone = '+111111111'
session_file = 'username' # use your username if unsure
password = 'passwords' # if you have two-step verification enabled
client = TelegramClient(session_file, api_id, api_hash, sequential_updates=True)
xyz = pd.DataFrame()
z = pd.DataFrame()
client.connect()
client.start()
for dialog in client.iter_dialogs():
x = dialog.id
y = dialog.title
#print('{:>14}: {}'.format(dialog.id, dialog.title))
xyz['dialog id'] = [x]
xyz['username'] = [y]
z = pd.concat([z,xyz],ignore_index= True)
print(xyz)
print(z)
EDIT:
below is the code i've tried which hasnt worked.
import time
from telethon import TelegramClient
from telethon import events
import pandas as pd
api_id = 123456
api_hash = 'enterownapihash'
# fill in your own details here
phone = '+111111111'
session_file = 'username' # use your username if unsure
password = 'passwords' # if you have two-step verification enabled
# content of the automatic reply
message = "Hello!"
client = TelegramClient(session_file, api_id, api_hash, sequential_updates=True)
xyz = pd.DataFrame()
z = pd.DataFrame()
#client.connect()
#client.start()
for dialog in client.iter_dialogs():
x = dialog.id
y = dialog.title
print('{:>14}: {}'.format(dialog.id, dialog.title))
xyz['x'] = [x]
xyz['y'] = [y]
z = pd.concat([z,xyz],ignore_index= True)
if __name__ == '__main__':
#client.on(events.NewMessage(incoming=True))
async def handle_new_message(event):
if event.is_private: # only auto-reply to private chats
from_ = await event.client.get_entity(event.from_id) # this lookup will be cached by telethon
if event.client.get_entity != z['x'].values:
if not from_.bot: # don't auto-reply to bots
print(time.asctime(), '-', event.message) # optionally log time and message
time.sleep(1) # pause for 1 second to rate-limit automatic replies
await event.respond(message)
client.start(phone, password)
client.run_until_disconnected()

One way to do this is to fetch all the IDs present in your dialogs on start up. When a new message arrives, if their ID is not there, you know it's a new person (the code assumes you run it inside an async def):
async def setup():
users = set()
async for dialog in client.iter_dialogs():
if dialog.is_user:
users.add(dialog.id)
# later, on a handler
async def handler(event):
if event.is_private and event.sender_id not in users:
# new user
Another option is to use messages.GetPeerSettingsRequest, which will have result.report_spam as True when it's a new chat. Ideally, you would also cache this:
user_is_new = {}
async def handler(event):
if event.is_private:
if event.sender_id not in user_is_new:
res = client(functions.messages.GetPeerSettingsRequest(event.sender_id))
# if you can report them for spam it's a new user
user_is_new[event.sender_id] = res.report_spam
if user_is_new[event.sender_id]:
# new user

Trying to create a Gmail API compatible message just does nothing and gives no errors

Gmail decided that SMTP was too simple so they had to block it and setup their own API with all the weird requirements around it. This script which I was trying to use has now become outdated and broken. In an attempt to use it anyway I tried to rewrite it:
"""
Checks stock on specified items at Microcenter store locations,
and sends email notifications when changes are detected.
Applicably, it helps the user obtain rare items during shortages.
"""
from aiohttp import ClientSession
from async_timeout import timeout
from getpass import getpass
from re import search
from smtplib import SMTP
import asyncio
import base64
from email.mime.audio import MIMEAudio
from email.mime.base import MIMEBase
from email.mime.image import MIMEImage
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText
import mimetypes
import os
import pickle
import os.path
from googleapiclient.discovery import build
from google_auth_oauthlib.flow import InstalledAppFlow
from google.auth.transport.requests import Request
class Item:
"""
Class for containing state of individual items; methods update state
by awaiting update().
Item does not need to be directly instantiated; Store will create one
per provided url.
"""
def __init__(self, storeNum, url):
self.storeNum, self.url = storeNum, url
self.sku = self.price = self.stock = None
self.stockChanged = self.priceChanged = False
self.loop = asyncio.get_event_loop()
def __str__(self):
stock = 'in' if self.stock else 'out of'
return f'SKU {self.sku} is {stock} stock for {self.price} at Microcenter {self.storeNum}\n{self.url}\n'
async def pull(self):
async with ClientSession() as session:
async with timeout(10):
async with session.get(self.url, params={'storeSelected': self.storeNum}) as response:
return await response.text()
#staticmethod
def parse_lines(page):
for var in ['SKU', 'inStock', 'productPrice']:
reply = search(f"(?<='{var}':').*?(?=',)", page)
if reply:
yield reply.group()
#staticmethod
def compare(new, old):
return (new != old and old is not None)
async def update(self):
data = tuple(self.parse_lines(await self.pull()))
if not data or any(data) is None:
raise ValueError('Data missing from request or store number invalid')
self.sku, stock, price = int(data[0]), data[1] is 'True', float(data[2])
self.stockChanged, self.priceChanged = self.compare(stock, self.stock), self.compare(price, self.price)
self.stock, self.price = stock, price
class Store:
"""
Periodically checks a given list of urls for stock changes
A store number is required to get accurate stock numbers.
The default store number is set to the North Dallas/Richardson, TX location.
Also required is valid email account information for notifications.
If a recipient address is not provided, the user will be prompted for one.
If the prompt is empty, notifications are sent from the sender
address to itself. Providing an empty string for recipient is a valid
argument to enable loopback operation, as only a value of None
will trigger a prompt.
The default time between checks is 15 minutes. This value should
be at least a few minutes, to avoid being blacklisted by the
server, though this class enforces no such limit. To change the
time period, provide a value in minutes to self.run(minutes).
Setting debug to True enables false positives for testing
"""
def __init__(
self, storeNum=131, sender=None,
recipient=None, debug=True, service=None
):
self.storeNum = storeNum
self.items, self.newInStock, self.totalInStock = set(), 0, 0
self.debug = debug
if not sender:
self.sender = input('Enter sender email address: ').lstrip().rstrip()
else:
self.sender = sender
if recipient is None:
prompted = input('Enter recipient email address (leave blank for loopback): ').lstrip().rstrip()
if not prompted:
self.recipient = self.sender
else:
self.recipient = prompted
else:
self.recipient = self.sender
#Google API BULLSHIT
SCOPES = ['https://www.googleapis.com/auth/gmail.compose','https://www.googleapis.com/auth/gmail.readonly']
creds = None
# The file token.pickle stores the user's access and refresh tokens, and is
# created automatically when the authorization flow completes for the first
# time.
if os.path.exists('token.pickle'):
with open('token.pickle', 'rb') as token:
creds = pickle.load(token)
# If there are no (valid) credentials available, let the user log in.
if not creds or not creds.valid:
if creds and creds.expired and creds.refresh_token:
creds.refresh(Request())
else:
flow = InstalledAppFlow.from_client_secrets_file(
'credentials.json', SCOPES)
creds = flow.run_local_server(port=0)
# Save the credentials for the next run
with open('token.pickle', 'wb') as token:
pickle.dump(creds, token)
self.service = build('gmail', 'v1', credentials=creds)
# Call the Gmail API
results = self.service.users().labels().list(userId='me').execute()
labels = results.get('labels', [])
if not labels:
print('No labels found.')
else:
print('Labels:')
for label in labels:
print((label['name']))
self.loop = asyncio.get_event_loop()
def __str__(self):
return '\n'.join(item.__str__() for item in self.items)
def __enter__(self):
return self
def __exit__(self, exc_type, exc_val, exc_tb):
self.loop.close()
#property
def storeNum(self):
return self._storeNum
#storeNum.setter
def storeNum(self, val):
"""
Check to see if value is formatted properly
storeNum must be sent as a string, but should contain an integer.
"""
assert isinstance(val, (int, str)), 'Store number must be an integer or string of integer'
try:
num = int(val)
except:
raise
else:
self._storeNum = str(num)
#property
def sender(self):
return self._sender
#sender.setter
def sender(self, val):
assert val is not None, 'Sender address cannot be empty'
assert isinstance(val, str), 'Must be str'
self._sender = val
def run(self, minutes=5):
run = asyncio.ensure_future(self.check(minutes))
self.loop.run_forever()
async def check(self, minutes=5):
assert isinstance(minutes, (int, float)), 'Minutes must be an integer or float'
seconds = minutes * 60
while True:
print('Checking stock...')
await self.update()
if self.newInStock:
print('New items available')
msg = email_message()
print("message created")
self.send_email(msg)
print("email send attempted")
#if sent:
#print('Recipient notified of stock changes')
else:
print('Stock unchanged')
await asyncio.sleep(seconds)
def add_interactive(self):
entry = True
while entry:
entry = eval(input('Add one or more URLs separated by spaces, or leave blank to complete: '))
try:
urls = entry.split()
except:
if entry and 'http' in entry:
self.add(entry.lstrip().rstrip())
else:
self.add(*urls)
def add(self, *urls):
for url in urls:
assert isinstance(url, str), 'URL must be a string'
if url not in (item.url for item in self.items):
new = Item(self.storeNum, url)
self.loop.run_until_complete(new.update())
self.items.add(new)
def remove(self, *urls):
for url in urls:
assert isinstance(url, str), 'URL must be a string'
self.items = set([item for item in self.items if item.url not in urls])
def email_message(self):
if self.debug:
new = self.items
else:
new = tuple([item for item in self.items if item.stockChanged])
message_text = '\n'.join(item.__str__() for item in new)
print(message_text)
#Create message container
message = MIMEMultipart('alternative') # needed for both plain & HTML (the MIME type is multipart/alternative)
message['Subject'] = self.email_subject()
print("set Subject")
message['From'] = self.sender
print("set sender")
message['To'] = self.recipient
print("set recipient")
#Create the body of the message (a plain-text and an HTML version)
message.attach(MIMEText(message_text, 'plain'))
print("attached plaintext")
message.attach(MIMEText(message_text, 'html'))
print("attached html")
raw_message_no_attachment = base64.urlsafe_b64encode(message.as_bytes())
print("encoded b64")
raw_message_no_attachment = raw_message_no_attachment.decode()
print("decoded raw")
body = {'raw': raw_message_no_attachment}
print("set body")
return body
def email_subject(self):
return f'({self.newInStock} new, {self.totalInStock} total) items in stock at Microcenter {self.storeNum}'
def send_email(self, msgOBJ):
message = msgOBJ
print("message encoded")
try:
message_sent = (self.service.users().messages().send(userId='me', body=message).execute())
message_id = message_sent['id']
# print(attached_file)
print (f'Message sent (without attachment) \n\n Message Id: {message_id}\n\n Message:\n\n {message_text_plain}')
# return body
return True
except errors.HttpError as error:
print (f'An error occurred: {error}')
return False
async def update(self):
for item in self.items:
await item.update()
if self.debug:
self.newInStock = self.totalInStock = len(self.items)
else:
self.newInStock = sum(item.stockChanged for item in self.items)
self.totalInStock = sum(item.stock for item in self.items)
class Clerk(Store):
"""
Further abstraction and automation of Store
Instantiate Clerk with a list of urls as arguments
and an optional store number as a keyword argument.
Clerk exists to be able to start and run a Store in one line.
The user will be prompted for email account information.
"""
def __init__(self, *urls, storeNum=131):
super().__init__(storeNum=storeNum)
if urls:
super().add(*urls)
else:
super().add_interactive()
super().run()
Clerk("https://www.microcenter.com/product/616858/amd-ryzen-9-3950x-35ghz-16-core-am4-boxed-processor", storeNum=155)
I wrote this in a way that is Python 3.6 compatible and Gmail API friendly so it'll actually work. However, upon calling the Store.email_message method (which is supposed to create and return the necessary b64 encoded message object) nothing happens, not one of the prints spaced throughout it is called and no error is returned either. It just stops there.
I initially tried the code from the examples in the Gmail API Documentation, but that didn't work, so then i went searching through the web until I decided to stop with the code I got here (code stolen from their send_Message_without_attachment and create_message_without_attachment functions) and ask for help.
Edit
I followed the advice of the answer I got and changed the email_message function to
def email_message(self):
if self.debug:
new = self.items
else:
new = tuple([item for item in self.items if item.stockChanged])
message_text = '\n'.join(item.__str__() for item in new)
print(message_text)
#Create message container
message = MIMEMultipart('alternative') # needed for both plain & HTML (the MIME type is multipart/alternative)
message['Subject'] = self.email_subject()
message['From'] = self.sender
message['To'] = self.recipient
#Create the body of the message (a plain-text and an HTML version)
message.attach(MIMEText(message_text, 'plain'))
message.attach(MIMEText(message_text, 'html'))
raw_message_no_attachment = urlsafe_b64encode(bytes(message))
raw_message_no_attachment = raw_message_no_attachment.decode()
body = {'raw': raw_message_no_attachment}
return body
That said it still gives no error and doesn't even get to print the message text when it gets to the point where it's called, so I'm still pretty lost.

For the encoding you have to change your import and use like this:
Import:
from base64 import urlsafe_b64encode
Use:
encode = urlsafe_b64encode(bytes(message))
For the scopes using this one is more than enough:
SCOPES = ['https://mail.google.com/']
Remember to delete and renew the token.pickle every time you change the scopes.
Be sure that the API credentials are Ok.