Develop Reference

SendGrid Random Duplicate Emails (Python Flask Heroku)

I'm currently working on an app that updates artists with times they are booked for events. Randomly the app will send duplicate emails a few times a day, over 90% of the time there are not duplicates.
Currently, there are 10+ emails that can be produced, but there is only one email send function. The duplicates occur across all emails which makes me think there is an issue with the email send function or the configuration of the web server to make multiple requests to sendgrid. PLEASE HELP ME FIND THE CAUSE OF THE DUPILCATES!
Stack:
Python (v3.10.6)
Flask (v2.2.2)
Heroku (buildstack-22)
Sendgrid python library (v6.9.7)
Email Send Function:
def send_email(to, subject, template, cc='None', attachment_location='None', attachment_name='None', private_email=False, **kwargs):
## Remove Guest DJ Emails Here
guest_dj_list = get_list_of_guest_djs()
if to in guest_dj_list:
return None
message = Mail(
from_email=From('test#test.com', current_app.config['MAIL_ALIAS']),
to_emails=[to],
subject=subject,
html_content=render_template(template, **kwargs))
## cc management
if private_email == False:
cc_list = ['test#test.com']
if cc != 'None':
cc_list.append(cc)
message.cc = cc_list
if attachment_location != 'None':
with open(attachment_location, 'rb') as f:
data = f.read()
f.close()
encoded_file = base64.b64encode(data).decode()
attachedFile = Attachment(
FileContent(encoded_file),
FileName(attachment_name),
FileType('application/xlsx'),
Disposition('attachment')
)
message.attachment = attachedFile
try:
sg = SendGridAPIClient(os.environ.get('SENDGRID_API_KEY'))
response = sg.send(message)
# print(response.status_code)
# print(response.body)
# print(response.headers)
print(f'Complete: Email Sent to {to}')
except Exception as e:
print(e.message)
Heroku Procfile
web: gunicorn test_app.app:app --preload --log-level=debug

According to your comments, the emails are triggered by user actions. From the code you have shared there is nothing that would cause an email to send twice, so my guess is that users are causing the occasional double sending by submitting forms more than once.
To discover the root of this, I would first attempt to disable your forms after the first submit. You can do this with a bit of JavaScript, something like:
document.querySelectorAll('form').forEach(form => {
form.addEventListener('submit', (e) => {
// Prevent if already submitting
if (form.classList.contains('is-submitting')) {
e.preventDefault();
}
// Add class to hook our visual indicator on
form.classList.add('is-submitting');
});
});
This comes from this article which has a good discussion of the issue too.
Once you have done that, you should likely also log the various actions that cause an email to send and try to chase back through your system to find what could be causing double submissions. I would pay attention to other things like callbacks or background jobs that may be the culprit here too.

python aiosmtpd server with basic authentication

Im trying to create an aiosmtpd server to process emails received.
It works great without authentication, yet i simply cannot figure out how to setup the authentication.
I have gone through the documents and searched for examples on this.
a sample of how im currently using it:
from aiosmtpd.controller import Controller
class CustomHandler:
async def handle_DATA(self, server, session, envelope):
peer = session.peer
mail_from = envelope.mail_from
rcpt_tos = envelope.rcpt_tos
data = envelope.content # type: bytes
# Process message data...
print('peer:' + str(peer))
print('mail_from:' + str(mail_from))
print('rcpt_tos:' + str(rcpt_tos))
print('data:' + str(data))
return '250 OK'
if __name__ == '__main__':
handler = CustomHandler()
controller = Controller(handler, hostname='192.168.8.125', port=10025)
# Run the event loop in a separate thread.
controller.start()
# Wait for the user to press Return.
input('SMTP server running. Press Return to stop server and exit.')
controller.stop()```
which is the basic method from the documentation.
could someone please provide me with an example as to how to do simple authentication?

Alright, since you're using version 1.3.0, you can follow the documentation for Authentication.
A quick way to start is to create an "authenticator function" (can be a method in your handler class, can be standalone) that follows the Authenticator Callback guidelines.
A simple example:
from aiosmtpd.smtp import AuthResult, LoginPassword
auth_db = {
b"user1": b"password1",
b"user2": b"password2",
b"user3": b"password3",
}
# Name can actually be anything
def authenticator_func(server, session, envelope, mechanism, auth_data):
# For this simple example, we'll ignore other parameters
assert isinstance(auth_data, LoginPassword)
username = auth_data.login
password = auth_data.password
# If we're using a set containing tuples of (username, password),
# we can simply use `auth_data in auth_set`.
# Or you can get fancy and use a full-fledged database to perform
# a query :-)
if auth_db.get(username) == password:
return AuthResult(success=True)
else:
return AuthResult(success=False, handled=False)
Then you're creating the controller, create it like so:
controller = Controller(
handler,
hostname='192.168.8.125',
port=10025,
authenticator=authenticator_func, # i.e., the name of your authenticator function
auth_required=True, # Depending on your needs
)

Send push notification to many users via python

For my project using Firebase messaging to send push notification. I have users's firebase tokens stored on the database. Using them I sent push to each user. Total time of sending is about 100 seconds for 100 users. Is there way to send push asynchronously(I mean at one time to send many push notifications)
# Code works synchronously
for user in users:
message = messaging.Message(
notification=messaging.Notification(
title="Push title",
body="Push body"
),
token = user['fcmToken']
)
response = messaging.send(message)

Sure, you could use one of the python concurrency libraries. Here's one option:
from concurrent.futures import ThreadPoolExecutor, wait, ALL_COMPLETED
def send_message(user):
message = messaging.Message(
notification=messaging.Notification(
title="Push title",
body="Push body"),
token = user['fcmToken'])
return messaging.send(message)
with ThreadPoolExecutor(max_workers=10) as executor: # may want to try more workers
future_list = []
for u in users:
future_list.append(executor.submit(send_message, u))
wait(future_list, return_when=ALL_COMPLETED)
# note: we must use the returned self to get the test count
print([future.result() for future in future_list])

If you want to send the same message to all tokens, you can use a single API call with a multicast message. The Github repo has this sample of sending a multicast message in Python:
def send_multicast():
# [START send_multicast]
# Create a list containing up to 500 registration tokens.
# These registration tokens come from the client FCM SDKs.
registration_tokens = [
'YOUR_REGISTRATION_TOKEN_1',
# ...
'YOUR_REGISTRATION_TOKEN_N',
]
message = messaging.MulticastMessage(
data={'score': '850', 'time': '2:45'},
tokens=registration_tokens,
)
response = messaging.send_multicast(message)
# See the BatchResponse reference documentation
# for the contents of response.
print('{0} messages were sent successfully'.format(response.success_count))
# [END send_multicast]

Pulling historical channel messages python

I am attempting to create a small dataset by pulling messages/responses from a slack channel I am a part of. I would like to use python to pull the data from the channel however I am having trouble figuring out my api key. I have created an app on slack but I am not sure how to find my api key. I see my client secret, signing secret, and verification token but can't find my api key
Here is a basic example of what I believe I am trying to accomplish:
import slack
sc = slack.SlackClient("api key")
sc.api_call(
"channels.history",
channel="C0XXXXXX"
)
I am willing to just download the data manually if that is possible as well. Any help is greatly appreciated.

messages
See below for is an example code on how to pull messages from a channel in Python.
It uses the official Python Slack library and calls
conversations_history with paging. It will therefore work with
any type of channel and can fetch large amounts of messages if
needed.
The result will be written to a file as JSON array.
You can specify channel and max message to be retrieved
threads
Note that the conversations.history endpoint will not return thread messages. Those have to be retrieved additionaly with one call to conversations.replies for every thread you want to retrieve messages for.
Threads can be identified in the messages for each channel by checking for the threads_ts property in the message. If it exists there is a thread attached to it. See this page for more details on how threads work.
IDs
This script will not replace IDs with names though. If you need that here are some pointers how to implement it:
You need to replace IDs for users, channels, bots, usergroups (if on a paid plan)
You can fetch the lists for users, channels and usergroups from the API with users_list, conversations_list and usergroups_list respectively, bots need to be fetched one by one with bots_info (if needed)
IDs occur in many places in messages:
user top level property
bot_id top level property
as link in any property that allows text, e.g. <#U12345678> for users or <#C1234567> for channels. Those can occur in the top level text property, but also in attachments and blocks.
Example code
import os
import slack
import json
from time import sleep
CHANNEL = "C12345678"
MESSAGES_PER_PAGE = 200
MAX_MESSAGES = 1000
# init web client
client = slack.WebClient(token=os.environ['SLACK_TOKEN'])
# get first page
page = 1
print("Retrieving page {}".format(page))
response = client.conversations_history(
channel=CHANNEL,
limit=MESSAGES_PER_PAGE,
)
assert response["ok"]
messages_all = response['messages']
# get additional pages if below max message and if they are any
while len(messages_all) + MESSAGES_PER_PAGE <= MAX_MESSAGES and response['has_more']:
page += 1
print("Retrieving page {}".format(page))
sleep(1) # need to wait 1 sec before next call due to rate limits
response = client.conversations_history(
channel=CHANNEL,
limit=MESSAGES_PER_PAGE,
cursor=response['response_metadata']['next_cursor']
)
assert response["ok"]
messages = response['messages']
messages_all = messages_all + messages
print(
"Fetched a total of {} messages from channel {}".format(
len(messages_all),
CHANNEL
))
# write the result to a file
with open('messages.json', 'w', encoding='utf-8') as f:
json.dump(
messages_all,
f,
sort_keys=True,
indent=4,
ensure_ascii=False
)

This is using the slack webapi. You would need to install requests package. This should grab all the messages in channel. You need a token which can be grabbed from apps management page. And you can use the getChannels() function. Once you grab all the messages you will need to see who wrote what message you need to do id matching(map ids to usernames) you can use getUsers() functions. Follow this https://api.slack.com/custom-integrations/legacy-tokens to generate a legacy-token if you do not want to use a token from your app.
def getMessages(token, channelId):
print("Getting Messages")
# this function get all the messages from the slack team-search channel
# it will only get all the messages from the team-search channel
slack_url = "https://slack.com/api/conversations.history?token=" + token + "&channel=" + channelId
messages = requests.get(slack_url).json()
return messages
def getChannels(token):
'''
function returns an object containing a object containing all the
channels in a given workspace
'''
channelsURL = "https://slack.com/api/conversations.list?token=%s" % token
channelList = requests.get(channelsURL).json()["channels"] # an array of channels
channels = {}
# putting the channels and their ids into a dictonary
for channel in channelList:
channels[channel["name"]] = channel["id"]
return {"channels": channels}
def getUsers(token):
# this function get a list of users in workplace including bots
users = []
channelsURL = "https://slack.com/api/users.list?token=%s&pretty=1" % token
members = requests.get(channelsURL).json()["members"]
return members

Using Tweepy to listen to stream and search for tweets. How to stop previous search and only listen for new stream?

I'm using Flask and Tweepy to search for live tweets. On the front-end I have a user text input, and button called "Search". Ideally, when a user gives a search-term into the input and clicks the "Search" button, the Tweepy should listen for the new search-term and stop the previous search-term stream. When the "Search" button is clicked it executes this function:
#app.route('/search', methods=['POST'])
# gets search-keyword and starts stream
def streamTweets():
search_term = request.form['tweet']
search_term_hashtag = '#' + search_term
# instantiate listener
listener = StdOutListener()
# stream object uses listener we instantiated above to listen for data
stream = tweepy.Stream(auth, listener)
if stream is not None:
print "Stream disconnected..."
stream.disconnect()
stream.filter(track=[search_term or search_term_hashtag], async=True)
redirect('/stream') # execute '/stream' sse
return render_template('index.html')
The /stream route that is executed in the second to last line in above code is as follows:
#app.route('/stream')
def stream():
# we will use Pub/Sub process to send real-time tweets to client
def event_stream():
# instantiate pubsub
pubsub = red.pubsub()
# subscribe to tweet_stream channel
pubsub.subscribe('tweet_stream')
# initiate server-sent events on messages pushed to channel
for message in pubsub.listen():
yield 'data: %s\n\n' % message['data']
return Response(stream_with_context(event_stream()), mimetype="text/event-stream")
My code works fine, in the sense that it starts a new stream and searches for a given term whenever the "Search" button is clicked, but it does not stop the previous search. For example, if my first search term was "NYC" and then I wanted to search for a different term, say "Los Angeles", it will give me results for both "NYC" and "Los Angeles", which is not what I want. I want just "Los Angeles" to be searched. How do I fix this? In other words, how do I stop the previous stream? I looked through other previous threads, and I know I have to use stream.disconnect(), but I'm not sure how to implement this in my code. Any help or input would be greatly appreciated. Thanks so much!!

Below is some code that will cancel old streams when a new stream is created. It works by adding new streams to a global list, and then calling stream.disconnect() on all streams in the list whenever a new stream is created.
diff --git a/app.py b/app.py
index 1e3ed10..f416ddc 100755
--- a/app.py
+++ b/app.py
## -23,6 +23,8 ## auth.set_access_token(access_token, access_token_secret)
app = Flask(__name__)
red = redis.StrictRedis()
+# Add a place to keep track of current streams
+streams = []
#app.route('/')
def index():
## -32,12 +34,18 ## def index():
#app.route('/search', methods=['POST'])
# gets search-keyword and starts stream
def streamTweets():
+ # cancel old streams
+ for stream in streams:
+ stream.disconnect()
+
search_term = request.form['tweet']
search_term_hashtag = '#' + search_term
# instantiate listener
listener = StdOutListener()
# stream object uses listener we instantiated above to listen for data
stream = tweepy.Stream(auth, listener)
+ # add this stream to the global list
+ streams.append(stream)
stream.filter(track=[search_term or search_term_hashtag],
async=True) # make sure stream is non-blocking
redirect('/stream') # execute '/stream' sse
What this does not solve is the problem of session management. With your current setup a search by one user will affect the searches of all users. This can be avoided by giving your users some identifier and storing their streams along with their identifier. The easiest way to do this is likely to use Flask's session support. You could also do this with a requestId as Pierre suggested. In either case you will also need code to notice when a user has closed the page and close their stream.

Disclaimer: I know nothing about Tweepy, but this appears to be a design issue.
Are you trying to add state to a RESTful API? You may have a design problem.
As JRichardSnape answered, your API shouldn't be the one taking care of canceling a request; it should be done in the front-end. What I mean here is in the javascript / AJAX / etc calling this function, add another call, to the new function
#app.route('/cancelSearch', methods=['POST'])
With the "POST" that has the search terms. So long as you don't have state, you can't really do this safely in an async call: Imagine someone else makes the same search at the same time then canceling one will cancel both (remember, you don't have state so you don't know who you're canceling). Perhaps you do need state with your design.
If you must keep using this and don't mind breaking the "stateless" rule, then add a "state" to your request. In this case it's not so bad because you could launch a thread and name it with the userId, then kill the thread every new search
def streamTweets():
search_term = request.form['tweet']
userId = request.form['userId'] # If your limit is one request per user at a time. If multiple windows can be opened and you want to follow this limit, store userId in a cookie.
#Look for any request currently running with this ID, and cancel them
Alternatively, you could return a requestId, which you would then keep in the front-end can call cancelSearch?requestId=$requestId. In cancelSearch, you would have to find the pending request (sounds like that's in tweepy since you're not using your own threads) and disconnect it.
Out of curiosity I just watched what happens when you search on Google, and it uses a GET request. Have a look (debug tools -> Network; then enter some text and see the autofill). Google uses a token sent with every request (every time you type something)). It doesn't mean it's used for this, but that's basically what I described. If you don't want a session, then use a unique identifier.

Well I solved it by using timer method But still I'm looking for pythonic way.
from streamer import StreamListener
def stream():
hashtag = input
#assign each user an ID ( for pubsub )
StreamListener.userid = random_user_id
def handler(signum, frame):
print("Forever is over")
raise Exception("end of time")
def main_stream():
stream = tweepy.Stream(auth, StreamListener())
stream.filter(track=track,async=True)
redirect(url_for('map_stream'))
def close_stream():
# this is for closing client list in redis but don't know it's working
obj = redis.client_list(tweet_stream)
redis_client_list = obj[0]['addr']
redis.client_kill(redis_client_list)
stream = tweepy.Stream(auth, StreamListener())
stream.disconnect()
import signal
signal.signal(signal.SIGALRM, handler)
signal.alarm(300)
try:
main_stream()
except Exception:
close_stream()
print("function terminate")