Although using Python, I set the headers as described here would result in flagging my e-mails as spam by SpamAssassin.
I am sending reminders for un-paid invoices, so I would like to do anything in my power to make the receiver aware of the e-mail - but this cannot happen if my e-mail ends up in the spam folder due to the urgent flag.
Using the X-MSMail-Priority in the header would add a positive spam score from MISSING_MIMEOLE, using X-Priority would add a positive score. Only using Priority in the header are not implemented in Thunderbird Mail client nor in RoundCube webinterface, thus the urgency is not shown.
What can I do to make my e-mails urgent, but simultaneously make SpamAssassin (and other filters) happy?
You cannot control how others filter their spam. If you find that anything in your mail triggers common spam filters, you have to remove that if it is important that your users get those emails. The priority header is abused by spammers, so you cannot use that.
Likewise, I would expect any other unauthenticated priority indicator to be abused by spammers, so there won’t be any way. Possibly signing these headers using DKIM and deploying DMARC (with a strict policy) on your domain might help, but I do not know for sure that filters such as spamassassin are smart enough to consider the priority header authenticated in such cases.
Deploying DMARC might be a good idea anyways for transactional mail, to prevent spoofing.
Related
I have written a Python script that iterates through rows of an Excel file and, for each row:
Gets an e-mail address, name, and name of attachment file to use
Composes an e-mail
Sends out the e-mail
I'm not sure if it's accurate to call this mass-emailing or if it is a candidate for being black-listed because it is sending out individualized e-mails. With a message submission rate of 5/minute, I want to throttle it (or have the limit increased to 100).
So my question is: Is the sort of scenario, assuming the limit is increased to 100, prone to black-listing?
It depends on how well you know the people you're mailing.
If you know them pretty well, it should be fine. If they're total strangers, the recipients might think it's spam and start blocking you.
I could help more if you told me how well you know the recipients.
Its not easy to answer your question as it depends hardly on the remote email environment used here and what you understand with individualized emails (only a different "Hello Mr. ZZ" or "Dear Ms. YY" isn´t really an individualized email these days). To give you three possible examples:
Situation 1:
All users are on the same email environment (e.g. Exchange Online / Office 365). Then the remote mail server might see here +100 similar emails and might mark them as spam. If all +100 users are on +100 different email servers that might be different however the following might be possible:
Situation 2:
One user think that this email is spam and report that as spam. Depending on the AntiSpam engine used here a hash value from that email might be created and other email server using the same AntiSpam engine might therefore detect your email as spam as well.
Situation 3:
The users are on different email environments but in front of them is an AntiSpam cloud solution. This solution will then see +100 similar emails from one eMail environment and might therefore clarify that as SPAM.
Offtopic: You might consider using services like from MailChimp which use different email servers to spread out a similar email. This might help to prevent such issues as the mass emails aren´t send from only one server. On top of that you do not risk your own email server from being blacklisted which might have a very bad business impact on your company.
Summary
How do I create a set of on-demand, mock/dummy/fake emails in numerous
IMAP folders? Email content needs to be share-able in a public forum via
a commonly-access IMAP-server account (typically for testers/developers
trying to debug MUA problems/configurations), with no privacy
risks.
I haven't yet found a solution to do this. Unless I can find something
(I'm totally looking for suggestions), I'm relegated to writing my own
software client. If so, I'd do it in Python, and I'm looking for general
pointers on which tools/libraries/methods/approaches I should employ to
most-quickly get a first, working prototype.
How should I solve the above, given the context below?
Purpose
I want to test various MUA deployments, sharing the same IMAP
account between many users/testers/developers (of any MUA) in a
public arena. Example: I might ask on a Notmuch email list:
"Why is my mbsync/Notmuch
config not working? Here's a shared gmail.com account we can
collectively use as a common IMAP server to minimize server-side
variables and thus collectively help debug stuff."
IMAP-Client Requirements
The IMAP-client program:
must be able to create a variable number of nested IMAP folders with any
number emails,
must prove all email content and folder names are share-able,
with no privacy concerns (any reasonable content will do,
and it doesn't have to make sense; eg #1: variants of Lorem
ipsum might work; eg #1:
provide input for 3 or more example emails, provided by the user/caller)
so long as the emails can be opened and read by MUAs, and their
attachments are "real" enough to be opened by the attachment-file's
corresponding application,
will include some number of emails with 1 or more attachments,
will optimally (but not required for initial versions) be capable
of generating GB's of content by creating many thousands of emails in
hundreds of nested IMAP folders. The client can leverage many or large
file attachments will help do this.
must be able to do all of the above on-demand, given any new/fresh
IMAP-server account credentials.
As an implementation shortcut, it's ok for the client to duplicate
much/most of the email conent, so long as there's significant variance
in Date:, To:, From:, and Subject: headers and email-folder names (all
of which are presumably easy to "randomize").
More Details
I've pondered trying to non-private-ize existing emails/folders from
IMAP accounts I already have (that serve the above requirements), but
that work appears way too hard. Too much personal, sensitive information
would need to be "converted"/"private-ized." However, I'd like to hear
options for ways to easily privatize (scramble, encrypt, something?)
this existing email content. Such a path might save me having to write
the software.
The only way I see to solve this properly: leverage an IMAP client
program (again, I'm presumably writing it) that can create emails and
email folders on any designated IMAP server/account. Program input can
include example (presumably private) email content, number of folders
and nesting levels, randomness, date ranges (of emails), etc.
I've not yet found anything that does this.
GreenMail appears to setup
the IMAP server, but not the IMAP server content--unless I'm overlooking
something?
I am working on an application that will use mqtt. I will be using the python library. I have been leaning towards using mosquitto but can find no way of programmatically setting access control limits for it. The application I'm writing needs to be able to differentiate between users, and only allow them to subscribe to certain topics. The current solution looks like this is done from a config file. Is there a scalable solution to access control limits in mosquitto? If not, do you know of a mqtt broker in which this exists?
Even if this might not concern you anymore, others could find it useful. I am following here mosquitto's man page.
There are two configuration files, a general one, say mosquitto.conf, and an ACL (Access Control List) one, say acl.conf.
mosquitto.conf enables the acl.conf file for access control:
acl_file acl.conf
acl.conf defines the access control behavior:
# users can anonymously publish to the topic 'in'
topic write in
# users can subscribe topics named 'out/%u', where %u is the user's name
pattern read out/%u
# an admin may subscribe to 'in'
# and publish to all subtopics of 'out/' (note the +)
user adminWithSecretName
topic read in
topic write out/+
We execute mosquitto -c mosquitto.conf to run mosquitto with the configuration file.
In this case, a dynamic authentication mechanism can be established by using randomly generated user names.
Example: Alice wants to subscribe so that she can read here private messages. She sends her credentials in combination with a nonceN1 to in. Furthermore, she also subscribes the topic out/N1, using N1 as user name. The pattern read out/%u allows that.
A third-party server application, connected as adminWithSecretName and subscribed to the topic in receives Alice' message. It verifies its authenticity and then generates a new nonce N2 and publishes it to out/N1 where Alice has subscribed to.
From now on -- at least for this session -- out/N2 is the regular topic where Alice respectively here devices will receive messages. Therefore, Alice unsubscribes and disconnects form out/N1 and subscribes to out/N2. The third-party server application publishes all new messages, belonging to Alice, to the topic out/N2.
Further Considerations: One may also want to reflect on other aspects of security such as TLS and/or per-message encryption. The configuration discussed here would, depending on the grade of targeted security/privacy, probably also require TLS.
On the other hand, this could be obsolete if the messages are encrypted separately. One, say Eve, could intercept (even subscribe!) the messages if she had access to the cable/WiFi stream, as she would see the secret user name as plain text. But: when one already has access to the data stream, he/she can intercept the bytes anyway. They are encrypted either way, using TLS or per-message encryption. Also, traffic analysis may be applied to both approaches.
I would suggest to use either TLS or per-message encryption. Both should, correctly implemented and applied, lead to comparable security.
You could write a plugin to handle this for you. See http://mosquitto.org/2013/07/authentication-plugins/ for some examples.
You may find more answers if you ask on the mosquitto mailing list.
If you are familiar with Java you should try the HiveMQ MQTT broker: http://www.hivemq.com.
There is an open PluginSDK, which enables you to write any kind of extensions to the broker.
You can implement the authentication or authorization method that fits your use case best, for example from database, file...
The authorization based on topic is a common use case and there is an example in the HiveMQ Plugin Guide.
As entry point into HiveMQ plugin development see the Get started with Plugins page: http://www.hivemq.com/documentations/getting-started-plugins/
Disclosure: I'm one of the developers of HiveMQ.
I'm trying to write a pop3 and imap clients in python using available libs, which will download email headers (and subsequently entire email bodies) from various servers and save them in a mongodb database. The problem I'm facing is that this client downloads emails in addition to a user's regular email client. So with the assumption that a user might or might not leave emails on the server when downloading using his mail client, I'd like to fetch the headers but only collect them from a certain date, to avoid grabbing entire mailboxes every time I fetch the headers.
As far as I can see the POP3 list call will get me all messages on the server, even those I probably already downloaded. IMAP doesn't have this problem.
How do email clients handle this situation when dealing with POP3 servers?
Outlook logs in to a POP3 server and issues the STAT, LIST and UIDL commands; then if it decides the user has no new messages it logs out. I have observed Outlook doing this when tracing network traffic between a client and my DBMail POP3 server. I have seen Outlook fail to detect new messages on a POP3 server using this method. Thunderbird behaves similarly but I have never seen it fail to detect new messages.
Issue the LIST and UIDL commands to the server after logging in. LIST gives you an index number (the message's linear position in the mailbox) and the size of each message. UIDL gives you the same index number and a computed hash value for each message.
For each user you can store the size and hash value given by LIST and UIDL. If you see the same size and hash value, assume it is the same message. When a given message no longer appears in this list, assume it has been deleted and clear it from your local memory.
For complete purity, remember the relative positions of the size/hash pairs in the message list, so that you can support the possibility that they may repeat. (My guess on Outlook's new message detection failure is that sometimes these values do repeat, at least for DBMail, but Outlook remembers them even after they are deleted, and forever considers them not new. If it were me, I would try to avoid this behavior.)
Footnote: Remember that the headers are part of the message. Do not trust anything in the header for this reason: dates, senders, even server hand-off information can be easily faked and cannot be assumed unique.
I'm looking into a possible feature for my little to-do application... I like the idea that I can send an email to a particular email address, containing a to-do task I need to complete, and this will be read by my web application and be put in the database... So, when I come to log into my application, the to-do task I emailed will be there as a entry in the app.
Is this possible? I have a slice with SliceHost (basically a dedicated server) so I have total control on what to install etc. I'm using Python/Django/MySQL for this.
Any ideas on what steps to take to make this happen?
If I were to implement this, I'd use a scheduler and a job to be scheduled.
That job would connect to the mail server (be it POP3 or IMAP) and parse the unread messages (or messages unread by the job). Based on that I would insert that record.
You'd get 2 types of records that way. A list of mail message ids which have been processed (so you don't reprocess mails) and a list of tasks.
Disadvantage is that it takes some time before you see the task, as the job only executes every X minutes, or seconds.
If that is not good enough I'd go for a permanent IMAP connection, but you'd have to implement more error handling; you don't just retry automatically every X minutes.
Googling for Django +scheduler will get you started.
also have a look at this StackOverflow thread, no need to reinvent the wheel :)
I needed the exact same thing. I use the Lamson project (which is written in python) to transform email, forward email based on rules to my www.evernote.com and thinking rock www.trgtd.com.au accounts, update firewall web filtering rules, update allow/deny lists for my spam filter, read and write databases etc....
I like to think of it as email server automation and email application development.
www.lamsonproject.org
Troy
One way that I've solved this in the past was using qmail's .qmail files (docs).
Basically you set up qmail and point your email address (for ease of use, lets assume proc#whatever.com is your email address) to your home directory. In that directory you set up a .qmail-proc file to handle the mail.
This allows you to use a full-fledged SMTP server on your server, including spam filtering, forwarding, aliases, all that fun stuff. You can then pipe the data from an email into an application. In your case, I would suggest making a Mangement Command in Django to process the email (I'll call it proc_email). Thus your .qmail-proc may look like:
/var/spool/mail/proc
| /www/django/myproject/manage.py proc_email
This stores a copy of the email in /var/spool/mail/proc, then passes the email to the script in the second line. The email itself is passed to proc_email via sys.stdin. Simply read the email from there, and store it through your Django Models.
If you need to process email for different addresses later, you can also set up aliases which point to your home directory, and use .qmail-<username> files for each alias. Allowing you to pass other flags (such as the username for each alias) to proc_email if needed.
I should note that this isn't the simplest solution, but it can scale, and is pretty darn bullet proof.
I would not focus on Django for this.
I would create a mail server to catch these emails. Use http://docs.python.org/library/smtpd.html.
I would then use just the Django ORM to update the database based on the emails received.