Our office uses 2 IMAP servers for e-mail, one is the incoming server and holds the recent e-mails and the other is an archive server. We mainly use Outlook 2010 and our current process is to periodically drag sent messages from the incoming server to the archive.
Today I was asked into looking into writing a script and that would periodically (probably using crontab) grab all sent messages and move them to archive.
I've looked into some example of SSL or telnet to access the server and poke around. However I don't know the best way to script this or how to move files cross server within the IMAP environment.
What's the best way to accomplish this? I'd prefer to use Python just from comfort level, but if there is already an existing solution in another language, I could deal with it.
Update:
Ok, here's some code. Currently It copies the messages just fine, however, it will duplicate exisiting messages on the archive server.
import imaplib
import sys
#copy from
f_server = 'some.secret.ip.address'
f_username = 'j#example.com'
f_password = 'password'
f_box_name = 'Sent Messages'
#copy to
t_server = 'archive.server.i.p'
t_username = 'username'
t_password = 'password'
t_box_name = 'test'
To = imaplib.IMAP4(t_server)
To.login(t_username, t_password)
print 'Logged into mail server'
From = imaplib.IMAP4(f_server)
From.login(f_username, f_password)
print 'Logged into archive'
From.select(f_box_name) #open box which will have its contents copied
print 'Fetching messages...'
typ, data = From.search(None, 'ALL') #get all messages in the box
msgs = data[0].split()
sys.stdout.write(" ".join(['Copying', str(len(msgs)), 'messages']))
for num in msgs: #iterate over each messages id number
typ, data = From.fetch(num, '(RFC822)')
sys.stdout.write('.')
To.append(t_box_name, None, None, data[0][1]) #add a copy of the message to the archive box specified above
sys.stdout.write('\n')
try:
From.close()
From.logout()
try:
To.close()
To.logout()
Some sources:
Doug Hellman's Blog: imaplib - IMAP4 Client Library
Tyler Lesmann's Blog: Copying IMAP Mailboxes with Python and imaplib
I still need to:
delete/expunge messages on the live server
not copy duplicates (actually this would be fixed by deleting originals after copying, but...)
error trapping
Update 2:
Anyone have any ideas on how to not create duplicates when copying? (excluding the option of deleting originals, for now) I thought about searching text, but realized nested replies could throw that off.
Here's what I ended up using. I don't claim that it's perfect, the way it uses msg_num and not id is a little risky. But this is fairly low volume moves, maybe a couple an hour (on cron).
import imaplib
#copy from
from_server = {'server': '1.1.1.1',
'username': 'j#example.com',
'password': 'pass',
'box_names': ['Sent', 'Sent Messages']}
#copy to
to_server = {'server': '2.2.2.2',
'username': 'archive',
'password': 'password',
'box_name': 'Sent'}
def connect_server(server):
conn = imaplib.IMAP4(server['server'])
conn.login(server['username'], server['password'])
print 'Logged into mail server # %s' % server['server']
return conn
def disconnect_server(server_conn):
out = server_conn.logout()
if __name__ == '__main__':
From = connect_server(from_server)
To = connect_server(to_server)
for box in from_server['box_names']:
box_select = From.select(box, readonly = False) #open box which will have its contents copied
print 'Fetching messages from \'%s\'...' % box
resp, items = From.search(None, 'ALL') #get all messages in the box
msg_nums = items[0].split()
print '%s messages to archive' % len(msg_nums)
for msg_num in msg_nums:
resp, data = From.fetch(msg_num, "(FLAGS INTERNALDATE BODY.PEEK[])") # get email
message = data[0][1]
flags = imaplib.ParseFlags(data[0][0]) # get flags
flag_str = " ".join(flags)
date = imaplib.Time2Internaldate(imaplib.Internaldate2tuple(data[0][0])) #get date
copy_result = To.append(to_server['box_name'], flag_str, date, message) # copy to archive
if copy_result[0] == 'OK':
del_msg = From.store(msg_num, '+FLAGS', '\\Deleted') # mark for deletion
ex = From.expunge() # delete marked
print 'expunge status: %s' % ex[0]
if not ex[1][0]: # result can be ['OK', [None]] if no messages need to be deleted
print 'expunge count: 0'
else:
print 'expunge count: %s' % len(ex[1])
disconnect_server(From)
disconnect_server(To)
I'm not sure what volume of messages you're dealing with, but you could extract the Message-ID from each one and use that to find out if it's a duplicate. Either generate a list of IDs already on the target server each time you prepare to move messages, or add them to a simple database as they are archived.
You could narrow things down by an additional message property like Date if the lookups are too expensive, then drop the older lists when you no longer need them.
Presumably too late to be helpful to the OP, but hopefully useful for anyone following along after now.
This looks like a generic requirement. You probably shouldn't be custom coding anything.
You would probably be better off using an MTA configured to send copies of everything to an archive as well as sending stuff to your IMAP server. If this is hard for you to set up, consider using a third party service, who would manage your archives, and forward mail on to your existing mail server.
If you really do want to do this by copying from IMAP, I'd suggest looking at offlineimap.
If you really do want to do it yourself, the way to track the messages you've already seen is by using the Message-ID header.
Related
I'm currently working on an app that updates artists with times they are booked for events. Randomly the app will send duplicate emails a few times a day, over 90% of the time there are not duplicates.
Currently, there are 10+ emails that can be produced, but there is only one email send function. The duplicates occur across all emails which makes me think there is an issue with the email send function or the configuration of the web server to make multiple requests to sendgrid. PLEASE HELP ME FIND THE CAUSE OF THE DUPILCATES!
Stack:
Python (v3.10.6)
Flask (v2.2.2)
Heroku (buildstack-22)
Sendgrid python library (v6.9.7)
Email Send Function:
def send_email(to, subject, template, cc='None', attachment_location='None', attachment_name='None', private_email=False, **kwargs):
## Remove Guest DJ Emails Here
guest_dj_list = get_list_of_guest_djs()
if to in guest_dj_list:
return None
message = Mail(
from_email=From('test#test.com', current_app.config['MAIL_ALIAS']),
to_emails=[to],
subject=subject,
html_content=render_template(template, **kwargs))
## cc management
if private_email == False:
cc_list = ['test#test.com']
if cc != 'None':
cc_list.append(cc)
message.cc = cc_list
if attachment_location != 'None':
with open(attachment_location, 'rb') as f:
data = f.read()
f.close()
encoded_file = base64.b64encode(data).decode()
attachedFile = Attachment(
FileContent(encoded_file),
FileName(attachment_name),
FileType('application/xlsx'),
Disposition('attachment')
)
message.attachment = attachedFile
try:
sg = SendGridAPIClient(os.environ.get('SENDGRID_API_KEY'))
response = sg.send(message)
# print(response.status_code)
# print(response.body)
# print(response.headers)
print(f'Complete: Email Sent to {to}')
except Exception as e:
print(e.message)
Heroku Procfile
web: gunicorn test_app.app:app --preload --log-level=debug
According to your comments, the emails are triggered by user actions. From the code you have shared there is nothing that would cause an email to send twice, so my guess is that users are causing the occasional double sending by submitting forms more than once.
To discover the root of this, I would first attempt to disable your forms after the first submit. You can do this with a bit of JavaScript, something like:
document.querySelectorAll('form').forEach(form => {
form.addEventListener('submit', (e) => {
// Prevent if already submitting
if (form.classList.contains('is-submitting')) {
e.preventDefault();
}
// Add class to hook our visual indicator on
form.classList.add('is-submitting');
});
});
This comes from this article which has a good discussion of the issue too.
Once you have done that, you should likely also log the various actions that cause an email to send and try to chase back through your system to find what could be causing double submissions. I would pay attention to other things like callbacks or background jobs that may be the culprit here too.
I have a bunch of functions to store all attachments from all mails in one imap folder. Attachments are json datasets for further processing. To avoid oubliettes after connection loss and re-connection I inserted an expunge command after each message in the loop.
This leads to sporadic errors while processing an emailbody or better say: I can't read the body of the email resulting in [none] for the body.
To solve the problem I placed the expunge command after the loop and inserted another expunge command before the loop to delete marked messages to prevent multiple processing.
Though I have my function working flawlessly maybe someone can tell me about the serverside behaviour of an expunge command ( can't say which imap-server is running on serverside, provider is ionos.de). I can't find related information in the RFC document. Maybe some kind of race condition.
This is the code that leads to errors:
def downloadAllAttachmentsInInbox( **args):
conf =args['conf']
m = connectImapServer( conf.imappoll.server, conf.imappoll.account, conf.imappoll.password)
resp, items = m.search( None, "(ALL)")
items = items[0].split()
for emailid in items:
downloaAttachmentsInEmail( m =m, emailid =emailid, conf =conf)
# Markiere Mail zum löschen
if conf.imappoll.clearmb =="True":
m.store( emailid, '+FLAGS', '\\Deleted')
# Loesche markierte Mail(s)
m.expunge()
# Schliesse Mailbox, logout aus Server
m.close()
m.logout()
And this is the working code:
def downloadAllAttachmentsInInbox( **args):
conf =args['conf']
m = connectImapServer( conf.imappoll.server, conf.imappoll.account, conf.imappoll.password)
# Loesche bereits zur Löschung markierte Mails vom Server vor Abruf aller Nachrichten
m.expunge()
resp, items = m.search( None, "(ALL)")
items = items[0].split()
for emailid in items:
downloaAttachmentsInEmail( m =m, emailid =emailid, conf =conf)
# Markiere Mail zum löschen
if conf.imappoll.clearmb =="True":
m.store( emailid, '+FLAGS', '\\Deleted')
# Loesche markierte Mail(s)
m.expunge()
# Schliesse Mailbox, logout aus Server
m.close()
m.logout()
If you use message sequence numbers, you may not use expunge, since it renumbers messages, making your search results out of date.
You have a couple options:
Use UIDs everywhere. m.uid(‘SEARCH’, ‘ALL’) and m.uid(‘FETCH’, …) and m.uid(‘STORE’, …). UIDs aren’t renumbered on expunge.
Don’t expunge, and search for UNDELETED instead, to ignore messages you’ve already marked deleted (eg, processed). m.search(‘UNDELETED’). Then expunge outside of the loop.
I have sleekXMPP for python and I have used the API to create functions to send messages, although when I researched into receiving them I can't find anything, can someone please help me to work this out, or disprove the possibility. Thanks.
Below is the code I used to send Messages, If its any help.
to = config.get('usermap', to[4:])
gmail_from_user = config.get('auth', 'email')
gmail_from_secret = config.get('auth', 'secret')
sys.stdout = stdouttmp
sys.stderr = stderrtmp
print "Sending chat message to " + to
xmpp = SendMsgBot(gmail_from_user, gmail_from_secret, to, message)
xmpp.register_plugin('xep_0030') # Service Discovery
xmpp.register_plugin('xep_0199') # XMPP Ping
sys.stdout = stdouttmp
if xmpp.connect(('talk.google.com', 5222)):
xmpp.process(block=True)
else:
sys.stdout = stdouttmp
print("Unable to connect.")
sys.stdout = stdouttmp
sys.stderr = stderrtmp
btw I'm using a .cfg text file for the users email and password, along with some contacts, which is then parsed in
I see that you're using the send_client.py example. The intent of that example is how to reliably log in, send a single message, and then log out. Your use case is to both send and receive messages, so you would be better served looking at the echo_client.py example.
Notably, in order to receive a message you would do:
# in your __init__ method:
def __init__(...):
# ...
self.add_event_handler('message', self.recv_message)
def recv_message(self, msg):
# You'll probably want to ignore error and headline messages.
# If you want to handle group chat messages, add 'groupchat' to the list.
if msg['type'] in ('chat', 'normal'):
print "%s says: %s" % (msg['from'], msg['body'])
Again, you will need to switch from using the SendMsgBot example because it automatically disconnects after sending its message.
Don't forget that there is the sleek#conference.jabber.org chat room if you need any help.
-- Lance
So I have this class that starts like this:
class emailreader():
def __init__(self, server, port, username, password):
self.imap_server = imaplib.IMAP4_SSL(server, int(port))
self.imap_server.login(username, password)
self.imap_server.select('INBOX')
def _get_raw_emails(self):
messages = []
typ, data = self.imap_server.search(None, 'UnSeen')
for num in data[0].split():
typ, data = self.imap_server.fetch(num, '(RFC822)')
messages.append(data[0][1])
return messages
It's working great for fetching messages like this:
mail = emailreader(server, port, username, password)
emails = mail._get_raw_emails()
But if I send a new message to the e-mail address I can't just run the last line again, the new mail won't show up until I delete the mail object and start all over again. Why is that? Do I have to reset the last search or something?
Edit: I think I found the solution. I had to do a imap_server.check() also...
I found the solution. I had to execute a method that's called check() from my imap object whenever new mails may have arrived. This is my "new" _get_raw_emails() method:
def _get_raw_emails(self):
messages = []
self.imap_server.check() # Fetch new mails since object is created.
typ, data = self.imap_server.search(None, 'UnSeen')
for num in data[0].split():
typ, data = self.imap_server.fetch(num, '(RFC822)')
messages.append(data[0][1])
return messages
It may be because I'm not a native english speaker, but when I looked att the documentation for imaplib it did'nt seem obvious what the check method did. It only says "Checkpoint mailbox on server." So I did'nt thought that it would solve my problem.
I have a python script that has to fetch unseen messages, process it, and mark as seen (or read)
I do this after login in:
typ, data = self.server.imap_server.search(None, '(UNSEEN)')
for num in data[0].split():
print "Mensage " + str(num) + " mark"
self.server.imap_server.store(num, '+FLAGS', '(SEEN)')
The first problem is that, the search returns ALL messages, and not only the UNSEEN.
The second problem is that messages are not marked as SEEN.
Can anybody give me a hand with this?
Thanks!
import imaplib
obj = imaplib.IMAP4_SSL('imap.gmail.com', '993')
obj.login('user', 'password')
obj.select('Inbox') <--- it will select inbox
typ ,data = obj.search(None,'UnSeen')
obj.store(data[0].replace(' ',','),'+FLAGS','\Seen')
I think the flag names need to start with a backslash, eg: \SEEN
I am not so familiar with the imaplib but I implement this well with the imapclient module
import imapclient,pyzmail,html2text
from backports import ssl
context=ssl.SSLContext(ssl.PROTOCOL_TLSv1_2)
iobj=imapclient.IMAPClient('outlook.office365.com', ssl=True, ssl_context=context)
iobj.login(uname,pwd)# provide your username and password
iobj.select_folder('INBOX',readonly=True)# Selecting Inbox.
unread=iobj.search('UNSEEN')# Selecting Unread messages, you can add more search criteria here to suit your purpose.'FROM', 'SINCE' etc.
print('There are: ',len(unread),' UNREAD emails')
for i in unread:
mail=iobj.fetch(i,['BODY[]'])#I'm fetching the body of the email here.
mcontent=pyzmail.PyzMessage.factory(mail[i][b'BODY[]'])#This returns the email content in HTML format
subject=mcontent.get_subject()# You might not need this
receiver_name,receiver_email=mcontent.get_address('from')
mail_body=html2text.html2text(mcontent.html_part.get_payload().decode(mcontent.html_part.charset))# This returns the email content as text that you can easily relate with.
Let's say I want to just go through the unread emails, reply the sender and mark the email as read. I'd call the smtp function from here to compose and send a reply.
import smtplib
smtpobj=smtplib.SMTP('smtp.office365.com',587)
smtpobj.starttls()
smtpobj.login(uname,pwd)# Your username and password goes here.
sub='Subject: '+str(subject)+'\n\n'# Subject of your reply
msg='Thanks for your email! You're qualified for the next round' #Some random reply :(
fullmsg=sub+new_result
smtpobj.sendmail(uname,test,fullmsg)# This sends the email.
iobj.set_flags(i,['\\Seen','\\Answered'])# This marks the email as read and adds the answered flag
iobj.append('Sent Items', fullmsg)# This puts a copy of your reply in your Sent Items.
iobj.logout()
smtpobj.logout()
I hope this helps