Develop Reference

Extract data from text file using Python (or any language)

I have a text file that looks like:
First Name Bob
Last name Smith
Phone 555-555-5555
Email bob#bob.com
Date of Birth 11/02/1986
Preferred Method of Contact Text Message
Desired Appointment Date 04/29
Desired Appointment Time 10am
City Pittsburgh
Location State
IP Address x.x.x.x
User-Agent (Browser/OS) Apple Safari 14.0.3 / OS X
Referrer http://www.example.com
First Name john
Last name Smith
Phone 555-555-4444
Email john#gmail.com
Date of Birth 03/02/1955
Preferred Method of Contact Text Message
Desired Appointment Date 05/22
Desired Appointment Time 9am
City Pittsburgh
Location State
IP Address x.x.x.x
User-Agent (Browser/OS) Apple Safari 14.0.3 / OS X
Referrer http://www.example.com
.... and so on
I need to extract each entry to a csv file, so the data should look like: first name, last name, phone, email, etc. I don't even know where to start on something like this.

first of all you'll need to open the text file in read mode.
I'd suggest using a context manager like so:
with open('path/to/your/file.txt', 'r') as file:
for line in file.readlines():
# do something with the line (it is a string)
as for managing the info you could build some intermediate structure, for example a dictionary or a list of dictionaries, and then translate that into a CSV file with the csv module.
you could for example split the file whenever there is a blank line, maybe like this:
with open('Downloads/test.txt', 'r') as f:
my_list = list() # this will be the final list
entry = dict() # this contains each user info as a dict
for line in f.readlines():
if line.strip() == "": # if line is empty start a new dict
my_list.append(entry) # and append the old one to the list
entry = dict()
else: # otherwise split the line and create new dict
line_items = line.split(r' ')
print(line_items)
entry[line_items[0]] = line_items[1]
print(my_list)
this code won't work because your text is not formatted in a consistent way: you need to find a way to make the split between "title" and "content" (like "first name" and "bob") in a consistent way. I suggest maybe looking at regex and fixing the txt file by making spacing more consistent.

assuming the data resides in a:
a="""
First Name Bob
Last name Smith
Phone 555-555-5555
Email bob#bob.com
Date of Birth 11/02/1986
Preferred Method of Contact Text Message
Desired Appointment Date 04/29
Desired Appointment Time 10am
City Pittsburgh
Location State
IP Address x.x.x.x
User-Agent (Browser/OS) Apple Safari 14.0.3 / OS X
Referrer http://www.example.com
First Name john
Last name Smith
Phone 555-555-4444
Email john#gmail.com
Date of Birth 03/02/1955
Preferred Method of Contact Text Message
Desired Appointment Date 05/22
Desired Appointment Time 9am
City Pittsburgh
Location State
IP Address x.x.x.x
User-Agent (Browser/OS) Apple Safari 14.0.3 / OS X
Referrer http://www.example.com
"""
line_sep = "\n" # CHANGE ME ACCORDING TO DATA
fields = ["First Name", "Last name", "Phone",
"Email", "Date of Birth", "Preferred Method of Contact",
"Desired Appointment Date", "Desired Appointment Time",
"City", "Location", "IP Address", "User-Agent","Referrer"]
records = a.split(line_sep * 2)
all_records = []
for record in records:
splitted_record = record.split(line_sep)
one_record = {}
csv_record = []
for f in fields:
found = False
for one_field in splitted_record:
if one_field.startswith(f):
data = one_field[len(f):].strip()
one_record[f] = data
csv_record.append(data)
found = True
if not found:
csv_record.append("")
all_records.append(";".join(csv_record))
one_record will have the record as dictionary and csv_record will have it as a list of fields (ordered as fields variable)

Edited to add: ignore this answer, the code from Koko Jumbo looks infinitely more sensible and actually gives you a CVS file at the end of it! It was a fun exercise though :)
Just to expand on fcagnola's code a bit.
If it's a quick and dirty one-off, and you know that the data will be consistently presented, the following should work to create a list of dictionaries with the correct key/value pairing. Each line is processed by splitting the line and comparing the line number (reset to 0 with each new dict) against an array of values that represent where the boundary between key and value falls.
For example, "First Name Bob" becomes ["First","Name","Bob"]. The function has been told that linenumber= 0 so it checks entries[linenumber] to get the value "2", which it uses to join the key name (items 0 & 1) and then join the data (items 2 onwards). The end result is ["First Name", "Bob"] which is then added to the dictionary.
class Extract:
def extractEntry(self,linedata,lineindex):
# Hardcoded list! The quick and dirty part.
# This is specific to the example data provided. The entries
# represent the index to be used when splitting the string
# between the key and the data
entries = (2,2,1,1,3,4,3,3,1,1,2,2,1)
return self.createNewEntry(linedata,entries[lineindex])
def createNewEntry(self,linedata,dataindex):
list_data = linedata.split()
key = " ".join(list_data[:dataindex])
data = " ".join(list_data[dataindex:])
return [key,data]
with open('test.txt', 'r') as f:
my_list = list() # this will be the final list
entry = dict() # this contains each user info as a dict
extr = Extract() # class for splitting the entries into key/value
x = 0
for line in f.readlines():
if line.strip() == "": # if line is empty start a new dict
my_list.append(entry) # and append the old one to the list
entry = dict()
x = 0
else: # otherwise split the line and create new dict
extracted_data = extr.extractEntry(line,x)
entry[extracted_data[0]] = extracted_data[1]
x += 1
my_list.append(entry)
print(my_list)

Google people.listDirectoryPeople() method on python returns a slightly different list everytime

My organisation uses Google G Suite and contact details of all the employees are saved on the workspace directory. I've enabled people API for my work email (since it's part of G Suite) and tried listing out all employee contact details using people.listDirectoryPeople method.
Here's what I'm doing:
service = build('people', 'v1', credentials=creds)
src = 'DIRECTORY_SOURCE_TYPE_DOMAIN_PROFILE'
results = service.people().listDirectoryPeople(
readMask='names,emailAddresses,phoneNumbers,organizations',
sources=src,
pageSize=1000
).execute()
directory_people = results.get('people', [])
## I'M SAVING THE NEXT PAGE TOKEN HERE TO USE IN THE WHILE LOOP ##
next_page_token = results.get('nextPageToken')
for i, person in enumerate(directory_people):
names = person.get('names', [])
emails = person.get('emailAddresses',[])
phones = person.get('phoneNumbers',[])
orgs = person.get('organizations',[])
### code to save contacts ###
with open('file.tsv', 'w') as f:
f.write("Name\tOrg\tPhone\tEmail\n")
f.write(f"{name}\t{org}\t{phone}\t{email}\n")
while next_page_token:
results = service.people().listDirectoryPeople(
readMask='names,emailAddresses,phoneNumbers,organizations',
sources=src,
pageSize=1000,
pageToken=next_page_token
).execute()
directory_people = results.get('people', [])
next_page_token = results.get('nextPageToken')
print(next_page_token)
for i, person in enumerate(directory_people):
names = person.get('names', [])
emails = person.get('emailAddresses',[])
phones = person.get('phoneNumbers',[])
orgs = person.get('organizations',[])
### same code to save contacts ###
with open('file.tsv', 'a+') as f:
f.write(f"{name}\t{org}\t{phone}\t{email}\n")
The subsequent pages are loaded using next_page_token in a while loop.
The problem I'm facing is that the list returned is slightly different every time. E.g. Running the script 3 times would result in 3 different lists of lengths like 20, 25, 18.
Most of the contacts are the same, but there are some which weren't there in the previous run, while some from the previous run are not present now.
Note: I've used source DIRECTORY_SOURCE_TYPE_DOMAIN_CONTACT too but it doesn't serve my purpose because the contacts I'm interested in aren't available on this source
I've also tried using people.connections().list() method but that simply returns None for my work email
Does anyone know why the method isn't returning all the contacts like it's supposed to (or at least I believe it's supposed to)?

Not getting correct list because with open statement is inside the directory_people list for loop, move this out of loop like this:
## I'M SAVING THE NEXT PAGE TOKEN HERE TO USE IN THE WHILE LOOP ##
next_page_token = results.get('nextPageToken')
with open('file.tsv', 'w') as f:
f.write("Name\tOrg\tPhone\tEmail\n")
for i, person in enumerate(directory_people):
name = person.get('names', [])
email = person.get('emailAddresses',[])
phone = person.get('phoneNumbers',[])
org = person.get('organizations',[])
f.write(f"{name}\t{org}\t{phone}\t{email}\n")

Python microsoft graph api

I am using microsoft graph api to pull my emails in python and return them as a json object. There is a limitation that it only returns 12 emails. The code is:
def get_calendar_events(token):
graph_client = OAuth2Session(token=token)
# Configure query parameters to
# modify the results
query_params = {
#'$select': 'subject,organizer,start,end,location',
#'$orderby': 'createdDateTime DESC'
'$select': 'sender, subject',
'$skip': 0,
'$count': 'true'
}
# Send GET to /me/events
events = graph_client.get('{0}/me/messages'.format(graph_url), params=query_params)
events = events.json()
# Return the JSON result
return events
The response I get are twelve emails with subject and sender, and total count of my email.
Now I want iterate over emails changing the skip in query_params to get the next 12. Any method of how to iterate it using loops or recursion.

I'm thinking something along the lines of this:
def get_calendar_events(token):
graph_client = OAuth2Session(token=token)
# Configure query parameters to
# modify the results
json_list = []
ct = 0
while True:
query_params = {
#'$select': 'subject,organizer,start,end,location',
#'$orderby': 'createdDateTime DESC'
'$select': 'sender, subject',
'$skip': ct,
'$count': 'true'
}
# Send GET to /me/events
events = graph_client.get('{0}/me/messages'.format(graph_url), params=query_params)
events = events.json()
json_list.append(events)
ct += 12
# Return the JSON result
return json_list
May require some tweaking but essentially you're adding 12 to the offset each time as long as it doesn't return an error. Then it appends the json to a list and returns that.
If you know how many emails you have, you could also batch it that way.

How to send a personalised email to multiple recipients depending on a condition?

I am making a program which imports an excel file with multiple vehicle registrations and the emails of people the vehicles belong to. The program searches the MOT database using the MOT API and extracts the days left on each vehicle's MOT. I have set a condition where if the days left are less than 30 the program will send an email to the person as reminder. At the moment, my program sends a generic email to everyone on the list whose MOT runs out in 30 days. I would like to personalise the email similar to this example:
"The MOT for your vehicle with registration _______ expires in __ days!"
So the program would take the registration and the days left from my dataframe and send it to the corresponding individual. I assume this will require some sort of a for loop but I am not sure how to implement it. Any help is much appreciated. Here is my code:
apiKey = 'abcdxyz'
import requests
from datetime import datetime
import smtplib
import time
import pandas as pd
import pprint as pp
import xlrd
df = pd.read_excel(r'vehicles.xlsx') # Reading the excel file with the vehicle database
registrations = df['registration'].tolist() # Converting the registrations column to a list
mobile = df['mobile'].tolist() # Converting the mobile numbers column to a list
email = df['email'].tolist() # Converting the email column to a list
list_days_left = [] # Empty list to store data later
# Function to extract days left on MOT using the given dataframe:
def check_days_left(regi):
headers = {
'Accept': 'application/json',
'x-api-key': apiKey,
}
params = (
('registration', regi),
)
response = requests.get('https://beta.check-mot.service.gov.uk/trade/vehicles/mot-tests',
headers=headers, params=params)
r = response.json() # This returns a list of dicts of the full MOT History data
carDetails = r[0] # Access the first dictionary of the list
motTests = carDetails['motTests'] # This returns a list of dicts again of the MOT tests
latestTest = motTests[0] # Access the first dictionary to get the details of the latest test
expDate = latestTest['expiryDate'] # This returns the expiry date key of the latest test dictionary
uk_date = datetime.strptime(expDate, '%Y.%m.%d') # Convert expDate to datetime format
difference = uk_date - datetime.now() # Gets the timedelta between now and expiry date
days_left = difference.days # Extract the number of days from the above as an int
return days_left
# Function to apply the above function to each registration in the dataframe and append it to the empty list creater earlier:
def get_days_left():
for reg in registrations:
list_days_left.append(check_days_left(reg))
# print(list_days_left)
df['days_left'] = list_days_left
# print(df)
df.drop(df[df.days_left > 30].index, axis=0, inplace=True)
print(df)
return df
# Function to send emails:
def send_mail(m):
server = smtplib.SMTP('smtp.gmail.com', 587)
server.ehlo()
server.starttls()
server.ehlo()
server.login('myemail#gmail.com', 'mypassword')
subject = 'Your MOT is expiring soon!'
body = f"The MOT for your vehicle is running out soon!"
msg = f'Subject: {subject}\n\n{body}'
server.sendmail(
'myemail#gmail.com',
m,
msg
)
print('EMAIL HAS BEEN SENT!')
server.quit()
# Calling the function to return a dataframe with just those vehicles whose MOT runs out in 30 days:
get_days_left()
# Calling the function to send email to each person on the list:
for mailid in df['email']:
send_mail(mailid)

In the get_days_left method, in the for loop you could get the days left from the check_days_left function and do a check there. As for the infomation such as their registration, you already have it because you're passing the registration to the check_days_left method.
def get_days_left():
for reg in registrations:
days_left = check_days_left(reg)
# Assuming days_left is an integer
if days_left < 30:
send_mail("ownersemail#email.com", reg=reg, days_left=days_left)
def send_mail(ownerEmail, reg, days_left):
# Send the mail here.
server.sendmail("me#mymail.com", ownerEmail, f"Your vehicle with registration: {reg} has {days_left} days left!")
I apologise for any formatting errors or if my response isn't top-notch, I haven't posted here before.
Edit: Added how send_mail could look for clarity.

for vehicle in r:
# do work
Inside the loop you'll access all the variables and send the mail.

send multiple record to a particular user by mail

I need to send mail to particular user from the database. i have data table where 1000 of records are available. each record have some email ids. or some id has assigned to multiple record . i need to send record to the particular email only.
I have form from where i used to select company name and strt date. from these i fetch all the value into list. now worries is how to send record to particular user
let my code example
def sendmail():
form = SQLFORM.factory(Field('select_company', db.company, represent=lambda c, row:get_company_name(c), requires = IS_IN_DB(db,'company.id','%(company)s')),
Field('choose_start_date','date',requires=IS_NOT_EMPTY(error_message='Please choose Start Date')),
Field('choose_end_date','date',requires=IS_NOT_EMPTY(error_message='Please choose end Date')))
if form.accepts(request,session):
session.company = form.vars.select_company
session.strtdate = form.vars.choose_start_date
session.enddate = form.vars.choose_end_date
mailidr()
return dict(form=form)
def mailidr():
from gluon.tools import Mail,datetime
mail=Mail()
#specify server
mail=auth.settings.mailer
mail.settings.server='smtp.gmail.com:587'
mail.settings.login='comdummy09#gmail.com:critical#009'
#specify address to send as
mail.settings.sender='comdummy09#gmail.com'
#send the message
list = []
for row in db((db.tracker.company == session.company) & (db.tracker.Complience_Status != 1) & (db.tracker.periodicity_due_date >= session.strtdate) & (db.tracker.periodicity_due_date <= session.enddate)).select():
list.append('<tr><td style="padding-top:5px;padding-bottom:5px;padding-right:5px;padding-left:5px;">'+row.Compliance_Area+'</td><td style="padding-top:5px;padding-bottom:5px;padding-right:5px;padding-left:5px;">'+row.law_description+'</td><td style="padding-top:5px;padding-bottom:5px;padding-right:5px;padding-left:5px;" >'''+row.activity_name+'''</td><td style="padding-top:5px;padding-bottom:5px;padding-right:5px;padding-left:5px;">'+str(row.periodicity_due_date)+'</td></tr>')
mail.send(to=[row.client_owner],
subject='Compliance and Due_Date',
message='<html>'
'<body>'
'<h1>Test Mails</h1>'
'<h2>Dear \t' +session.company+ '</h2>'
'</br><h3>REMINDER</h3>'
'<p>Please note that the fol1ing records are due between ' +session.strtdate+ ' to ' +session.enddate+ '</p>'
'<table border=1>'
'<thead >'
'<tr>'
'<th style="padding-top:5px;padding-bottom:5px;padding-right:5px;padding-left:5px;">Compliance Area</th>'
'<th>record</th><th>date (Due Date)</th>'
'</tr>'
'</thead>'
'<tbody>'
+str(list)+
'</tbody>'
'</table>'
'</body>'
'</html>')
response.flash='mail send'
return ''
from this code i a able to send mail. but problem is that user get n no mails
if i used to take mail.send to out side the function its not work

from the docs (http://www.web2py.com/books/default/chapter/29/08/emails-and-sms#Setting-up-email)
Mail returns True if it succeeds in sending the email and False
otherwise. A complete argument list for mail.send() is as follows:
So, for a test you should try capturing the response - e.g. change:
mail.send(to=[row.client_owner],
to
mail_response = mail.send(to=[row.client_owner],
and then return locals() instead of "" so you can see in a test page what (if any) failure you are getting in the send. this is just for testing...
A permanent alternate method is shows here: i am trying send email using web2py with gmail and using smtp setting i have attached all code