I'm currently writing a script to find emails based on the domain name using Hunter.io API
The thing is that my script returns me a JSON with a lot's a details and I only want the mail address in it.
Here's my code :
"""script to find emails based on the domain name using Hunter.io"""
import requests # To make get and post requests to APIs
import json # To deal with json responses to APIs
from pprint import pprint # To pretty print json info in a more readable format
from pyhunter import PyHunter # Using the hunter module
# Global variables
hunter_api_key = "API KEY" # API KEY
contacts = [] # list where we'll store our contacts
contact_info = {} # dictionary where we'll store each contact information
limit = 1 # Limit of emails adresses pulled by the request
value = "value" # <- seems to be the key in Hunter's API of the email adress founded
# TODO: Section A - Ask Hunter to find emails based on the domain name
def get_email_from_hunter(domain_name,limit):
url = "https://api.hunter.io/v2/domain-search"
params = {
"domain" : domain_name,
"limit" : 1,
"api_key" : hunter_api_key,
}
response = requests.get(url, params= params,)
json_data = response.json()
email_adress = json_data["data"]["emails"] # <- I have to find witch is the good key in order to return only the mail adress
#pprint(email_adress)
contact_info["email_adress"] = email_adress
contact_info["domain_name"] = domain_name
pprint(contact_info)
return contact_info
get_email_from_hunter("intercom.io","1")
and here's the JSON returned :
JSON exemple extracted from the documentation
Thanks per advance for the help provided :)
email_addresses = [item['value'] for item in json_data["data"]["emails"]]
note, not tested as you post image, not json data as text.
Related
I want to send Send direct message to specific user in ms teams through python with Odoo?
This is a little more complicated task, but definitely doable. Sending an individual message using python is not as easy as using Webhook to send the message in Teams. You have to go through Authentication, get your token and then send the chat. Here are the steps that I used to get this to work using selenium to get the Authentication and then using requests.post to send individual massage.
By the way if you thought getting authentication with Microsoft is easy you are 100% wrong, here is the link to getting access token from microsoft https://learn.microsoft.com/en-us/graph/auth-v2-user
After you register your app with Azure you can use the following cod to get the Authentication token using slenium.
Set up Selenium
import gc
gc.disable() # 03/23/2022 added this one to prevent trash collection and avoide crashing the notebooks
from selenium import webdriver
from selenium.webdriver.firefox.options import Options
from selenium.webdriver.firefox.service import Service
from selenium.common.exceptions import WebDriverException # to catch WebDriverException after changing the host from postman to local
Options.binary_location = "/usr/bin/firefox"
ser = Service("/tmp/geckodriver")
options = Options()
options.binary_location = "location for firefix bynary .. /firefox_binary.py"
options.headless = True
driver = webdriver.Firefox(options=options, service=ser)
Use the following code to go through Authentication. ( You need some selenium skills to code the manual process), I'll share the starting point as an example and you can code yours as it is different than the links that I am using.
import time
import re
import json
import requests
import pandas as pd
Options.binary_location = "/usr/bin/firefox"
ser = Service("/tmp/geckodriver")
options = Options()
options.headless = True
driver = webdriver.Firefox(options=options, service=ser)
tenant_id = 'your tenant id for Microsoft graph'
client_id = "your client id for Microsoft graph"
url = f"http://login.microsoftonline.com/{tenant_id}/oauth2/v2.0/authorize?client_id={client_id}&response_type=code&response_mode=query&scope=user.read%20chat.read&state=12345"
driver.get(url)
print(driver.current_url)
#open_tab
time.sleep(3)
This is the section that you are going to use your selenium skills, I just put the first step that is required by Microsoft, depending on your/your company setup you may need few more steps to sign in.
element_id = webdriver.common.by.By.ID
email_locator=(element_id,"i0116")
driver.find_element(*email_locator).send_keys("your email address")
next_button_locator = (element_id,"idSIButton9")
driver.find_element(*next_button_locator).click()
time.sleep(9)
print(driver.current_url)
I recommend printing driver.current_url after each steps so you can manually check the link and then code it accordingly.
After you logged in, now you can use the following code to get your Authentication token. I used localhost:5000 as return Url, but you can set up any URL that you want in your App registration page.
This authentication tokens are only valid for one hour, so I used the following while loop to refresh the token every 30 minutes
while True:
try:
driver.get(url)
except WebDriverException:
time.sleep(3)
url_pattern = 'http://localhost:5000/\?code=(?P<code>[^&]*)&state=12345.*'
re_match = re.match(url_pattern,driver.current_url)
code = re_match.group('code')
headers = { 'Content-Type': 'application/x-www-form-urlencoded'}
body = {'grant_type': 'authorization_code',
'code': code,
'redirect_url':'your redirect url',
'client_id': 'your client id',
'client_secret':'your client secret'}
response = requests.post('https://login.microsoftonline.com/yor tenant id/oauth2/v2.0/token',headers=headers,data = body)
access_token = json.loads(response.text)['access_token']
pd_response = pd.DataFrame([json.loads(response.text)])
# I am saving the new token in spark dataframe where I can read it with other code, but you can save it different way
sf_response = spark.createDataFrame(pd_response)
sf_response.write.mode("overwrite").saveAsTable('graphTest')
time.sleep(30*60)
Now if you got through all the obstacle that Microsoft throws at you to just authenticate yoursel, you can use the following line to post a message
#used this function to get updated token, but you can pass your token anyway that you want.
def getGraphAccessToken():
"""
Gets access token that is saved in graphTest table
"""
dataFrame = spark.sql("select * from graphTest")
return dataFrame.collect()[0]['access_token']
And Finally you can use the following codes to send the message to anyone or group of people as long as you have their chat id.
def sendIndividualChat(text,chatIdList,contentType="html"):
"""
sends individual chat through Microsoft Teams.
Parameters:
----------
text : str, message content
chatIdList : list, list of chat id(id is in string format) for individual chat messages
"""
headers = {'Content-type':'application/json',
"Authorization": f"Bearer {getGraphAccessToken()}"
}
body = {
"body": {
"contentType": contentType,
"content": text
}}
for chatId in chatIdList:
requests.post(f"https://graph.microsoft.com/v1.0/chats/{chatId}/messages",headers=headers,data =json.dumps(body) )
If you dont know how to get the chat id for the individuals or group chat there are few ways, you can use Graph Explorer https://developer.microsoft.com/en-us/graph/graph-explorer or the following code to get list of your most recent chat with their according chat id. Make sure you send the person/group that you want to get the chat id a message, so it shows up on recent chat messages.
def getCurrentChats():
"""
Gets list of current chats for current user.
Returns:
--------
Pandas DataFrame with information about the current chats
"""
headers = {'Content-type':'application/json',
"Authorization": f"Bearer {getGraphAccessToken()}"}
connection = http.client.HTTPSConnection("graph.microsoft.com")
connection.request("GET","/v1.0/me/chats",headers=headers)
resp = connection.getresponse()
text = resp.read()
return pd.DataFrame(json.loads(text)['value'])
def getCurrentChatMembers(currentChatsDataFrame,idColumn = 'id',debug=False):
"""
Get dictionary of member emails for each chat id
Parameters:
----------
currentChatsDataFrame : Pandas DataFrame returned from getCurrentChats()
idColumn : str, name of column with chat id (default= id)
Returns:
--------
Pandas DataFrame with ['memberName','memberEmail','chatId'] columns
"""
memberList = []
chatIdList = []
emailList = []
headers = {'Content-type':'application/json',
"Authorization": f"Bearer {getGraphAccessToken()}"}
connection = http.client.HTTPSConnection("graph.microsoft.com")
for chat_id in currentChatsDataFrame[idColumn]:
connection.request("GET",f"/v1.0/me/chats/{chat_id}/members",headers=headers)
resp = connection.getresponse()
text = resp.read()
#chatIdList.append(chat_id)
respJson = json.loads(text)['value']
dprint(respJson,debug=debug)
if respJson[1]['email'] =='your email address':# This returns the information about other chat member not the current user.
chatIdList.append(chat_id)
memberList.append(respJson[0]['displayName'])
emailList.append(respJson[0]['email'])
else:
chatIdList.append(chat_id)
memberList.append(respJson[1]['displayName'])
emailList.append(respJson[1]['email'])
dprint(text,debug=debug)
dprint('---------------------------')
#member_list.append(json.loads(text)['value'])
dprint(f"chatIdList is {chatIdList}",debug=debug)
dprint(f"memberlist is {memberList}",debug=debug)
dprint(f"emaillist is {emailList}",debug=debug)
return pd.DataFrame({'memberName':memberList,'memberEmail':emailList,'chatId': chatIdList})
Firstly :
Create a new channel in your Team if desired. A new channel will prevent Python generated notices from taking over other conversation channels.
img1
Add a new connector on your desired channel.
img 2
Find the Webhook connector and configure.
img 3
The configuration required is just a name for the webhook and optionally an image.
img4
img 5
Click create and copy the resulting webhook URL.
img6
Add this code your Python project so that it can write a message to Teams.
Install pymsteams with pip.
pip install pymsteams
Add this code to your Python project to enable writing messages to Teams, substitute the URL for your webhook:
import pymsteams
myTeamsMessage = pymsteams.connectorcard("INSERT WEBHOOK URL HERE")
Use this code to generate messages:
myTeamsMessage.text("This message was generated from Python!")
myTeamsMessage.send()
output
I am trying to access an API from this website. (https://www.eia.gov/opendata/qb.php?category=717234)
I am able to call the API but I am getting only headers. Not sure if I am doing correctly or any additions are needed.
Code:
import urllib
import requests
import urllib.request
locu_api = 'WebAPI'
def locu_search(query):
api_key = locu_api
url = 'https://api.eia.gov/category?api_key=' + api_key
locality = query.replace(' ', '%20')
response = urllib.request.urlopen(url).read()
json_obj = str(response, 'utf-8')
data = json.loads(json_obj)
When I try to print the results to see whats there in data:
data
I am getting only the headers in JSON output. Can any one help me figure out how to do extract the data instead of headers.
Avi!
Look, the data you posted seems to be an application/json response. I tried to reorganize your snippet a little bit so you could reuse it for other purposes later.
import requests
API_KEY = "insert_it_here"
def get_categories_data(api_key, category_id):
"""
Makes a request to gov API and returns its JSON response
as a python dict.
"""
host = "https://api.eia.gov/"
endpoint = "category"
url = f"{host}/{endpoint}"
qry_string_params = {"api_key": api_key, "category_id": category_id}
response = requests.post(url, params=qry_string_params)
return response.json()
print(get_categories_data(api_key=API_KEY, category_id="717234"))
As far as I can tell, the response contains some categories and their names. If that's not what you were expecting, maybe there's another endpoint that you should look for. I'm sure this snippet can help you if that's the case.
Side note: isn't your API key supposed to be private? Not sure if you should share that.
Update:
Thanks to Brad Solomon, I've changed the snippet to pass query string arguments to the requests.post function by using the params parameter which will take care of the URL encoding, if necessary.
You haven't presented all of the data. But what I see here is first a dict that associates category_id (a number) with a variable name. For example category_id 717252 is associated with variable name 'Import quantity'. Next I see a dict that associates category_id with a description, but you haven't presented the whole of that dict so 717252 does not appear. And after that I would expect to see a third dict, here entirely missing, associating a category_id with a value, something like {'category_id': 717252, 'value': 123.456}.
I think you are just unaccustomed to the way some APIs aggressively decompose their data into key/value pairs. Look more closely at the data. Can't help any further without being able to see the data for myself.
I am unable to retrieve tickets from service now by matching assignment_group. I am using pysnow and using table api to retrieve ticket from service now.
Below is my code :
import pysnow
c = pysnow.Client(instance='myinstance', user='username', password='password')
incident = c.resource(api_path='/table/incident')
#getting the assignment group of a ticket
print ("tickets are")
response = incident.get(query={'number': 'INC0010020'}, stream=True)
a = response['assignment_group']
print (a)
#using the same assigment group above to fetch more tickets
response = incident.get(query={'assignment_group' : a}, stream=True)
print (response.one())
I get the below result :
Try adding the following after line 2:
c.parameters.exclude_reference_link = True
your 'a' variable is a JSON object. Try using a.value
Also be mindful when using the responses. you may need to use JSON.parse if the response comes as a string
I am working a project, where in I am suppose to get some user input through web application, and send that data to cloudant DB. I am using python for the use case. Below is the sample code:
import requests
import json
dict_key ={}
key = frozenset(dict_key.items())
doc={
{
"_ID":"1",
"COORD1":"1,1",
"COORD2":"1,2",
"COORD3":"2,1",
"COORD4":"2,2",
"AREA":"1",
"ONAME":"abc",
"STYPE":"black",
"CROPNAME":"paddy",
"CROPPHASE":"initial",
"CROPSTARTDATE":"01-01-2017",
"CROPTYPE":"temp",
"CROPTITLE":"rice",
"HREADYDATE":"06-03-2017",
"CROPPRICE":"1000",
"WATERRQ":"1000",
"WATERSRC":"borewell"
}
}
auth = ('uid', 'pwd')
headers = {'Content-type': 'application/json'}
post_url = "server_IP".format(auth[0])
req = requests.put(post_url, auth=auth,headers=headers, data=json.dumps(doc))
#req = requests.get(post_url, auth=auth)
print json.dumps(req.json(), indent=1)
When I am running the code, I am getting the below error:
"WATERSRC":"borewell"
TypeError: unhashable type: 'dict'
I searched a bit, and found below stackflow link as a prospective resolution
TypeError: unhashable type: 'dict'
It says that "To use a dict as a key you need to turn it into something that may be hashed first. If the dict you wish to use as key consists of only immutable values, you can create a hashable representation of it like this:
key = frozenset(dict_key.items())"
I have below queries:
1) I have tried using it in my code above,but I am not sure if I have used it correctly.
2) To put the data in the cloudant DB, I am using Python module "requests". In the code, I am using the below line to put the data in the DB:
req = requests.put(post_url, auth=auth,headers=headers, data=json.dumps(doc))
But I am getting below error:
"reason": "Only GET,HEAD,POST allowed"
I searched on that as well, and I found IBM BLuemix document about it as follows
https://console.ng.bluemix.net/docs/services/Cloudant/basics/index.html#cloudant-basics
As I referred the document, I can say that I am using the right option. But may be I am wrong.
If you are adding a document to the database and you know the the _id, then you need to do an HTTP POST. Here's some slightly modified code:
import requests
import json
doc={
"_id":"2",
"COORD1":"1,1",
"COORD2":"1,2",
"COORD3":"2,1",
"COORD4":"2,2",
"AREA":"1",
"ONAME":"abc",
"STYPE":"black",
"CROPNAME":"paddy",
"CROPPHASE":"initial",
"CROPSTARTDATE":"01-01-2017",
"CROPTYPE":"temp",
"CROPTITLE":"rice",
"HREADYDATE":"06-03-2017",
"CROPPRICE":"1000",
"WATERRQ":"1000",
"WATERSRC":"borewell"
}
auth = ('admin', 'admin')
headers = {'Content-type': 'application/json'}
post_url = 'http://localhost:5984/mydb'
req = requests.post(post_url, auth=auth,headers=headers, data=json.dumps(doc))
print json.dumps(req.json(), indent=1)
Notice that
the _id field is supplied in the doc and is lower case
the request call is a POST not a PUT
the post_url contains the name of the database being written to - in this case mydb
N.B in the above example I am writing to local CouchDB, but replacing the URL with your Cloudant URL and adding correct credentials should get this working for you.
Basically i need to get all messages of a page using facebook SDK in python.
Following some tutorial i arrived to this point:
import facebook
def main():
cfg = {
"page_id" : "MY PAGE ID",
"access_token" : "LONG LIVE ACCESS TOKEN"
}
api = get_api(cfg)
msg = "Hre"
status = api.put_wall_post(msg) #used to post to wall message Hre
x = api.get_object('/'+str(MY PAGE ID)+"/conversations/") #Give actual conversations
def get_api(cfg):
graph = facebook.GraphAPI(cfg['access_token'])
resp = graph.get_object('me/accounts')
page_access_token = None
for page in resp['data']:
if page['id'] == cfg['page_id']:
page_access_token = page['access_token']
graph = facebook.GraphAPI(page_access_token)
return graph
if __name__ == "__main__":
main()
The first problem is that api.get_object('/'+str(MY PAGE ID)+"/conversations/")returns a dictionary containing many informations, but what i would like to see is the messages they sent to me, while for now it print the user id that sent to me a message.
The output look like the following:
{u'paging': {u'next': u'https://graph.facebook.com/v2.4/571499452991432/conversations?access_token=Token&limit=25&until=1441825848&__paging_token=enc_AdCqaKAP3e1NU9MGSsvSdzDPIIDtB2ZCe2hCYfk7ft5ZAjRhsuVEL7eFYOOCdQ8okvuhZA5iQWaYZBBbrZCRNW8uzWmgnKGl69KKt4catxZAvQYCus7gZDZD', u'previous': u'https://graph.facebook.com/v2.4/571499452991432/conversations?access_token=token&limit=25&since=1441825848&__paging_token=enc_AdCqaKAP3e1NU9MGSsvSdzDPIIDtB2ZCe2hCYfk7ft5ZAjRhsuVEL7eFYOOCdQ8okvuhZA5iQWaYZBBbrZCRNW8uzWmgnKGl69KKt4catxZAvQYCus7gZDZD&__previous=1'}, u'data': [{u'link': u'/communityticino/manager/messages/?mercurythreadid=user%3A1055476438&threadid=mid.1441825847634%3Af2e0247f54f5c4d222&folder=inbox', u'id': u't_mid.1441825847634:f2e0247f54f5c4d222', u'updated_time': u'2015-09-09T19:10:48+0000'}]}
which is basically paging and data.
Given this is there a way to read the conversation?
In order to get the messages content you need first to request the single messages in the conversation, accessible with the 'id' field in the dictionary you copied, result of
x = api.get_object('/'+str(MY PAGE ID)+"/conversations/") #Give actual conversations
you can request the messages in the conversation by calling
msg = api.get_object('/'+<message id>)
Here it gets tricky, because following the graph api documentation you should receive back a dictionary with ALL the possible fields, including the 'message' (content) field. The function however returns only the fields 'created_time' and 'id'.
Thanks to this other question Request fields in Python Facebook SDK I found that you can request for those fields by adding a dict with such fields specified in the arguments of the graph.get_object() function. As far as I know this is undocumented in the facebook sdk reference for python.
The correct code is
args = {'fields' : 'message'}
msg = api.get_object('/'+<message id>, **args)
Similar question: Read facebook messages using python sdk