I have a JSON POST data that a user is going to send me every time to fetch some data from a third party service.I plan to cache the data based on a scope id so that I don't keep inserting the data each time the user requests for something.Futhermore I am keeping a time stamp for each user request.Below is the POST data that user is going to send me everytime.
{
"scope_id": "user1",
"tool_id": "appdynamics",
"api_id": "get metrics",
"input_params": {"user": "myuser", "pwd": "mypwd", "acc_id": "myaccount", "app_id": "TestApp", "metric-path": "ars",
"time-range-type": "BEFORE_NOW", "duration-in-mins": 10},
"output_filters": {}
}
Below is the code snippet to handle the insertion of data
def post(self):
data = ServiceAPI.parser.parse_args()
print("First data", data)
scope_id = data["scope_id"]
tool_id = data["tool_id"]
api_id = data["api_id"]
input_params = data["input_params"]
output_filter = data["output_filter"]
if all([scope_id, tool_id, api_id]) and all(input_params.values()):
check_id = [j for i in users.find({}) for j in i if j == scope_id]
if check_id and check_id[0] == scope_id:
users.update({scope_id: [tool_id, api_id, input_params]},
{scope_id: [tool_id, api_id, input_params],
"timestamp": datetime.now().strftime('%Y-%m-%d %H:%M:%S')}, upsert=True)
else:
users.insert_one(
{scope_id: [tool_id, api_id, input_params],
"timestamp": datetime.now().strftime('%Y-%m-%d %H:%M:%S')})
Here the update statement works great if the user request is exactly the same as last time but makes a new entry if the user demands a new information for example in the POST request api_id = "get logs" when ideally it should have updated the user's data with the latest one.
For the first time when the user makes a POST request, below is the data that gets stored in my database
[{'user1': ['appdynamics', 'get metrics', {'pwd': 'mypwd', 'metric-path': 'ars', 'user': 'myuser', 'time-range-type': 'BEFORE_NOW', 'acc_id': 'myaccount', 'app_id': 'TestApp', 'duration-in-mins': 10}], 'timestamp': '2018-03-24 21:49:28', '_id': ObjectId('5ab67a901899db6d8a266558')}]
Now I make the same request again, it ensures no new entry is made since its made by the same scope id
However now if the user requests some new information for example
{
"scope_id": "user1",
"tool_id": "appdynamics",
"api_id": "get logs",
"input_params": {"user": "myuser", "pwd": "mypwd", "acc_id": "myaccount", "app_id": "TestApp", "metric-path": "ars",
"time-range-type": "BEFORE_NOW", "duration-in-mins": 10},
"output_filters": {}
}
Notice I have changed "api_id": "get logs", it makes a new entry instead of just modifying the existing data in my database.Here is the data now
[{'user1': ['appdynamics', 'get metrics', {'pwd': 'mypwd', 'metric-path': 'ars', 'user': 'myuser', 'time-range-type': 'BEFORE_NOW', 'acc_id': 'myaccount', 'app_id': 'TestApp', 'duration-in-mins': 10}], 'timestamp': '2018-03-24 21:49:28', '_id': ObjectId('5ab67a901899db6d8a266558')}, {'user1': ['appdynamics', 'get logs', {'pwd': 'mypwd', 'metric-path': 'ars', 'user': 'myuser', 'time-range-type': 'BEFORE_NOW', 'acc_id': 'myaccount', 'app_id': 'TestApp', 'duration-in-mins': 10}], 'timestamp': '2018-03-24 21:55:29', '_id': ObjectId('5ab67bf9089b16e9e77037f4')}]
So here the update seems to fail.What could be going wrong?
Note: This is a flask app and I suggest not to get into the details of the implementation.I just need to update the given data based on the scope id each time a user makes a request irrespective of whether it is the same request or a different one.
You are passing upsert=True to update(). Upsert tells MongoDB to update an existing document if one matching the query is found, insert a new document otherwise. The first parameter to update() is a query filter to find documents to apply the update to. The update query filter where api_id == "get logs" isn't matching any existing document, so a new document is being created.
Related
I am working on a program that includes a strawpoll. As far as I can tell this code should work to create a poll however it is returning an error page instead of a poll. Here is the api
json = json.dumps({"title": "Question", "options": ["option1", "option2", "option3"]})
poll = requests.post("http://strawpoll.me/api/v2/polls", data = json, headers = {"Content-Type": "application/json"})
This is the url it returns
https://www.strawpoll.me/error?aspxerrorpath=/api/v2/polls
Unfortunately, strawpoll.me' api have been broken for quite some time now. You can still retrieve the polls but creating one redirects you to the error landing page.
Try posting your request to https://www.strawpoll.me/api/v2/polls:
data = {"title": "Question", "options": ["option1", "option2", "option3"]}
poll = requests.post("https://www.strawpoll.me/api/v2/polls", json=data,
headers={"Content-Type": "application/json"})
print(poll.url)
# https://www.strawpoll.me/api/v2/polls
print(poll.json())
# {'multi': False, 'title': 'Question', 'votes': [0, 0, 0], 'id': 16578754,
# 'captcha': False, 'dupcheck': 'normal', 'options': ['option1', 'option2', 'option3']}
I'm querying the onename api in an effort to get the bitcoin addresses of all the users.
At the moment I'm getting all the user information as a json-esque list, and then piping the output to a file, it looks like this:
[{'0': {'owner_address': '1Q2Tv6f9vXbdoxRmGwNrHbjrrK4Hv6jCsz', 'zone_file': '{"avatar": {"url": "https://s3.amazonaws.com/kd4/111"}, "bitcoin": {"address": "1NmLvYVEZqPGeQNcgFS3DdghpoqaH4r5Xh"}, "cover": {"url": "https://s3.amazonaws.com/dx3/111"}, "facebook": {"proof": {"url": "https://facebook.com/jasondrake1978/posts/10152769170542776"}, "username": "jasondrake1978"}, "graph": {"url": "https://s3.amazonaws.com/grph/111"}, "location": {"formatted": "Mechanicsville, Va"}, "name": {"formatted": "Jason Drake"}, "twitter": {"username": "000001"}, "v": "0.2", "website": "http://1642.com"}', 'verifications': [{'proof_url': 'https://facebook.com/jasondrake1978/posts/10152769170542776', 'service': 'facebook', 'valid': False, 'identifier': 'jasondrake1978'}], 'profile': {'website': 'http://1642.com', 'cover': {'url': 'https://s3.amazonaws.com/dx3/111'}, 'facebook': {'proof': {'url': 'https://facebook.com/jasondrake1978/posts/10152769170542776'}, 'username': 'jasondrake1978'}, 'twitter': {'username': '000001'}, 'bitcoin': {'address': '1NmLvYVEZqPGeQNcgFS3DdghpoqaH4r5Xh'}, 'name': {'formatted': 'Jason Drake'}, 'graph': {'url': 'https://s3.amazonaws.com/grph/111'}, 'location': {'formatted': 'Mechanicsville, Va'}, 'avatar': {'url': 'https://s3.amazonaws.com/kd4/111'}, 'v': '0.2'}}}]
what I'm really interested in is the field {"address": "1NmLvYVEZqPGeQNcgFS3DdghpoqaH4r5Xh"}, the rest of the stuff I don't need, I just want the addresses of every user.
Is there a way that I can just write only the addresses to a file using python?
I'm trying to write it as something like:
1NmLvYVEZqPGeQNcgFS3DdghpoqaH4r5Xh,
1GA9RVZHuEE8zm4ooMTiqLicfnvymhzRVm,
1BJdMS9E5TUXxJcAvBriwvDoXmVeJfKiFV,
1NmLvYVEZqPGeQNcgFS3DdghpoqaH4r5Xh,
...
and so on.
I've tried a number of different ways using dump, dumps, etc. but I haven't yet been able to pin it down.
My code looks like this:
import os
import json
import requests
#import py2neo
import csv
# set up authentication parameters
#py2neo.authenticate("46.101.180.63:7474", "neo4j", "uni-bonn")
# Connect to graph and add constraints.
neo4jUrl = os.environ.get('NEO4J_URL',"http://46.101.180.63:7474/db/data/")
#graph = py2neo.Graph(neo4jUrl)
# Add uniqueness constraints.
#graph.run("CREATE CONSTRAINT ON (q:Person) ASSERT q.id IS UNIQUE;")
# Build URL.
apiUrl = "https://api.onename.com/v1/users"
# apiUrl = "https://raw.githubusercontent.com/s-matthew-english/26.04/master/test.json"
# Send GET request.
Allusersjson = requests.get(apiUrl, headers = {"accept":"application/json"}).json()
#print(json)])
UsersDetails=[]
for username in Allusersjson['usernames']:
usernamex= username[:-3]
apiUrl2="https://api.onename.com/v1/users/"+usernamex+"?app-id=demo-app-id&app-secret=demo-app-secret"
userinfo=requests.get(apiUrl2, headers = {"accept":"application/json"}).json()
# try:
# if('bitcoin' not in userinfo[usernamex]['profile']):
# continue
# else:
# UsersDetails.append(userinfo)
# except:
# continue
try:
address = userinfo[usernamex]["profile"]["bitcoin"]["address"]
UsersDetails.append(address)
except KeyError:
pass # no address
out = "\n".join(UsersDetails)
print(out)
open("out.csv", "w").write(out)
# f = csv.writer(open("test.csv", "wb+"))
# Build query.
query = """
RETURN {json}
"""
# Send Cypher query.
# py2neo.CypherQuery(graph, query).run(json=json)
# graph.run(query).run(json=json)
#graph.run(query,json=json)
anyway, in such a situation, what's the best way to write out those addresses as csv :/
UPDATE
I ran it, and at first it worked, but then I got the following error:
Instead of adding all the information to the UsersDetails list
UsersDetails.append(userinfo)
you can add just the relevant part (address)
try:
address = userinfo[usernamex]["profile"]["bitcoin"]["address"]
UsersDetails.append(address)
except KeyError:
pass # no address
except TypeError:
pass # illformed data
To print the values to the screen:
out = "\n".join(UsersDetails)
print(out)
(replace "\n" with "," for comma separated output, instead of one per line)
To save to a file:
open("out.csv", "w").write(out)
You need to reformat the list, either through map() or a list comprehension, to get it down to just the information you want. For example, if the top-level key used in the response from the api.onename.com API is always 0, you can do something like this
UsersAddresses = [user['0']['profile']['bitcoin']['address'] for user in UsersDetails]
I am using Django 1.9.5 and have a problem with the data that python social auth gives. I got the data, but I cant' access it. This is my code
def save_auth_data_session(request):
if request.user:
twitter = request.user.social_auth.get(provider ='twitter')
print twitter.extra_data['access_token']
The terminal gives me this error
""TypeError: string indices must be integers""
If I do a "print vars(twitter)", this the data I get
{'user_id': 123, 'uid': u'2413671427750', '_user_cache': <User: quintana>, '_state': <django.db.models.base.ModelState object at 0x7fa3d86fc2d0>, 'provider': u'twitter', 'extra_data': u'{"access_token": {"oauth_token_secret": "Afkasdasdas21zxbxcbn", "oauth_token": "2468cxcpqsldkenI56", "x_auth_expires": "0", "user_id": "455793463", "screen_name": "puertoRico"}, "id": 455793463}', 'id': 123}
As i can see from output of print vars(twitter) - extra_data stores serialized dictionary (string) 'extra_data': u'{...}'.
So you need to deserialize it first.
import json
twitter = request.user.social_auth.get(provider ='twitter')
extra_data = json.loads(twitter.extra_data) # will give you a python dictionary
print extra_data['access_token']
I have around 1000 unique mails with unique content. I want to bulk send them to mandril i.e. I dont want to make individual network calls for every message I am sending. The following document shows the api. But it does not define what I want to.
https://mandrillapp.com/api/docs/messages.python.html
What I want
mandrill_client = mandrill.Mandrill('YOUR_API_KEY')
messages ={ message1, message2 ....}
result = mandrill_client.messages.send(messages=messages, async=True, ip_pool='Main Pool', send_at='example send_at')
I could not find the above. But this is what I want.
Try this
message = {
#other details
'from_email': 'message.from_email#example.com',
'from_name': 'Example Name',
'to': [{'email': 'recipient.email#example.com',
'name': 'Recipient Name',
'type': 'to'},2nd reciepient,....],
}
result = mandrill_client.messages.send(messages=messages, async=True)
I have been reading through this document. Most of the document is based on finding an email's uid. From the article:
"The way this works is pretty simple: use the uid function, and pass in the string of the command in as the first argument. The rest behaves exactly the same.
result, data = mail.uid('search', None, "ALL") # search and return uids instead
latest_email_uid = data[0].split()[-1]
result, data = mail.uid('fetch', latest_email_uid, '(RFC822)')
raw_email = data[0][1]
I'm working with a django app called django-mailbox (http://django-mailbox.readthedocs.org/en/latest/index.html) the purpose of which is to consume emails.
The app creates a "Message" model that looks like:
u'django_mailbox.message': {
'Meta': {'object_name': 'Message'},
'body': ('django.db.models.fields.TextField', [], {}),
'encoded': ('django.db.models.fields.BooleanField', [], {'default': 'False'}),
'from_header': ('django.db.models.fields.CharField', [], {'max_length': '255'}),
u'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
'in_reply_to': ('django.db.models.fields.related.ForeignKey', [], {'blank': 'True', 'related_name': "'replies'", 'null': 'True', 'to': u"orm['django_mailbox.Message']"}),
'mailbox': ('django.db.models.fields.related.ForeignKey', [], {'related_name': "'messages'", 'to': u"orm['django_mailbox.Mailbox']"}),
'message_id': ('django.db.models.fields.CharField', [], {'max_length': '255'}),
'outgoing': ('django.db.models.fields.BooleanField', [], {'default': 'False'}),
'processed': ('django.db.models.fields.DateTimeField', [], {'auto_now_add': 'True', 'blank': 'True'}),
'read': ('django.db.models.fields.DateTimeField', [], {'default': 'None', 'null': 'True', 'blank': 'True'}),
'subject': ('django.db.models.fields.CharField', [], {'max_length': '255'}),
'to_header': ('django.db.models.fields.TextField', [], {})
using the python "email" library I can select a record from a django queryset and turn it into an email object:
qs = Message.objects.filter("my criteria")
first = qs[0]
one = first.get_email_object() // one is an email object
Does the existing data in the db have a uid, and if so how can I grab it.
The strict answer to your question is "no". The document you quote is about looping through an IMAP folder (in this case, a Gmail account), which will certainly get a unique ID (uid) from the server which tracks the unique message ID for each Email message.
Because you are constructing a mail message object using Django, you won't have such a UID. The "ID" field you do get from django.db.models.fields.AutoField is the sequential auto-increment ID that the Gmail/IMAP web page you quote says is "unacceptable".
You may want to look at the "uuid" library (http://docs.python.org/2/library/uuid.html) to generate unique ID values for your messages, but unless you also store those in your database, you'll be re-generating them over and over.
If you care to share more exact information about what you're trying to build (a web-based Email reader, perhaps?) then we as a community might have some better ideas for you.
you will get the uid of your mail in response
email_user = 'your gmail'
email_pass = 'your app password'
mail = imaplib.IMAP4_SSL('imap.gmail.com')
mail.login(email_user, email_pass)
mail.select('inbox')
status, response = mail.uid('search', None, r'(X-GM-RAW "subject:\"your latest mail subject\"")')
response = response[0].decode('utf-8').split()
response.reverse()
response = response[:min(10, len(response))]
print (response)