I have some custom flask methods in an eve app that need to communicate with a telnet device and return a result, but I also want to pre-populate data into some resources after retrieving data from this telnet device, like so:
#app.route("/get_vlan_description", methods=['POST'])
def get_vlan_description():
switch = prepare_switch(request)
result = dispatch_switch_command(switch, 'get_vlan_description')
# TODO: populate vlans resource with result data and return status
My settings.py looks like this:
SERVER_NAME = '127.0.0.1:5000'
DOMAIN = {
'vlans': {
'id': {
'type': 'integer',
'required': True,
'unique': True
},
'subnet': {
'type': 'string',
'required': True
},
'description': {
'type': 'boolean',
'default': False
}
}
}
I'm having trouble finding docs or source code for how to access a mongo resource directly and insert this data.
Have you looked into the on_insert hook? From the documentation:
When documents are about to be stored in the database, both on_insert(resource, documents) and on_insert_<resource>(documents) events are raised. Callback functions could hook into these events to arbitrarily add new fields, or edit existing ones. on_insert is raised on every resource being updated while on_insert_<resource> is raised when the <resource> endpoint has been hit with a POST request. In both circumstances, the event will be raised only if at least one document passed validation and is going to be inserted. documents is a list and only contains documents ready for insertion (payload documents that did not pass validation are not included).
So, if I get what you want to achieve, you could have something like this:
def telnet_service(resource, documents):
"""
fetch data from telnet device;
update 'documents' accordingly
"""
pass
app = Eve()
app.on_insert += telnet_service
if __name__ == "__main__":
app.run()
Note that this way you don't have to mess with the database directly as Eve will take care of that.
If you don't want to store the telnet data but only send it back along with the fetched documents, you can hook to on_fetch instead.
Lastly, if you really want to use the data layer you can use app.data.driveras seen in this example snippet.
use post_internal
Usage example:
from run import app
from eve.methods.post import post_internal
payload = {
"firstname": "Ray",
"lastname": "LaMontagne",
"role": ["contributor"]
}
with app.test_request_context():
x = post_internal('people', payload)
print(x)
Related
Campaign Monitor is a service where we can send emails to a set of subscribers. We can create multiple lists within Campaign Monitor and add the required users to these lists as subscribers(to whom we can send personalised emails). So, here I am trying to send a set of customers' details like their name, emails, total_bookings, and first_booking to the campaign monitor's list using the API in Python so that I can send emails to this set of users.
More details on campaign monitor: https://www.campaignmonitor.com/api/v3-3/subscribers/
I am new to using Campaign Monitor. I have searched documentation, a lot of posts and blogs for examples on how to push data with multiple custom fields to Campaign Monitor using Python. By default, a list in Campaign Monitor will have a name and an email that can be added, but I want to add other details for each subscriber(here I want to add total_bookings and first_booking data) and Campaign Monitor provides custom fields to achieve this.
For instance:
I have my data stored in a redshift table named customer_details with the fields name, email, total_bookings, first_booking. I was able to retrieve this data from redshift table using Python with the following code.
# Get the data from the above table:
cursor = connection.cursor()
cursor.execute("select * from customer_details")
creator_details = cursor.fetchall()
# Now I have the data as a list of sets in creator_details
Now I want to push this data to a list in the Campaign Monitor using API like request.put('https://api.createsend.com/api/../.../..'). But I am not sure on how to do this. Can someone please help me here.
400 indicated invalid parameters
we can first see the request is POST not PUT
so first change requests.put to requests.post
the next thing is that all the variables need to be sent either as www-formdata or as json body data not sure which
and lastly you almost certainly cannot verify with basic auth ... but maybe
something like the following
some_variables = some_values
...
header = {"Authorization": f"Bearer {MY_API_KEY}"}
data = {"email":customer_email,"CustomFields":[{"key":"total_bookings","value":customer_details2}]}
url = f'https://api.createsend.com/api/v3.3/subscribers/{my_list_id}.json'
res = requests.post(url,json=data,headers=header)
print(res.status_code)
try:
print(res.json())
except:
print(res.content)
after looking more into the API docs it looks like this is the expected request
{
"EmailAddress": "subscriber#example.com",
"Name": "New Subscriber",
"MobileNumber": "+5012398752",
"CustomFields": [
{
"Key": "website",
"Value": "http://example.com"
},
{
"Key": "interests",
"Value": "magic"
},
{
"Key": "interests",
"Value": "romantic walks"
}
],
"Resubscribe": true,
"RestartSubscriptionBasedAutoresponders": true,
"ConsentToTrack":"Yes"
}
which we can see has "EmailAddress" not "email" so you would need to do
data = {"EmailAddress":customer_email,"CustomFields":[{"key":"total_bookings","value":customer_details2}]}
Im not sure if all of the fields are required or not ... so you may also need to provide "Name","MobileNumber",Resubscribe",etc
and looking at "Getting Started" it looks like the publish a python package to make interfacing simpler
http://campaignmonitor.github.io/createsend-python/
which makes it as easy as
import createsend
cli = createsend.CreateSend({"api_key":MY_API_KEY})
cli.subscriber.add(list_id,"user#email.com","John Doe",custom_fields,True,"No")
(which I found here https://github.com/campaignmonitor/createsend-python/blob/master/test/test_subscriber.py#L70)
I am trying to make Python Eve create different collections programmatically,
Let's say I want to expose an endpoint receiving a schema to be able to create that collection in mongo:
i.e
DOMAIN = {}
app.route('/gen')
def gen(schema):
app.config['DOMAIN'][schema.name] = schema.def # <-- This obviously does not work, don't know how to focus it
So that via curl I could post this schema def:
curl -H "Content-Type: application/json" -d '[{"name":"test", "def": "{\"age\":\"int\"}"}]' http://localhost:5000/gen
And POST objects of this new collection(test) created
curl -H "Content-Type: application/json" -d '[{"age":5]' http://localhost:5000/test
Obviously this is just the initial problem. In order to persist it in the future I will need to save this data in mongo, and load it once application starts, so that "mongo autodefines python eve DOMAIN itself". I hope this will also be possible to achieve
My approach is to use custom settings for Eve() object:
app = eve.Eve(settings=settings)
where settings contains DOMAIN definition:
settings = {
"SERVER_NAME": None,
"DEBUG": True,
# MongoDB params
"MONGO_HOST": '...',
"MONGO_PORT": 27017,
"MONGO_USERNAME": '...',
"MONGO_PASSWORD": '...',
.....
# add to DOMAIN all collections defined in `schema`
"DOMAIN": {
# doc
'app_doc': {
'item_title': 'app_doc',
'resource_methods': ['GET', 'POST', 'DELETE'],
'allow_unknown': True,
'schema': {
'name': {'type': 'string', 'required': True},
....
}
The settings variable could be modified in order to receive parameters from database (I use a collection named app_schema where I keep these custom definitions for endpoints).
Just connect to Mongo (I use pymongo) before instantiation of Eve(), then fill settings["DOMAIN"] with all data from app_schema collection, then pass this settings variable to Eve(settings=settings). Exemple here:
# setup Mongo connection (#see config.py - store here default schemas and DOMAIN)
client = MongoClient(settings["MONGO_HOST"], settings["MONGO_PORT"])
db = client.docfill
# check app_schema collection
tab_schemas = db.app_schema.find({})
def load_settings_from_schema_collection():
"""
Defines a new settings DOMAIN variable loaded with metadata regarding "ent_"-collections
create other API endpoints by definitions found in schema
this is a huge workload, as it analyzes each schemadef and create endpoints for various operations
like GET/POST/DEL/PUT/PATCH and search
:return:
"""
i = 0
# add to `settings` new definitions from app_schema collection
global settings
# now parse each doc and create settings table for it, the insert settings table into DOMAIN definition
for doc in tab_schemas:
i = i + 1
# this name should be unique for each collection
this_collection_name = "ent_" + doc["collection"]
# only allow "ent_" prefixed schemas to be overridden
this_schema_setting = {
"datasource": {
"source": this_collection_name # important ca tabela sa fie definita cu prefix
},
"resource_methods": ['GET', 'POST', 'DELETE'],
"item_methods": ['GET', 'DELETE', 'PUT', 'PATCH'],
"schema": {}
}
for fld_meta in doc["content"]:
this_schema_setting["schema"][fld_meta] = {"type": doc["content"][fld_meta]["type"]}
# is there a required option ?
if "required" in doc["content"][fld_meta]:
this_schema_setting["schema"][fld_meta] = {"required": bool(doc["content"][fld_meta]["required"])}
settings["DOMAIN"][this_collection_name] = this_schema_setting
# output everything in settings variable to config.js (just for viewing what happens in settings)
file = "config.js"
with open(file, 'w') as filetowrite:
filetowrite.write('settings = ' + json.dumps(settings, indent=4, sort_keys=True))
return 1
# load settings from schema collections in MongoDB: collection=app_schema
load_settings_from_schema_collection()
And finally, start server:
app = eve.Eve(settings=settings)
app.run(...)
Hope it helps!
I'm trying to use the batch functionality for the Mailchimp API. My current set up is like this
operations = []
for idx, row in users_df.iterows():
payload = {
'email': row['email'],
'last_updated': row['new_time'],
'custom_value': row['new_value']
}
operation_item = {
"method": "POST", # I changed this to PUT
"path": '/lists/members/12345',
"body": json.dumps(payload),
}
operations.append(operation_item)
client = mc.MailChimp(MAILCHIMP_TOKEN, MAILCHIMP_USER)
batch = client.batches.create(data={"operations": operations})
Whenever I use the POST method I get this error: old_user#gmail.com is already a list member. Use PUT to insert or update list members.
But whenever I change my method to PUT, I get this other error: The requested method and resource are not compatible. See the Allow header for this resource's available methods.
Is there a way to overcome this? I have already looked at this and this is a similar problem in Ruby.
you can't do PUT to the list resource "/lists/members/12345", you need to change path to be "/lists/members/12345/{email}"
this code works for us:
def add_users_to_mailchimp_list(list_id, users):
operations = []
client = get_mailchimp_client()
for user in users:
member = {
'email_address': user.email,
'status_if_new': 'subscribed',
'merge_fields': {
'FNAME': user.first_name or '',
'LNAME': user.last_name or '',
},
}
operation_item = {
"method": "PUT",
"path": client.lists.members._build_path(list_id, 'members', user.email),
"body": json.dumps(member),
}
operations.append(operation_item)
return client.batches.create(data={"operations": operations})
using protected method client.lists.members._build_path is a bit hacky I guess, if you like you can build the url manually, it will be f'lists/{list_id}/members/{user.email}'
I am trying to write some test cases for some code I've developed using Elasticsearch and Django. The concept is straightforward - I just want to test a get request, which will be an Elasticsearch query. However, I am constructing the query as a nested dict. When I pass the nested dict to the Client object in the test script it gets passed through Django until it ends up at the urlencode function which doesn't look like it can handle nested dicts only MultivalueDicts. Any suggestions or solutions? I don't want to use any additional packages as I don't want to depend on potentially non-supported packages for this application.
Generic Code:
class MyViewTest(TestCase):
es_connection = elasticsearch.Elasticsearch("localhost:9200")
def test_es_query(self):
client = Client()
query = {
"query": {
"term": {
"city": "some city"
}
}
}
response = client.get("", query)
print(response)
Link for urlencode function: urlencode Django
The issue is clearly at the conditional statement when the urlencode function checks if the dictionary value is a str or bytes object. If it isn't it creates a generator object which can never access the nested portions of the dictionary.
EDIT: 07/25/2018
So I was able to come up with a temporary work around to at least run the test. However, it is ugly and I feel like there must be a better way. The first thing I tried was specifying the content_type and converting the dict to a json string first. However, Django still kicked back and error in the urlencode function.
class MyViewTest(TestCase):
es_connection = elasticsearch.Elasticsearch("localhost:9200")
def test_es_query(self):
client = Client()
query = {
"query": {
"term": {
"city": "some city"
}
}
}
response = client.get("", data=json.dumps(query), content_type="application/json")
print(response)
So instead I had to do:
class MyViewTest(TestCase):
es_connection = elasticsearch.Elasticsearch("localhost:9200")
def test_es_query(self):
client = Client()
query = {
"query": {
"term": {
"city": "some city"
}
}
}
query = json.dumps(query)
response = client.get("", data={"q": query}, content_type="application/json")
print(response)
This let me send the HttpRequest to my View and parse it back out using:
json.loads(request.GET["q"])
Then I was able to successfully get the requested data from Elasticsearch and return it as an HttpResponse. I feel like in Django though there has to be a way to just pass a json formatted string directly to the Client object's get function. I thought specifying the content_type as application/json would work but it still calls the urlencode function. Any ideas? I really don't want to implement this current system into production.
I am trying to make a cache flow, that on a user request it cache a big dict of 870 records and it should stay in cache for some time. When the defined time pass on next request the dict should be updated in na cache memory.
So I have created such a function:
from django.core.cache import get_cache
def update_values_mapping():
cache_en = get_cache('en')
values_dict = get_values_dict() <- this make a request to obtain the dict with values
cache_en.set_many(values_dict, 120) # 120s for testing
cache_en.set('expire', datetime.datetime.now() + datetime.timedelta(seconds=120))
Then in the second function I try to get values from cache
from django.core.cache import get_cache
def get_value_details(_id):
cache = get_cache('en')
details = cache.get(_id, {}) # Values in cache has expire date so they should eventually be gone
expire = cache.get('expire', None)
if not details and expire and expire < datetime.now():
update_values_mapping()
value = cache.get(_id, {})
return details
During rendering a view get_value_details() is called many times to obtain all needed values.
The problem is that some of the values are missing e.g. cache.get('b', {}) return {} even if the value 'b' was saved to cache (and expire date does not pass yet). The missing values are changing, sometimes it is 'a', sometimes 'b', other time 'c' etc.
I have been testing it on LocMemCache and DummyCache so far.
My example cache settings:
CACHES = {
'default': {
'BACKEND': 'django.core.cache.backends.locmem.LocMemCache',
'LOCATION': 'cache-default'
},
'en': {
'BACKEND': 'django.core.cache.backends.locmem.LocMemCache',
'LOCATION': 'cache-en'
},
'pl': {
'BACKEND': 'django.core.cache.backends.locmem.LocMemCache',
'LOCATION': 'cache-pl'
}
}
When I was playing with that in a console some of the values was disappearing from cache after next call of update_values_mapping(), but some were missing from the beginning.
Does anyone have any clue what it could be ?
Or maybe how to solve described flow in another way ?
LocMemCache is exactly that - a local memory cache. That means it's local to the particular server process, and won't be visible either in other processes or in the console.
If you need something that is shared across all processes, you should use a proper cache backend like memcached or redis.