I am trying to make Python Eve create different collections programmatically,
Let's say I want to expose an endpoint receiving a schema to be able to create that collection in mongo:
i.e
DOMAIN = {}
app.route('/gen')
def gen(schema):
app.config['DOMAIN'][schema.name] = schema.def # <-- This obviously does not work, don't know how to focus it
So that via curl I could post this schema def:
curl -H "Content-Type: application/json" -d '[{"name":"test", "def": "{\"age\":\"int\"}"}]' http://localhost:5000/gen
And POST objects of this new collection(test) created
curl -H "Content-Type: application/json" -d '[{"age":5]' http://localhost:5000/test
Obviously this is just the initial problem. In order to persist it in the future I will need to save this data in mongo, and load it once application starts, so that "mongo autodefines python eve DOMAIN itself". I hope this will also be possible to achieve
My approach is to use custom settings for Eve() object:
app = eve.Eve(settings=settings)
where settings contains DOMAIN definition:
settings = {
"SERVER_NAME": None,
"DEBUG": True,
# MongoDB params
"MONGO_HOST": '...',
"MONGO_PORT": 27017,
"MONGO_USERNAME": '...',
"MONGO_PASSWORD": '...',
.....
# add to DOMAIN all collections defined in `schema`
"DOMAIN": {
# doc
'app_doc': {
'item_title': 'app_doc',
'resource_methods': ['GET', 'POST', 'DELETE'],
'allow_unknown': True,
'schema': {
'name': {'type': 'string', 'required': True},
....
}
The settings variable could be modified in order to receive parameters from database (I use a collection named app_schema where I keep these custom definitions for endpoints).
Just connect to Mongo (I use pymongo) before instantiation of Eve(), then fill settings["DOMAIN"] with all data from app_schema collection, then pass this settings variable to Eve(settings=settings). Exemple here:
# setup Mongo connection (#see config.py - store here default schemas and DOMAIN)
client = MongoClient(settings["MONGO_HOST"], settings["MONGO_PORT"])
db = client.docfill
# check app_schema collection
tab_schemas = db.app_schema.find({})
def load_settings_from_schema_collection():
"""
Defines a new settings DOMAIN variable loaded with metadata regarding "ent_"-collections
create other API endpoints by definitions found in schema
this is a huge workload, as it analyzes each schemadef and create endpoints for various operations
like GET/POST/DEL/PUT/PATCH and search
:return:
"""
i = 0
# add to `settings` new definitions from app_schema collection
global settings
# now parse each doc and create settings table for it, the insert settings table into DOMAIN definition
for doc in tab_schemas:
i = i + 1
# this name should be unique for each collection
this_collection_name = "ent_" + doc["collection"]
# only allow "ent_" prefixed schemas to be overridden
this_schema_setting = {
"datasource": {
"source": this_collection_name # important ca tabela sa fie definita cu prefix
},
"resource_methods": ['GET', 'POST', 'DELETE'],
"item_methods": ['GET', 'DELETE', 'PUT', 'PATCH'],
"schema": {}
}
for fld_meta in doc["content"]:
this_schema_setting["schema"][fld_meta] = {"type": doc["content"][fld_meta]["type"]}
# is there a required option ?
if "required" in doc["content"][fld_meta]:
this_schema_setting["schema"][fld_meta] = {"required": bool(doc["content"][fld_meta]["required"])}
settings["DOMAIN"][this_collection_name] = this_schema_setting
# output everything in settings variable to config.js (just for viewing what happens in settings)
file = "config.js"
with open(file, 'w') as filetowrite:
filetowrite.write('settings = ' + json.dumps(settings, indent=4, sort_keys=True))
return 1
# load settings from schema collections in MongoDB: collection=app_schema
load_settings_from_schema_collection()
And finally, start server:
app = eve.Eve(settings=settings)
app.run(...)
Hope it helps!
Related
#db_task()
def test_db_access(tenant_id, batch_obj):
print('DBAccess')
print(tenant_id)
print(batch_obj.id)
files = File.objects.filter(batch_id=batch_obj.id)
print(files)
If I run this in django without django-huey, I get a filtered queryset but if I start using django-huey, I'm always getting an empty queryset. Only 'DBAccess' is getting printed and files is always '[]'.
Do I have to add other settings in settings.py?
This is my current huey settings
# Huey - Task Queue
HUEY = {
'name': 'appname',
'consumer': {
'workers': 4,
'worker_type': 'thread'
},
'immediate': False,
'connection': {
'host': RedisConfig.HOST,
'port': RedisConfig.PORT,
},
}
You're trying to pass an object as an argument to that function and it's probably not getting serialized - huey uses pickle to serialize function call data that is passed to the consumer. Instead, change your function to accept a batch_obj identifier like this:
#db_task()
def test_db_access(tenant_id, batch_obj_id):
print('DBAccess')
print(tenant_id)
print(batch_obj_id)
files = File.objects.filter(batch_id=batch_obj_id)
print(files)
and pass in batch_obj_id=batch_obj.id when you're calling test_db_access. Alternatively, you can write a custom serializer, but it should be much simpler to just pass the numerical identifier.
I'm trying to use the batch functionality for the Mailchimp API. My current set up is like this
operations = []
for idx, row in users_df.iterows():
payload = {
'email': row['email'],
'last_updated': row['new_time'],
'custom_value': row['new_value']
}
operation_item = {
"method": "POST", # I changed this to PUT
"path": '/lists/members/12345',
"body": json.dumps(payload),
}
operations.append(operation_item)
client = mc.MailChimp(MAILCHIMP_TOKEN, MAILCHIMP_USER)
batch = client.batches.create(data={"operations": operations})
Whenever I use the POST method I get this error: old_user#gmail.com is already a list member. Use PUT to insert or update list members.
But whenever I change my method to PUT, I get this other error: The requested method and resource are not compatible. See the Allow header for this resource's available methods.
Is there a way to overcome this? I have already looked at this and this is a similar problem in Ruby.
you can't do PUT to the list resource "/lists/members/12345", you need to change path to be "/lists/members/12345/{email}"
this code works for us:
def add_users_to_mailchimp_list(list_id, users):
operations = []
client = get_mailchimp_client()
for user in users:
member = {
'email_address': user.email,
'status_if_new': 'subscribed',
'merge_fields': {
'FNAME': user.first_name or '',
'LNAME': user.last_name or '',
},
}
operation_item = {
"method": "PUT",
"path": client.lists.members._build_path(list_id, 'members', user.email),
"body": json.dumps(member),
}
operations.append(operation_item)
return client.batches.create(data={"operations": operations})
using protected method client.lists.members._build_path is a bit hacky I guess, if you like you can build the url manually, it will be f'lists/{list_id}/members/{user.email}'
I'm building an API for a new web service using Python, Flask-Restful w/ pymongo.
A sample MongoDB document should look like this:
{ domain: 'foobar.com',
attributes: { web: [ akamai,
google-analytics,
drupal,
... ] } }
The imports:
from flask import Flask, jsonify
from flask.ext.restful import Api, Resource, reqparse
from pymongo import MongoClient
The class:
class AttributesAPI(Resource):
def __init__(self):
self.reqparse = reqparse.RequestParser()
self.reqparse.add_argument('domain', type = str, required = True, help = 'No domain given', location='json')
self.reqparse.add_argument('web', type = str, action='append', required = True, help = 'No array/list of web stuff given', location = 'json')
super(AttributesAPI, self).__init__()
def post(self):
args = self.reqparse.parse_args()
post = db.core.update( {'domain': args['domain']},
{'$set':{'attr': { 'web': args['web'] }}},
upsert=True)
return post
When I CURL post, I use this:
curl -i -H "Content-Type: application/json" -X POST -d '{"domain":"foobar", "web":"akamai", "web":"drupal", "web":"google-analytics"}' http://localhost:5000/v1/attributes
However, this is what gets saved in my document:
{ "_id" : ObjectId("5313a9006759a3e0af4e548a"), "attr" : { "web" : [ "google-analytics" ] }, "domain" : "foobar.com"}
It only stores the last value given in the curl for 'web'. I also tried to use the CLI command with multiple -d params as described in the reqparse documentation but that throws a 400 - BAD REQUEST error.
Any ideas how why it is only saving the last value instead of all values as a list?
In JSON objects and in Python dictionaries, names are unique; you cannot repeat the web key here and expect it to work. Use one web key instead and make the value a list:
{"domain": "foobar", "web": ["akamai", "drupal", "google-analytics"]}
and it should be processed as such.
In addition to #Martin Pieters answer, you would need to set your location parameter on your self.reqparse.add_argument to a tuple of json and values and the store parameter is append
self.reqparse.add_argument('domain',store='append', type = str, required = True, help = 'No domain given', location=('json','values'))
`
I have some custom flask methods in an eve app that need to communicate with a telnet device and return a result, but I also want to pre-populate data into some resources after retrieving data from this telnet device, like so:
#app.route("/get_vlan_description", methods=['POST'])
def get_vlan_description():
switch = prepare_switch(request)
result = dispatch_switch_command(switch, 'get_vlan_description')
# TODO: populate vlans resource with result data and return status
My settings.py looks like this:
SERVER_NAME = '127.0.0.1:5000'
DOMAIN = {
'vlans': {
'id': {
'type': 'integer',
'required': True,
'unique': True
},
'subnet': {
'type': 'string',
'required': True
},
'description': {
'type': 'boolean',
'default': False
}
}
}
I'm having trouble finding docs or source code for how to access a mongo resource directly and insert this data.
Have you looked into the on_insert hook? From the documentation:
When documents are about to be stored in the database, both on_insert(resource, documents) and on_insert_<resource>(documents) events are raised. Callback functions could hook into these events to arbitrarily add new fields, or edit existing ones. on_insert is raised on every resource being updated while on_insert_<resource> is raised when the <resource> endpoint has been hit with a POST request. In both circumstances, the event will be raised only if at least one document passed validation and is going to be inserted. documents is a list and only contains documents ready for insertion (payload documents that did not pass validation are not included).
So, if I get what you want to achieve, you could have something like this:
def telnet_service(resource, documents):
"""
fetch data from telnet device;
update 'documents' accordingly
"""
pass
app = Eve()
app.on_insert += telnet_service
if __name__ == "__main__":
app.run()
Note that this way you don't have to mess with the database directly as Eve will take care of that.
If you don't want to store the telnet data but only send it back along with the fetched documents, you can hook to on_fetch instead.
Lastly, if you really want to use the data layer you can use app.data.driveras seen in this example snippet.
use post_internal
Usage example:
from run import app
from eve.methods.post import post_internal
payload = {
"firstname": "Ray",
"lastname": "LaMontagne",
"role": ["contributor"]
}
with app.test_request_context():
x = post_internal('people', payload)
print(x)
I'd like to use a standalone instance of CherryPy to serve several domains from a single server. I'd like each domain to be served from a completely separate CherryPy application, each with its own configuration file.
I played with cherrypy.dispatch.VirtualHost, but it seems like separate configuration files aren't possible.
A similar question (here) suggests that this is quite difficult, but doesn't explain why and might have been due to the fact that no one answered the question.
This CherryPy recipe for multiple apps shows how to load multiple sandboxed apps with separate configuration files, but it looks like they are being served form the same domain.
I can understand that the answer might be, "use CherryPy as a WSGI server behind Nginx or Apache," but I'd rather only deal with CherryPy on this particular server.
In the same repo, there's vhost recipe. However it uses a shared app. I don't see the way to get cherrypy.dispatch.VirtualHost working with separately mounted apps. This is because cherrypy.serving.request.app is set before invocation of the dispatcher. Say you have the following.
hostmap = {
'api.domain.com' : '/app1',
'www.domain.com' : '/app2'
}
cherrypy.tree.mount(App1(), '/app1', appConfig1)
cherrypy.tree.mount(App2(), '/app2', appConfig2)
All what cherrypy.dispatch.VirtualHost does is prepending domain prefix to current url, e.g. requesting http://www.domain.com/foo will result in /app2/foo/ as internal path that is sent to a next dispatcher which is usually cherrypy.dispatch.Dispatcher. However the latter will try to find a page handler using current cherrypy.serving.request.app which is set to empty app because there's nothing in CherryPy tree that corresonded to /foo path. So it will find nothing.
All you need here is to replace prefixing to changing the current app. That is to say changing that line to this.
cherrypy.serving.request.app = cherrypy.tree.apps[prefix]
But because cherrypy.dispatch.VirtualHost is pretty small, you can rewrite in your code easily.
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import cherrypy
from cherrypy._cpdispatch import Dispatcher
config = {
'global' : {
'server.socket_host' : '127.0.0.1',
'server.socket_port' : 80,
'server.thread_pool' : 8
},
'hostmap' : {
'api.domain.com' : '/app1',
'www.domain.com' : '/app2'
}
}
appConfig1 = {
'/' : {
'tools.json_out.on' : True
}
}
appConfig2 = {
'/' : {
'tools.encode.encoding' : 'utf-8'
}
}
def VirtualHost(nextDispatcher = Dispatcher(), useXForwardedHost = True, **domains):
def dispatch(pathInfo):
request = cherrypy.serving.request
domain = request.headers.get('Host', '')
if useXForwardedHost:
domain = request.headers.get('X-Forwarded-Host', domain)
prefix = domains.get(domain, '')
if prefix:
request.app = cherrypy.tree.apps[prefix]
result = nextDispatcher(pathInfo)
# Touch up staticdir config. See
# https://bitbucket.org/cherrypy/cherrypy/issue/614.
section = request.config.get('tools.staticdir.section')
if section:
section = section[len(prefix):]
request.config['tools.staticdir.section'] = section
return result
return dispatch
class App1:
#cherrypy.expose
def index(self):
return {'bar': 42}
class App2:
#cherrypy.expose
def index(self):
return '<em>foo</em>'
if __name__ == '__main__':
config['/'] = {'request.dispatch': VirtualHost(**config['hostmap'])}
cherrypy.tree.mount(App1(), '/app1', appConfig1)
cherrypy.tree.mount(App2(), '/app2', appConfig2)
cherrypy.quickstart(config = config)