I have an ICatalogTool, and catalog which I could query using AdvancedQuery and I want to learn how to use this tool, which queries I could use to find something in that Catalog.
I have an example of usage of this tool:
results = ICatalogTool(dc).search(query=Eq('id', self._object.ip))
# Eq - is an "EQUALS" in AdvancedQuery
# dc - instance of DeviceList class
# self._object.ip - some IP for search
I have read a documentation and found that each function like Eq takes some index. So I want to know which other indexes except 'id' are in my catalog. How to look for that? Are there some tools for introspection?
Look in the Zope Management Interface in the Indexes tab. Otherwise, you can list index names programmatically by calling the indexes() method of the catalog object.
IMHO, you should familiarize with the basic query interface (calling searchResults() method using queries specified as mappings) before attempting to use the AdvancedQuery add-on.
Related
I am trying to update some queries in a web application because as stated in Flask-SQLAlchemy
You may see uses of Model.query or session.query to build queries. That query interface is
considered legacy in SQLAlchemy. Prefer using the session.execute(select(...)) instead.
I have a query:
subnets = db.session.query(Subnet).order_by(Subnet.id).all()
Which is translated into:
SELECT subnet.id AS subnet_id, subnet.name AS subnet_name, subnet.network AS subnet_network, subnet.access AS subnet_access, subnet.date_created AS subnet_date_created
FROM subnet ORDER BY subnet.id
And I take the subnets variable and loop it over in my view in two different locations. And it works.
However, when I try to update my query and use the new SQLAlchemy interface:
subnets = db.session.execute(db.select(Subnet).order_by(Subnet.id)).scalars()
I can only loop once and there is nothing left to loop over in the second loop?
How can I achieve the same result with the new query interface?
As noted in the comments to the question, your second example is not directly comparable to your first example because your second example is missing the .all() at the end.
Side note:
session.scalars(select(Subnet).order_by(Subnet.id)).all()
is a convenient shorthand for
session.execute(select(Subnet).order_by(Subnet.id)).scalars().all()
and is the recommended approach for SQLAlchemy 1.4+.
Check out the 2.0 migration docs for ORM:
https://docs.sqlalchemy.org/en/14/changelog/migration_20.html#migration-orm-usage
It lists some examples to show your how to migrate your code from 1.x style to 2.x style. For example:
get()
1.x:
session.query(User).get(42)
2.x:
session.get(User, 42)
all()
1.x:
session.query(User).all()
2.x:
session.execute(select(User)).scalars().all()
Note
For Flask-SQLAlchemy, you also need to:
Replace session with db.session.
Replace select with db.select or import the select directly from sqlalchemy:
from sqlalchemy import select
session.scalars() is not supported (yet), use session.execute().scalars() instead.
I'm using bulbflow (python) with Neo4j and I'm trying to add an index only on a subset of my keys (for now, simply keys named 'name' for optional index-based lookup).
I don't love the bulbflow Models (too restrictive) and I couldn't figure out how to do selective indexing without changing code since the 'autoindex' is a global setting -- I don't see how to configure it based on the key.
Has anyone done something like this?
-Andrew
You can disable Bulbs auto-indexing by setting g.config.autoindex to False.
See https://github.com/espeed/bulbs/blob/master/bulbs/config.py#L62
>>> from bulbs.neo4jserver import Graph
>>> g = Graph()
>>> g.config.autoindex = False
>>> g.vertices.create(name="James")
In the example above, this will cause the name property not to be indexed automatically.
Setting autoindex to False will switch to using the low-level client's create_vertex() method instead of the create_indexed_vertex() method:
See https://github.com/espeed/bulbs/blob/master/bulbs/neo4jserver/client.py#L422
The create_indexed_vertex() method has a keys arg, which you can use for selective indexing:
See https://github.com/espeed/bulbs/blob/master/bulbs/neo4jserver/client.py#L424
This is the low-level client method used by Bulbs models. You generally don't need to explicitly call the low-level client methods, but if you do, you can selectively index properties by including the property name in the keys arg.
To selectively index properties in a Model, simply override get_index_keys() in your Model definition:
See https://github.com/espeed/bulbs/blob/master/bulbs/model.py#L383
By default, Bulbs models index all properties. If no keys are provided, then all properties are indexed (like in TinkerPop/Blueprints).
See the Model _create() and get_bundle() methods:
_create() https://github.com/espeed/bulbs/blob/master/bulbs/model.py#L583
get_bundle() https://github.com/espeed/bulbs/blob/master/bulbs/model.py#L363
get_index_keys() https://github.com/espeed/bulbs/blob/master/bulbs/model.py#L383
To enable selective indexing for generic vertices and edges, I updated the Bulbs generic vertex/edge methods to include a _keys arg where you can supply a list of property names (keys) to index.
See https://github.com/espeed/bulbs/commit/4fe39d5a76675020286ec9aeaa8e71d58e3a432a
Now, to selectively index properties on generic vertices/edges, you can supply a list of property names to index:
>>> from bulbs.neo4jserver import Graph
>>> g = Graph()
>>> g.config.autoindex = False
>>> james = g.vertices.create(name="James", city="Dallas", _keys=["name"])
>>> julie = g.vertices.create(name="Julie", city="Dallas", _keys=["name"])
>>> g.edges.create(james, "knows", julie, timestamp=12345, someprop="somevalue", _keys=["someprop"])
In the example above, the name property will be indexed for each vertex, and someprop will be indexed for the edge. Note that city and timestamp will not be indexed because those property names were not explicitly included in the list of index keys.
If g.config.autoindex is True and _keys is None (the default), all properties will be indexed (just like before).
If g.config.autoindex is False and _keys is None, no properties will be indexed.
If _keys is explicitly set to a list of property names, only those properties will be indexed, regardless if g.config.autoindex is True or False.
See https://github.com/espeed/bulbs/blob/master/bulbs/neo4jserver/client.py#L422
NOTE: How auto-indexing works differs somewhat if you're using Neo4j Server, Rexster, or Titan Server, and the indexing architecture for all the graph-database servers has been in a state of flux for the past few months. It appears that all are moving from a manual-indexing system to auto-indexing.
For graph-database servers that did not have auto-indexing capability until recently (e.g. Neo4j Server), Bulbs enabled auto-indexing via custom Gremlin scripts that used the database's low-level manual indexing methods:
https://github.com/espeed/bulbs/blob/master/bulbs/neo4jserver/client.py#L1008
https://github.com/espeed/bulbs/blob/master/bulbs/neo4jserver/gremlin.groovy#L11
However, manual indexing has been deprecated among Neo4j Server, TinkerPop/Rexster, and Titan Server so Bulbs 0.4 indexing architecture will change accordingly. Selective indexing will still be possible by declaring your index keys upfront, like you would in an SQL create table statement.
BTW: What about did you find restrictive about Models? Bulbs Models (actually the entire library) is designed to be flexible so you can modify it to whatever you need.
See the Lightbulb example for how to customize Bulbs Models: Is there a equivalent to commit in bulbs framework for neo4j
Let me know if you have any questions.
I'm just using SQLAlchemy core, and cannot get the sql to allow me to add where clauses. I would like this very generic update code to work on all my tables. The intent is that this is part of a generic insert/update function that corresponds to every table. By doing it this way it allows for extremely brief test code and simple CLI utilities that can simply pass all args & options without the complexity of separate sub-commands for each table.
It'll take a few more tweaks to get it there, but should be doing the updates now just fine. However, while SQLAlchemy refers to generative queries it doesn't distinguish between selects & updates. I've reviewed SQLAlchemy documentation, Essential SQLAlchemy, stackoverflow, and several source code repositories, and have found nothing.
u = self._table.update()
non_key_kw = {}
for column in self._table.c:
if column.name in self._table.primary_key:
u.where(self._table.c[column.name] == kw[column.name])
else:
col_name = column.name
non_key_kw[column.name] = kw[column.name]
print u
result = u.execute(kw)
Which fails - it doesn't seem to recognize the where clause:
UPDATE struct SET year=?, month=?, day=?, distance=?, speed=?, slope=?, temp=?
FAIL
And I can't find any examples of building up an update in this way. Any recommendations?
the "where()" method is generative in that it returns a new Update() object. The old one is not modified:
u = u.where(...)
My app serves multiple domains which I understand should be done by namespaces which I'm researching. Since multiple domains should have multiple analytics ID:s I get the analytics ID from the code but I want to make it even more configurable:
if os.environ.get('HTTP_HOST').endswith('.br') \
or os.environ['SERVER_NAME'].endswith('.br'):
data[u'analytics'] = 'UA-637933-12'
else:
data[u'analytics'] = 'UA-637933-18'
self.response.out.write(template.render(os.path.join(os.path.dirname(__file__),
'templates', name + '.html'), data))
The above sets analytics ID to ..-12 if it's my brazilian domain and to the other ID ...-18 if it is my dot com. But this is only for 2 domains and it's not easiliy generalizable. How can I achieve this function in a more scientific and scalable way so that it becomes easy to add my application to a domain without manually adding the domain to my application?
I suppose namespaces is the way to go here since the domains are google apps domains but I don't understand how to use namespaces:
def namespace_manager_default_namespace_for_request():
"""Determine which namespace is to be used for a request.
The value of _NAMESPACE_PICKER has the following effects:
If _USE_SERVER_NAME, we read server name
foo.guestbook-isv.appspot.com and set the namespace.
If _USE_GOOGLE_APPS_DOMAIN, we allow the namespace manager to infer
the namespace from the request.
If _USE_COOKIE, then the ISV might have a gateway page that sets a
cookie called 'namespace', and we set the namespace to the cookie's value
"""
name = None
if _NAMESPACE_PICKER == _USE_SERVER_NAME:
name = os.environ['SERVER_NAME']
elif _NAMESPACE_PICKER == _USE_GOOGLE_APPS_DOMAIN:
name = namespace_manager.google_apps_namespace()
elif _NAMESPACE_PICKER == _USE_COOKIE:
cookies = os.environ.get('HTTP_COOKIE', None)
if cookies:
name = Cookie.BaseCookie(cookies).get('namespace')
return name
I suppose I should use the namespace manager, get the namespace and set the analytics ID according to the namespace but how?
Thank you
The simplest way to do this is with a Python dict:
analytics_ids = {
'mydomain.br': 'UA-637933-12',
'mydomain.com': 'UA-637933-18',
}
data['analytics'] = analytics_ids[self.request.host]
If you have other per-domain stats, you may want to make each dictionary entry a tuple, a nested dict, or a configuration object of some sort, then fetch and store it against the current request for easy reference.
If you want to be able to reconfigure this at runtime, you could use a datastore model, but that will impose extra latency on requests that need to fetch it; it seems likely to me that redeploying each time you add a domain isn't likely to be a problem in your case.
Namespaces are tangential to what you're doing. They're a good way to divide up the rest of your data between different domains, but they're not useful for dividing up configuration data.
I presume you have two instances of the same application running.
Instead of fiddling with namespaces, I suggest you turn the Analytics ID into a configuration variable.
That is, either store it in a config file or a database your web is using. Then set one ID for each deployment (in each place your web is running from) and fetch it in the runtime.
For example:
Config file:
analyticsId="UA-637933-12"
Code:
data[u'analytics'] = getValueFromConfig("analyticsId")
where getValueFromConfig is a function you define to read the appropriate value. (To use configuration files effortlessly, you may use the ConfigParser module.)
Now you've gained a lot more flexibility - you don't have to do any checking and switching at runtime. You only have to define the value once per web site and be done with it.
So im working with redis in a python app. any advice on key management? i try and keep all redis calls in one location but something seems off with hardcoding keys everywhere. tips?
I use a set of wrapper classes that handle key generation, something like:
public User Get(string username) {
return redis.Get("user:"+username);
}
I have an instance of each of these classes globally available, so just need to call
Server.Users.Get(username);
That's in .NET of course, but something similar should work in any language. An additional advantage of this approach over using a generic mapping tool is that it provides a good place to put things like connection sharing and indexing.
Well, key names are comparable to classes and attributes in your application, so some degree of 'repetition' will happen.
However, if you find you're repeating a lot of code to actually build your keys, then you might want to look at a mapper. In Python there's Redisco.
Or if you need something simpler, in Ruby there's Nest (porting it to Python should be trivial). It allows you to DRY up your references to keys:
users = Nest.new("User")
user = users[1]
# => "User:1"
user[:name]
# => "User:1:name"
redis.set(user[:name], "Foo")
# or even:
user[:name].set("Foo")
user[:name].get
# => "Foo"