My app serves multiple domains which I understand should be done by namespaces which I'm researching. Since multiple domains should have multiple analytics ID:s I get the analytics ID from the code but I want to make it even more configurable:
if os.environ.get('HTTP_HOST').endswith('.br') \
or os.environ['SERVER_NAME'].endswith('.br'):
data[u'analytics'] = 'UA-637933-12'
else:
data[u'analytics'] = 'UA-637933-18'
self.response.out.write(template.render(os.path.join(os.path.dirname(__file__),
'templates', name + '.html'), data))
The above sets analytics ID to ..-12 if it's my brazilian domain and to the other ID ...-18 if it is my dot com. But this is only for 2 domains and it's not easiliy generalizable. How can I achieve this function in a more scientific and scalable way so that it becomes easy to add my application to a domain without manually adding the domain to my application?
I suppose namespaces is the way to go here since the domains are google apps domains but I don't understand how to use namespaces:
def namespace_manager_default_namespace_for_request():
"""Determine which namespace is to be used for a request.
The value of _NAMESPACE_PICKER has the following effects:
If _USE_SERVER_NAME, we read server name
foo.guestbook-isv.appspot.com and set the namespace.
If _USE_GOOGLE_APPS_DOMAIN, we allow the namespace manager to infer
the namespace from the request.
If _USE_COOKIE, then the ISV might have a gateway page that sets a
cookie called 'namespace', and we set the namespace to the cookie's value
"""
name = None
if _NAMESPACE_PICKER == _USE_SERVER_NAME:
name = os.environ['SERVER_NAME']
elif _NAMESPACE_PICKER == _USE_GOOGLE_APPS_DOMAIN:
name = namespace_manager.google_apps_namespace()
elif _NAMESPACE_PICKER == _USE_COOKIE:
cookies = os.environ.get('HTTP_COOKIE', None)
if cookies:
name = Cookie.BaseCookie(cookies).get('namespace')
return name
I suppose I should use the namespace manager, get the namespace and set the analytics ID according to the namespace but how?
Thank you
The simplest way to do this is with a Python dict:
analytics_ids = {
'mydomain.br': 'UA-637933-12',
'mydomain.com': 'UA-637933-18',
}
data['analytics'] = analytics_ids[self.request.host]
If you have other per-domain stats, you may want to make each dictionary entry a tuple, a nested dict, or a configuration object of some sort, then fetch and store it against the current request for easy reference.
If you want to be able to reconfigure this at runtime, you could use a datastore model, but that will impose extra latency on requests that need to fetch it; it seems likely to me that redeploying each time you add a domain isn't likely to be a problem in your case.
Namespaces are tangential to what you're doing. They're a good way to divide up the rest of your data between different domains, but they're not useful for dividing up configuration data.
I presume you have two instances of the same application running.
Instead of fiddling with namespaces, I suggest you turn the Analytics ID into a configuration variable.
That is, either store it in a config file or a database your web is using. Then set one ID for each deployment (in each place your web is running from) and fetch it in the runtime.
For example:
Config file:
analyticsId="UA-637933-12"
Code:
data[u'analytics'] = getValueFromConfig("analyticsId")
where getValueFromConfig is a function you define to read the appropriate value. (To use configuration files effortlessly, you may use the ConfigParser module.)
Now you've gained a lot more flexibility - you don't have to do any checking and switching at runtime. You only have to define the value once per web site and be done with it.
Related
Is there a prebuilt way to include prefixes on resource names when you create them? I am looking for something similar to terraform, but I'm not sure if we need to create it programmatically...
In terraform I had something like:
variable "org" {
type = string
validation {
condition = length(var.org) <= 3
error_message = "The org variable cannot be larger than 3 characters."
}
}
variable "tenant" {
type = string
validation {
condition = length(var.tenant) <= 4
error_message = "The tenant variable cannot be larger than 4 characters."
}
}
variable "environment" {
type = string
validation {
condition = length(var.environment) <= 4
error_message = "The environment variable cannot be larger than 4 characters."
}
}
And I use the above variables to name an Azure resource group like:
module "resource_group_name" {
source = "gsoft-inc/naming/azurerm//modules/general/resource_group"
name = "main"
prefixes = [var.org, var.tenant, var.environment]
}
It is possible to do something similar in Pulumi? I saw a similar issue reported here, but it looks like this is more under programmatic control(?)
You could use Python's formatting functions directly, like
resource_group = = azure_native.resources.ResourceGroup("main",
location="eastus",
resource_group_name="{0}-{1}-{2}-{3}".format(org, tenant, environment, rgname))
You could also define a helper function and use it in multiple places.
Following the answer from #Mikhail Shilkov I created a helper function to give format to the name of an storage account resource on azure. But before I used the configuration of my dev stack at Pulumi.dev.yaml to read the values I want to assign to the name of the storage account.
Taking as a reference the way of setting and getting configuration values, I set it up the following values to be included in my dev stack:
pulumi config set org rhd
pulumi config set application wmlab
pulumi config set environment dev
As long those values are set, I can see them at the Pulumi.dev.yaml stack file: (* Pulumi give it the name of the project wmlab-infrastructure to those values)
config:
azure-native:location: westeurope # This one was set it up when creating the pulumi python project
wmlab-infrastructure:application: wmlab
wmlab-infrastructure:environment: dev
wmlab-infrastructure:org: rhd
Then from python I use Config.require to get the value by giving the key in this way:
def generate_storage_account_name(name: str, number: int, org: str, app: str, env: str):
return f"{name}{number}{org}{app}{env}"
config = pulumi.Config()
organization = config.require('org')
application = config.require('application')
environment = config.require('environment')
Then when creating the storage account name, I called the generate_storage_account_name helper function:
(* I am using random.randint(a,b) function to provide an integer value to the name of the storage account, it will make easier things when assigning it a name)
# Create an Azure Resource Group
resource_group = azure_native.resources.ResourceGroup(
'resource_group',
resource_group_name="{0}-{1}-{2}".format(organization, application, environment)
)
# Create an Azure resource (Storage Account)
account = storage.StorageAccount(
'main',
resource_group_name=resource_group.name,
account_name=generate_storage_account_name('sa', random.randint(1,100000), organization, application, environment),
sku=storage.SkuArgs(
name=storage.SkuName.STANDARD_LRS,
),
kind=storage.Kind.STORAGE_V2)
And it works. When creating the resources, the name of the storage account is using the helper function:
> pulumi up
Previewing update (rhdhv/dev)
View Live: https://app.pulumi.com/myorg/wmlab-infrastructure/dev/previews/549c2c34-853f-4fe0-b9f2-d5504525b073
Type Name Plan
+ pulumi:pulumi:Stack wmlab-infrastructure-dev create
+ ├─ azure-native:resources:ResourceGroup resource_group create
+ └─ azure-native:storage:StorageAccount main create
Resources:
+ 3 to create
Do you want to perform this update? details
+ pulumi:pulumi:Stack: (create)
[urn=urn:pulumi:dev::wmlab-infrastructure::pulumi:pulumi:Stack::wmlab-infrastructure-dev]
+ azure-native:resources:ResourceGroup: (create)
[urn=urn:pulumi:dev::wmlab-infrastructure::azure-native:resources:ResourceGroup::resource_group]
[provider=urn:pulumi:dev::wmlab-infrastructure::pulumi:providers:azure-native::default_1_29_0::04da6b54-80e4-46f7-96ec-b56ff0331ba9]
location : "westeurope"
resourceGroupName: "rhd-wmlab-dev"
+ azure-native:storage:StorageAccount: (create)
[urn=urn:pulumi:dev::wmlab-infrastructure::azure-native:storage:StorageAccount::main]
[provider=urn:pulumi:dev::wmlab-infrastructure::pulumi:providers:azure-native::default_1_29_0::04da6b54-80e4-46f7-96ec-b56ff0331ba9]
accountName : "sa99180rhdwmlabdev" # HERE THE NAME GENERATED
kind : "StorageV2"
location : "westeurope"
resourceGroupName: output<string>
sku : {
name: "Standard_LRS"
}
To read more about accessing config values from code read here
Pulumi has a way to autonaming resources, it is explained here,but alter this scheme looks like is not possible or at least is not recommended, doing it can cause some issues and resources will be recreated.
Overriding auto-naming makes your project susceptible to naming collisions. As a result, for resources that may need to be replaced, you should specify deleteBeforeReplace: true in the resource’s options. This option ensures that old resources are deleted before new ones are created, which will prevent those collisions.
If I understood well, I can override those auto-named resources which allow the name attribute on their API specification, but then when doing so is when naming collisions might be presented (?)
In my case I am using the StorageAccount resource on python azure API, and it does not allow override the property name so the helper function works well.
I want to access a property exist in the self.context using a variable. I have a variable name "prop" and it contains a value and it is already set in the self.context. I am using Flask Restplus framework.
prop = 'binding'
If I try to access this property like below then it gives me an error:
Object is not subscriptable
I want to know if there is any way to get the value? Like doing this:
print(self.context[prop])
I only get one solution don't know if its correct or not, I tried this :
self.context.__getattribute__(prop)
There are two ways to do this, the simplest is using getattr:
getattr(self.context, prop)
This function internally calls __getattribute__ so it's the same as your code, just a little neater.
However, you still have the problem that you probably only want to access some of the context, while not risking editing values set by different parts of your application. A much better way would be to store a dict in context:
self.context.attributes = {}
self.context.attributes["binding"] = False
print(self.context.attributes[prop])
This way, only certain context variables can be accessed and those that are meant to be dynamic don't mess with any used by your application code directly.
I'm using Notion to store my data {client name, subscription...} in tables.
I want to extract some data using python but I can't figure out how.
For example count the total number of clients, get the total amount of subscriptions...
Could you please suggest a way to help me.
If you need to do this only once - you can export a notion page (database) to HTML, which will probably be easier to extract from.
If you want this as a weekly/daily/monthly thing - I can't help with doing that in python but zapier and automate.io would be perfect.
For fetching the data you can use notion-client. The great thing about it is that it supports both sync and async interfaces. What it lacks, though, is an easy way to navigate the data's structure (which is quite complicated in Notion)
For that you can use basic-notion. It allows you to use model classes to easily access all the properties and attributes of your Notion objects - kind of like you would with an ORM.
In your case the code might look something like this:
from notion_client import Client
from basic_notion.query import Query
from basic_notion.page import NotionPage, NotionPageList
from basic_notion.field import SelectField, TitleField, NumberField
# First define models
class MyRow(NotionPage):
name = TitleField(property_name='Name')
subscription = SelectField(property_name='Subscription')
some_number = NumberField(property_name='Some Number')
# ... your other fields go here
# See your database's schema and the available field classes
# in basic_notion.field to define this correctly.
class MyData(NotionPageList[MyRow]):
ITEM_CLS = MyRow
# You need to create an integration and get an API token from Notion:
NOTION_TOKEN = '<your-notion-api-token>'
DATABASE_ID = '<your-database-ID>'
# Now you can fetch the data
def get_data(database_id: str) -> MyData:
client = Client(auth=NOTION_TOKEN)
data = client.databases.query(
**Query(database_id=database_id).filter(
# Some filter here
MyRow.name.filter.starts_with('John')
).sorts(
# You can sort it here
MyRow.name.sort.ascending
).serialize()
)
return MyData(data=data)
my_data = get_data()
for row in my_data.items():
print(f'{row.name.get_text()} - {row.some_number.number}')
# Do whatever else you may need to do
For more info, examples and docs see:
notion-client: https://github.com/ramnes/notion-sdk-py
basic-notion: https://github.com/altvod/basic-notion
Notion API Reference: https://developers.notion.com/reference/intro
I am trying to figure out what the 'Not tagged' AND 'Empty value' fields are for EC2 instances when filtering them. The function below works for Instances tagged Owner with a value of 'test'. However, the moment I try anything in place of 'test' (Null, None, Not tagged, Empty value, 0, etc...) the function either returns every instance or the function return none of the instances. I have gone through countless pages of documentation and any assistance is greatly appreciated. If I cannot get this to work, my next step will be to use Lambda, Tag Editor and Beautiful Soup to locate Specific Un-Tagged and Empty Instances and shut them down.
import boto3
import logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)
ec2 = boto3.resource('ec2')
def lambda_handler(event, context):
filters1 = [{'Name': 'instance-state-name',
'Values': ['running']}]
base = ec2.instances.filter(Filters=filters1)
filters2 = [{'Name': 'tag:Owner',
'Values': ['test']}]
instances = base.filter(Filters=filters2)
for instance in instances:
print(instance.id)
RunningInstances = [instance.id for instance in instances]
ec2.instances.filter(InstanceIds=RunningInstances).stop()
A better solution might use the concept of Events or reactive programming, instead of running such scan occasionally.
In AWS there are a few options to get relevant events to react to. It can be using CloudTrail that is keeping track of your API calls, or CloudWatch Events that is sending out events on changes in your infrastructure. But for this use case (tracking instances that are not tagged), you can use AWS Config Rules: https://aws.amazon.com/blogs/aws/aws-config-rules-dynamic-compliance-checking-for-cloud-resources/
There is a template in AWS Config that is ready for you to configure your rule, and define the reaction to the event.
Just use describe_instances() to list all the tags, running state and build control from there.
ec2 = boto3.client("ec2")
reservations = ec2.describe_instances()["Reservations"]
for each_item in reservations :
for instance in each_item["Instances"]:
print "{} {} {}".format(instance["InstanceId"], instance["State"]["Name"], instance["Tags"] )
Config Rules incur costs: $2 per active rule per month. So use it sparingly, you don't want to create too many redundant rules that get trigger.
I am facing similar situation. The best option i can see for now is.
Tag all instances with a custom tag : tagged-by:boto3
Filter all instances with this tag
Run a loop through the instances to see if this is the only tag avilable
if yes, delete the instance
I am still writing this in code, shall update the answer once its done.
Just to point out, if you choose the "(not tagged)" option in the Create Resource Group console screen, it lists nothing in the resulting resource group. From that point of view, it's just the same as using "(not tagged)" as one of the values in the AWS cli.
aws resource-groups create-group --name Non-Project-Linked-Resources --description "All resources NOT tagged with a ProjectNumber" --resource-query '{"Type":"TAG_FILTERS_1_0", "Query":"{\"ResourceTypeFilters\":[\"AWS::AllSupported\"],\"TagFilters\":[{\"Key\":\"ProjectNumber\",\"Values\":[\"\",\"(not tagged)\"]}]}"}'
has exactly the same effect as choosing ProjectNumber as the key and adding (empty value) and (not tagged) as the values. The consensus from all I have read is that server-side queries will not support searching for the absence of a tag.
I have an ICatalogTool, and catalog which I could query using AdvancedQuery and I want to learn how to use this tool, which queries I could use to find something in that Catalog.
I have an example of usage of this tool:
results = ICatalogTool(dc).search(query=Eq('id', self._object.ip))
# Eq - is an "EQUALS" in AdvancedQuery
# dc - instance of DeviceList class
# self._object.ip - some IP for search
I have read a documentation and found that each function like Eq takes some index. So I want to know which other indexes except 'id' are in my catalog. How to look for that? Are there some tools for introspection?
Look in the Zope Management Interface in the Indexes tab. Otherwise, you can list index names programmatically by calling the indexes() method of the catalog object.
IMHO, you should familiarize with the basic query interface (calling searchResults() method using queries specified as mappings) before attempting to use the AdvancedQuery add-on.