Using inventory group variables in dynamic inventory script - python

This question is exactly like this stackoverflow question. And I am hoping I get some different answer with this one. I have been trying to achieve this for over a year. but can't seem to achieve it.
Stripped down version of my ansible directory looks like this:
[root#python-test ansible]# tree
.
├── files
├── inventory
│   ├── prod
│   │   ├── group_vars
│   │   │   └── all
│   │   └── hosts -> ../../scripts/inventory.py
│   └── staging
│   ├── group_vars
│   │   └── all
│   └── hosts -> ../../scripts/inventory.py
├── roles
│   ├── ant
│   ├── build
│   ├── jdk
│   └── python
├── scripts
│   └── inventory.py
├── templates
└── vars
└── all.yaml
I would like to use some of the variables declared in group_vars/all file. It has some endpoint details that I can use during the script execution. A striped down version of this group_vars/all looks like this:
inv_cloudprovider: aws
inv_environment: prod
inv_environment_type: production
inv_vpc_cidr: 10.0.0.0/16
inv_build_url: 'https://build.{{inv_environment}}.local'
inv_cloud_vpc_name: '{{inv_environment}}-vpc'
inv_vpc_id: '{{inv_environment_type}}-{{inv_cloud_vpc_name}}'
inv_glassfish_version: 4
At this point I am loading this file using yaml.safe_loads() and then using them in script execution. but problem with that is there are variables which are recursive jinja templates. and I am having hard time getting to make those variables work. So I was wondering if I can use ansible to do this for me.
I am using ansible==2.2.3.0 and python 2.7.
The closest I have been to achieving this in python by doing this:
from ansible.parsing.dataloader import DataLoader
from ansible.vars import VariableManager
from ansible.inventory import Inventory
inventory_path = '/root/ansible/inventory/prod/hosts'
inventory = Inventory(DataLoader(), VariableManager(), inventory_path)
group = inventory.get_group('all')
group_vars = group.get_vars()
Which is probably not the correct way because my script tries to execute itself and that goes on recursively.
Is it possible to parse group vars variable using ansible? If no, how do I best get final value of variables from that file?


The stripped down version of code that can do what this question asked is:
#!/usr/bin/env python
import json
import os
import sys
import yaml
from jinja2 import Environment
class VarLoader(yaml.SafeLoader):
#staticmethod
def construct_python_string(text):
return str(text.value)
def yaml_remove_parse(self, text):
self.construct_python_string(text)
env_name = os.environ.get('ENV_NAME')
script_path = os.path.dirname(os.path.abspath(__file__))
ansible_dir_path = os.path.abspath(script_path + '/../')
global_vars_path = ansible_dir_path + '/vars/all.yaml'
inventory_vars_path = ansible_dir_path + '/inventory/' + env_name + '/group_vars/all'
with open(global_vars_path) as f1, open(inventory_vars_path) as f2:
global_vars_data = f1.read()
inventory_vars_data = f2.read()
all_vars_data = global_vars_data + '\n' + inventory_vars_data
VarLoader.add_constructor('tag:yaml.org,2002:float', VarLoader.yaml_remove_parse)
data = yaml.load(all_vars_data, Loader=VarLoader)
jinja_env = Environment()
template_file = jinja_env.from_string(all_vars_data)
for key in data:
val = data[key]
if isinstance(val, str) and '{{' in val:
template = jinja_env.from_string(val)
data[key] = template.render(**data)
yaml_data = yaml.load(template_file.render(data), Loader=VarLoader)
print(json.dumps(yaml_data))
This works with both python3 and python2. Output:
[root#python-test ansible]# python scripts/inventory.py | jq .
{
"inv_environment": "prod",
"inv_environment_type": "production",
"inv_cloudprovider": "aws",
"gv_redis_port": 6379,
"inv_glassfish_version": 4,
"inv_vpc_id": "production-prod-vpc",
"gv_redis_version": "4.0.11",
"inv_build_url": "https://build.prod.local",
"gv_glassfish_admin_user": "admin",
"inv_vpc_cidr": "10.0.0.0/16",
"gv_glassfish_asadmin_path": "/usr/local/glassfish/bin/asadmin",
"gv_java_home": "/usr/local/java/default",
"inv_cloud_vpc_name": "prod-vpc",
"gv_tomcat_home": "/usr/local/tomcat",
"gv_java_path": "/usr/local/java/default/bin/java"
}

Related

Building Hierarchy Graph from Strings

I am trying to build a hierarchy graph from a list of strings I have. Each string just consists of its absolute hierarchy seperated by dots. Example Strings:
memberA.memberB.memberC
memberA.memberE.memberG
memberA.memberE
memberA.memberB
memberA.memberF.memberX
memberA.memberF
memberA.memberF.memberG #in this case this should be treated as a seperate leaf node and not the same as in memberA.memberE.memberG
I tried using Anytree and Treelib to achieve this but I could not come up with a working solution. Although this problem looks simple (might not be) I just can not figure it out.

You'd need to keep track of which node objects correspond to a certain path. For that you can use a dictionary, that maps a given path to a node object.
With AnyTree it could look like this:
from anytree import Node, RenderTree
strings = [
"memberA.memberB.memberC",
"memberA.memberE.memberG",
"memberA.memberE",
"memberA.memberB",
"memberA.memberF.memberX",
"memberA.memberF",
"memberA.memberF.memberG"
]
d = {}
root = Node("root")
for s in strings:
path = "root"
parent = root
for name in s.split("."):
path += "." + name
if path not in d:
d[path] = Node(name, parent=parent)
parent = d[path]
print(RenderTree(root))
Output:
Node('/root')
└── Node('/root/memberA')
├── Node('/root/memberA/memberB')
│ └── Node('/root/memberA/memberB/memberC')
├── Node('/root/memberA/memberE')
│ └── Node('/root/memberA/memberE/memberG')
└── Node('/root/memberA/memberF')
├── Node('/root/memberA/memberF/memberX')
└── Node('/root/memberA/memberF/memberG')
In case you want "memberA" to be the root, then you need to make sure your input data only has strings that start with "memberA". And then at the end of the above script do:
root = root.children[0]
root.parent = None
print(RenderTree(root))
Output:
Node('/memberA')
├── Node('/memberA/memberB')
│ └── Node('/memberA/memberB/memberC')
├── Node('/memberA/memberE')
│ └── Node('/memberA/memberE/memberG')
└── Node('/memberA/memberF')
├── Node('/memberA/memberF/memberX')
└── Node('/memberA/memberF/memberG')

Why does the "package" has a "package" in the Doxygen tree?

I generate the documentation of my Python code from the docstrings via Doxygen (1.9.1) in addition with doxypypy (git version from today).
My project is called Project and the packages name in it (which should be imported) is mypackage. When I look into the tree sidebar of the generated html it looks like this:
└── Project
   └── Packages
      └── Packages
└──mypackages
The two packages are linking to the same target: ../html/namespaces.html.
The files and folders are structured like this
Project
├── LICENSE
├── README.md
├── docs
└── ...
└── src
├── mypackage
│   ├── a.py
│   ├── __init__.py
│   └── _mypackage.py
├── setup.cfg
└── setup.py
The Doxyfile is located in Project/docs, doxygen is run in there and use ../src/mypackage as INPUT directory.
More details
__init__.py
__version__ = '0.0.1a'
from ._mypackage import *
_mypackage.py
# -*- coding: utf-8 -*-
"""Example __init__.py short.
Now some longer with multiple lines. Here
comes the scond line.
"""
def foo(bar):
"""
This is foo() in mypackage.
Args:
bar (str): A paramenter.
Returns:
(int): Fixed seven.
"""
print(bar)
return 7
a.py
# -*- coding: utf-8 -*-
"""This is mypackage.a
"""
import mypackage
def bar(bar):
"""
This is the function named bar.
The function calls `mypackage.foo()` and returns an 'a'.
Paramters:
bar (str): Just a parameter.
Returns:
str: Just an 'a'.
"""
mypackage.foo(bar)
return('a')
Some (maybe) related Doxyfile settings
BRIEF_MEMBER_DESC = YES
REPEAT_BRIEF = YES
ALWAYS_DETAILED_SEC = NO
FULL_PATH_NAMES = YES
JAVADOC_AUTOBRIEF = NO
PYTHON_DOCSTRING = YES
OPTIMIZE_OUTPUT_FOR_C = NO
OPTIMIZE_OUTPUT_JAVA = YES
OPTIMIZE_FOR_FORTRAN = NO
OPTIMIZE_OUTPUT_VHDL = NO
OPTIMIZE_OUTPUT_SLICE = NO
MARKDOWN_SUPPORT = YES
EXTRACT_ALL = YES
EXTRACT_PRIVATE = YES
EXTRACT_PACKAGE = YES
EXTRACT_STATIC = YES
EXTRACT_LOCAL_CLASSES = YES
EXTRACT_LOCAL_METHODS = YES
EXTRACT_ANON_NSPACES = NO
RESOLVE_UNNAMED_PARAMS = YES
HIDE_UNDOC_MEMBERS = NO
HIDE_UNDOC_CLASSES = NO
HIDE_FRIEND_COMPOUNDS = NO
HIDE_IN_BODY_DOCS = NO
INPUT = ../src/mypackage
FILE_PATTERNS =
RECURSIVE = YES
FILTER_PATTERNS = *.py=./py_filter
GENERATE_HTML = YES
GENERATE_TREEVIEW = YES
I opened an Issue about that.

ModuleNotFoundError no module named in Terminal on Mac

I am very new to python and still learning how it all works. I'm building a tiny card game for a python class and decided to build it using Sublime Text and Github to teach myself how it all works.
The (very small) directory looks like this:
/GitHub
├── python-practice
└── war
├── __init_.py
├── __main__.py
├── config
│   ├── __init__.py
│   ├── __pycache__
│   │   ├── card.cpython-38.pyc
│   │   ├── deck.cpython-38.pyc
│   │   └── player.cpython-38.pyc
│   ├── card.py
│   ├── card.pyc
│   ├── deck.py
│   ├── deck.pyc
│   ├── deck_test.py
│   └── player.py
└── game
├── __init__.py
└── logic.py
But no matter if I use the compiler in terminal or the system interpreter in sublime to run logic.py it always comes back:
Traceback (most recent call last):
File "game/logic.py", line 1, in <module>
import card, deck, player
ModuleNotFoundError: No module named 'card'
...even though it's clearly there. I've tried from config import card, deck, player and I've also tried import config.
It would be easy and simple to just put all my modules in one folder but I'm trying to teach myself directories so it feels important that I figure this out. How do I get my script to correctly import modules from another folder adjacent to it in the same directory? What am I missing?
edit: adding my code from the modules below.
card.py
values = {'Two':2, 'Three':3, 'Four':4, 'Five':5, 'Six':6, 'Seven':7, 'Eight':8, 'Nine':9, 'Ten':10, 'Jack':11, 'Queen':12, 'King':13, 'Ace':14}
suits = ('Hearts', 'Diamonds', 'Spades', 'Clubs')
ranks = ('Two','Three','Four','Five','Six','Seven','Eight','Nine','Ten','Jack','Queen','King','Ace')
class Card():
def __init__(self,suit,rank):
self.suit = suit
self.rank = rank
self.value = values[rank]
def __str__(self):
return self.rank+" of "+ self.suit
logic.py (unfinished, WIP)
from config.card import Card
from config.deck import Deck
from config.player import Player
#Game setup
player_one = player.Player("One")
player_two = player.Player("Two")
new_deck = deck.Deck()
new_deck.shuffle()
for x in range(26):
player_one.add_cards(new_deck.deal_one())
player_two.add_cards(new_deck.deal_one())
game_on = True
round_num = 0
while game_on:
if len(player_one.all_cards) == 0:
print('Player One, out of cards! Player Two Wins!')
game_on = False
break
elif len(player_two.all_cards) == 0:
print('Player Two, out of cards! Player One Wins!')
game_on = False
break
else:
round_num+=1
print(f'Round {round_num}')
#NEW ROUND
player_one_cards = []
player_one_cards.append(player_one.remove_one())
player_two_cards = []
player_two_cards.append(player_two.remove_one())
pass

Something like this should work:
import sys
from pathlib import Path
sys.path.append(str(Path('.').absolute().parent))
from config.card import Card
from config.deck import Deck
from config.player import Player
This basically allows you to import stuff from the parent directory, which is what you need.

Bazel: Reading a file with relative path to package, not workspace

Suppose we have a project like this:
project-path
├── root
│   ├── BUILD
│   ├── gen
│   │   ├── a2.txt
│   │   └── a.txt
│   └── use.py
└── WORKSPACE
And in use.py:
f = open("gen/a.txt", "r")
f2 = open("gen/a2.txt", "r")
print(f.read())
print(f2.read())
And BUILD:
py_binary(
name = "use",
srcs = ["use.py"],
data = ["gen/a.txt", "gen/a2.txt"],
)
when I bazel run root:use, it errors:
FileNotFoundError: [Errno 2] No such file or directory: 'gen/a.txt'
It expects paths relative to the WORKSPACE directory, not the current package (root/gen/a.txt here). But I want to access files relative to each package.

Add Skylib to your project, i.e. extend you WORKSPACE file:
WORKSPACE
load("#bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
http_archive(
name = "bazel_skylib",
sha256 = "74d544d96f4a5bb630d465ca8bbcfe231e3594e5aae57e1edbf17a6eb3ca2506",
urls = [
"https://mirror.bazel.build/github.com/bazelbuild/bazel-skylib/releases/download/1.3.0/bazel-skylib-1.3.0.tar.gz",
"https://github.com/bazelbuild/bazel-skylib/releases/download/1.3.0/bazel-skylib-1.3.0.tar.gz",
],
)
load("#bazel_skylib//:workspace.bzl", "bazel_skylib_workspace")
bazel_skylib_workspace()
Replace the path in the bash script using text replacement:
load("#bazel_skylib//rules:expand_template.bzl", "expand_template")
expand_template(
name = "modifiy_use_for_bazel",
out = "prepared_for_bazel_use.py",
substitutions = {
"gen/a.txt": "root/gen/a.txt",
"gen/a2.txt": "root/gen/a.txt",
},
template = "use.py",
)
py_binary(
name = "use",
main = "prepared_for_bazel_use.py",
srcs = ["prepared_for_bazel_use.py"],
data = [
"gen/a.txt",
"gen/a2.txt",
],
)
You can run now the script via bazel run root:use
Note: Tested with Bazel 6.0.0
To make this all a bit more convenient to use you could implement your own rule for it since this seems to be a common problem (e.g. when supporting two build systems at the same time where Bazel is not the primary build system and the other build system can cope with relative path names.)
A similar problem is described here: Change test execution directory in Bazel?

Building a QT project with WAF

I am trying to building my project using waf.The Project is complied properly but there is some link error.Its fails to link to with the Resources file (.qrc file)
main.cpp.1.o: In function main':
/home/sobingt/project/masterdetail/build/../main.cpp:20: undefined reference
toqInitResources_masterdetail()'
WAF output --zones=deps
sobingt#tuneb03-System-Product-Name:~/projects/masterdetail$ ./waf --zones=deps
Waf: Entering directory `/home/sobingt/projects/masterdetail/build'
10:49:34 deps deps for [/home/sobingt/projects/masterdetail/main.cpp]: [/home/sobingt/projects/masterdetail/mainwindow.h, /home/sobingt/projects/masterdetail/listmodel.h, /home/sobingt/projects/masterdetail/cache.h, /home/sobingt/projects/masterdetail/tablemodel.h, /home/sobingt/projects/masterdetail/mytablemodel.h]; unresolved ['QApplication', 'QPalette', 'QPixmap', 'QSplashScreen', 'qthread.h', 'boost/scoped_ptr.hpp', 'QMainWindow', 'QModelIndex', 'QAbstractListModel', 'QList', 'QVariant', 'tr1/unordered_map', 'QAbstractTableModel', 'QStringList', 'iostream', 'sstream', 'vector']
10:49:34 deps scanner for qxx: main.cpp -> build/main.cpp.1.o returned [/home/sobingt/projects/masterdetail/mainwindow.h, /home/sobingt/projects/masterdetail/listmodel.h, /home/sobingt/projects/masterdetail/cache.h, /home/sobingt/projects/masterdetail/tablemodel.h, /home/sobingt/projects/masterdetail/mytablemodel.h] ['QApplication', 'QPalette', 'QPixmap', 'QSplashScreen', 'qthread.h', 'boost/scoped_ptr.hpp', 'QMainWindow', 'QModelIndex', 'QAbstractListModel', 'QList', 'QVariant', 'tr1/unordered_map', 'QAbstractTableModel', 'QStringList', 'iostream', 'sstream', 'vector']
[14/15] qxx: main.cpp -> build/main.cpp.1.o
[15/15] cxxprogram: build/main.cpp.1.o build/mainwindow.cpp.1.o build/masterdetail_rc.o build/cache.cpp.1.o build/listmodel.cpp.1.o build/model.cpp.1.o build/mysortfilterproxymodel.cpp.1.o build/mytablemodel.cpp.1.o build/song.cpp.1.o build/songitem.cpp.1.o build/songs.cpp.1.o build/tablemodel.cpp.1.o build/tableproxymodel.cpp.1.o -> build/app
main.cpp.1.o: In function `main':
/home/sobingt/projects/masterdetail/build/../main.cpp:20: undefined reference to `qInitResources_masterdetail()'
main.cpp.1.o: In function `~MainWindow':
/home/sobingt/projects/masterdetail/build/../mainwindow.h:22: undefined reference to `vtable for MainWindow'
/home/sobingt/projects/masterdetail/build/../mainwindow.h:22: undefined reference to `vtable for MainWindow'
mainwindow.cpp.1.o: In function `MainWindow':
/home/sobingt/projects/masterdetail/build/../mainwindow.cpp:22: undefined reference to `vtable for MainWindow'
/home/sobingt/projects/masterdetail/build/../mainwindow.cpp:22: undefined reference to `vtable for MainWindow'
mainwindow.cpp.1.o: In function `ListItem':
/home/sobingt/projects/masterdetail/build/../listmodel.h:14: undefined reference to `vtable for ListItem'
mainwindow.cpp.1.o: In function `MainWindow::tr(char const*, char const*)':
/home/sobingt/projects/masterdetail/build/../mainwindow.h:24: undefined reference to `MainWindow::staticMetaObject'
mainwindow.cpp.1.o: In function `SongItem':
/home/sobingt/projects/masterdetail/build/../songitem.h:17: undefined reference to `vtable for SongItem'
listmodel.cpp.1.o: In function `ListModel':
/home/sobingt/projects/masterdetail/build/../listmodel.cpp:7: undefined reference to `vtable for ListModel'
listmodel.cpp.1.o: In function `~ListModel':
/home/sobingt/projects/masterdetail/build/../listmodel.cpp:27: undefined reference to `vtable for ListModel'
mysortfilterproxymodel.cpp.1.o: In function `MySortFilterProxyModel':
/home/sobingt/projects/masterdetail/build/../mysortfilterproxymodel.cpp:4: undefined reference to `vtable for MySortFilterProxyModel'
mytablemodel.cpp.1.o: In function `MyTableModel':
/home/sobingt/projects/masterdetail/build/../mytablemodel.cpp:3: undefined reference to `vtable for MyTableModel'
tablemodel.cpp.1.o: In function `TableModel':
/home/sobingt/projects/masterdetail/build/../tablemodel.cpp:9: undefined reference to `vtable for TableModel'
collect2: ld returned 1 exit status
Waf: Leaving directory `/home/sobingt/projects/masterdetail/build'
Build failed
-> task failed (exit status 1):
{task 14658768: cxxprogram main.cpp.1.o,mainwindow.cpp.1.o,masterdetail_rc.o,cache.cpp.1.o,listmodel.cpp.1.o,model.cpp.1.o,mysortfilterproxymodel.cpp.1.o,mytablemodel.cpp.1.o,song.cpp.1.o,songitem.cpp.1.o,songs.cpp.1.o,tablemodel.cpp.1.o,tableproxymodel.cpp.1.o -> app}
['/usr/bin/g++', 'main.cpp.1.o', 'mainwindow.cpp.1.o', 'masterdetail_rc.o', 'cache.cpp.1.o', 'listmodel.cpp.1.o', 'model.cpp.1.o', 'mysortfilterproxymodel.cpp.1.o', 'mytablemodel.cpp.1.o', 'song.cpp.1.o', 'songitem.cpp.1.o', 'songs.cpp.1.o', 'tablemodel.cpp.1.o', 'tableproxymodel.cpp.1.o', '-o', '/home/sobingt/projects/masterdetail/build/app', '-Wl,-Bstatic', '-Wl,-Bdynamic', '-lQtCore', '-lQtGui', '-lQtCore', '-lQtOpenGL', '-lQtGui', '-lQtCore', '-lQtSvg', '-lQtGui', '-lQtCore', '-lQtSql', '-lQtCore', '-lQtUiTools', '-lQtXml', '-lQtCore', '-lQtScript', '-lQtCore']
wscript file
VERSION='0.0.1'
APPNAME='qt4_test'
top = '.'
out = 'build'
def options(opt):
opt.load('compiler_cxx qt4')
def configure(conf):
conf.load('compiler_cxx qt4')
conf.env.append_value('CXXFLAGS', ['-g']) # test
def build(bld):
def build(bld):
bld(
features = 'qt4 cxx',
uselib = 'QTCORE QTGUI QTOPENGL QTSVG QWIDGET QTSQL QTUITOOLS QTSCRIPT',
includes = bld.env.INCLUDES_QTGUI,
source = 'mainwindow.cpp masterdetail.qrc cache.cpp listmodel.cpp model.cpp mysortfilterproxymodel.cpp mytablemodel.cpp song.cpp songitem.cpp songs.cpp tablemodel.cpp tableproxymodel.cpp',
target = 'mainwindow.o',
)
bld(
features = 'qt4 cxx cxxprogram',
uselib = 'QTCORE QTGUI QTOPENGL QTSVG QWIDGET QTSQL QTUITOOLS QTSCRIPT',
includes = bld.env.INCLUDES_QTGUI,
source = 'main.cpp',
target = 'app',
use = 'mainwindow.o',
)
tree structure
├── build
│   ├── c4che
│   │   ├── build.config.py
│   │   └── _cache.py
│   ├── config.log
│   ├── main.cpp.1.o
│   ├── masterdetail_rc.cpp
│   └── masterdetail_rc.o
├── cache.cpp
├── cache.h
├── images
│   ├── betaLogo.png
│   ├── exitButton.gif
│   ├── icon.png
│   ├── image.png
│   ├── mas.png
│   ├── play.png
│   ├── saveButton.png
│   ├── stopbutton.png
│   └── stop.png
├── listmodel.cpp
├── listmodel.h
├── main.cpp
├── mainwindow.cpp
├── mainwindow.h
├── Makefile
├── masterdetail64.png
├── masterdetail80.png
├── masterdetail.desktop
├── masterdetail_harmattan.desktop
├── masterdetail.pro
├── masterdetail.pro.user
├── masterdetail.qrc
├── masterdetail.svg
├── model.cpp
├── model.h
├── mysortfilterproxymodel.cpp
├── mysortfilterproxymodel.h
├── mytablemodel.cpp
├── mytablemodel.h
├── repository.db
├── song.cpp
├── song.h
├── songitem.cpp
├── songitem.h
├── songs.cpp
├── songs.h
├── tablemodel.cpp
├── tablemodel.h
├── tableproxymodel.cpp
├── tableproxymodel.h
├── tableUnit.cpp
├── waf
├── waf-1.6.11
├── wscript
I know it must be a stupid mistake.Please help.
Link to Question posted in qtcentre
I know it must be a stupid mistake.Please help. Link to Question posted in qtcentre
Solution i am trying out now:
I am running moc on all headers (or source) file containing a Q_OBJECT macro.
moc -o moc_something.cpp something.h
and then the moc_something.cpp is added to the sources being built.
I am trying for resource compiler:
rcc -o masterdetail.cpp masterdetail.qrc
and then the masterdetail.cpp has to be added to the sources being built.
I think the above is done by the qmake automatically..i am compiling it manually to try to find a proper solution

Add moc file at the end of the cpp file manually to all cpp file with a header i.e for mainwindow.cpp add
#include "mainwindow.moc"
and also include -DWAF = 1 in the compiler flag

It seems waf is not collecting moc files. According with waf docs, try adding something like this:
from waflib.TaskGen import feature, before_method, after_method
#feature('cxx')
#after_method('process_source')
#before_method('apply_incpaths')
def add_includes_paths(self):
incs = set(self.to_list(getattr(self, 'includes', '')))
for x in self.compiled_tasks:
incs.add(x.inputs[0].parent.path_from(self.path))
self.includes = list(incs)
process_source should take care of the moc files processing

Categories

Resources