tl;dr: If you manage hundreds of terraform files, is it worth writing a high level engine in a fully fledged programming language to manages those files ?
I spend a lot of time, editing, copying, and changing terraform files for infrastructure changes. Although the cicd used to make changes to the infrastructure, deploying a new release, uses cp, and seds to change terraform files and create new ones.
I find that a bit complicated to introduce changes to our cicd pipelines.
I thought of having a set of template that I could manage using a python engine, to create files, plan and apply, ...
Is it a good idea ? Do you have experience, or have you tried that too ?
Generic templating of Terraform HCL modules (.tf files) seems like an almost insurmountable goal and you would lose a lot of smart editor and IDE support in the process.
You would be better off moving the changing values into .tfvars or terragrunt.hcl files and sticking to you sed script or subscribing to Gruntwork and using their terraform-update-variable script which has a lot of testing and functionality that might prove useful.
Related
I have a lot of automated test cases with Robot Framework and, consequently, I have more and more keywords. It's a bit difficult for me to bring order.
My question is if I can include my keywords in a library. If this is possible, how can I do it?
Thank you.
Marta
This is how you create libraries - Creating test libraries.
However, moving keywords into a library will not bring order to your system. You will only move the disorder to another place.
Keeping your test scripts maintainable deals for the most part in having a certain structure to your work. This applies to Robot Framework as much as it does to any other language.
In Robot Framework we use Resource Files to store keywords that we want to reuse across multiple Test Case files. Thought these links you should be able to learn more on how to do this. You can import Resource Files in a Resource file, so chaining them.
As for what to put in these files that is often a personal preference. However, typically adhering to development principles like DRY, Separation of Concerns and most importantly Common Sense works best.
I'd advise adhering to principles instead of to fixed structure. Separate Data from Process logic, abstract the UI from your Process logic and model your process logic as close to Business Processes as possible.
As for converting keywords to Python Code. If the logic in your resource files means you use a lot of keywords for a specific feature automation, then perhaps this makes sense. But keep in mind that for maintainability you will then be more heavily dependent on the Python skills in your organisation.
So, I'm looking at writing an app with python2 django(-rest-framework), postgres and angular.
I'm aware there are lots of things that can be done
multi-server setup behind load balancer
DB replication/sharding?
caching (in various ways)
swapping DRF serialiser for serpy
running on python3
running on pypy
my question is - Which of these (or other things) should really be done right at the start of the project?
Write with scalability in mind.
scaliability is not limited just to production servers/environments but also to development environments.
Always write with scalability in mind.
At development
Scalability at development let you develop the product seemlessly.
Structure your repository
Use git branching models like GitFlow so that developers can work on parallel, or a single developer can switch working on different features. Use a bug tracker.
Design your apps.
Before actually writing a single line of code, write down what apps you are going to write. Design apps as to minimize Relations (ManyToMany, ForeignKey, etc..), imports. Django offers pulggable app architecture feel free to use it wisely.
Write your tests first.
This ensures that you can migrate(production environments), upgrade and downgrade with less pain and hairloss. Trust me writing tests feels so boring, but it is worth to.
Abstract models, managers
Use Abstract Models and Managers, it can eliminate bolier plate model code and help you maintain the code.
Name variables, classes and methods descriptive.
Name the variables, classes and methods descriptive, as you would be able to know what it represents without looking documentation.
Document code.
Feel free to document classes and methods, so you or other peers who look into the code get an idea of what it is indented for than stacktracing to see what a method is doing.
Use debug toolbar
Use django debug toolbar on developement, as you test your API, use prefectch_related() and select_related() to minimize/eliminate duplicated queries.
Modularize code.
Modularize the code. Python and django in general encourage use of modules. Modules are easy to manage. Use classes, more inheritance and abstract base classes to reuse code.
Use Continuous Integration
use continuous integration test your repo and make sure new pushes don't break the system.
At production.
scalability at production let you serve the product to infinite users seemlessly.
For multi server setup
Stick with rest design principles.
Eliminate sessions.
Use distributed cache like Redis.
Swapping DRF serializer for serpy
Start with serpy if you need more speed and if you are comfortable with. It is better to stick with Serpy than rewriting DRF serializer as writing both looks smiliar, but make sure you are not wasting time by optimizing for the lost 1 or 2ms.
Running on python3
Depends on the libraries that you plan to use.
Running on pypy
pypy is faster than the standard implementations. It depends on the library compatibility to use pypy. A list of compatibile package and status of compatibility.
now the question,
Which of these (or other things) should really be done right at the start of the project?
Ans: Developemet (1,2,3,4,5,6,7,8) Production(1,2)
I don't think you need to start worrying about the setup right away. I would discourage premature optimizations. Rather, run the app in production, profile it. See what affects the performance when you hit scale - you would know what's the bottleneck.
The first and main things you have to get right are a clean and correct db schema and clear, readable and correctly factored (DRY... unless it's accidental duplication) and decoupled code. If you know to design a relational DB schema and learn to use Python and Django properly you shouldn't have much problems so far, and if you get both these things right it will (well it should) be easy to scale - by adding cache where needed (Redis, Memcache, or an intermediary NoSQL document database storing "pre-processed" versions of your often accessed data), adding servers, load-balancing etc, depending on your application's needs. Django is built to scale easily, and unless you do stupid things it does scale easily.
I am going to write configuration tool for my Ubuntu based system. Next I would like to write frontends (text, GUI and web). But it is the most complicated project I wanted to write and I am not sure about general architecture I should use.
At the current I have functions and classes for changing system config. But these functions will probably grow & change. #Abki gave me advice how to write interface for frontends. I am going to make base classes for this interface but I don't know how to connect it with backend and next with frontends. Probably I should use design patterns like fasade, wrapper or something else.
It looks like (without interface_to_backend layer):
I don't care about UI and functions to change system config now. But I don't know how to write middle layer so It would be easy to connect it with the rest and extend functionality i the future.
I need general ideas, design patterns, advices how to implement this in Python.
I'm not sure this is entirely appropriate for SO but I'm intrigued and so I'll bite. As a rubyist I can't help much with the Python but here is some opinion on pattens from my experience.
My initial suggestion is you should review a few of the contenders out there. Specifically I'd be looking at cfengine, chef and bcfg2. They each tell a different story but if I'd summarise I'd say:
Chef has a lovely dsl syntax but is let down by a complicated architecture
bcfg2 is written in python but seems to have an annoying tendency to use XML :(
cfengine has the strongest theoretical underpinnings in promise theory (which is v.interesting BTW) but is C based.
Wikipedia also provides a pretty impressive list of configuration management tools that you will find useful.
In regard to designing your own tool I'd suggest there are three principles you want to pursue:
Simplicity, the simpler you make this the better. Simple in terms of scope, configuration and use are all important.
You'll need a single way to store data, you need to be able to trace the choices as they are made and not trample other people's changes (especially in a team environment).
Security, most configuration management tools need root privileges at some point. So you need to make sure that users can trust the code they're running.
You could use Fabric with Python as described in the article Ubuntu Server Setup with Python Fabric
The Wikipedia article at Comparison of open source configuration management software has several other tools that use Python to do this.
I like the approach taken by SALT.
If you write the GUI, text/CLI, and Web interfaces using Python, they can all use the same Python module. That way a change in one interface transparently affects the others. Plus all of those are in Python's area of strength.
Anyone know of any good ways to do puppet like pp files for python. Almost like config files that can inherit other configs files and also have limited logic in them like if statements and variables?
Thanks!
Kokki is an interesting project which attempts to be Chef / Puppet but for Python. It looks a little immature to use in production, but if you're looking for a Python-based configuration management system and don't want to use something more well established (and heavy-handed) like Bcfg2, I'd go with Kokki.
Consider using Python, with in-house rules limiting what can appear in a config file.
The advantages are that no one has to learn another language, it's really, really cheap to implement, and you don't have to worry about maintenance of the config parser. The disadvantages are that it's not a custom-designed language for your particular use-case, and that if you're not strict in limiting what syntax may be used you'll end up with mushrooming complexity in your config files.
This is a pretty good list of open source configuration management tools. Good Python implementations include Synctool, Salt, Pacha, and Bcfg2.
We have an in house developed web-based admin console that uses a combination of C CGI and Perl scripts to administer our mail server stack. Of late we have been thinking of cleaning up the code (well, replacing most of it), making the implementation more secure, and improving the overall behavior.
I don't have much programming knowledge, but I use Ruby on and off (mainly for writing erb templates), and hence was thinking of using ruby/rails for developing such an app (off-duty for now, I also need to learn stuff !).
Before blindly picking up a language though, what would you folks suggest ? Please let me know if this is too vague a question, I'll try to supply more information, if needed.
Have you considered writing your applications as Webmin modules?
You get a lot of stuff for free when you do so (users and groups, tons of security features, a pretty big variety of helper functions related to config files, and tons of existing code for most aspects of a UNIX/Linux system). You also get a lot of stuff for nearly free, like action logging, packages and updates via wbm or apt or yum, an online help system, etc.
There are some cons, as well. It's an old codebase, so it has some clunky bits in the API among other places. A lot of the old modules can be a bit hard to grok if you're not an old-school Perl programmer. But, it's a well-maintained codebase, and it's been banged on by millions of users for over a dozen years. It's pretty robust. The UI isn't beautiful, but it is relatively theme-able, and if you're distributing a minimized version it becomes easier to customize the UI.
I suspect you can be up and running a lot faster than starting from scratch or using most existing frameworks that aren't targeted specifically to building systems management interfaces the way Webmin is.
Also, it's BSD licensed, so you can do whatever you want with it, including building a custom commercial app with it (hundreds of companies have done so over the years).
If you already know a bit of ruby, then there's no reason not to use that.
If you're interested specifically in learning another language, then what you're trying to do could be done in pretty much any language/framework, it's just a matter of which one you want to learn.
Without knowing much about your existing application I'd say that this effectively boils down to "which language do you like to work with?".
Python and Ruby are both mature languages with ample library infrastructure. They also boast popular, similar web application frameworks namely Django and Ruby-on-Rails respectively.
Since you are porting an existing Perl app(lets) it may be worthwhile to note that Ruby is relatively more similar to Perl. Not surprising given that Ruby was influenced "primarily by Perl, Smalltalk, Eiffel and Lisp".
django has a nice admin interface