Develop Reference

Python wildcard search

I have a Lambda python function that I inherited which searches and reports on installed packages on EC2 instances. It pulls this information from SSM Inventory where the results are output to an S3 bucket. All of the installed packages have specific names until now. Now we need to report on Palo Alto Cortex XDR. The issue I'm facing is that this product includes the version number in the name and we have different versions installed. If I use the exact name (i.e. Cortex XDR 7.8.1.11343) I get reporting on that particular version but not others. I want to use a wild card to do this. I import regex (import re) on line 7 and then I change line 71 to xdr=line['Cortex*']) but it gives me the following error. I'm a bit new to Python and coding so any explanation as to what I'm doing wrong would be helpful.
File "/var/task/SoeSoftwareCompliance/RequiredSoftwareEmail.py", line 72, in build_html
xdr=line['Cortex*'])
import configparser
import logging
import csv
import json
from jinja2 import Template
import boto3
import re
# config
config = configparser.ConfigParser()
config.read("config.ini")
# logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)
# #TODO
# refactor common_csv_header so that we use one with variable
# so that we write all content to one template file.
def build_html(account=None,
ses_email_address=None,
recipient_email=None):
"""
:param recipient_email:
:param ses_email_address:
:param account:
"""
account_id = account["id"]
account_alias = account["alias"]
linux_ec2s = []
windows_ec2s = []
ec2s_not_in_ssm = []
excluded_ec2s = []
# linux ec2s html
with open(f"/tmp/{account_id}_linux_ec2s_required_software_report.csv", "r") as fp:
lines = csv.DictReader(fp)
for line in lines:
if line["platform-type"] == "Linux":
item = dict(id=line['instance-id'],
name=line['instance-name'],
ip=line['ip-address'],
ssm=line['amazon-ssm-agent'],
cw=line['amazon-cloudwatch-agent'],
ch=line['cloudhealth-agent'])
# skip compliant linux ec2s where are values are found
compliance_status = not all(item.values())
if compliance_status:
linux_ec2s.append(item)
# windows ec2s html
with open(f"/tmp/{account_id}_windows_ec2s_required_software_report.csv", "r") as fp:
lines = csv.DictReader(fp)
for line in lines:
if line["platform-type"] == "Windows":
item = dict(id=line['instance-id'],
name=line['instance-name'],
ip=line['ip-address'],
ssm=line['Amazon SSM Agent'],
cw=line['Amazon CloudWatch Agent'],
ch=line['CloudHealth Agent'],
mav=line['McAfee VirusScan Enterprise'],
trx=line['Trellix Agent'],
xdr=line['Cortex*'])
# skip compliant windows ec2s where are values are found
compliance_status = not all(item.values())
if compliance_status:
windows_ec2s.append(item)
# ec2s not found in ssm
with open(f"/tmp/{account_id}_ec2s_not_in_ssm.csv", "r") as fp:
lines = csv.DictReader(fp)
for line in lines:
item = dict(name=line['instance-name'],
id=line['instance-id'],
ip=line['ip-address'],
pg=line['patch-group'])
ec2s_not_in_ssm.append(item)
# display or hide excluded ec2s from report
display_excluded_ec2s_in_report = json.loads(config.get("settings", "display_excluded_ec2s_in_report"))
if display_excluded_ec2s_in_report == "true":
with open(f"/tmp/{account_id}_excluded_from_compliance.csv", "r") as fp:
lines = csv.DictReader(fp)
for line in lines:
item = dict(id=line['instance-id'],
name=line['instance-name'],
pg=line['patch-group'])
excluded_ec2s.append(item)
# pass data to html template
with open('templates/email.html') as file:
template = Template(file.read())
# pass parameters to template renderer
html = template.render(
linux_ec2s=linux_ec2s,
windows_ec2s=windows_ec2s,
ec2s_not_in_ssm=ec2s_not_in_ssm,
excluded_ec2s=excluded_ec2s,
account_id=account_id,
account_alias=account_alias)
# consolidated html with multiple tables
tables_html_code = html
client = boto3.client('ses')
client.send_email(
Destination={
'ToAddresses': [recipient_email],
},
Message={
'Body': {
'Html':
{'Data': tables_html_code}
},
'Subject': {
'Charset': 'UTF-8',
'Data': f'SOE | Software Compliance | {account_alias}',
},
},
Source=ses_email_address,
)
print(tables_html_code)

If I understand your problem correctly, you are getting a KeyError exception because Python does not support wildcards out of the box. A csv.DictReader creates a standard Python dictionary for each row in csv. Python's dictionary is just an associative array without pattern matching.
You can implement this by regex, though. If you have a dictionary line and you don't know the full name of a key you are looking for, you can solve it by re.search function.
line = {'Cortex XDR 7.8.1.11343': 'Some value you are looking for'}
val = next(v for k, v in line.items() if re.search('Cortex.+', k))
print(val) # 'Some value you are looking for'
Be aware that this assumes that a line dictionary contains at least one item that matches the 'Cortex.+' pattern and returns the first match. You would have to refactor this a bit to change this.

1. import os - missing in the code
2. def build_html(account=None -> When the account is pass with Nonetype and below error will thrown in account["id"] and account["alias"].
Ex:
Traceback (most recent call last):
File "C:\Users\pro\Documents\project\pywilds.py", line 134, in <module>
build_html(account=None)
File "C:\Users\pro\Documents\project\pywilds.py", line 33, in build_html
account_id = account["id"]
TypeError: 'NoneType' object is not subscriptable
I hope it helps..

Converting the comma separated values to Python dictionary

I am getting the XML data in below format
<?xml version="1.0"?>
<localPluginManager>
<plugin>
<longName>Plugin Usage - Plugin</longName>
<pinned>false</pinned>
<shortName>plugin-usage-plugin</shortName>
<version>0.3</version>
</plugin>
<plugin>
<longName>Matrix Project Plugin</longName>
<pinned>false</pinned>
<shortName>matrix-project</shortName>
<version>4.5</version>
</plugin>
</localPluginManager>
Used below program to fetch the "longName" and "version" from the XML
import xml.etree.ElementTree as ET
import requests
import sys
response = requests.get(<url1>,stream=True)
response.raw.decode_content = True
tree = ET.parse(response.raw)
root = tree.getroot()
for plugin in root.findall('plugin'):
longName = plugin.find('longName').text
shortName = plugin.find('shortName').text
version = plugin.find('version').text
master01 = longName, version
print (master01,version)
Which gives me below output which I want to convert in dictionary format to process further
('Plugin Usage - Plugin', '0.3')
('Matrix Project Plugin', '4.5')
Expected Output -
dictionary = {"Plugin Usage - Plugin": "0.3", "Matrix Project Plugin": "4.5"}

import xml.etree.ElementTree as ET
import requests
import sys
response = requests.get(<url1>,stream=True)
response.raw.decode_content = True
tree = ET.parse(response.raw)
root = tree.getroot()
mydict = {}
for plugin in root.findall('plugin'):
longName = plugin.find('longName').text
shortName = plugin.find('shortName').text
version = plugin.find('version').text
master01 = longName, version
print (master01,version)
mydict[longName]=version

I think you should create a dictionary at the beginning:
my_dict = {}
then assign values to this dictionary in the loop:
my_dict[longName] = version

Assuming you have all your tuples stored in a list, you can iterate over it like this:
tuple_list = [('Plugin Usage - Plugin', '0.3'), ('Matrix Project Plugin', '4.5')]
dictionary = {}
for item in tuple_list:
dictionary[item[0]] = item[1]
Or, in Python 3, a dict comprehension instead.

It's very simple, actually. First, you initialize the dictionary before the loop, then you add key-value pairs as you get them:
dictionary = {}
for plugin in root.findall('plugin'):
...
dictionary[longName] = version # In place of the print call

How to parse ini files and choose appropriate function with python

I try to use an ini file to configure the resolution to use in my script and need help to know how to do this.
"Fontion script":
#RECUP QUALITE FHD
import re, os
def FHD(RFHD):
mykey = open("/home/gaaara/adn/tmp/ajax.json", "r")
for text in mykey:
match = re.search('"FHD":"(.+?).mp4', text)
if match:
s = 'http://www.website.fr:1935/' + match.group(1) + '.mp4?audioindex=0.smil'
return s
In fact it has 2 other similar functions in the file HD and SD which are the others function of resolution. How do I programmatically select the right function?
Edit
import ConfigParser
import sys
sys.path.append('files/')
from xrez import FHD
from xrez import HD
from xrez import SD
#variables
x1080 = FHD('RFHD')
x720 = HD('RHD')
x480 = SD('RSD')
#fin
config = ConfigParser.ConfigParser()
config.read('config.ini')
try:
val = config.get('resolution', 'Write the resolution wish', 'x1080' , 'x720' , 'x480' )
except:
sys.exit(1)
print val

You can use the Python ConfigParser library. This will read your INI file and give you the parameters you need (e.g. resolution), which you can then use in your JSON downloading code.

some ini file like that:
[section1]
var1=value1
Would be read by that:
import ConfigParser
config = ConfigParser.ConfigParser()
config.read('conf.ini')
try:
val = config.get('section1', 'var1')
except:
sys.exit(1)
print val

Writing to an existing file without overwriting/erasing

My goal is quite simple, but I couldn't find it on the guide for configobj.
When I run my code I want it to write to a file but not erase what there's in the file already.
I would like everytime I run this it should write underneath what's already in the file
This is my current code: That erase/overwrite what's inside the dasd.ini already
from configobj import ConfigObj
config = ConfigObj()
config.filename = "dasd.ini"
#
config['hey'] = "value1"
config['test'] = "value2"
#
config['another']['them'] = "value4"
#
config.write()

this would be remarkably simpler if configobj accepted a file-like object instead of a file name. This is a solution i offered in comments.
import tempfile
with tempfile.NamedTemporaryFile() as t1, tempfile.NamedTemporaryFile() as t2, open('dasd.ini', 'w') as fyle:
config = ConfigObj()
config.filename = t1.file.name
config['hey'] = "value1"
config['test'] = "value2"
config['another']['them'] = "value4"
config.write()
do_your_thing_with_(t2)
t1.seek(0)
t2.seek(0)
fyle.write(t2.read())
fyle.write(t1.read())

If I understand your question correctly, doing what you want is a very simple change. Use the following syntax to create your initial config object. This reads in keys and values from the existing file.
config = ConfigObj("dasd.ini")
Then you can add new settings or change the existing ones as in your example code.
config['hey'] = "value1"
config['test'] = "value2"
After you write it out using config.write(), you'll find that your dasd.ini file contains the original and new keys/values merged. It also preserves any comments you had in your original ini file, with new keys/values added to the end of each section.
Check out this link, I found it to be quite helpful: An Introduction to ConfigObj

try it:
You have to read all keys and values of the section if the section existed already
and then write the whole section data
# -*- coding: cp950 -*-
import configobj
import os
#-------------------------------------------------------------------------
# _readINI(ini_file, szSection, szKey)
# read KeyValue from a ini file
# return True/False, KeyValue
#-------------------------------------------------------------------------
def _readINI(ini_file, szSection, szKey=None):
ret = None
keyvalue = None
if os.path.exists(ini_file) :
try:
config = configobj.ConfigObj(ini_file, encoding='UTF8')
if not szKey==None :
keyvalue = config[szSection][szKey]
else:
keyvalue = config[szSection]
ret = True
print keyvalue
except Exception, e :
ret = False
return ret, keyvalue
#-------------------------------------------------------------------------
# _writeINI(ini_file, szSection, szKey, szKeyValue):
# write key value into a ini file
# return True/False
# You have to read all keys and values of the section if the section existed already
# and then write the whole section data
#-------------------------------------------------------------------------
def _writeINI(ini_file, szSection, szKey, szKeyValue):
ret = False
try:
ret_section = _readINI(ini_file, szSection)
if not os.path.exists(ini_file) :
# create a new ini file with cfg header comment
CreateNewIniFile(ini_file)
config = configobj.ConfigObj(ini_file, encoding='UTF8')
if ret_section[1] == None :
config[szSection] = {}
else :
config[szSection] = ret_section[1]
config[szSection][szKey] = szKeyValue
config.write()
ret = True
except Exception, e :
print str(e)
return ret
#-------------------------------------------------------------------------
# CreateNewIniFile(ini_file)
# create a new ini with header comment
# return True/False
#-------------------------------------------------------------------------
def CreateNewIniFile(ini_file):
ret = False
try:
if not os.path.exists(ini_file) :
f= open(ini_file,'w+')
f.write('########################################################\n')
f.write('# Configuration File for Parallel Settings of Moldex3D #\n')
f.write('# Please Do Not Modify This File #\n')
f.write('########################################################\n')
f.write('\n\n')
f.close()
ret = True
except Exception, e :
print e
return ret
#----------------------------------------------------------------------
if __name__ == "__main__":
path = 'D:\\settings.cfg'
_writeINI(path, 'szSection', 'szKey', u'kdk12341 他dkdk')
_writeINI(path, 'szSection', 'szKey-1', u'kdk123412dk')
_writeINI(path, 'szSection', 'szKey-2', u'kfffk')
_writeINI(path, 'szSection', 'szKey-3', u'dhhhhhhhhhhhh')
_writeINI(path, 'szSection-333', 'ccc', u'555')
#_writeINI(path, 'szSection-222', '', u'')
print _readINI(path, 'szSection', 'szKey-2')
print _readINI(path, 'szSection-222')
#CreateNewIniFile(path)

Convert ConfigParser.items('') to dictionary

How can I convert the result of a ConfigParser.items('section') to a dictionary to format a string like here:
import ConfigParser
config = ConfigParser.ConfigParser()
config.read('conf.ini')
connection_string = ("dbname='%(dbname)s' user='%(dbuser)s' host='%(host)s' "
"password='%(password)s' port='%(port)s'")
print connection_string % config.items('db')

Have you tried
print connection_string % dict(config.items('db'))
?

How I did it in just one line.
my_config_parser_dict = {s:dict(config.items(s)) for s in config.sections()}
No more than other answers but when it is not the real businesses of your method and you need it just in one place use less lines and take the power of dict comprehension could be useful.

This is actually already done for you in config._sections. Example:
$ cat test.ini
[First Section]
var = value
key = item
[Second Section]
othervar = othervalue
otherkey = otheritem
And then:
>>> from ConfigParser import ConfigParser
>>> config = ConfigParser()
>>> config.read('test.ini')
>>> config._sections
{'First Section': {'var': 'value', '__name__': 'First Section', 'key': 'item'}, 'Second Section': {'__name__': 'Second Section', 'otherkey': 'otheritem', 'othervar': 'othervalue'}}
>>> config._sections['First Section']
{'var': 'value', '__name__': 'First Section', 'key': 'item'}
Edit: My solution to the same problem was downvoted so I'll further illustrate how my answer does the same thing without having to pass the section thru dict(), because config._sections is provided by the module for you already.
Example test.ini:
[db]
dbname = testdb
dbuser = test_user
host = localhost
password = abc123
port = 3306
Magic happening:
>>> config.read('test.ini')
['test.ini']
>>> config._sections
{'db': {'dbname': 'testdb', 'host': 'localhost', 'dbuser': 'test_user', '__name__': 'db', 'password': 'abc123', 'port': '3306'}}
>>> connection_string = "dbname='%(dbname)s' user='%(dbuser)s' host='%(host)s' password='%(password)s' port='%(port)s'"
>>> connection_string % config._sections['db']
"dbname='testdb' user='test_user' host='localhost' password='abc123' port='3306'"
So this solution is not wrong, and it actually requires one less step. Thanks for stopping by!

I know this was asked a long time ago and a solution chosen, but the solution selected does not take into account defaults and variable substitution. Since it's the first hit when searching for creating dicts from parsers, thought I'd post my solution which does include default and variable substitutions by using ConfigParser.items().
from ConfigParser import SafeConfigParser
defaults = {'kone': 'oneval', 'ktwo': 'twoval'}
parser = SafeConfigParser(defaults=defaults)
parser.set('section1', 'kone', 'new-val-one')
parser.add_section('section1')
parser.set('section1', 'kone', 'new-val-one')
parser.get('section1', 'ktwo')
parser.add_section('section2')
parser.get('section2', 'kone')
parser.set('section2', 'kthree', 'threeval')
parser.items('section2')
thedict = {}
for section in parser.sections():
thedict[section] = {}
for key, val in parser.items(section):
thedict[section][key] = val
thedict
{'section2': {'ktwo': 'twoval', 'kthree': 'threeval', 'kone': 'oneval'}, 'section1': {'ktwo': 'twoval', 'kone': 'new-val-one'}}
A convenience function to do this might look something like:
def as_dict(config):
"""
Converts a ConfigParser object into a dictionary.
The resulting dictionary has sections as keys which point to a dict of the
sections options as key => value pairs.
"""
the_dict = {}
for section in config.sections():
the_dict[section] = {}
for key, val in config.items(section):
the_dict[section][key] = val
return the_dict

For an individual section, e.g. "general", you can do:
dict(parser['general'])

Another alternative would be:
config.ini
[DEFAULT]
potato=3
[foo]
foor_property=y
potato=4
[bar]
bar_property=y
parser.py
import configparser
from typing import Dict
def to_dict(config: configparser.ConfigParser) -> Dict[str, Dict[str, str]]:
"""
function converts a ConfigParser structure into a nested dict
Each section name is a first level key in the the dict, and the key values of the section
becomes the dict in the second level
{
'section_name': {
'key': 'value'
}
}
:param config: the ConfigParser with the file already loaded
:return: a nested dict
"""
return {section_name: dict(config[section_name]) for section_name in config.sections()}
main.py
import configparser
from parser import to_dict
def main():
config = configparser.ConfigParser()
# By default section names are parsed to lower case, optionxform = str sets to no conversion.
# For more information: https://docs.python.org/3/library/configparser.html#configparser-objects
# config.optionxform = str
config.read('config.ini')
print(f'Config read: {to_dict(config)}')
print(f'Defaults read: {config.defaults()}')
if __name__ == '__main__':
main()

In Python +3.6 you could do this
file.ini
[SECTION1]
one = 1
two = 2
[SECTION2]
foo = Hello
bar = World
[SECTION3]
param1 = parameter one
param2 = parameter two
file.py
import configparser
cfg = configparser.ConfigParser()
cfg.read('file.ini')
# Get one section in a dict
numbers = {k:v for k, v in cfg['SECTION1'].items()}
If you need all sections listed you should use cfg.sections()

Combining Michele d'Amico and Kyle's answer (no dict),
produces a less readable but somehow compelling:
{i: {i[0]: i[1] for i in config.items(i)} for i in config.sections()}

Here is another approach using Python 3.7 with configparser and ast.literal_eval:
game.ini
[assets]
tileset = {0:(32, 446, 48, 48),
1:(96, 446, 16, 48)}
game.py
import configparser
from ast import literal_eval
config = configparser.ConfigParser()
config.read('game.ini')
# convert a string to dict
tileset = literal_eval(config['assets']['tileset'])
print('tileset:', tileset)
print('type(tileset):', type(tileset))
output
tileset: {0: (32, 446, 48, 48), 1: (96, 446, 16, 48)}
type(tileset): <class 'dict'>