Develop Reference

Python unit testing with PyTest, stuck on more advanced testing

I have been using PyTest for some time now to write some simple tests (like the ones you find in tutorials and youtube video's) and I thought now it was time to start writing actual test for our python scripts. The scripts are way more advanced than any shown in tutorials so I am getting a bit stuck. I do not want the entire correct answer, but rather a nudge in the right direction if possible. Here is my issue:
We have a script that reads a .md text file and converts it to a pdf file based on an external template. Part of the script is here below (I removed most of it because I first just want to have 1 running test)
class DocumentationEngine:
def __init__(self, title, subtitle, series, style='TIIStyle_Digital_Aug_2020', templateFile='template.docet', tableOfContents=True, listOfFigures=False, listOfTables=False):
self.title = title
self.subtitle = subtitle
self.series = series
self.style = style
self.template = {}
self.hasTOC = tableOfContents
self.hasLOF = listOfFigures
self.hasLOT = listOfTables
self.loadTemplate(templateFile)
def loadTemplate(self, file='template.docet'):
with open(file, "r") as templatefile:
lines = templatefile.readlines()
key = "dummy"
value = ""
for line in lines:
line = line.strip()
if line.startswith('[') and line.endswith(']'):
self.template[key] = value
key = line[1:-1]
value = ""
else:
value += line + '\n'
def build(self, versions=[], content='', filename='Documenter\\_Autogenerated'):
document = self.template["doc"]
document = document.replace("%%style%%", self.style)
document = document.replace("%%body%%",
self.buildFirstPage() +
self.buildTableOfContents() +
self.buildListOfFigures() +
self.buildListOfTables() +
self.buildVersionTable(versions, filename) +
self.buildContentPages(content=content) +
self.buildLastPage()
)
return document
def buildLastPage(self):
return self.template["last_page"]
I am trying to write a simple unit test for the buildLastPage method and have been stuck for several days now.
I am not sure whether or not I need to mock the template file, use a fixture and/or if I can actually test only that method with all dependencies.
I started with the following:
from doceng import DocumentationEngine
import pytest
class Test:
def test_buildLastPage(self):
build_last_page = DocumentationEngine()
assert build_last_page.template(1) == 1
which gives me an error regarding 3 required arguments. When adding the arguments like this:
from doceng import DocumentationEngine
import pytest
class Test:
def test_buildLastPage(self, title, subtitle, series):
build_last_page = DocumentationEngine()
assert build_last_page.template(1) == 1
which gives me an error that the fixture is not found.
I added a fixture in conftest.py file like this:
import pytest
from doceng import DocumentationEngine
#pytest.fixture
def title(title):
return title("test")
which will get me another error, recursive dependency involving fixture 'title' detected
I'm quite stuck so any nudge in the right direction for a newbie would be highly appreciated

The error of the fixtures is regarding your test function test_buildLastPage. The way you are using it, it only needs the self argument.
A test function in pytest without any decorators always expects to find fixtures, that have the same name as the arguments. You did not define any fixtures and also do not use the arguments in your function. Therefore, you can remove them safely.
The actual error points DocumentationEngine(). The class expect 3 arguments when initializing the object. You set no arguments. Check your __init__ function again to find the proper arguments.

How to access the variables of a class - Python

I have the following code to enable the file browser using blender:
import bpy
import os
from bpy.props import StringProperty
from bpy_extras.io_utils import ImportHelper
from bpy.types import Operator
sel = ''
class OpenBrowser(bpy.types.Operator):
bl_idname = "open.file"
bl_label = "Select Excel File"
bli_description = "Simulation output excel file"
filter_glob: StringProperty(default = '*.xls;*.xlsx',options = {'HIDDEN'})
filepath: bpy.props.StringProperty(subtype="FILE_PATH")
#somewhere to remember the address of the file
def execute(self, context):
global sel
sel = self.filepath
#self.selected_file = self.filepath
#display = "filepath= "+self.filepath
#print(display) #Prints to console
#Window>>>Toggle systen console
return {'FINISHED'}
def invoke(self, context, event): # See comments at end [1]
context.window_manager.fileselect_add(self)
global sel
sel = self.filepath
#Open browser, take reference to 'self'
#read the path to selected file,
#put path in declared string type data structure self.filepath
return {'RUNNING_MODAL'}
# Tells Blender to hang on for the slow user input
bpy.utils.register_class(OpenBrowser)
#Tell Blender this exists and should be used
# [1] In this invoke(self, context, event) is being triggered by the below command
#but in your script you create a button or menu item. When it is clicked
# Blender runs invoke() automatically.
#execute(self,context) prints self.filepath as proof it works.. I hope.
bpy.ops.open.file('INVOKE_DEFAULT')
print(sel)
The issue I am facing is that I have declared a global variable sel to which I want to save the filepath selected from the user when running the code. However, when I run the script I see that sel has not changed and it is as it was initialized. Could someone please help me on how to access from the class the self.filepath variable? What am I doing wrong here?

If I understand correctly, you want to store that value for later.
I'm not sure why 'sel' doesn't even update in your case, but I think the more correct way would be to use a property like so:
import bpy
# Assign a custom property to an existing type.
bpy.types.Scene.my_sel_value = bpy.props.StringProperty(name="Sel")
# Set property value.
bpy.context.scene.my_sel_value = "Foo"
# Get property value.
print(bpy.context.scene.my_sel_value)
Properties can be added to all ID types, but for "global" values, bpy.types.scene us ussualy used. Though there can be multiple scenes in one project and they will have separate values. Property values are stored when Blender closes.
If you are making an addon, you can also store your value in Addon Preferences. This value will be the same for all blender projects.

How can I reference a variable from another file

I have two files: client.py and ItemDatabase.py. I'm trying to store item values in a separate source file, similar to a database, and then call on those item values when I need them in other source files. Here I have the classes Item and WeaponData, which both hold information about the item, and then the instance itself is created in the ItemDatabase. When I want to get that information in the other source file through a print statement I am getting an error: NameError: name 'tinder_sword' is not defined, even though my IDE recognizes that tinder_sword is apart of ItemDatabase.
In client.py:
from ItemDatabase import *
import pygame as pg
class Item:
def __init__(self):
self.name = ""
self.description = ""
self.max_stack_size = 0
self.icon = ""
class WeaponData(Item):
def __init__(self):
super().__init__()
self.damage = 0
self.range = 0
self.cooldown = 0
self.max_condition = 0
self.max_stack_size = 1
print(tinder_sword.name)
In ItemDatabase.py:
from client import *
tinder_sword = WeaponData()
tinder_sword.name = "Tinder Sword"
tinder_sword.description = "Weak and flimsy, but it's not bad for a starter weapon."
tinder_sword.damage = 10
tinder_sword.cooldown = 0.2
tinder_sword.max_condition = 75

Hey! So the issue with your code is not that the variable is not getting imported. The issue is that you are importing client into ItemDatabase.py and then importing ItemDatabase into client.
When you import a python file all the code that is not in a scope gets run. So if you follow the logic:
in client.py
from ItemDatabase import *
so now we run the code in ItemDatabase
from client import *
so now we run the code in client
But now python has already imported ItemDatabase so it skips it.
Now we create the 2 classes in client.py and we run the last line of code:
print(tinder_sword.name)
Except. We have not actually run the code to create tinder_sword yet. Because we are still in the phase of importing client into ItemDatabase.
Importing files needs to flow one way, like a river. If you need two files to use data from each other then you can create a class called Populater in file2. File 1 imports Populator, gives it the values that file 2 needs, and then exits. File 2 never needs to import file 1 but it's populate class will still have file 1's values. This is like sending a power boat up the river to fetch some logs and the driving the boat back down the river to be used there.
This is however not a good way to write programs! It usually means that the way you are thinking about your application is too intertwined and will probably lead to more bugs like this down the line.
The "Nice" way to write a program like this is to do what is called "Separation of concerns". One file focuses on doing one thing only. But doing that one thing well. For example: You might have a SwordBuilder file. It creates amazing swords. Then you might have a storage class. It is focused on storing and reading any items you might have to the filesystem so your games can be saved.
Hope this helps!

trying to help here, usually I would create a config.py file contain classes and stuff as what you have in client.py
then import it in doSomething.py then do the printing in there
import pygame as pg
class Item:
def __init__(self):
self.name = ""
self.description = ""
self.max_stack_size = 0
self.icon = ""
class WeaponData(Item):
def __init__(self):
super().__init__()
self.damage = 0
self.range = 0
self.cooldown = 0
self.max_condition = 0
self.max_stack_size = 1
import client.py
tinder_sword = client.WeaponData()
tinder_sword.name = "Tinder Sword"
tinder_sword.description = "Weak and flimsy, but it's not bad for a starter weapon."
tinder_sword.damage = 10
tinder_sword.cooldown = 0.2
tinder_sword.max_condition = 75
print(tinder_sword.name)

When using Python classes as program configuration structures (which includes inherited class attributes), a good way to save/restore?

Let's say I have a (simplified) class as below. I am using it for a program configuration (hyperparameters).
# config.py
class Config(object): # default configuration
GPU_COUNT = 1
IMAGES_PER_GPU = 2
MAP = {1:2, 2:3}
def display(self):
pass
# experiment1.py
from config import Config as Default
class Config(Default): # some over-written configuration
GPU_COUNT = 2
NAME='2'
# run.py
from experiment1 import Config
cfg = Config()
...
cfg.NAME = 'ABC' # possible runtime over-writing
# Now I would like to save `cfg` at this moment
I'd like to save this configuration and restore later. The member functions must be out of concern when restoring.
1. When I tried pickle:
import pickle
with open('cfg.pk', 'rb') as f: cfg = pickle.load(f)
##--> AttributeError: Can't get attribute 'Config' on <module '__main__'>
I saw a solution using class_def of Config, but I wish I can restore the configuration without knowing the class definition (eg, export to dict and save as JSON)
2. I tried to convert class to dict (so that I can export as JSON)
cfg.__dict__ # {'NAME': 'ABC'}
vars(cfg) # {'NAME': 'ABC'}
In both cases, it was difficult to access attributes. Is it possible?

The question's title is "how to convert python class to dict", but I suspect you are really just looking for an easy way to represent (hyper)parameters.
By far the easiest solution is to not use classes for this. I've seen it happen on some machine learning tutorials, but I consider it a pretty ugly hack. It breaks some semantics about classes vs objects, and the difficulty pickling is a result from that. How about you use a simple class like this one:
class Params(dict):
__getattr__ = dict.__getitem__
__setattr__ = dict.__setitem__
__delattr__ = dict.__delitem__
def __getstate__(self):
return self
def __setstate__(self, state):
self.update(state)
def copy(self, **extra_params):
return Params(**self, **extra_params)
It can do everything the class approach can. Predefined configs are then just objects you should copy before editing, as follows:
config = Params(
GPU_COUNT = 2,
NAME='2',
)
other_config = config.copy()
other_config.GPU_COUNT = 4
Or alternatively in one step:
other_config = config.copy(
GPU_COUNT = 4
)
Works fine with pickle (although you will need to have the Params class somewhere in your source), and you could also easily write load and save methods for the Params class if you want to use JSON.
In short, do not use a class for something that really is just an object.

Thankfully, #evertheylen's answer was great to me. However, the code returns error when p.__class__ = Params, so I slightly changed as below. I think it works in the same way.
class Params(dict):
__getattr__ = dict.__getitem__
__setattr__ = dict.__setitem__
__delattr__ = dict.__delitem__
def __getstate__(self):
return self
def __setstate__(self, state):
self.update(state)
def copy(self, **extra_params):
lhs = Params()
lhs.update(self)
lhs.update(extra_params)
return lhs
and you can do
config = Params(
GPU_COUNT = 2,
NAME='2',
)
other_config = config.copy()
other_config.GPU_COUNT = 4

Reading YAML config file in python and using variables

Say I have a yaml config file such as:
test1:
minVolt: -1
maxVolt: 1
test2:
curr: 5
volt: 5
I can read the file into python using:
import yaml
with open("config.yaml", "r") as f:
config = yaml.load(f)
Then I can access the variables with
config['test1']['minVolt']
Style-wise, what is the best way to use variables from the config file? I will be using the variables in multiple modules. If I simply access the variables as shown above, if something is renamed, I will need to rename every instance of the variable.
Just wondering what the best or common practices for using variables from a config file in different modules.

You can do this:
class Test1Class:
def __init__(self, raw):
self.minVolt = raw['minVolt']
self.maxVolt = raw['maxVolt']
class Test2Class:
def __init__(self, raw):
self.curr = raw['curr']
self.volt = raw['volt']
class Config:
def __init__(self, raw):
self.test1 = Test1Class(raw['test1'])
self.test2 = Test2Class(raw['test2'])
config = Config(yaml.safe_load("""
test1:
minVolt: -1
maxVolt: 1
test2:
curr: 5
volt: 5
"""))
And then access your values with:
config.test1.minVolt
When you rename the values in the YAML file, you only need to change the classes at one place.
Note: PyYaml also allows you to directly deserialize YAML to custom classes. However, for that to work, you'd need to add tags to your YAML file so that PyYaml knows which classes to deserialize to. I expect that you do not want to make your YAML input more complex.

See Munch, Load YAML as nested objects instead of dictionary in Python
import yaml
from munch import munchify
c = munchify(f)yaml.safe_load(…))
print(c.test1.minVolt)
# -1
# Or
f = open(…)
c = Munch.fromYAML(f)