Develop Reference

Python class attribute 'is not defined' when referenced by another class attribute

Using the following, I am able to successfully create a parser and add my arguments to self._parser through the __init()__ method.
class Parser:
_parser_params = {
'description': 'Generate a version number from the version configuration file.',
'allow_abbrev': False
}
_parser = argparse.ArgumentParser(**_parser_params)
Now I wish to split the arguments into groups so I have updated my module, adding some classes to represent the argument groups (in reality there are several subclasses of the ArgumentGroup class), and updating the Parser class.
class ArgumentGroup:
_title = None
_description = None
def __init__(self, parser) -> ArgumentParser:
parser.add_argument_group(*self._get_args())
def _get_args(self) -> list:
return [self._title, self._description]
class ArgumentGroup_BranchType(ArgumentGroup):
_title = 'branch type arguments'
class Parser:
_parser_params = {
'description': 'Generate a version number from the version configuration file.',
'allow_abbrev': False
}
_parser = argparse.ArgumentParser(**_parser_params)
_argument_groups = [cls(_parser) for cls in ArgumentGroup.__subclasses__()]
However, I'm now seeing an error.
Traceback (most recent call last):
...
File "version2/args.py", line 62, in <listcomp>
_argument_groups = [cls(_parser) for cls in ArgumentGroup.__subclasses__()]
NameError: name '_parser' is not defined
What I don't understand is why _parser_params do exist when they are referred by another class attribute, but _parser seemingly does not exist in the same scenario? How can I refactor my code to add the parser groups as required?

This comes from the confluence of two quirks of Python:
class statements do not create a new local scope
List comprehensions do create a new local scope.
As a result, the name _parser is in a local scope whose closest enclosing scope is the global scope, so it cannot refer to the about-to-be class attribute.
A simple workaround would be to replace the list comprehension with a regular for loop.
_argument_groups = []
for cls in ArgumentGroup.__subclasses()__:
_argument_groups.append(cls(_parser))
(A better solution would probably be to stop using class attributes where instance attributes make more sense.)

Having trouble mocking a request in a class method using patch context manager in Python

I am trying to unit test a class method that makes an API request and returns a json file that is loaded into a variable as a dictionary. The request returns a json file as requested. Another test that I would like to implement is that it is accessing the correct link. I am using patch as a context manager with the mock module.
In summary, my app snowReport.py accesses a weather API that returns a weatherJson, and then it accesses that Json to determine if there will be snow in the forecast. The class is called Resort because it is meant specifically for ski resorts.
In my module, this is my __init__ function.
class Resort():
# kwargs is created so the user can pass in "96hr", "realtime", and or "360min"
def __init__(self, resortKey, *args):
# Check if you are in the current directory, if not, set it to the current directory
currentDir = os.getcwd()
if currentDir != D_NAME:
os.chdir(D_NAME)
else:
pass
# Checks if the user enters arguments to initiate json files or not
self.dataJSON = SKI_RESORT_JSON
self.args = args
if not args:
raise Exception("Invalid arg passed. Function arguments must contain one of '360min' and/or '96hr' and/or 'realtime'") # Note there is a hidden arg that is called test that does not access the api to do a mock test
# Opens json file to get location parameters
with open(SKI_RESORT_JSON, "r") as f:
resortDictList = json.load(f)
resortDict = resortDictList[resortKey]
self.name = resortDict["name"]
self.lon = resortDict["lon"]
self.lat = resortDict["lat"]
self.country = resortDict["country"]
self.weatherJsonRealTime = {}
self.weatherJson360Min = {}
self.weatherJson96hr = {}
The purpose of this __init__ function is to initialize variables and access a .json file where it pulls the location data.
The function I am trying to test is a class method that is the following:
def requestRealtime(self):
querystring = {
"lat": str(self.lat),
"lon": str(self.lon),
"unit_system": "si",
"fields": "precipitation,precipitation_type,temp,feels_like,wind_speed,wind_direction,sunrise,sunset,visibility,cloud_cover,cloud_base,weather_code",
"apikey": CLIMACELL_KEY,
}
response = requests.request("GET", URL_REALTIME, params=querystring)
if response.ok:
self.weatherJsonRealTime = json.load(response.text)
else:
return "Bad response"
self.nowTime = localTime(self.weatherJsonRealTime["observation_time"]["value"])
self.nowTemp = self.weatherJsonRealTime["temp"]["value"]
self.nowFeelsLike = self.weatherJsonRealTime["feels_like"]["value"]
self.nowPrecipitation = self.weatherJsonRealTime["precipitation"]["value"]
self.nowPrecipitationType = self.weatherJsonRealTime["precipitation_type"]["value"]
self.nowWindSpeed = self.weatherJsonRealTime["wind_speed"]["value"]
self.nowWindDirection = self.weatherJsonRealTime["wind_direction"]["value"]
self.nowCloudCover = self.weatherJsonRealTime["cloud_cover"]["value"]
return self.weatherJsonRealTime
I am trying to test the request, more specifically, I am trying to test that the request. To do so, I am using the patch context manager to mock the request.requests and set the return value to be a static test json file. My test code is as follows:
with open(".\\Resources\\test_realtimeJson.json", "r") as f:
testrealtimeDict = json.load(f)
#classmethod
def setUpClass(cls):
cls.testResort = snowReport.Resort("test_Location (Banff)", "96hr", "realtime", "360min")
os.chdir(D_NAME) # Set the directory back to D_NAME because that the snowReport.Resort class changes it's class
def test_requestRealtime(self):
with patch("snowApp.snowReport.requests.request") as mocked_request:
mocked_request.return_value.ok = True
mocked_request.return_value.request = testrealtimeDict
self.testResort.requestRealtime()
Assuming the mock works, I would then like to use the assertEqual function to test and see if the attributes that are created in the snowReport function returns the expected value based on the dictionary that I pass it.
When I run the test script, it throws me the error:
File "c:\users\steve\appdata\local\programs\python\python39\lib\json\__init__.py", line 339, in loads
raise TypeError(f'the JSON object must be str, bytes or bytearray, '
TypeError: the JSON object must be str, bytes or bytearray, not MagicMock
Since I am mocking the request and setting the return value to the testrealtimeDict that I have created, shouldn't self.weatherJsonRealtime = testrealtimeDict? Why is it throwing the type error? Also - is the unit test that I am planning appropriate for this application or is there a better or easier way to complete this?

BDD behave Python need to create a World map to hold values

I'm not too familiar with Python but I have setup a BDD framework using Python behave, I now want to create a World map class that holds data and is retrievable throughout all scenarios.
For instance I will have a world class where I can use:
World w
w.key.add('key', api.response)
In one scenario and in another I can then use:
World w
key = w.key.get('key').
Edit:
Or if there is a built in way of using context or similar in behave where the attributes are saved and retrievable throughout all scenarios that would be good.
Like lettuce where you can use world http://lettuce.it/tutorial/simple.html
I've tried this between scenarios but it doesn't seem to be picking it up
class World(dict):
def __setitem__(self, key, item):
self.__dict__[key] = item
print(item)
def __getitem__(self, key):
return self.__dict__[key]
Setting the item in one step in scenario A: w.setitem('key', response)
Getting the item in another step in scenario B: w.getitem('key',)
This shows me an error though:
Traceback (most recent call last):
File "C:\Program Files (x86)\Python\lib\site-packages\behave\model.py", line 1456, in run
match.run(runner.context)
File "C:\Program Files (x86)\Python\lib\site-packages\behave\model.py", line 1903, in run
self.func(context, *args, **kwargs)
File "steps\get_account.py", line 14, in step_impl
print(w.__getitem__('appToken'))
File "C:Project\steps\world.py", line 8, in __getitem__
return self.__dict__[key]
KeyError: 'key'
It appears that the World does not hold values here between steps that are run.
Edit:
I'm unsure how to use environment.py but can see it has a way of running code before the steps. How can I allow my call to a soap client within environment.py to be called and then pass this to a particular step?
Edit:
I have made the request in environment.py and hardcoded the values, how can I pass variables to environment.py and back?

It's called "context" in the python-behave jargon. The first argument of your step definition function is an instance of the behave.runner.Context class, in which you can store your world instance. Please see the appropriate part of the tutorial.

Have you tried the
simple approach, using global var, for instance:
def before_all(context):
global response
response = api.response
def before_scenario(context, scenario):
global response
w.key.add('key', response)
Guess feature can be accessed from context, for instance:
def before_feature(context, feature):
feature.response = api.response
def before_scenario(context, scenario):
w.key.add('key', context.feature.response)
You are looking for:
Class variable: A variable that is shared by all instances of a class.
Your code in Q uses Class Instance variable.
Read about: python_classes_objects
For instance:
class World(dict):
__class_var = {}
def __setitem__(self, key, item):
World.__class_var[key] = item
def __getitem__(self, key):
return World.__class_var[key]
# Scenario A
A = World()
A['key'] = 'test'
print('A[\'key\']=%s' % A['key'] )
del A
# Scenario B
B = World()
print('B[\'key\']=%s' % B['key'] )
Output:
A['key']=test
B['key']=test
Tested with Python:3.4.2
Come back and Flag your Question as answered if this is working for you or comment why not.

Defining global var in before_all hook did not work for me.
As mentioned by #stovfl
But defining global var within one of my steps worked out.
Instead, as Szabo Peter mentioned use the context.
context.your_variable_name = api.response
and just use
context.your_variable_name anywhere the value is to be used.

For this I actually used a config file [config.py] I then added the variables in there and retrieved them using getattr. See below:
WSDL_URL = 'wsdl'
USERNAME = 'name'
PASSWORD = 'PWD'
Then retrieved them like:
import config
getattr(config, 'USERNAME ', 'username not found')

Self-Updating Code?

I have a module that needs to update new variable values from the web, about once a week. I could place those variable values in a file & load those values on startup. Or, a simpler solution would be to simply auto-update the code.
Is this possible in Python?
Something like this...
def self_updating_module_template():
dynamic_var1 = {'dynamic var1'} # some kind of place holder tag
dynamic_var2 = {'dynamic var2'} # some kind of place holder tag
return
def self_updating_module():
dynamic_var1 = 'old data'
dynamic_var2 = 'old data'
return
def updater():
new_data_from_web = ''
new_dynamic_var1 = new_data_from_web # Makes API call. gets values.
new_dynamic_var2 = new_data_from_web
# loads self_updating_module_template
dynamic_var1 = new_dynamic_var1
dynamic_var2 = new_dynamic_var2
# replace module place holders with new values.
# overwrite self_updating_module.py.
return

I would recommend that you use configparser and a set of default values located in an ini-style file.
The ConfigParser class implements a basic configuration file parser
language which provides a structure similar to what you would find on
Microsoft Windows INI files. You can use this to write Python programs
which can be customized by end users easily.
Whenever the configuration values are updated from the web api endpoint, configparser also lets us write those back out to the configuration file. That said, be careful! The reason that most people recommend that configuration files be included at build/deploy time and not at run time is for security/stability. You have to lock down the endpoint that allows updates to your running configuration in production and have some way to verify any configuration value updates before they are retrieved by your application:
import configparser
filename = 'config.ini'
def load_config():
config = configparser.ConfigParser()
config.read(filename)
if 'WEB_DATA' not in config:
config['WEB_DATA'] = {'dynamic_var1': 'dynamic var1', # some kind of place holder tag
'dynamic_var2': 'dynamic var2'} # some kind of place holder tag
return config
def update_config(config):
new_data_from_web = ''
new_dynamic_var1 = new_data_from_web # Makes API call. gets values.
new_dynamic_var2 = new_data_from_web
config['WEB_DATA']['dynamic_var1'] = new_dynamic_var1
config['WEB_DATA']['dynamic_var2'] = new_dynamic_var2
def save_config(config):
with open(filename, 'w') as configfile:
config.write(configfile)
Example usage::
# Load the configuration
config = load_config()
# Get new data from the web
update_config(config)
# Save the newly updated configuration back to the file
save_config(config)

Python: File being saved as empty when it should be saved with text in it, but only when an unrelated method is run before it

Observe the following Python file:
# configmanager.py
"""
ConfigManager controls the modification and validation of config files.
"""
import os
from ruamel import yaml
from voluptuous import Schema
class ConfigManager():
"""
Controls all interaction with configuration files
"""
def __init__(self):
super().__init__()
self.configvalidator = ConfigValidator()
# The config directory inside users home directory.
# Config files will be stored here.
config_dir = os.path.expanduser('~')+'/.config/MyProject/'
# The default config file
config_file = config_dir+'myproject.conf'
# The default configuration
default_config = {
'key1': {},
'key2': {}
}
def _get_config(self):
"""
Get the config file and return it as python dictionary.
Will create the config directory and default config file if they
do not exist.
"""
# Create config directory if it does not exist
if not os.path.exists(self.config_dir):
os.makedirs(self.config_dir)
# Create default config file if it does not exist
if not os.path.isfile(self.config_file):
config_file = open(self.config_file, 'w')
config_file.write(yaml.dump(self.default_config))
# Open config file, and load from YAML
config_file = open(self.config_file, 'r')
config = yaml.safe_load(config_file)
# Validate config
self.configvalidator.validate(config)
return config
def _save_config(self, config):
"""
Save the config file to disk as YAML
"""
# Open current config file
config_file = open(self.config_file, 'w')
# Validate new config
# THE ERROR IS HERE
# If this runs then the config file gets saved as an empty file.
self.configvalidator.validate(config)
# This shows that the 'config' variable still holds the data
print(config)
# This shows that yaml.dump() is working correctly
print(yaml.dump(config))
config_file.write(yaml.dump(config))
def edit_config(self):
"""
Edits the configuration file
"""
config = self._get_config()
# Perform manipulation on config here
# No actual changes to the config are necessary for the bug to occur
self._save_config(config)
class ConfigValidator():
def __init__(self):
super().__init__()
# Config file schema
# Used for validating the config file with voluptuous
self.schema = Schema({
'key1': {},
'key2': {},
})
def validate(self, config):
"""
Validates the data against the defined schema.
"""
self.schema(config)
app = ConfigManager()
app.edit_config()
-
# ~/.config/MyProject/myproject.conf
key1: {}
key2: {}
Description of my module
This is a module I am working on which is for modifying the config file for my project. It accesses the file in ~/.config/MyProject/myproject.conf, which is saved in YAML format, and stores various pieces of information that are used by my program. I have removed as much of the code as I can, leaving only that necessary for understanding the bug.
ConfigManager
ConfigManager is the class containing methods for manipulating my config file. Here it contains three methods: _get_config(), _save_config(), and edit_config(). When instantised, it will get an instance of ConfigValidator (described below), and assign it to self.configvalidator.
_get_config
_get_config() simply opens the file defined by the class variables, specifically ~/.config/MyProject/myproject.conf, or creates the file with default values if it does not exist. The file is saved in YAML format, so this method loads it into a python object, using ruamel.yaml, validates it using self.configvalidator.validate(config) and returns it for use by other pieces of code.
_save_config
_save_config() is where the error occurs, which is described in detail below. It's purpose is to validate the given data, and if it is valid, save it to disk in YAML format.
edit_config
This a generic function, which, in my program, would make specific changes to my config file, depending on the arguments given. In my example, this function simply gets the config file with self._get_config(), and then saves it using self._save_config, without making any changes.
ConfigValidator
This class is for validating my config file using voluptuous. When instantised, it will create the schema that is to be used, and assign it to self.schema. When the validate method is run, it validates the given data using voluptuous.
The error
Observe the line self.configvalidator.validate(config) in ConfigManager._save_config(). This will validate the given data against the schema, and raise an error if it does not pass validation.
But, in the following line config_file.write(yaml.dump(config)), which simply saves the given data to a file as YAML, it will instead save an empty file to disk. (Note: the file is empty, not deleted)
If I disable the validation, by commenting out self.configvalidator.validate(config), then the file is written correctly as YAML.
If self.configvalidator.validate(config) is run, then the config file is saved as an empty file.
My testing
As can be seen with the line print(config), the data in the variable config does not change after being used for validation, yet when being saved to disk, it would seem that config is an empty variable.
print(yaml.dump(config)) shows that ruamel.yaml does work correctly.
If I change edit_config to give invalid data to _save_config, then self.configvalidator.validate(config) will raise an error, as expected. self.configvalidator.validate(config) is running correctly.
End
If self.configvalidator.validate(config) is run, then config_file.write(yaml.dump(config)) saves the config file as an empty file, despite the data in the variable config not changing.
If self.configvalidator.validate(config) is not run, then config_file.write(yaml.dump(config)) saves the file correctly.
That is my error, and it makes absolutely no sense to me.
If your keen to help, then configmanager.py should run correctly (with the error) on your machine, as long as it has access to ruamel.yaml and voluptuous. It will create ~/.config/MyProject/myproject.conf, then save it as empty. Save my example myproject.conf to see how it is then saved as empty when configmanager.py is run. If configmanager.py is run again, when myproject.conf is empty, then a validation error will be raised in _get_config, as expected.
I am so confused by this bug, so if you have any insight it would be greatly appreciated.
Cheers