API

Configuration

tree-config provides the tools to configure applications where the objects to be configured are nested in a tree-like fashion such that they can be represented as nested dicts in e.g. a yaml file.

See the guide for complete examples.

class tree_config.Configurable

Bases: object

tree_config uses a duck typing approach. E.g. when applying config, apply_config() will call apply_config_property if it is defined for the objects being configured, otherwise it directly sets the properties.

The Configurable can be used as a base-class instead of the duck typing approach and it defines all the special config methods. It can also be used as a guide to the names of the special config hooks available when using the duck typing approach.

There are only two possible benefits to using Configurable. (1) it defines _config_props / _config_children which caches the names of the configurable properties / children on a per-class basis. Unlike the duck typing approach that gathers them afresh every time. (2) You can use super for properties you want to use the default handling approach as in the following example.

However, the main reason for the class’ existance is to define the configuration API in one place so we can refer to the method names.

Consider a duck typing example and the inheritance based example:

class DuckApp:

    _config_props_ = ('frame', 'color')

    frame = 'square'
    color = 'blue'

class ConfigurableApp(Configurable):

    _config_props_ = ('frame', 'color')

    frame = 'square'
    color = 'blue'

Both behave identically with respect to configuration:

>>> read_config_from_object(DuckApp())
{'frame': 'square', 'color': 'blue'}
>>> read_config_from_object(ConfigurableApp())
{'frame': 'square', 'color': 'blue'}

However, if we wanted to customize setting the frame property, using Configurable will be slighlty simpler since you can call super to set the remaining properties in the default way. E.g.:

class DuckApp:

    _config_props_ = ('frame', 'color')

    frame = 'square'

    color = 'blue'

    def apply_config_property(self, name, value):
        if name == 'frame':
            self.frame = value * 2
        else:
            setattr(self, name, value)

class ConfigurableApp(Configurable):

    _config_props_ = ('frame', 'color')

    frame = 'square'

    color = 'blue'

    def apply_config_property(self, name, value):
        if name == 'frame':
            self.frame = value * 2
        else:
            super().apply_config_property(name, value)
property _config_children: Dict[str, str]

A property, which if defined will be used to get all the configurable children instead of using _config_children_. It returns the dict of configurable children like _config_children_.

property _config_props: List[str]

A property, which if defined will be used to get all the configurable properties instead of using _config_props_. It returns the list of configurable properties like _config_props_.

apply_config_child(name: str, prop: str, obj: Any, config: Union[Dict[str, Any], List[Dict[str, Any]]]) → Any

If defined, it is called to apply the configuration for all the properties of the child, for each child.

When defined, it must either call apply_config() which also implicitly dispatches post_config_applied(). If you manually configure the child, you also have to call post_config_applied() manually.

Parameters

  • name – The human friendly name of the child that will be configured. It is the same as the keys in _config_children_.

  • prop – The property name storing the child that will be configured. It is the same as the values in _config_children_.

  • obj – The configurable child object or a list of objects if the property is a list.

  • config – The config dict or list of config dicts to be applied to the child(ern).

apply_config_property(name: str, value: Any) → None

If defined, it is called to set the value of a configurable property of the class.

Parameters

  • name – The name of the property.

  • value – The value of the property.

get_config_property(name: str) → Any

If defined, it is called by the configuration system to get the value of the named property.

Parameters

name – The name of the property to get.

post_config_applied() → None

If defined, it is called by the configuration system when it finishes applying all the configuration properties/children of this object.

tree_config.apply_config(obj, config: typing.Dict[str, typing.Any], get_attr=<built-in function getattr>, set_attr=<built-in function setattr>) → None

Takes the config data read with e.g. read_config_from_object() or read_config_from_file() and applies them to the object and its children.

Calls post_config_applied on the object and/or its children after they are configured if all/any have such a method.

Parameters

  • obj – The object to which to apply the config.

  • config – The config dict.

  • get_attr – The function to use to get the child value from the object and its children. defaults to getattr.

  • set_attr – The function to use to set the property values for the object and its children. defaults to setattr.

The object’s config is applied before its children’s config is applied.

E.x.:

class App:

    _config_props_ = ('name', )

    _config_children_ = {'the box': 'box'}

    name = 'chair'

    box = None

class Box:

    _config_props_ = ('volume', )

    volume = 12

    def apply_config_property(self, name, value):
        print('applying child', name)
        setattr(self, name, value)

    def post_config_applied(self):
        print('done applying child')

then:

>>> app = App()
>>> app.box = Box()
>>> d = read_config_from_object(app)
>>> d
{'the box': {'volume': 12}, 'name': 'chair'}
>>> d['name'] = 'bed'
>>> d['the box']['volume'] = 34
>>> apply_config(app, d)
applying child volume
done applying child
>>> app.name
'bed'
>>> app.box.volume
34
tree_config.dump_config(filename: typing.Union[str, pathlib.Path], data: typing.Dict[str, typing.Any], yaml_dump_str=<function yaml_dumps>) → None

Dumps config data gotten with e.g. read_config_from_object() to a yaml file.

Parameters

  • filename – The yaml filename.

  • data – The config data.

  • yaml_dump_str – The function to encode the config to yaml. Defaults to yaml_dumps().

E.g.:

>>> class App:
>>>     _config_props_ = ('name', )
>>>     name = 'chair'
>>> app = App()
>>> d = read_config_from_object(app)
>>> dump_config('config.yaml', d)
>>> with open('config.yaml') as fh:
...     print(fh.read())

Which prints name: chair.

tree_config.get_config_children_items(obj_or_cls, get_attr=<built-in function getattr>) → List[Tuple[str, str, Any]]

Returns a list of 3-tuples of all the configurable children of obj_or_cls. Each item is is a 3-tuple of (friendly_name, prop_name, value), where friendly_name is the name as listed in the keys of _config_children_, prop_name is the property name in the class as listed in the values of _config_children_, and value is the value taken from obj_or_cls.

We get the list of children of the object from get_config_children_names().

Parameters

  • obj_or_cls – The object or class on which to search for children.

  • get_attr – The function to use to get the child value from the object. defaults to getattr.

E.x.:

class App:

    _config_children_ = {'the box': 'box'}

    box = None

class Box:
    pass

then:

>>> get_config_children_items(App())
[('the box', 'box', None)]
>>> get_config_children_items(App)
[('the box', 'box', None)]
>>> app = App()
>>> app.box = Box()
>>> get_config_children_items(app)
[('the box', 'box', <Box at 0x264581a2a90>)]
tree_config.get_config_children_names(obj_or_cls) → Dict[str, str]

Returns a dict of the names friendly and property names of all the configurable children of obj_or_cls.

If obj_or_cls has a _config_children property, that is used to get the list of configurable children, otherwise we walk the class and its super classes hierarchy using _config_children_ to get the configurable children names.

Parameters

obj_or_cls – The object or class on which to search for children.

E.x.:

class App:

    _config_children_ = {'the box': 'box'}

    box = None

class Box:
    pass

then:

>>> get_config_children_names(App())
{'the box': 'box'}
>>> get_config_children_names(App)
{'the box': 'box'}
tree_config.get_config_prop_items(obj_or_cls, get_attr=<built-in function getattr>) → Dict[str, Any]

Returns a dict mapping the names of all the configurable properties of obj_or_cls to its values as gotten from the object or class.

We get the list of properties of the object from get_config_prop_names(). For each property, if the object has a get_config_property method, that is used to get the value. Otherwise, or if it’s a class get_attr is used instead.

Parameters

  • obj_or_cls – The object or class on which to search for properties.

  • get_attr – The function to use to get the property value from the object. defaults to getattr.

E.x.:

class App:

    _config_props_ = ('name', )

    name = 'chair'

class MyApp(App):

    _config_props_ = ('location', )

    location = 0

    def get_config_property(self, name):
        if name == 'location':
            return len(name)
        return getattr(self, name)

then:

>>> get_config_prop_items(MyApp())
{'location': 8, 'name': 'chair'}
>>> get_config_prop_items(MyApp)
{'location': 0, 'name': 'chair'}
tree_config.get_config_prop_names(obj_or_cls) → List[str]

Returns a list of the names of all the configurable properties of obj_or_cls.

If obj_or_cls has a _config_props property, that is used to get the list of configurable properties, otherwise we walk the class and its super classes hierarchy using _config_props_ to get the configurable property names.

Parameters

obj_or_cls – The object or class on which to search for properties.

E.x.:

class App:

    _config_props_ = ('name', )

    name = 'chair'

class MyApp(App):

    _config_props_ = ('location', )

    location = 0

    def get_config_property(self, name):
        if name == 'location':
            return len(name)
        return getattr(self, name)

then:

>>> get_config_prop_names(MyApp())
['location', 'name']
>>> get_config_prop_names(MyApp)
['location', 'name']
tree_config.load_apply_save_config(obj, filename: typing.Union[str, pathlib.Path], get_attr=<built-in function getattr>, set_attr=<built-in function setattr>, yaml_dump_str=<function yaml_dumps>, yaml_load_str=<function yaml_loads>) → Dict[str, Any]

Applies the config to the object from the yaml file (if the file doesn’t exist it creates it), and then dumps to the yaml file the current config from the object. It also returns the final config dict.

This can be used to set the object from the config, but also making sure the file contains the current config including any new properties not previously there or properties that changed during config application.

Parameters

  • obj – The configurable object.

  • filename – The yaml filename.

  • get_attr – The function to use to get the property/children values from the object. defaults to getattr.

  • set_attr – The function to use to set the property values of the object and its children. defaults to setattr.

  • yaml_dump_str – The function to encode the config to yaml. Defaults to yaml_dumps().

  • yaml_load_str – The function to parse the yaml string read from the file. Defaults to yaml_loads().

E.x.:

class App:

    _config_props_ = ('name', )

    name = 'chair'

class AppV2:

    _config_props_ = ('name', 'side')

    name = 'chair'

    side = 'left'

then:

>>> app = App()
>>> app.name = 'tree'
>>> load_apply_save_config(app, 'config_app.yaml')
{'name': 'tree'}
>>> app.name
'tree'
>>> # then later for v2 of the app
>>> app_v2 = AppV2()
>>> app_v2.name
'chair'
>>> load_apply_save_config(app_v2, 'config_app.yaml')
{'name': 'tree', 'side': 'left'}
>>> app_v2.name
'tree'
>>> with open('config_app.yaml') as fh:
...     print(fh.read())

this prints:

name: tree
side: left
tree_config.load_config(obj, filename: typing.Union[str, pathlib.Path], get_attr=<built-in function getattr>, yaml_dump_str=<function yaml_dumps>, yaml_load_str=<function yaml_loads>) → Dict[str, Any]

Loads and decodes the config from a yaml file. If the config file doesn’t exist, it first dumps the config to the file using read_config_from_object() and dump_config() before loading it.

Parameters

  • obj – The object from which to dump the config when the files doesn’t exist.

  • filename – The yaml filename.

  • get_attr – The function to use to get the property/children values from the object. defaults to getattr.

  • yaml_dump_str – The function to encode the config to yaml. Defaults to yaml_dumps().

  • yaml_load_str – The function to parse the yaml string read from the file. Defaults to yaml_loads().

E.g.:

>>> class App:
>>>     _config_props_ = ('name', )
>>>     name = 'chair'
>>> load_config(App(), 'app_config.yaml')
{'name': 'chair'}
tree_config.read_config_from_file(filename: typing.Union[str, pathlib.Path], yaml_load_str=<function yaml_loads>) → Dict[str, Any]

Reads and returns the yaml config data dict from a file that was previously dumped with dump_config().

Parameters

  • filename – The yaml filename.

  • yaml_load_str – The function to parse the yaml string read from the file. Defaults to yaml_loads().

E.g.:

>>> class App:
>>>     _config_props_ = ('name', )
>>>     name = 'chair'
>>> app = App()
>>> d = read_config_from_object(app)
>>> dump_config('config.yaml', d)
>>> read_config_from_file('config.yaml')
{'name': 'chair'}
tree_config.read_config_from_object(obj, get_attr=<built-in function getattr>) → Dict[str, Any]

Returns a recursive dict containing all the configuration options of the obj and its configurable children.

Parameters

  • obj – The object from which to get the config.

  • get_attr – The function to use to get the child value from the object and its children. defaults to getattr.

E.x.:

class App:

    _config_props_ = ('name', )

    _config_children_ = {'the box': 'box'}

    name = 'chair'

    box = None

class Box:

    _config_props_ = ('volume', )

    volume = 12

then:

>>> app = App()
>>> app.box = Box()
>>> read_config_from_object(app)
{'the box': {'volume': 12}, 'name': 'chair'}

Yaml module

Provides the functions to create yaml objects, and to dump and load yaml using these objects.

tree_config.yaml.get_yaml() → ruamel.yaml.main.YAML

Creates and returns a yaml object that can be used by yaml_dumps() and yaml_loads() to dump and load from yaml files.

The default flow style (default_flow_style) is set to False so the file is formatted as expanded. The yaml type instantiated is safe.

tree_config.yaml.register_kivy_yaml_support() → None

Registers the Kivy properties so it can be encoded to yaml.

E.g. Some properties create custom list sub-classes.

tree_config.yaml.register_numpy_yaml_support() → None

Registers the numpy data types so it can be encoded to yaml and then decoded.

When it is encountered in an object it is converted to a list with a special numpy tag. When decoded, an appropriate numpy type is constructed and returned.

tree_config.yaml.register_pathlib_yaml_support() → None

Registers pathlib.Path so it can be encoded and decoded back to Path.

tree_config.yaml.register_torch_yaml_support() → None

Registers the torch data types so it can be encoded to yaml and then decoded.

When it is encountered in an object it is converted to a list with a special torch tag. When decoded, an appropriate torch type is constructed and returned.

tree_config.yaml.yaml_dumps(value: typing.Any, get_yaml_obj: typing.Callable[[], ruamel.yaml.main.YAML] = <function get_yaml>) → str

Converts the object to yaml.

Parameters

  • value – the object to convert.

  • get_yaml_obj – A function such as get_yaml() that will be called to get a yaml object. Defaults to get_yaml().

Returns

a string yaml representation.

tree_config.yaml.yaml_loads(value: str, get_yaml_obj: typing.Callable[[], ruamel.yaml.main.YAML] = <function get_yaml>) → Any

Loads a yaml file and converts it to the objects it represents.

Parameters

  • value – the string to decode.

  • get_yaml_obj – A function such as get_yaml() that will be called to get a yaml object. Defaults to get_yaml().

Returns

the decoded object.

Documentation generation

When generating docs, the documentation of the properties listed in __config_props__ can dumped to a yaml file using create_doc_listener().

write_config_props_rst() can load the docstrings from this yaml file and then generate a nicely formatted rst file listing all the configurable properties of the project. See the user guide for details and complete examples.

Command line

This module provides two command line options that can call either download_merge_yaml_doc() or merge_yaml_doc() and forwards the arguments to these functions. See the guide for examples.

tree_config.doc_gen.create_doc_listener(sphinx_app, package_name, filename, yaml_dump_str=<function yaml_dumps>)

Creates a listener for the __config_props__ attributes and dumps the docs of any props listed, to filename. If the file already exists, it extends it with new data and overwrites any exiting properties that we see again in this run.

To use, in the sphinx conf.py file do something like:

def setup(app):
    import package
    create_doc_listener(app, package, 'config_attrs.yaml')

where package is the package for which the docs are generated.

See the guide for a full example.

tree_config.doc_gen.download_merge_yaml_doc(filename, url, out_filename)

Downloads a yaml file with previously saved configurable properties docstrings, optionally merges it with an existing docstrings yaml file, and outputs the merged yaml file. See the guide for an example.

Parameters

  • filename – The optional yaml filename into which to merge the remote yaml docstrings.

  • url – A url to a docstrings containing yaml file that will be downloaded and optionally merged into filename and output to out_filename.

  • out_filename – The filename of the output yaml file.

On the command line it’s invocated as:

$ python -m tree_config.doc_gen download --help
usage: Config Docs generation download [-h] [-f FILENAME] -u URL -o
                                       OUT_FILENAME

optional arguments:
  -h, --help            show this help message and exit
  -f FILENAME, --filename FILENAME
                        The optional yaml filename with which to merge
                        the downloaded file.
  -u URL, --url URL
  -o OUT_FILENAME, --out_filename OUT_FILENAME
tree_config.doc_gen.merge_yaml_doc(filename1, filename2, out_filename)

Merges two yaml files containing configurable properties docstrings into a single output yaml file.

Parameters

  • filename1 – First yaml input filename into which the second is merged.

  • filename2 – Second yaml input filename.

  • out_filename – The output filename of the resulting yaml file.

On the command line it’s invocated as:

$ python -m tree_config.doc_gen merge --help
usage: Config Docs generation merge [-h] -f1 FILENAME1 -f2 FILENAME2 -o
                                    OUT_FILENAME

optional arguments:
  -h, --help            show this help message and exit
  -f1 FILENAME1, --filename1 FILENAME1
  -f2 FILENAME2, --filename2 FILENAME2
  -o OUT_FILENAME, --out_filename OUT_FILENAME
tree_config.doc_gen.write_config_props_rst(obj, project, app, exception, filename, rst_filename, get_attr=<built-in function getattr>, yaml_dump_str=<function yaml_dumps>)

Walks through all the configurable classes of obj, recursively by looking at _config_props_ and _config_children_ and using the type hints of these children properties if they are None (e.g. if obj is a class). For each property it loads their docs from the yaml file filename and it generates a rst output file at rst_filename with all the tokens.

For example in the sphinx conf.py file do:

def setup(app):
    app.connect('build-finished', partial(write_config_props_rst, ProjectApp, project_name, filename='config_prop_docs.yaml', rst_filename='source/config.rst'))

where project_name is the name of project and ProjectApp is the App of the package the contains all the configurable objects.

See the guide for a complete example.

Utils

class tree_config.utils.class_property(fget=None, fset=None, fdel=None, doc=None)

Bases: property

Can be used as @class_property on a method to make it a class property in the same way @property can be used on a normal method to make it a object property.

tree_config.utils.get_class_annotations(obj_or_cls: Union[Any, Type[Any]]) → Dict[str, Any]

Gets a dict of all the annotations of the object/class, including from all superclasses. The values are actual object, even if the type is given as a string (for forward declarations).

tree_config.utils.get_class_bases(cls: Type) → Generator[Type, None, None]

Yields all the base-classes of the class, not including the passed class or object.