root/releases/pkgcore/0.2.5/dev-notes/config.rst @ marienz%2540gentoo.org-20070218170719-tdrfk0mqanfsemxa

=====================================
 Config use and implementation notes
=====================================

Using the manager
=================

Normal use
----------

To get at the user's configuration::

 from pkgcore.config import load_config
 config = load_config()
 main_repo = config.get_default('repo')
 spork_repo = config.repo['spork']

Usually this is everything you need to know about the manager. Some
things to be aware of:

- Some of the managed sources of configuration data may be slow, so
  accessing a source is delayed for as long as possible. Some things
  require accessing all sources though and should therefore be
  avoided. The easiest one to trigger is config.repo.keys() or the
  equivalent list(config.sections('repo')). This has to get the
  "class" setting for every available config section, which might be
  slow.
- For the same reason the manager does not know what type names exist
  (there is no hardcoded list of them, so the only way to get that
  information would be loading all config sections). This is why you
  can get this::

   >>> load_config().section('repo') # typo, should be "sections"
   Traceback (most recent call last):
     File "<stdin>", line 1, in ?
   TypeError: '_ConfigMapping' object is not callable

  This constructed a dictlike object for accessing all config sections
  of the type "section", then tried to call it.

Testcase use
------------

For testing of high-level scripts it can be useful to construct a
config manager containing hardcoded values::

 from pkgcore.config import basics, central

 config = central.ConfigManager([{
     'repo' = basics.HardCodedConfigSection({'class': my_repo,
                                             'data': ['1', '2']}),
     'cont' = basics.ConfigSectionFromStringDict({'class': 'pkgcore.my.cont',
                                                  'ref': 'repo'}),
     }])

What this does should be fairly obvious. Be careful you do not use the
same ConfigSection object in more than one place: caching will not
behave the way you want. See `Adding a config source`_ for details.

Adding a configurable
=====================

You often do not really *have* to do anything to make something a
valid "class" value, but it is clearer and it is necessary in certain
cases.

Adding a class
--------------

To make a class available, do this::

 from pkgcore.config import ConfigHint, errors

 class MyRepo(object):

     pkgcore_config_type = ConfigHint({'cache': 'section_ref'},
                                      typename='repo')

     def __init__(self, repo):
         try:
             self.initialize(repo)
         except SomeRandomException:
             raise errors.InstantiationError('eep!')

The first ConfigHint arg tells the config system what kind of
arguments you take. Without it it assumes arguments with no default
are strings and guesses for the other args based on the type of the
default value. So if you have no default values or they are just None
you should tell the system about your args.

The second one tells it you fulfill the repo "protocol", meaning your
instances will show up in load_config().repo.

ConfigHint takes some more arguments, see the api docs for details.

Adding a callable
-----------------

To make a callable available you can do this::

 from pkgcore.config import configurable, errors

 @configurable({'cache': 'section_ref'}, typename=repo)
 def my_repo(repo):
     # do stuff

configurable is just a convenience function that applies a ConfigHint.

Exception handling
------------------

If you raise an exception when the config system calls you it will
catch the exception and wrap it in an InstantiationError. This is good
for calling code since catching and printing those provides the user
with a readable description of what happened. It is less good for
developers since the raising of a new exception kills the traceback
printed in debug mode. You will have a traceback that "ends" in the
config code handling instantiation.

You can improve this by raising an InstantiationError yourself. If you
do this the config system will be able to add the extra information
needed for a user-friendly error message to it without raising a new
exception, meaning debug mode will give a traceback leading right back
to your code raising the InstantiationError.

Adding a config source
======================

Config sources are pretty straightforward: they are mappings from a
section name to a ConfigSection subclass. The only tricky thing is the
combination of section references and caching. The general rule is "do
not expose the same ConfigSection in more than one way". If you do it
will be collapsed and instantiated once for every way it is exposed,
which is usually not what you want. An example::

 from pkgcore.config import basics, configurable

 def example():
     return object()

 @configurable({'ref': 'section_ref'})
 def nested(ref):
     return ref

 multi = basics.HardCodedConfigSection({'class': example})

 myconf = {
     'multi': multi,
     'bad': basics.HardCodedConfigSection({'class': nested, 'ref': multi})
     'good': basics.ConfigSectionFromStringDict({'class': 'nested',
                                                 'ref': 'multi'})

If you feed this to the ConfigManager and instantiate everything
"multi" and "good" will be identical but "bad" will be a different
object. For an explanation of why this happens see the implementation
notes in the next section.

You trigger a similar problem if you create a custom ConfigSection
subclass that bypasses central's collapse_named_section for named
section refs. If you somehow get at the referenced ConfigSection and
hand it to collapse_section you will most likely circumvent caching.
Only use collapse_section for unnamed sections.

ConfigManager tries not to extract more things from this mapping than
it has to. Specifically, it will not call __getitem__ before it needs
to instantiate the section or needs to know its type. However it
*will* iterate over the keys (section names) immediately to find
autoloads. If this is a problem (getting those names is slow) then
make sure the manager knows your config is "remote".

Implementation notes
====================

This code has evolved quite a bit over time. The current code/design
tries among other things to:

- Allow sections to contain both named and nameless/inline references
  to other sections.
- Allow serialization of the loaded config.
- Not do unnecessary work (if possibly not recollapse configs,
  definitely not trigger unnecessary imports, access configs
  unnecessarily, reinstantiate configs)
- Provide both end-user error messages that are complete enough to
  track down a problem in a complex nested config and tracebacks that
  reach back to actual buggy code for developers.

Overview from load_config() to instantiated repo
------------------------------------------------

When you call load_config() it looks up what config files are
available (/etc/pkgcore.conf, ~/.pkgcore.conf, /etc/make.conf) and
loads them. This produces a dict mapping section names to
ConfigSection instances. For the ini-format pkgcore.conf files this is
straightforward, for make.conf this is a lot of work done in
pkgcore.config.portage_conf. I'm not going to describe that module
here, read the source for details.

The ConfigSections have a pretty straightforward api: they work like
dicts but get passed a string describing what "type" the value should
be and a central.ConfigManager instance for reasons described later.
Passing in this "type" string when getting the value is necessary
because the way things like lists of strings are stored depends on the
format of the configuration file but the parser does not have enough
information to know it should parse as a list instead of a string. For
example, an ini-format pkgcore.conf could contain::

  [my-overlay-cache]
  class=pkgcore.cache.flat_hash.database
  auxdbkeys=DEPEND RDEPEND

We want to turn that auxdbkeys value into a list of strings in the ini
file parser code instead of in the central.ConfigManager or even
higher up because more exotic config sections may want to store this
in a different way (perhaps as a comma-separated list, or even as
"<el>DEPEND</el><el>RDEPEND</el>". But there is obviously not enough
information in the ini file for the parser to know this is meant as a
list instead of a string with a space in it.

central.ConfigManager gets instantiated with one or more of those
dicts mapping section names to ConfigSections. They're split up into
normal and "remote" configs which I'll describe later, let's assume
they're all "remote" for now. In that case no work is done when the
ConfigManager is instantiated.

Getting an actual configured object out of the ConfigManager is split
in two phases. First the involved config sections are "collapsed":
inherits are processed, values are converted to the right type,
presence of required arguments is checked, etc. Everything up to
actually instantiating the target class and actually instantiating any
section references it needs. The result of this work is bundled in a
CollapsedConfig instance. Actual instantiation is handled by the
CollapsedConfig instance.

The ConfigManager manages CollapsedConfig instances. It creates new
ones if required and makes sure that if a cached instance is available
it is used.

For the remainder of the example let's assume our config looks like
this::

  [spork]
  inherit=cache
  auxdbkeys=DEPEND RDEPEND

  [cache]
  class=pkgcore.cache.flat_hash.database

Running config.repo['spork'] runs
config.collapse_named_section('spork'). This first checks if this
section was already collapsed and returns the CollapsedConfig if it is
available. If it is not in the cache it looks up the ConfigSection
with that name in the dicts handed to the ConfigManager on
instantiation and calls collapse_section on it.

collapse_section first recursively finds any inherited sections (just
the "cache" section in this case). It then grabs the 'class' setting
(which is always of type 'callable'). In this case that's
"pkgcore.cache.flat_hash.database", which the ConfigSection imports
and returns. This is then wrapped in a config.basics.ConfigType. A
ConfigType contains the information necessary to validate arguments
passed to the callable. It uses the magic pkgcore_config_type
attribute if the callable has it and introspection for everything
else. In this case
pkgcore.cache.flat_hash.database.pkgcore_config_type is a ConfigHint
stating the "auxdbkeys" argument is of type "list".

Now that collapse_section has a ConfigType it uses it to retrieve the
arguments from the ConfigSections and passes the ConfigType and
arguments to CollapsedConfig's __init__. Then it returns the
CollapsedConfig instance to collapse_named_section.
collapse_named_section caches it and returns it.

Now we're back in the __getattr__ triggered by config.repo['spork'].
This checks if the ConfigType on the CollapsedConfig is actually
'repo', and returns collapsedConfig.instantiate() if this matches.

Lazy section references
-----------------------

The main reason the above is so complicated is to support various
kinds of references to other sections. Example::

  [spork]
  class=pkgcore.Spork
  ref=foon

  [foon]
  class=pkgcore.Foon

Let's say pkgcore.Spork has a ConfigHint stating the type of the "ref"
argument is "lazy_ref:foon" (lazy reference to a foon) and its typename is
"repo", and pkgcore.Foon has a ConfigHint stating its typename is
"foon". a "lazy reference" is an instance of basics.LazySectionRef,
which is an object containing just enough information to produce a
CollapsedConfig instance. This is not the most common kind of
reference, but it is simpler from the config point of view so I'm
describing this one first.

When collapse_section runs on the "spork" section it calls
section.get_value(self, 'ref:repo', 'section_ref'). "lazy_ref" in the
type hint is converted to just "ref" here because the ConfigSections
do not have to distinguish between lazy and "normal" references.
Because this particular ConfigSection only supports named
references it returns a LazyNamedSectionRef(central, 'ref:repo',
'foon'). This just gets handed to Spork's __init__. If the Spork
decides to call instantiate() on the LazyNamedSectionRef it calls
central.collapse_named_section('foon'), checks if the result is of
type foon, instantiates it and returns it.

The same thing using a dhcp-style config::

  spork {
      class pkgcore.Spork;
      ref {
          class pkgcore.Foon;
      };
  }

In this format the reference is an inline unnamed section. When
get_value(central, 'ref:repo', 'foon') is called it returns a
LazyUnnamedSectionRef(central, 'ref:repo', section) where section is a
ConfigSection instance for the nested section (knowing just that
"class" is "pkgcore.Foon" in this case). This is handed to
Spork.__init__. If Spork calls instantiate() on it it calls
central.collapse_section(self.section) and does the same type checking
and instantiating LazyNamedSectionRef did.

Notice neither Spork nor ConfigManager care if the reference is inline
or named. get_value just has to return a LazySectionRef instance
(LazyUnnamedSectionRef and LazyNamedSectionRef are subclasses of
this). How this actually gets a referenced config section is up to the
ConfigSection whose get_value gets called.

Normal section references
-------------------------

If Spork's ConfigHint defines the type of its "ref" argument as
"ref:foon" instead of "lazy_ref:foon" it gets handed an actual Foon
instance instead of a LazySectionRef to one. This is built on top of
the lazy reference code. For the ConfigSections nothing changes (the
same get_value call is made). But the ConfigManager now immediately
calls collapse() on the LazySectionRef, retrieving a CollapsedConfig
instance (for the "foon"). This is handed to the CollapsedConfig for
"spork", and when this one is instantiated the referenced
CollapsedConfig is also instantiated.

Miscellaneous details
---------------------

The support for nameless sections means neither ConfigSection nor
CollapsedConfig have a name attribute. This makes the error handling
code a bit tricky as it has to tag in the name at various points, but
this works better than enforcing names where it does not make sense
(means lots of unnecessary duplication of names when dealing with
dicts of HardCoded/StringBasedConfigSections).

The suppport for serialization of the loaded config means section_refs
cannot be instantiated straight away. The object used for
serialization is the CollapsedConfig which gives you both the actual
values and the type they have. If the CollapsedConfig contained
arbitrary instantiated objects serializing them would be impossible.
So it contains nested CollapsedConfigs instead.

Not doing unnecessary work is done by caching in two places. The
simple one is CollapsedConfig caching its instantiated value. This is
pretty straightforward. The more subtle one is ConfigManager caching
CollapsedConfigs by name. It is obviously a good idea to cache these
(if we didn't we would have to cache the instantiated value in the
ConfigManager). An alternative would be caching them by their
ConfigSection. This has the minor disadvantage of keeping the
ConfigSection in memory, and the larger one that it may break caching
for weird config sources that generate ConfigSections on demand. The
downside of caching by name is we have to make sure nothing generates
a CollapsedConfig for a named section in a way other than
collapse_named_section (handing the ConfigSection to collapse_section
bypasses caching).

This means a ConfigSection cannot return a raw ConfigSection from a
section_ref get_value call. If it was a ConfigSection that central
then collapsed and the reference was actually to a named section
caching is bypassed.

The need for a section name starting with "autoload" is also there to
avoid unnecessary work. Without this we would have to figure out the
typename of every section. While we can do that without entirely
collapsing the config we cannot avoid importing the "class", which
means load_config() would import most of pkgcore. That should
definitely be avoided.