Date:	Wed, 24 May 2000 13:39:17 -0400
From:	"Eric S. Raymond" <esr@snark.thyrsus.com>
To:	linux-kernel@vger.rutgers.edu, linux-kbuild@torque.net
Subject: Announcing CML2, a replacement for the kbuild system

For some weeks now, I have been developing a replacement for the kbuild 
system used to configure Linux kernels.  This effort has had the support of
Michael Elizabeth Chastain, the principal kbuild maintainer, and has
benefited from input by others on the kbuild list.

The project is not yet complete, but it has reached a beta stage at
which it is usable and in significant ways functionally superior to the
present system.  I am confident that it will complete.  I am announcing
now rather than holding off until I'm completely done because there
are some preparations which, if begun now, will significantly reduce
total transition costs.  These preparations will *not* break the
present kbuild system.

Why this project at all?  It all started when I realized that building
kernels is way too hard.  I wanted to simplify the configuration task
enough to make configuration accessible to non-gurus.  It needs to
have more policy options.  Rather than hundreds of questions like
"Include FOOBAR2317 driver?", the novice should see stuff like
"Compile in all modular drivers as modules without prompting?"

This just can't be done with the existing kbuild system. The existing
config-language programs are hard to read and modify, and the code
that interprets them has become a huge, unmaintainable hairball of
Tcl/Tk, C, makefiles, and shell.  It has all become terminally
brittle, and the maintainers agree that it needs to be nuked and
rebuilt from scratch.

It happens that I love writing domain-specific minilanguages, so I have
tackled this problem head-on.  I have designed a new configuration
language I call CML2 (the existing language I have retrospectively
named CML1).  The implementation has two parts:

1. I have implemented a CML2 compiler that validates CML2 rulesets and
generates a rulebase that can be used to drive a configuration
process.  I have translated almost all of the 7049 lines of CML1 in
the 2.3.99-pre9 source tree to validated CML2 (and *that*, believe me,
was hard work -- it took longer than the CML2 design and coding!).

2. I have written a configurator that is ready for testing. This
program reads in the rulebase and uses it to do the actual config-file
generation from a dialog with the user. Though not yet equipped with a
Tk interface, this program fully demonstrates the capabilities of
CML2.  It runs in either line-oriented or curses mode depending on the
display environment (line-oriented mode can be forced with a
command-line switch).

The line-oriented mode of the new configurator is much more powerful
than the original Configure. It's possible to move backward or jump
around in the configuration sequence; the constraints that were
expressed by if-then-else logic in CML1 are now checked every time the
value of a relevant symbol is changed.  It also has full access to the
help system.

The curses mode, unlike the old menuconfig code, also has full access
to the help text. It reports attempts to set symbol combinations that
would result in an invalid configuration.

The configurator should be able to present a Tk-based menu interface
when it detects that it's running on an X display.  This is the part
I haven't written yet.

The code needs more testing, which is one reason I am announcing now.
It would be useful for configure maintainers to begin running through
odd configurations to see if they can get it to misbehave.

The first alpha of CML2 is available at

	http://www.tuxedo.org/~esr/kbuild/

It includes the alpha implementation, documentation, and a transition
guide for maintainers of CML1 code.

OK, here's the bad news: the new system will not be an instant,
painless replacement for the old.  I tried hard, but there was just
too much cruft to clean up for that to be possible.

The major source of problems is, as you might expect, that 6747-line
mass of old code -- the new language is nontrivially different than
the old, and the CML1 corpus is so tangled and nasty that I am certain
I have made at least a few mistakes in the translation.  There are
a couple of places where I didn't understand the author's intentions
well enough to translate some particularly grotty CML1 code.  I'll need
some help untangling these knots.

I apologize for this, but the translation overhead would only have
been avoidable if CML1 had been good enough not to replace.  It's a
cost we have to pay to clean up a mess that would otherwise only have
gotten worse, and eventually have become a serious drag on kernel
development and porting.

There are some other minor problems, which we can fix up front.  Mostly
they have to do with cleaning up the configuration-symbol namespace (which
would be a good idea even if we planned to keep CML1).

Now the good news: we will win big by changing over.  Here are some
of the advantages of the new language:

1. Single parser and front end

CML1 had three different interpreters, none perfectly compatible with
any of the others.  CML2 has one rule compiler and one rulebase-interpreter
front end.  This will be good for consistency and economy.

2. A more expressive, easier-to-program configuration language

The rather spiky and cluttered shell-like syntax of CML1 is replaced
with a much simpler and more regular format resembling that of .netrc or
.fetchmailrc.  More importantly, the semantics of the language are
declarative rather than imperative -- a better match for the problem
domain, and thus more expressive and easier to code in.

3. Drastic reduction in code size and complexity

The 7049 lines of CML1 in the 2.3.99-pre9 kernel translate to a hair
less than 2400 lines of CML2, a reduction by a factor of about three.
The CML2 compiler and prototype interpreter are the same factor of
three smaller than the nearly 10,000 lines of code in the CML1
interpreters and tools.  Where CML1 is a complex mixture of C, shell,
Tcl/Tk, and Makefiles, CML2 is all be written in a single language
(Python).

4. Eliminates (or at least drastically reduces) lag between port configurations

The fact that the top-level CML1 files of the nine ports in the kernel
tree are separate means there have been plenty of opportunities for
the common code in them to suffer from version skew -- I point out
about a dozen bugs of this kind in the list of errors at the end of
this post.  CML2's design and compilation rules should effectively
prevent future bugs of this kind.

5. Clean separation between configuration language and configuration UI

CML decouples the configuration language from the configuration user
interface (they communicate with each other only through the compiled
rulebase).  This means that it will be relatively easy to improve the
UI and the language separately.

6. Internationalization

CML2 query prompts and menu banners are separated from the symbol
dependency declarations.  Thus CML2 system definitions can be 
internationalized and localized.

7. Language is fully documented

CML2 has a complete, explicit description.  Syntax, language semantics,
and front-end policy options are all discussed in detail.  

8. Policy-based options

The declarative semantics of CML2 makes it much easier to set up
and check interdependencies among symbols.  I have done only 
enough of this in the CML1 translation for demonstration purposes (there
are new symbols TUNING, EXPERT and WIZARD that change some visibilities).
Once CML2 is in place, it should be a relatively small effort to
give the user a rich set of policy and don't-bother-me options.

So, how do we get there from here?

Obviously, I have to finish the CML2 front end.  This is not a large
job; I already have a demonstrable prototype that runs in tty and
curses modes, and even on my heavy travel schedule I expect to have
the Tk version ready in about two weeks.

I have designed the CML2 implementation to coexist with CML1, so both
methods can be used while CML2 is being field-tested and debugged.  I
anticipate a phase-in over three or four point releases during 2.5.x,
followed by a back-port to 2.4.x.  Once CML2 is reported OK by the
various porting groups, Linus can quietly nuke the CML1 machinery.

There are a couple of preparation steps that can fruitfully begin now
and should happen before 2.4 in order to minimize backporting hassles
later.  

1. Notably, I would appreciate it if config-file maintainers made
the following changes in those files and relevant C code:

CONFIG_6xx			-> CONFIG_PPC_6xx
CONFIG_4xx			-> CONFIG_PPC_4xx
CONFIG_PPC64			-> CONFIG_PPC_64
CONFIG_8260			-> CONFIG_PPC_8260
CONFIG_8xx			-> CONFIG_PPC_88x
CONFIG_060_WRITETHROUGH		-> CONFIG_M68060_WRITETHROUGH
CONFIG_21285_WATCHDOG		-> CONFIG_DC21285_WATCHDOG
CONFIG_3C515			-> CONFIG_ISA3C515
CONFIG_8139TOO			-> CONFIG_RTL8139
CONFIG_82C710_MOUSE		-> CONFIG_CT82C710_MOUSE
CONFIG_977_WATCHDOG		-> CONFIG_WB83C977_WATCHDOG
CONFIG_3215			-> CONFIG_IBM3215
CONFIG_3215_CONSOLE		-> CONFIG_IBM3215_CONSOLE

The reason for these is that CML2 symbol names drop the CONFIG_
prefix.  It's unneeded clutter, and made CML1 programs harder to read
(the eye-brain systems that handle spelling look for prefix matches to
recognize things).

Also, I had to change the KEYBOARD and MOUSE symbols used in the MIPS branch
to MIPS_KEYBOARD and MIPS_MOUSE.  This is because the MOUSE symbol
seems to be used in different ways on different architectures (notably
in the Intel branch).

2. I also found some apparent errors.  I need these explained so I'll
know how to handle them in the translation.  A summary of these apparent
errors is included at the end of this post.

3. Those are the easy parts.  The hard part is that I'd like to ask
config maintainers to eyeball-check the CML2 translation of their work
*now*.  Where I am most likely to have erred is in setting visibility
constraints by architecture.  Ideally, I'd like everyone to have
confidence that the translation is correct by the time the Tk-based
front end comes out.

Presently the entire CML2 translation lives in a single file, rather than
being distributed into per-subdirectory files like the CML1 corpus.  This
is a temporary expedient to make the transition easier.  CML2's "source"
facility is quite powerful enough to support distributing the information
later on.

The current CML2 menu tree is ugly and poorly organized -- that is to
say, it has changed relatively little from the CML1 version.  I am
deliberately refraining from large changes yet.  Once we have tested
and switched over to CML2, it will be possible to do a complete
redesign of the kbuild user experience.  The most important feature of
CML2 is that it will give us the capability to explore that design
space without risking breaking the ability to build kernels at all.

Here are the apparent errors I found in the CML1 corpus:

-----------------------------------------------------------------------------
There is what appears to be an error in the M68K configuration sequence.
Inside a PARPORT guard, the question 'Q40 Parallel port' sets PARPORT again.
I created a PARPORT_Q40 symbol and set it from this question.

The symbols SGI occurs in conditionals in config files but are never
set or associated with a query, nor are they used in C code anywere.

The symbol SUN3 is used in conditionals and set to n at one point, but there
is no place where it is set to y.

The symbol FB_CONSOLE is set at one point but
never used in either C or config language code.

The symbol ABSTRACT_CONSOLE is not used in C code, nor set anywhere in config
code.

The symbol AIC7XXX_TAGGED_QUEUEING is set in the sparc64 configuration
code, but not used in C code.  I suspect it should be
AIC7XXX_TCQ_ON_BY_DEFAULT.

The symbol ADB_PMU68K defined in the M68K driver is not referenced anywhere 
in the config code and not used in the C code.  It seems to be a misspelling
of ADB_PMU.  I have eliminated it.

In the ARM port configuration, symbols ARCH_TBOX and ARCH_SHARK and
ARCH_NEXUSPCI and ARCH_NEXUSPCI are referenced but never associated
with a query or defined.

There are two symbols in the configuration code that seem to relate
to endianness on processors that can operate in either big-endian 
or little-endian mode.  One is CPU_LITTLE_ENDIAN (in the MIPS ports)
and the other is LITTLE_ENDIAN (in the SuperH port).  Neither is used 
in the C code; CPU_LITTLE_ENDIAN is used in a guard, once, in the
MIPS32 config.  I have changed LITTLE_ENDIAN to CPU_LITTLE_ENDIAN.

The symbol PRINTER_READBACK is queried for once, but never used in the
config or C code.  I suspect it should have been asking for PARPORT_1284.
-----------------------------------------------------------------------------

Note: this announcement was crossposted to the linux-kernel and linux-kbuild
lists.  You may want to use group reply in respnding to it to reach both
populations.

Port maintainers and others with a continuing interest in the development
of CML2 should probably join the kbuild list -- subscribe in the usual
way via linux-kbuild-request@torque.net
-- 
		<a href="http://www.tuxedo.org/~esr">Eric S. Raymond</a>

The Bible is not my book, and Christianity is not my religion.  I could never
give assent to the long, complicated statements of Christian dogma.
	-- Abraham Lincoln

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.rutgers.edu
Please read the FAQ at http://www.tux.org/lkml/