The dragon book says separating lexical analysis and parsing is beneficial, so why doesn't ANTLR separate them?

Hi Folks,


Page 84-85 of the dragon book [1] says:


There are several reasons for separating the analysis phase of compiling into
lexical analysis and parsing.


1. Simpler design is perhaps the most important consideration. The separation
of lexical analysis from syntax analysis often allows us to simplify one or
the other of these phases. For example, a parser embodying the conventions for
comments and white space is significantly more complex than one that can
assume comments and white space have already been removed by a lexical
analyzer. It we are designing a new language, separating the lexical and
syntactic conventions can lead to a cleaner overall language design.


2. Compiler efficiency is improved. A separate lexical analyzer allows us to
construct a specialized and potentially more efficient processor for the task.
A large amount of time is spent reading the source program and partitioning it
into tokens. Specialized buffering techniques for reading input characters and
processing tokens can significantly speed up the performance of a compiler.


3. Compiler portability is enhanced. Input alphabet peculiarities and other
device-specific anomalies can be restricted to the lexical analyzer. The
representation of special or non-standard symbols, such as the up-arrow in
Pascal, can be isolated in the lexical analyzer.


Specialized tools have been designed to help automate the construction of
lexical analyzers and parsers when they are separated.
----------------------------
Those seem like compelling reasons for separating the lexical analysis from
parsing, so why does ANTLR not do so; i.e., why does ANTLR combine them?


/Roger


[1] Compilers: Principles, Techniques, and Tools by Aho, Sethi, and Ullman.

Return to the comp.compilers page.
Search the comp.compilers archives again.