Re: The dragon book says separating lexical analysis and parsing is beneficial, so why doesn't ANTLR separate them?

On Thursday, June 9, 2022 at 9:50:50 AM UTC-7, Roger L Costello wrote:
> 2. ...
> A large amount of time is spent reading the source program and partitioning it
> into tokens. Specialized buffering techniques for reading input characters and
> processing tokens can significantly speed up the performance of a compiler.


In any decent compiler this amount of time is relatively small compared to optimizations.
Unless we're talking about JIT, which is a different kind of compiling.


> 3. Compiler portability is enhanced. Input alphabet peculiarities and other
> device-specific anomalies can be restricted to the lexical analyzer. The
> representation of special or non-standard symbols, such as the up-arrow in
> Pascal, can be isolated in the lexical analyzer.


ASCII is supported out of the box nowadays. Unicode is available and simply
parsing and storing Unicode code points is straightforward (processing
Unicode text, especially displaying, is problematic, but that should hardly
affect a programming language or its compiler much).


> Those seem like compelling reasons for separating the lexical analysis from
> parsing,
...
> [1] Compilers: Principles, Techniques, and Tools by Aho, Sethi, and Ullman.


The book is old and doesn't quite reflect the current state of things, IMO.


Alex
[My impression is that the lexer can often take significant time since it has to look
at every character in the input, but the parser is fast unless you're doing something
strange like very ambiguous Earley parsing. -John]

Return to the comp.compilers page.
Search the comp.compilers archives again.