Why does the lexer convert text integer lexemes to binary integers? I thought that lexers should be simple?

Hi Folks,


A common example in books on Lex/Flex and Yacc/Bison is evaluating arithmetic
expressions. When the lexer encounters an integer lexeme, it casts the lexeme
to a binary integer and returns the value to the parser. The lexer contains a
rule that looks something like this:


{INTEGER} { yylval.intval = atoi(yytext); return NUMBER; }


But, but, but, ...


Countless times on this list I have been told: Keep the lexer simple!


By converting the lexeme to an integer, the lexer has assumed that the parser
needs/wants a binary integer, not a text number. How does the lexer know what
the parser needs/wants? That seems like knowledge the lexer shouldn't have if
the lexer is to be simple. Further, even if one parser needs/wants a binary
integer value, that parser might be swapped out at a later date and replaced
with a different parser that wants the text number.


It seems to me that the lexer should return to the parser the text number and
it is the responsibility of the parser to convert the value to an integer data
type if it desires.


What do you think?


/Roger
[I think the lexer should provide the tokens that the parser needs. If
integers are always handled as numbers, convert them, if not, don't.
If the parser does one and later changes to do the other, you can
change the lexer, too. -John]

Return to the comp.compilers page.
Search the comp.compilers archives again.