Re: Why does the lexer convert text integer lexemes to binary integers? I thought that lexers should be simple?

You are asking the wrong question. You are optimizing at the wrong level.
Stop.


Not that long ago, I wrote an article on Quora about this exact phenomenon.


https://www.quora.com/What-do-most-programmers-do-when-optimizing-code-that-is-essentially-wrong/answer/Christopher-F-Clark-1?ch=10&oid=337787257&share=2807f9fb&srid=20qA&target_type=answer


You are focusing on the trivial, the irrelevant. It is unlikely that
having the lexer convert integers (or floats or quaternions) into a binary
representation is a sufficiently expensive operation to make sense fretting
about it. And I mean that both in terms of runtime and time spent thinking
about it. Just do whatever lexer you are following from as an example does
and assume they made the right choice. The only time you should pay
attention to it, is once you have something working correctly and you have
measured its performance and determined that the code in question is
actually a bottleneck that is significantly impacting performance. Then,
you have a reason to revisit that choice.


And, when you do, you are as likely to find that the call to atoi is as
much a problem as having the lexer do it. atoi is going to rescan those
digits causing twice as much work as you would have done by building the
numeric representation while you lexed it. Of course, that alternative has
its own issues. It's clearly more complex code in the lexer.


But, again, you shouldn't be worrying about any of that, until you have
isolated it as an actual problem. Otherwise, figure out something simple
that works. Having the lexer call atoi seems pretty simple. It is also
likely to work over a rather large range of inputs. So, you haven't likely
bought yourself any problems by doing that.


Now, if you are doing SQL (or PL/I) float decimal numbers or handling of
BIGINTs (arbitrarily large integers) or reduced rationals, you might want
something more complex. But that's isn't the problem in this case.


Don't spend your time worrying about details until you know they are
important details.


Kind regards,
Chris


--
******************************************************************************


Chris Clark email:
christopher.f.clark@compiler-resources.com
Compiler Resources, Inc. Web Site: http://world.std.com/~compres
23 Bailey Rd voice: (508) 435-5016
Berlin, MA 01503 USA twitter: @intel_chris
------------------------------------------------------------------------------

Return to the comp.compilers page.
Search the comp.compilers archives again.