I was wondering if the following was possible in lex.
Let's say we have the following text:

+
A+B+C

Can we generated a rule that looks at the first delimiter token (+) and

use that delimiter to read the rest of the message?  The delimiter can
be dynamic meaning it can also be sent like the following and still be
read using the same lex rule.

*
A*B*C

or

#
A#B#C

Any help will be appreciated.  Thanks.

[ Post a follow-up to this message ]

David Shin wrote:
>
> I was wondering if the following was possible in lex.
> Let's say we have the following text:
>
> +
> A+B+C
>
> Can we generated a rule that looks at the first delimiter token (+) and
>
> use that delimiter to read the rest of the message?  The delimiter can
> be dynamic meaning it can also be sent like the following and still be
> read using the same lex rule.

This is probablky difficult with plain lex, but if you're
using GNU flex, you could use lexer contexts.

Erik
--
+-----------------------------------------------------------+
Erik de Castro Lopo
+-----------------------------------------------------------+
"If POSIX threads are a good thing, perhaps I don't want to know what
they're better than."                                   -- Rob Pike

[ Post a follow-up to this message ]

David Shin wrote:
> I was wondering if the following was possible in lex.
> Let's say we have the following text:
>
> +
> A+B+C
>
> Can we generated a rule that looks at the first delimiter token (+) and
>
> use that delimiter to read the rest of the message?  The delimiter can
> be dynamic meaning it can also be sent like the following and still be
> read using the same lex rule.
>
> *
> A*B*C
>
> or
>
> #
> A#B#C
>
> Any help will be appreciated.  Thanks.

If the delimeter can be literally anything (as opposed to one of a
special set of characters), then this is extremely difficult to do with
lex directly. Your best bet is a pre-processor that passes tokens to
lex. Define a special token that indicates your delimeter and pass all
delimeters to the lexer as that special token.

A very simple solution would be to always use '' as the delimeter and
translate all ''s in the input to '\' (unless '' was the delimeter
chosen by the input as well). Lex can handle this with no problem -- C
uses constructs much like this.

DS

[ Post a follow-up to this message ]

davids@webmaster.com writes:

> David Shin wrote: 
>
> If the delimeter can be literally anything (as opposed to one of a
> special set of characters), then this is extremely difficult to do with
> lex directly. Your best bet is a pre-processor that passes tokens to
> lex. Define a special token that indicates your delimeter and pass all
> delimeters to the lexer as that special token.
>
> A very simple solution would be to always use '' as the delimeter and
> translate all ''s in the input to '\' (unless '' was the delimeter
> chosen by the input as well). Lex can handle this with no problem -- C
> uses constructs much like this.

Well, I don't remember how lex works exactly, but with flex, you could
hack the character -> character-class map (changing temporarily the
character class of the delimiter character).  The main problem would
be to make sure that no lookahead occurs too soon.

--
__Pascal Bourguignon__                     http://www.informatimago.com/
The rule for today:
Touch my tail, I shred your hand.
New rule tomorrow.

[ Post a follow-up to this message ]

Thanks for the prompt response guys.  Does anybody know a parser that
could achieve this?

And Erik, can you please elaborate on how I can use lexer contexts in
GNU flex?

Thanks
David.



Erik de Castro Lopo wrote:

> This is probablky difficult with plain lex, but if you're
> using GNU flex, you could use lexer contexts.
>
> Erik
> --
> +-----------------------------------------------------------+
>   Erik de Castro Lopo
> +-----------------------------------------------------------+
> "If POSIX threads are a good thing, perhaps I don't want to know what
> they're better than."                                   -- Rob Pike

[ Post a follow-up to this message ]

David Shin wrote:
>
> And Erik, can you please elaborate on how I can use lexer contexts in
> GNU flex?

On this page:

http://www.gnu.org/software/flex/ma..._mono/flex.html


read the section titles "Start Conditions".

Erik
--
+-----------------------------------------------------------+
Erik de Castro Lopo
+-----------------------------------------------------------+
"This is like creating laws against blasphemy and then complaining that
unbelievers can't come up with any logical argument against the existence
of God"  -- www.infoanarchy.org on the Digital Millenium Copyright Act

[ Post a follow-up to this message ]