Com 212 intro to system programming book Theory

Download 0.65 Mb.

View original pdf

Page	45/72
Date	13.05.2021
Size	0.65 Mb.
	#56617

1 ... 41 42 43 44 45 46 47 48 ... 72

com-212-introduction-to-system-programming-theory
9833 SS1 FISHERY LESSON NOTE

lexeme token type
sum
IDENT
=
ASSIGN_OP
3
NUMBER
+
ADD_OP
2
NUMBER
;
SEMICOLON
Tokens are frequently defined by regular expressions, which are understood by a lexical analyzer generator such as lex. The lexical analyzer (either generated automatically by a tool like lex, or handcrafted) reads in a stream of characters, identifies the lexemes in the stream, and categorizes them into tokens. This is called "tokenizing." If the lexer finds an invalid token, it will report an error.

Page | 48 Following tokenizing is parsing. From there, the interpreted data maybe loaded into data structures, for general use, interpretation, or compiling. Consider a text describing a calculation
46 - number of (cows);
The lexemes here might be "
46
", "
-
", "
number_of ", "
(
" , "cows " , "
)
" and "
;
". The lexical analyzer will denote lexemes "
46
" as 'number, "
-
" as 'character' and "
number_of " as a separate token. Even the lexeme "
;
" in some languages (such as C) has some special meaning.

Download 0.65 Mb.

Share with your friends:

1 ... 41 42 43 44 45 46 47 48 ... 72