In computer science, lexical analysis is the process of converting a sequence of characters into a sequence of tokens, i. e. meaningful character strings. A program or function that performs lexical analysis is called a lexical analyzer, lexer, tokenizer,[1] or scanner, though "scanner" is also used for the first stage of a lexer. A lexer is generally combined with a parser, which together analyze the syntax of computer languages, such as in compilers for programming languages, but also HTML parsers in web browsers, among other examples.
Strictly speaking, a lexer is itself a kind of parser – the (context-free) syntax of the language is divided into two pieces: the lexical syntax (word structure), which is processed by the lexer; and the phrase structure, which is processed by the (phrase-level) parser. The lexical syntax is usually a regular language, whose atoms are individual characters, while the phrase syntax is usually a context-free language, whose atoms are words (tokens produced by the lexer). While this is a common separation, alternatively, a lexer can be combined with the parser in scannerless parsing.
Source:
1. Wikipedia ( http://en.wikipedia.org/wiki/Lexical_analysis )
2. WikiBooks (http://en.wikibooks.org/wiki/Compiler_Construction/Lexical_analysis)
1. Wikipedia ( http://en.wikipedia.org/wiki/Lexical_analysis )
2. WikiBooks (http://en.wikibooks.org/wiki/Compiler_Construction/Lexical_analysis)
1 comment:
Lexical Analysis or Linear Analysis or Scanning, in which the stream of characters making up the source program is read from left-to-right and grouped in to tokens, sequence of characters having a collective meaning. The blanks separating the characters of the tokens and the comments statements appearing within the program would normally be eliminated during lexical analysis. There are several reasons for separating the analysis phase of compiling into lexical analysis and parsing.
1)Simpler design is perhaps the most important consideration. The separation of lexical analysis from syntax analysis often allows us to simplify one or the other of these phases.
For example, a parser embodying the conventions for comments and white spaces is significantly more complex than one that can assume comments and white space have already been removed by a lexical analyzer. if we are designing a new language, separating the lexical and syntactic conventions can lead to a cleaner overall language design.
2)Compiler efficiency is improved. A separate lexical analyzer allows us to construct a specialized and potentially more efficient processor for the task. A large amount of time
is spent reading the source program and partitioning it into tokens. Specialized buffering techniques for reading input characters and processing tokens can significantly speed up the performance of a compiler.
3)Compiler portability is enhanced. Input alphabets peculiarities and other device-specific anomalies can be restricted to the lexical analyzer. The representation of special or non-standard symbols can be isolated in the lexical analyzer.
Specialized tools have been designed to help automate the construction of lexical analyzers and parsers when they are separated. LEX is a widely used tool to specify lexical analyzers for a variety of languages. We refer to the tool as the LEX compiler, and to its input specification as the LEX language. LEX is generally used in the manner of a lexical analyzer, is prepared by creating a program lex.l in the LEX language. Then, lex.l is run through the LEX compiler to produce a C program lex.yy.c. The program lex.yy.c consists of a tabular representation of a transition diagram constructed from the regular expressions of lex.l, together with a standard routine that uses the table to recognize lexemes. The actions associated with regular expressions in lex.l are pieces of C code and are carried over directly to lex.yy.c. Finally, lex.yy.c is run through the compiler to produce an object program a.out, which is the lexical analyzer that transforms an input stream into sequence of tokens.
Post a Comment