Parser analyzes a code syntactically according to the rules of the language's grammar. The parsing phase determines if the input code can be used to form a string of tokens according to the defined grammar. A parse tree is constructed in this phase. Parser defines functions to organize language into a data structure called AST. The parser defined in this recipe uses a recursive decent parser technique which is a top-down parser, and uses mutually recursive functions to build the AST.
We must have the custom-defined language, that is the TOY language in this case, and also a stream of tokens generated by the lexer.
Define some basic value holders in our TOY parser as shown in the following:
- Open the
toy.cppfile as follows:$ vi toy.cpp
- Define a global static variable to hold the current token from the lexer as follows:
static int Current_token;
- Define a function to get the next token from the input stream from the lexer as follows:
static void next_token() { Current_token = get_token(); } - The next step is to define functions for expression parsing by using the AST data structure defined in the previous section.
- Define a generic function to call specific parsing functions according to the types of tokens determined by the lexer, as shown in the following:
static BaseAST* Base_Parser() { switch (Current_token) { default: return 0; case IDENTIFIER_TOKEN : return identifier_parser(); case NUMERIC_TOKEN : return numeric_parser(); case '(' : return paran_parser(); } }
The stream of input is tokenized and fed to the parser. Current_token holds the token to be processed. The type of token is known at this stage and the corresponding parser functions are called to initialize ASTs.
- In next few recipes, you will learn how to parse different expressions. For more detailed parsing of the C++ language implemented in Clang, refer to it works: http://clang.llvm.org/doxygen/classclang_1_1Parser.html.