Lesson 3: Writing Your First ANTLR Grammar

Welcome back! Now that you understand the fundamentals of language parsing, it’s time to dive into writing your first ANTLR grammar. In this lesson, we’ll explore how to craft a grammar for a simple DSL, focusing on defining lexer and parser rules. By the end of this lesson, you will have a working grammar that can parse and validate input, which we will eventually use to define and generate ERP/CRM features.

Table of Contents

1. Defining a DSL Syntax

2. Writing Lexer Rules

3. Writing Parser Rules

4. Testing the Grammar with ANTLR

5. Practical Example: A Basic CRUD Language

6. Conclusion and Next Steps

1. Defining a DSL Syntax

Before writing any code, we must define the syntax of our DSL. Since the goal of our DSL is to describe the features of an ERP/CRM system or multi-sided marketplace, we need to focus on the basic entities, actions, and relationships that our system will support. Let’s start simple and define a DSL that allows us to describe models, attributes, and relationships.

Example DSL Syntax:

entity User {

id: Integer;

name: String;

email: String;

}

entity Order {

id: Integer;

date: Date;

total: Decimal;

user: User;

}

In this simple DSL, we define two entities: User and Order. Each entity has attributes with types (Integer, String, Date, etc.) and relationships (e.g., Order is related to User through the user attribute).

Key Elements of the DSL:

• Entities: Represent tables in a database, such as User and Order.

• Attributes: Define the columns for each table, such as name, email, and date.

• Relationships: Represent associations between entities, such as an Order belonging to a User.

2. Writing Lexer Rules

Lexer rules are responsible for breaking the input into tokens. For our DSL, we need to define tokens for keywords like entity, data types (String, Integer, etc.), identifiers (names of entities and attributes), and punctuation (curly braces, colons, semicolons).

Basic Lexer Rules for the DSL:

Here’s a simple ANTLR grammar with the lexer rules:

lexer grammar EntityLexer;

ENTITY : ‘entity’;

INTEGER : ‘Integer’;

STRING : ‘String’;

DATE : ‘Date’;

DECIMAL : ‘Decimal’;

COLON : ’:’;

SEMICOLON : ’;’;

LBRACE : ’{’;

RBRACE : ’}’;

ID : [a-zA-Z_][a-zA-Z0-9_]*;

WS : [ \t\r\n]+ -> skip; // Skip whitespace and newlines

Explanation of Lexer Rules:

• Keywords (ENTITY, INTEGER, STRING, etc.): These are tokens representing reserved words in the DSL.

• Punctuation (COLON, SEMICOLON, LBRACE, RBRACE): Tokens representing symbols used in the language, like {, }, :, and ;.

• Identifiers (ID): Matches variable names like User, Order, name, etc.

• Whitespace (WS): Skips spaces, tabs, and newlines, so they don’t interfere with parsing.

3. Writing Parser Rules

Parser rules define the grammar’s structure, ensuring that the input follows a valid sequence of tokens. For example, an entity definition must include the entity keyword, a name, and attributes enclosed in curly braces.

Parser Rules for the DSL:

Let’s define a simple parser grammar to parse entity definitions and their attributes.

parser grammar EntityParser;

options { tokenVocab=EntityLexer; } // Import lexer tokens

program : entityDefinition+ EOF ;

entityDefinition : ENTITY ID LBRACE attribute* RBRACE ;

attribute : ID COLON type SEMICOLON ;

type : INTEGER | STRING | DATE | DECIMAL ;

Explanation of Parser Rules:

• program: Represents the overall structure of the DSL. It consists of one or more entity definitions followed by the end of the file (EOF).

• entityDefinition: Parses an entity definition, which consists of the entity keyword, an identifier (the entity name), and a list of attributes enclosed in curly braces.

• attribute: Defines the attributes of an entity. Each attribute has a name, followed by a colon, a type, and a semicolon.

• type: Defines the possible data types for attributes (Integer, String, Date, and Decimal).

How ANTLR Interprets This Grammar:

Given the following input:

entity User {

id: Integer;

name: String;

email: String;

}

The lexer breaks it down into tokens: ENTITY, ID, LBRACE, ID, COLON, INTEGER, SEMICOLON, etc.

The parser ensures that these tokens follow the rules defined in the grammar. It checks that each entity has a valid structure and that the attributes are correctly typed.

4. Testing the Grammar with ANTLR

Now that we’ve defined the lexer and parser rules, let’s test the grammar to see if it works as expected.

Step 1: Write the Grammar File

Create a new file called Entity.g4:

grammar Entity;

lexer grammar EntityLexer;

ENTITY : ‘entity’;

INTEGER : ‘Integer’;

STRING : ‘String’;

DATE : ‘Date’;

DECIMAL : ‘Decimal’;

COLON : ’:’;

SEMICOLON : ’;’;

LBRACE : ’{’;

RBRACE : ’}’;

ID : [a-zA-Z_][a-zA-Z0-9_]*;

WS : [ \t\r\n]+ -> skip;

parser grammar EntityParser;

options { tokenVocab=EntityLexer; }

program : entityDefinition+ EOF ;

entityDefinition : ENTITY ID LBRACE attribute* RBRACE ;

attribute : ID COLON type SEMICOLON ;

type : INTEGER | STRING | DATE | DECIMAL ;

Step 2: Generate Lexer and Parser Files

Use ANTLR to generate the necessary Python files:

antlr4 -Dlanguage=Python3 Entity.g4

This will generate several files, including EntityLexer.py, EntityParser.py, and other necessary components.

Step 3: Write a Python Script to Test the Parser

Now, write a Python script that uses the generated lexer and parser to process input from the DSL.

from antlr4 import *

from EntityLexer import EntityLexer

from EntityParser import EntityParser

def main():

Input DSL code to parse

input_stream = InputStream('''

entity User {

id: Integer;

name: String;

email: String;

}

''')

Lexer and parser setup

lexer = EntityLexer(input_stream)

token_stream = CommonTokenStream(lexer)

parser = EntityParser(token_stream)

Parse the input

tree = parser.program()

print(tree.toStringTree(recog=parser)) # Output the parse tree structure

if name == ‘main’:

main()

Step 4: Run the Python Script

python test_parser.py

Expected Output:

(program (entityDefinition entity User { (attribute id : Integer ;) (attribute name : String ;) (attribute email : String ;) }) EOF)

The parser successfully processes the input DSL and constructs the correct parse tree, indicating that the input follows the rules defined in the grammar.

5. Practical Example: A Basic CRUD Language

Let’s expand the DSL slightly to support basic CRUD operations (Create, Read, Update, Delete) for entities. This will give us a more practical and useful language.

Updated DSL Syntax:

entity User {

id: Integer;

name: String;

email: String;

}

create User {

name: “John Doe”;

email: “john.doe@example.com”;

}

In this expanded version, we’ve added a create statement that allows us to insert data into an entity.

Updating the Lexer and Parser Rules:

lexer grammar EntityLexer;

ENTITY : ‘entity’;

CREATE : ‘create’;

INTEGER : ‘Integer’;

STRING : ‘String’;

DATE : ‘Date’;

DECIMAL : ‘Decimal’;

COLON : ’:’;

SEMICOLON: ’;’;

LBRACE : ’{’;

RBRACE : ’}’;

EQUAL : ’=’;

QUOTE : ’”’;

ID : [a-zA-Z_][a-zA-Z0-9_]*;

STRING_LITERAL : ’”’ (~[“\r\n])* ’”’;

WS : [ \t\r\n]+ -> skip;

parser grammar EntityParser;

options { tokenVocab=EntityLexer; }

program : (entityDefinition | createStatement)+ EOF ;

entityDefinition : ENTITY ID LBRACE attribute* RBRACE ;

attribute : ID COLON type SEMICOLON ;

createStatement : CREATE ID LBRACE keyValuePair* RBRACE ;

keyValuePair : ID EQUAL STRING_LITERAL SEMICOLON ;

type : INTEGER | STRING | DATE | DECIMAL ;

This grammar supports both defining entities and creating instances of those

entities.

Testing the Expanded Grammar:

Update your Python script to test the expanded grammar.

def main():

input_stream = InputStream('''

entity User {

id: Integer;

name: String;

email: String;

}

create User {

name: “John Doe”;

email: “john.doe@example.com”;

}

''')

lexer = EntityLexer(input_stream)

token_stream = CommonTokenStream(lexer)

parser = EntityParser(token_stream)

tree = parser.program()

print(tree.toStringTree(recog=parser))

if name == ‘main’:

main()

Running this should show that the DSL can now handle entity definitions and create statements.

6. Conclusion and Next Steps

In this lesson, we’ve:

• Defined the syntax of a simple DSL for describing entities and CRUD operations.

• Written lexer and parser rules to handle this DSL.

• Tested our grammar by generating lexer and parser code with ANTLR and running it in Python.

This basic framework will serve as the foundation for our DSL, which will grow to support more complex operations in an ERP/CRM system.

Practical Exercise

Task: Extend the grammar to support Read, Update, and Delete operations, following the same pattern we used for create. Ensure that the DSL can handle these operations and test it using Python.

Hints:

• Define read, update, and delete statements.

• Update the lexer to recognize new keywords.

• Add parser rules to handle these statements.

Sample DSL:

entity User {

id: Integer;

name: String;

email: String;

}

create User {

name: “John Doe”;

email: “john.doe@example.com”;

}

update User {

id: 1;

email: “new.email@example.com”;

}

delete User {

id: 1;

}

Looking Ahead

In the next lesson, we will focus on integrating ANTLR with Python more deeply, using the visitor pattern to traverse the parse tree and generate Python code that integrates with Flask-AppBuilder.

Feel free to reach out with any questions. See you in the next lesson!