Nyimbi Odero

Series 01 · a numbered run · kept in public

Lesson 5: Semantic Analysis, Validation, and Advanced Code Generation

Welcome back! So far, we’ve written a DSL grammar, used the visitor pattern to traverse the parse tree, and generated Flask-AppBuilder models. In this lesson, we’ll dive deeper into semantic analysis, which includes validating the DSL input, performing type checking, and ensuring consistent entity relationships. We’ll also extend our code generation logic to handle more advanced scenarios such as generating APIs and views in Flask-AppBuilder.

Table of Contents

  1. Introduction to Semantic Analysis

  2. Implementing Semantic Validation

  3. Type Checking and Relationship Validation

  4. Extending the Visitor for API and View Generation

  5. Practical Example: Generating Full Flask-AppBuilder Applications

  6. Conclusion and Next Steps

1. Introduction to Semantic Analysis

What is Semantic Analysis?

Semantic analysis is the phase of language processing where we verify that the input adheres to the semantic rules of the language. While the parser ensures the input is syntactically correct, semantic analysis ensures that it “makes sense.” For example:

  • Ensuring that types match in assignments.

  • Checking for undefined entities or attributes.

  • Validating relationships between entities (e.g., one-to-many, many-to-many).

Why is Semantic Analysis Important in DSLs?

  • Error Detection: It allows us to catch logical errors in the DSL code.

  • Consistency: It ensures that relationships between entities are correctly defined and consistent.

  • Type Safety: It prevents assigning incompatible types to attributes or relationships.

In our case, semantic analysis will help ensure that:

  • Attributes are defined with valid types.

  • Relationships between entities are properly defined.

  • Unique constraints and primary keys are respected.

2. Implementing Semantic Validation

Basic Semantic Checks

In our DSL, we need to validate several aspects of the entity definitions. For example, we want to ensure that:

  1. Unique Entity Names: No two entities have the same name.

  2. Unique Attribute Names: Within an entity, attribute names should be unique.

  3. Valid Types: All attributes must use valid types (Integer, String, Date, Decimal).

  4. Valid Relationships: Ensure that relationships reference existing entities.

Modifying the Visitor for Semantic Checks

We can modify the visitor to perform these semantic checks while traversing the parse tree. Here’s how to implement the basic validations:

class SemanticVisitor(ParseTreeVisitor):

def init(self):

Store entities and their attributes

self.entities = {}

self.errors = []

def visitEntityDefinition(self, ctx):

Get the entity name

entity_name = ctx.ID().getText()

Check for duplicate entity names

if entity_name in self.entities:

self.errors.append(f”Error: Duplicate entity name ‘{entity_name}’”)

return

Initialize a dictionary to store attributes

attributes = {}

Visit each attribute in the entity

for attr_ctx in ctx.attribute():

attr_name, attr_type = self.visit(attr_ctx)

Check for duplicate attributes within the entity

if attr_name in attributes:

self.errors.append(f”Error: Duplicate attribute ‘{attr_name}’ in entity ‘{entity_name}’”)

else:

attributes

attrnameattr_name

= attr_type

Store the entity and its attributes

self.entities

entitynameentity_name

= attributes

def visitAttribute(self, ctx):

Extract the attribute name and type

attr_name = ctx.ID().getText()

attr_type = ctx.type().getText()

Check if the type is valid

valid_types =

′Integer′,′String′,′Date′,′Decimal′'Integer', 'String', 'Date', 'Decimal'

if attr_type not in valid_types:

self.errors.append(f”Error: Invalid type ‘{attr_type}’ for attribute ‘{attr_name}’”)

return attr_name, attr_type

def visitProgram(self, ctx):

Visit all entity definitions in the program

self.visitChildren(ctx)

Print any semantic errors found

if self.errors:

for error in self.errors:

print(error)

else:

print(“No semantic errors found.”)

Explanation:

  • self.entities: Stores the entities and their attributes.

  • self.errors: Collects semantic errors found during the tree traversal.

  • visitEntityDefinition: Checks for duplicate entity names and duplicate attribute names within an entity.

  • visitAttribute: Ensures that the attribute type is valid.

Step 1: Testing the Semantic Analysis

def main():

input_stream = InputStream('''

entity User {

id: Integer;

name: String;

email: String;

name: String; // Duplicate attribute

}

entity Order {

id: Integer;

total: Float; // Invalid type

user: User;

}

entity User { // Duplicate entity

id: Integer;

}

''')

Lexical analysis

lexer = EntityLexer(input_stream)

token_stream = CommonTokenStream(lexer)

Parsing

parser = EntityParser(token_stream)

tree = parser.program()

Semantic validation

visitor = SemanticVisitor()

visitor.visit(tree)

if name == ‘main’:

main()

Expected Output:

Error: Duplicate attribute ‘name’ in entity ‘User’

Error: Invalid type ‘Float’ for attribute ‘total’

Error: Duplicate entity name ‘User’

Now we can catch semantic errors in the DSL input, making the language much safer and more robust.

3. Type Checking and Relationship Validation

Validating Relationships

In addition to checking attribute types, we also need to validate that relationships between entities are correctly defined. For example:

  • If an entity has a relationship with another entity, we must ensure the referenced entity exists.

  • We should also enforce relationship constraints such as one-to-many or many-to-many associations.

Extending the Visitor for Relationship Validation

Let’s modify the visitor to handle relationship validation:

def visitAttribute(self, ctx):

attr_name = ctx.ID().getText()

attr_type = ctx.type().getText()

Check if the type is a valid primitive or a reference to another

entity

valid_types =

′Integer′,′String′,′Date′,′Decimal′'Integer', 'String', 'Date', 'Decimal'

if attr_type not in valid_types and attr_type not in self.entities:

self.errors.append(f”Error: Undefined entity ‘{attr_type}’ used as a type for attribute ‘{attr_name}’”)

return attr_name, attr_type

This ensures that relationships between entities are defined with valid types.

Testing Relationship Validation

def main():

input_stream = InputStream('''

entity User {

id: Integer;

name: String;

}

entity Order {

id: Integer;

total: Decimal;

user: Customer; // Undefined entity ‘Customer’

}

''')

Lexical analysis

lexer = EntityLexer(input_stream)

token_stream = CommonTokenStream(lexer)

Parsing

parser = EntityParser(token_stream)

tree = parser.program()

Semantic validation

visitor = SemanticVisitor()

visitor.visit(tree)

if name == ‘main’:

main()

Expected Output:

Error: Undefined entity ‘Customer’ used as a type for attribute ‘user’

4. Extending the Visitor for API and View Generation

Now that we have a robust semantic analysis, let’s extend the visitor pattern to generate APIs and views for Flask-AppBuilder. This allows us to quickly create CRUD interfaces for our entities, automating a significant part of application development.

Generating API Code

Flask-AppBuilder makes it easy to define APIs for models. Here’s an example of an API class for the User entity:

class UserApi(ModelRestApi):

resource_name = ‘user’

datamodel = SQLAInterface(User)

appbuilder.add_api(UserApi)

Modifying the Visitor to Generate APIs

Let’s extend the visitor to generate this kind of API code for each entity.

class ApiVisitor(ParseTreeVisitor):

def init(self):

self.api_code = ""

def visitEntityDefinition(self, ctx):

entity_name = ctx.ID().getText()

Generate API class for the entity

api_class = f”class {entity_name}Api(ModelRestApi):\n”

api_class += f” resource_name = ‘{entity_name.lower()}‘\n”

api_class += f” datamodel = SQLAInterface({entity_name})\n\n”

api_class += f”appbuilder.add_api({entity_name}Api)\n\n”

self.api_code += api_class

return self.api_code

Generating View Code

Flask-AppBuilder also allows us to generate views for each entity. Here’s an example of a view for the User entity:

class UserModelView(ModelView):

datamodel = SQLAInterface(User)

appbuilder.add_view(UserModelView, “User”, icon=“fa-user”, category=“Users”)

We can extend the visitor to generate this code in a similar way:

class ViewVisitor(ParseTreeVisitor):

def init(self):

self.view_code = ""

def visitEntityDefinition(self, ctx):

entity_name = ctx.ID().getText()

Generate ModelView class for the entity

view_class = f”class {entity_name}ModelView(ModelView):\n”

view_class += f” datamodel = SQLAInterface({entity_name})\n\n

”

view_class += f”appbuilder.add_view({entity_name}ModelView, ‘{entity_name}’, icon=‘fa-user’, category=‘Users’)\n\n”

self.view_code += view_class

return self.view_code

Integrating API and View Generation with the Existing Visitor

We can combine both the API and view generation into a single visitor class:

class FullAppVisitor(ParseTreeVisitor):

def init(self):

self.model_code = ""

self.api_code = ""

self.view_code = ""

def visitEntityDefinition(self, ctx):

entity_name = ctx.ID().getText()

Generate Model code

class_code = f”class {entity_name}(Model):\n”

for attr_ctx in ctx.attribute():

class_code += self.visit(attr_ctx)

self.model_code += class_code + “\n”

Generate API code

api_class = f”class {entity_name}Api(ModelRestApi):\n”

api_class += f” resource_name = ‘{entity_name.lower()}‘\n”

api_class += f” datamodel = SQLAInterface({entity_name})\n\n”

api_class += f”appbuilder.add_api({entity_name}Api)\n\n”

self.api_code += api_class

Generate View code

view_class = f”class {entity_name}ModelView(ModelView):\n”

view_class += f” datamodel = SQLAInterface({entity_name})\n\n”

view_class += f”appbuilder.add_view({entity_name}ModelView, ‘{entity_name}’, icon=‘fa-user’, category=‘Users’)\n\n”

self.view_code += view_class

return class_code

def visitAttribute(self, ctx):

attr_name = ctx.ID().getText()

attr_type = ctx.type().getText()

type_mapping = {

‘Integer’: ‘Integer’,

‘String’: ‘String(50)’,

‘Date’: ‘Date’,

‘Decimal’: ‘Decimal’

}

if attr_type in type_mapping:

column_code = f” {attr_name} = Column({type_mapping

attrtypeattr_type

}, nullable=False)\n”

else:

column_code = f” {attr_name}_id = Column(Integer, ForeignKey(‘{attr_type}.id’), nullable=False)\n”

column_code += f” {attr_name} = relationship(‘{attr_type}’)\n”

return column_code

Step 1: Testing the Full Application Code Generation

def main():

input_stream = InputStream('''

entity User {

id: Integer;

name: String;

email: String;

}

entity Order {

id: Integer;

date: Date;

total: Decimal;

user: User;

}

''')

lexer = EntityLexer(input_stream)

token_stream = CommonTokenStream(lexer)

parser = EntityParser(token_stream)

tree = parser.program()

visitor = FullAppVisitor()

visitor.visit(tree)

print(visitor.model_code)

print(visitor.api_code)

print(visitor.view_code)

if name == ‘main’:

main()

Expected Output:

class User(Model):

id = Column(Integer, nullable=False)

name = Column(String(50), nullable=False)

email = Column(String(50), nullable=False)

class Order(Model):

id = Column(Integer, nullable=False)

date = Column(Date, nullable=False)

total = Column(Decimal, nullable=False)

user_id = Column(Integer, ForeignKey(‘User.id’), nullable=False)

user = relationship(‘User’)

class UserApi(ModelRestApi):

resource_name = ‘user’

datamodel = SQLAInterface(User)

appbuilder.add_api(UserApi)

class OrderApi(ModelRestApi):

resource_name = ‘order’

datamodel = SQLAInterface(Order)

appbuilder.add_api(OrderApi)

class UserModelView(ModelView):

datamodel = SQLAInterface(User)

appbuilder.add_view(UserModelView, ‘User’, icon=‘fa-user’, category=‘Users’)

class OrderModelView(ModelView):

datamodel = SQLAInterface(Order)

appbuilder.add_view(OrderModelView, ‘Order’, icon=‘fa-user’, category=‘Users’)

Now the visitor generates complete Flask-AppBuilder models, APIs, and views for each entity!

5. Practical Example: Generating Full Flask-AppBuilder Applications

Let’s take it a step further and generate a complete Flask application with all the models, APIs, and views defined in the DSL. We’ll also generate the app.py file, which serves as the entry point for the Flask application.

Step 1: Creating the app.py Template

Here’s a basic template for app.py:

from flask import Flask

from flask_appbuilder import SQLA, AppBuilder

from flask_sqlalchemy import Model

app = Flask(name)

app.config

′SQLALCHEMYDATABASEURI′'SQLALCHEMY_DATABASE_URI'

= ‘sqlite:///:memory:’

app.config

′SECRETKEY′'SECRET_KEY'

= ‘mysecret’

db = SQLA(app)

appbuilder = AppBuilder(app, db.session)

Model, API, and View imports go here

if name == “main”:

app.run(debug=True)

Step 2: Modifying the Visitor to Generate the Full Application

Extend the FullAppVisitor class to generate the app.py file, adding all the generated model, API, and view classes.

class FullAppVisitor(ParseTreeVisitor):

def init(self):

self.model_code = ""

self.api_code = ""

self.view_code = ""

def generate_app_file(self):

app_code = """

from flask import Flask

from flask_appbuilder import SQLA, AppBuilder

from flask_sqlalchemy import Model

app = Flask(name)

app.config

′SQLALCHEMYDATABASEURI′'SQLALCHEMY_DATABASE_URI'

= ‘sqlite:///:memory:’

app.config

′SECRETKEY′'SECRET_KEY'

= ‘mysecret’

db = SQLA(app)

appbuilder = AppBuilder(app, db.session)

Models

{models}

APIs

{apis}

Views

{views}

if name == “main”:

app.run(debug=True)

""".format(models=self.model_code, apis=self.api_code, views=self.view_code)

return app_code

The rest of the visitor methods remain the same

Step 3: Writing the Generated Code to Files

Modify the test script to write the generated code to separate Python files:

def main():

input_stream = InputStream('''

entity User {

id: Integer;

name: String;

email: String;

}

entity Order {

id: Integer;

date: Date;

total: Decimal;

user: User;

}

''')

lexer = EntityLexer(input_stream)

token_stream = CommonTokenStream(lexer)

parser = EntityParser(token_stream)

tree = parser.program()

visitor = FullAppVisitor()

visitor.visit(tree)

Write the generated models, APIs, and views to app.py

with open(‘app.py’, ‘w’) as f:

f.write(visitor.generate_app_file())

Step 4: Running the Flask Application

After running the script, an app.py file will be generated. You can run this Flask application using:

python app.py

This will start a Flask application with all the models, APIs, and views automatically generated from your DSL!

6. Conclusion and Next Steps

In this lesson, we’ve:

  • Performed semantic analysis to validate the DSL input.

  • Implemented type checking and relationship validation.

  • Extended the visitor pattern to generate Flask-AppBuilder models, APIs, and views.

  • Generated a full Flask application based on the DSL.

This approach greatly simplifies the process of defining and implementing ERP/CRM systems by automating large parts of the development process using a custom DSL.

Practical Exercise

Task: Extend the DSL to support many-to-many relationships and implement the necessary code generation for these relationships in Flask-AppBuilder.

Hints:

  • Add syntax for defining many-to-many relationships.

  • Use SQLAlchemy’s Table construct to manage many-to-many relationships.

Example DSL:

entity User {

id: Integer;

name: String;

email: String;

}

entity Group {

id: Integer;

name: String;

users: ManyToMany(User);

}

Looking Ahead

In the next lesson, we’ll explore optimizations, security implications, and best practices for building robust DSLs and Flask applications. We’ll also look at how to package the generated code for deployment in production environments.

Feel free to reach out if you have any questions. See you in the next lesson!